Avatars are slow as molasses

spamless · May 18, 2022, 11:45pm

Steps to reproduce

Any user including the admin user tries to personalize his avatar by uploading an image.

Expected behaviour

The avatar personalization should be relatively effortless and quick.

I have a friend who put a similar ownCloud build on his server for me to test. His setup and specs are also similar. He does not have this issue. I changed my avatar on his server and had no problem of the type I’m having on my own server.

Actual behaviour

It works – after close to two minutes of nothing visible happening. There is also a long wait after confirming the desired framing of the uploaded image – and the page does not reload on its own.

Server configuration

Operating system:
Ubuntu 20.04 VPS

Web server:
Server version: Apache/2.4.41 (Ubuntu)

Database:
mysql/MariaDB

PHP version:
PHP 7.4.3 (cli) (built: Mar 2 2022 15:36:52) ( NTS )

ownCloud version: (see ownCloud admin page)
ownCloud 10.9.1

Updated from an older ownCloud or fresh install:
New manual command-line install on fresh server instance

Where did you install ownCloud from:
Official repository (tarball)

Signing status (ownCloud 9.0 and above):

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results into https://gist.github.com/ and put the link here.

Nothing to share but “No errors have been found.”

The content of config/config.php:

Log in to the web-UI with an administrator account and click on
'admin' -> 'Generate Config Report' -> 'Download ownCloud config report'
This report includes the config.php settings, the list of activated apps
and other details in a well sanitized form.

or 

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

*ATTENTION:* Do not post your config.php file in public as is. Please use one of the above
methods whenever possible. Both, the generated reports from the web-ui and from occ config:list
consistently remove sensitive data. You still may want to review the report before sending.
If done manually then it is critical for your own privacy to dilligently
remove *all* host names, passwords, usernames, salts and other credentials before posting.
You should assume that attackers find such information and will use them against your systems.

Weirdly, your suggestion to run ‘Generate Config Report’ is not available from my admin panel.

From the CLI method, with hostname/IP manually munged as well:

{
    "system": {
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "localhost",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "\/var\/www\/owncloud\/data",
        "overwrite.cli.url": "http:\/\/localhost",
        "dbtype": "mysql",
        "version": "10.9.1.2",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "allow_user_to_change_mail_address": "",
        "logtimezone": "UTC",
        "apps_paths": [
            {
                "path": "\/var\/www\/owncloud\/apps",
                "url": "\/apps",
                "writable": false
            },
            {
                "path": "\/var\/www\/owncloud\/apps-external",
                "url": "\/apps-external",
                "writable": true
            }
        ],
        "installed": true,
        "instanceid": "ocq08k5pbusj",
        "maintenance": false,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "127.0.0.1",
            "port": "***REMOVED SENSITIVE VALUE***"
        },
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_smtpsecure": "ssl",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "465",
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated apps:

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your ownCloud installation folder.

$  sudo -u www-data php occ app:list
Enabled:
  - activity:
    - Version: 2.7.0
    - Path: /var/www/owncloud/apps-external/activity
  - announcementcenter:
    - Version: 1.2.2
    - Path: /var/www/owncloud/apps/announcementcenter
  - camerarawpreviews:
    - Version: 0.7.13
    - Path: /var/www/owncloud/apps-external/camerarawpreviews
  - checksum:
    - Version: 0.3.5
    - Path: /var/www/owncloud/apps-external/checksum
  - comments:
    - Version: 0.3.0
    - Path: /var/www/owncloud/apps/comments
  - configreport:
    - Version: 0.2.0
    - Path: /var/www/owncloud/apps/configreport
  - customgroups:
    - Version: 0.6.2
    - Path: /var/www/owncloud/apps/customgroups
  - dav:
    - Version: 0.7.0
    - Path: /var/www/owncloud/apps/dav
  - diagnostics:
    - Version: 0.1.4
    - Path: /var/www/owncloud/apps-external/diagnostics
  - duo:
    - Version: 2.5.2
    - Path: /var/www/owncloud/apps-external/duo
  - encryption:
    - Version: 1.5.1
    - Path: /var/www/owncloud/apps/encryption
  - external:
    - Version: 1.4.0
    - Path: /var/www/owncloud/apps/external
  - federatedfilesharing:
    - Version: 0.5.0
    - Path: /var/www/owncloud/apps/federatedfilesharing
  - federation:
    - Version: 0.1.0
    - Path: /var/www/owncloud/apps/federation
  - files:
    - Version: 1.5.2
    - Path: /var/www/owncloud/apps/files
  - files_antivirus:
    - Version: 1.0.0
    - Path: /var/www/owncloud/apps/files_antivirus
  - files_clipboard:
    - Version: 1.0.3
    - Path: /var/www/owncloud/apps-external/files_clipboard
  - files_external:
    - Version: 0.8.0
    - Path: /var/www/owncloud/apps/files_external
  - files_external_dropbox:
    - Version: 1.2.0
    - Path: /var/www/owncloud/apps/files_external_dropbox
  - files_external_ftp:
    - Version: 0.2.1
    - Path: /var/www/owncloud/apps/files_external_ftp
  - files_mediaviewer:
    - Version: 1.0.5
    - Path: /var/www/owncloud/apps/files_mediaviewer
  - files_pdfviewer:
    - Version: 1.0.1
    - Path: /var/www/owncloud/apps/files_pdfviewer
  - files_sharing:
    - Version: 0.14.0
    - Path: /var/www/owncloud/apps/files_sharing
  - files_texteditor:
    - Version: 2.4.1
    - Path: /var/www/owncloud/apps/files_texteditor
  - files_textviewer:
    - Version: 1.0.3
    - Path: /var/www/owncloud/apps-external/files_textviewer
  - files_trashbin:
    - Version: 0.9.1
    - Path: /var/www/owncloud/apps/files_trashbin
  - files_versions:
    - Version: 1.3.0
    - Path: /var/www/owncloud/apps/files_versions
  - firstrunwizard:
    - Version: 1.2.0
    - Path: /var/www/owncloud/apps/firstrunwizard
  - gallery:
    - Version: 16.1.2
    - Path: /var/www/owncloud/apps-external/gallery
  - graphapi:
    - Version: 0.2.0
    - Path: /var/www/owncloud/apps/graphapi
  - guests:
    - Version: 0.9.3
    - Path: /var/www/owncloud/apps/guests
  - impersonate:
    - Version: 0.5.1
    - Path: /var/www/owncloud/apps-external/impersonate
  - market:
    - Version: 0.6.3
    - Path: /var/www/owncloud/apps-external/market
  - music:
    - Version: 1.5.2
    - Path: /var/www/owncloud/apps-external/music
  - nextbackup:
    - Version: 21.1.1
    - Path: /var/www/owncloud/apps-external/nextbackup
  - notifications:
    - Version: 0.5.4
    - Path: /var/www/owncloud/apps/notifications
  - provisioning_api:
    - Version: 0.5.0
    - Path: /var/www/owncloud/apps/provisioning_api
  - search_elastic:
    - Version: 2.1.0
    - Path: /var/www/owncloud/apps-external/search_elastic
  - systemtags:
    - Version: 0.3.0
    - Path: /var/www/owncloud/apps/systemtags
  - templateeditor:
    - Version: 0.4.0
    - Path: /var/www/owncloud/apps/templateeditor
  - twofactor_backup_codes:
    - Version: 0.1.0
    - Path: /var/www/owncloud/apps-external/twofactor_backup_codes
  - updatenotification:
    - Version: 0.2.1
    - Path: /var/www/owncloud/apps/updatenotification
  - wallpaper:
    - Version: 0.2.2
    - Path: /var/www/owncloud/apps-external/wallpaper
  - web:
    - Version: 5.4.0
    - Path: /var/www/owncloud/apps-external/web
Disabled:
  ...

Are you using external storage, if yes which one: local/smb/sftp/…
No.

Are you using encryption: yes/no
No.

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…
[Edited.]
I have these checked. But I have no users yet except me! I’m using the ownCloud App in Windows.

Client configuration

Browser:

Current Google Chrome under Windows 11 Pro 64-bit.

Operating system:
Windows 11 Pro 64-bit

Logs

Web server error log

Insert your webserver log here

root@cloud:/var/log/apache2# cat error.log
[Thu May 19 00:00:04.724524 2022] [ssl:warn] [pid 584] AH01909: 127.0.0.1:443:0 server certificate does NOT include an ID which matches the server name
[Thu May 19 00:00:04.725016 2022] [mpm_prefork:notice] [pid 584] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1f configured -- resuming normal operations
[Thu May 19 00:00:04.725032 2022] [core:notice] [pid 584] AH00094: Command line: '/usr/sbin/apache2'
[Thu May 19 00:01:27.802793 2022] [authz_core:error] [pid 182303] [client ***REMOVED SENSITIVE VALUE***:***REMOVED SENSITIVE VALUE***] AH01630: client denied by server configuration: /var/www/owncloud/data/htaccesstest.txt

ownCloud log (data/owncloud.log)

Insert your ownCloud log here

I found it, but it appears to contain lots of sensitive lines that would take some effort to sanitize for posting. Do you really want to see it?

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log 
c) ...

I found the network log in Chrome (had to Google for how). I don’t see how to copy it for insertion here or what might be useful here, but feel free to give me a clue if I need one.

jvillafanez · May 19, 2022, 7:27am

I’d recommend to use tail -f owncloud.log and then reproduce the issue. This way you can see the relevant portion of the logs. It might be mixed with logs from other requests, but at least it isn’t the whole file.
However, if the request is slow, I don’t think there will be relevant logs.

Another thing you can try is to disable all the apps and keep a minimum set of them. You can use core/apps at master · owncloud/core · GitHub as base. With the minimum set, try to reproduce the issue. If you don’t reproduce the issue, start enabling the apps one by one and recheck. This way we can narrow down which app could be causing the issue.

spamless · May 19, 2022, 9:12pm

Okay, no new log material was generated while reproducing the issue. By the way, while I didn’t get out my stopwatch, it took something on the order of three minutes for the profile JPG I chose to appear in the framing selector, and after I framed it and clicked OK, it took 4-5 minutes to finish and show up on the screen. There are no users logged in but me, and nothing is happening on the server right now that would cause a load.

When I logged in and went to the profile area to do this, my avatar was not even displaying on the profile-picture area of the account page. After about a minute, it appeared. And see the log message below.

The tail-f log contains this from a bit earlier. Note the clamav error and my further remarks down below:

()\\n#10 \\\/var\\\/www\\\/owncloud\\\/console.php(116): OC\\\\Console\\\\Application->run()\\n#11 \\\/var\\\/www\\\/owncloud\\\/occ(11): require_once('\\\/var\\\/www\\\/ownclo...')\\n#12 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php\",\"Line\":96}"}
{"reqId":"[munged]","level":2,"time":"2022-05-19T20:40:55+00:00","remoteAddr":"[munged]","user":"[munged]","app":"files_antivirus","method":"GET","url":"\/index.php\/avatar\/[munged]\/154","message":"Failed to write a chunk. Check if Stream Length matches StreamMaxLength in ClamAV daemon settings"}

Now I have to add that I had initially installed clamav when I installed ownCloud, but after an out-of-memory message in dmesg, I disabled clamav. Is that what’s going on here? But I’m sure this issue was happening already from the very start, including when I had clamav running.

Here is the old dmesg log extract:

[ 4149.564248] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems
_allowed=0,global_oom,task_memcg=/system.slice/apache2.service,task=clamscan,pid
=9208,uid=33
[ 4149.564312] Out of memory: Killed process 9208 (clamscan) total-vm:1320240kB, anon-rss:444848kB, file-rss:0kB, shmem-rss:0kB, UID:33 pgtables:2528kB oom_score_adj:0
[ 4149.685442] oom_reaper: reaped process 9208 (clamscan), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

After that, I thought, well, I’m not using the server as a mail server, and I only plan to have a handful of trusted users, so I don’t need clamscan. So I searched around the web for how to disable it to stop its memory footprint. I found this and implemented it:

systemctl mask clamav-daemon clamav-freshclam

That’s from here.

Output:

root@cloud:~# systemctl mask clamav-daemon clamav-freshclam
Created symlink /etc/systemd/system/clamav-daemon.service → /dev/null.
Created symlink /etc/systemd/system/clamav-freshclam.service → /dev/null.

So, is that what’s producing that error message now in my ownCloud error log? And is that what’s causing the profile pictures to be rendered so slowly?

Thank you. I looked there, but I’m not sure how to infer from there which apps are the crucial couple I should pare down to to try the experiment. I admit I enabled too many apps. I just went to the smorgasbord, so to speak, and enabled most anything that hat an open-source license that looked at all interesting. I wasn’t thinking they would create a load.

Thanks for any further ideas.

spamless · May 19, 2022, 10:29pm

More data, @jvillafanez:

I disabled all apps except these eight “official” ones, and the problem is gone. So now I’ll re-enable some of the apps bit-by-bit and see. I don’t know if it’s the apps themselves or the memory footprint they require.

I rebooted the machine after I disabled the apps. Also, the tail of the owncloud.log is still showing some kind of error.

{"reqId":"BzuWIJHFQ4PLPBEfurh9","level":3,"time":"2022-05-19T22:00:02+00:00","remoteAddr":"","user":"--","app":"core","method":"--","url":"--","message":"Exception: {\"Exception\":\"OCP\\\\AppFramework\\\\QueryException\",\"Message\":\"Could not resolve OCA\\\\Search_Elastic\\\\Jobs\\\\DeleteJob! Class OCA\\\\Search_Elastic\\\\Jobs\\\\DeleteJob does not exist\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php(110): OC\\\\AppFramework\\\\Utility\\\\SimpleContainer->resolve()\\n#1 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/ServerContainer.php(86): OC\\\\AppFramework\\\\Utility\\\\SimpleContainer->query()\\n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/BackgroundJob\\\/JobList.php(253): OC\\\\ServerContainer->query()\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/BackgroundJob\\\/JobList.php(209): OC\\\\BackgroundJob\\\\JobList->buildJob()\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/BackgroundJob\\\/JobList.php(213): OC\\\\BackgroundJob\\\\JobList->getNext()\\n#5 \\\/var\\\/www\\\/owncloud\\\/core\\\/Command\\\/System\\\/Cron.php(116): OC\\\\BackgroundJob\\\\JobList->getNext()\\n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Command\\\/Command.php(255): OC\\\\Core\\\\Command\\\\System\\\\Cron->execute()\\n#7 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Application.php(1009): Symfony\\\\Component\\\\Console\\\\Command\\\\Command->run()\\n#8 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Application.php(273): Symfony\\\\Component\\\\Console\\\\Application->doRunCommand()\\n#9 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Application.php(149): Symfony\\\\Component\\\\Console\\\\Application->doRun()\\n#10 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Console\\\/Application.php(165): Symfony\\\\Component\\\\Console\\\\Application->run()\\n#11 \\\/var\\\/www\\\/owncloud\\\/console.php(116): OC\\\\Console\\\\Application->run()\\n#12 \\\/var\\\/www\\\/owncloud\\\/occ(11): require_once('\\\/var\\\/www\\\/ownclo...')\\n#13 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php\",\"Line\":96}"}

My questions that remain are:

Should I re-enable clamAV (see my last message)?

How can I tell if apps themselves are requiring too much memory or leaking memory, or if it’s just that I loaded too many?

Further: and now I have put most apps back in bunches of 4-5 at once, changing the avatar in between bunches. I’ve got most of 'em back now. It started to get slow with the last batch, which included two different media viewers. I disabled one of the two, and now things still are fast enough.

Am I done? Should I do more tests? And how can I easily list the apps that are running?

Much obliged.

jvillafanez · May 20, 2022, 7:00am

It seems the files_antivirus app isn’t properly configured and it might be the cause. If you don’t plan to use it, you should keep it disabled (or ensure it’s configured correctly).

Other than that, it’s up to you. The usual recommendation is to disable the apps that you won’t use.

How can I tell if apps themselves are requiring too much memory or leaking memory, or if it’s just that I loaded too many?

Without causing too much fuss, the basic idea is to enable only the apps you want to use, anything else should be disabled. If you need to enable a new app, you can compare the performance before and after enabling the app. Having a general idea should be enough to check if the app is giving any problem.
If you prefer something more profesional, you should check monitoring software for all the components you’re using.

And how can I easily list the apps that are running?

The app listing in the settings page should be enough. You can also use the occ app:list command to get a similar list.

system · August 18, 2022, 7:00am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.