Full scan on every request

Steps to reproduce

I have noticed that my owncloud server very slow responses when I work with files by web interface. I traced apache’s working processes and saw the process scans all files in the storage by each users’ click. And of couse it takes a lot of time.
I don’t know how long it happened but I think it has become a problem after massive uploading new files by my users.

My storage has 333G and 220953 files.

Expected behaviour

PHP work processes should not scan all files in the storage

Server configuration

Operating system: Debian 7

Web server: Apache/2.2.22 (Debian)

Database: MySQL 5.5.55-0+deb7u1

PHP version: 5.6.30-1~dotdeb+7.1 (with OPcache)

ownCloud version: 9.1.5.2

The content of config/config.php:

{
    "system": {
        "updatechecker": false,
        "instanceid": "ockfkpyha3od",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "\/var\/www\/owncloud\/data",
        "overwrite.cli.url": "http:\/\/***REMOVED SENSITIVE VALUE***\/owncloud",
        "dbtype": "mysql",
        "version": "9.1.5.2",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "mail_from_address": "owncloud",
        "mail_smtpmode": "php",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpsecure": "ssl",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "443",
        "mail_smtpauthtype": "LOGIN",
        "mail_smtpauth": 1,
        "theme": "",
        "loglevel": 2,
        "maintenance": false,
        "integrity.check.disabled": true,
        "singleuser": false
    }
}

List of activated apps:

Enabled:
  - activity: 2.3.2
  - audioplayer: 2.0.2
  - comments: 0.3.0
  - configreport: 0.1.1
  - dashboard: 2.0
  - dav: 0.2.7
  - direct_menu: 0.9.2
  - encryption: 1.3.0
  - external: true
  - federatedfilesharing: 0.3.0
  - federation: 0.1.0
  - files: 1.5.1
  - files_external: 0.6.0
  - files_pdfviewer: 0.8.1
  - files_sharing: 0.10.0
  - files_texteditor: 2.1
  - files_trashbin: 0.9.0
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 15.0.0
  - notifications: 0.3.0
  - onlyoffice: 1.3.0
  - ownnote: 1.08
  - passwordpolicy: 1.2
  - provisioning_api: 0.5.0
  - registration: 0.2.3
  - serverinfo: 1.1.1.1
  - shorten: 0.0.15
  - storagecharts2: 2.7.0
  - systemtags: 0.3.0
  - templateeditor: 0.1
  - twofactor_totp: 0.4.2
  - updatenotification: 0.2.1
Disabled:
  - files_antivirus
  - user_external
  - user_ldap

Are you using encryption: yes

Logs

I have full strace log of a request, it is about 2.5 Gb. I think it does not inlcude some useful for debug. For example, after read the last file in my account renton:

stat("/var/www/owncloud/data/renton//avatar.28.png", {st_mode=S_IFREG|0644, st_size=2064, ...}) = 0
stat("/var/www/owncloud/data/renton//avatar.png", {st_mode=S_IFREG|0644, st_size=28642, ...}) = 0
stat("/var/www/owncloud/data/renton//files_versions", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/owncloud/data/renton/files_versions", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/www/owncloud/data/renton/files_versions/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 20
close(21)                               = 0
getdents(20, /* 4 entries */, 32768)    = 128
stat("/var/www/owncloud/data/renton/files_versions//p.txt.v1532599440", {st_mode=S_IFREG|0644, st_size=8424, ...}) = 0
stat("/var/www/owncloud/data/renton/files_versions//t.txt.v1528553369", {st_mode=S_IFREG|0644, st_size=8304, ...}) = 0
getdents(20, /* 0 entries */, 32768)    = 0
getdents(19, /* 0 entries */, 32768)    = 0
close(19)                               = 0

process begins to read files next user

stat("/var/www/owncloud/data", {st_mode=S_IFDIR|0770, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/www/owncloud/data/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 19
close(20)                               = 0
getdents(19, /* 50 entries */, 32768)   = 1704
stat("/var/www/owncloud/data//vinogradovka", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/owncloud/data/vinogradovka", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/www/owncloud/data/vinogradovka/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 20
getdents(20, /* 3 entries */, 32768)    = 88
stat("/var/www/owncloud/data/vinogradovka//files_encryption", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/owncloud/data/vinogradovka/files_encryption", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/www/owncloud/data/vinogradovka/files_encryption/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 21
getdents(21, /* 3 entries */, 32768)    = 88
stat("/var/www/owncloud/data/vinogradovka/files_encryption//OC_DEFAULT_MODULE", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/var/www/owncloud/data/vinogradovka/files_encryption/OC_DEFAULT_MODULE", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
openat(AT_FDCWD, "/var/www/owncloud/data/vinogradovka/files_encryption/OC_DEFAULT_MODULE/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 22

Hey,

this looks to me like a quite outdated and already unsupported version of ownCloud. Maybe it could make sense to upgrade a recent version first:

After night digging I found culprit. It is 3rd-party Storage Charts 2.

1 Like