Using High CPU after Adding Local Storage

High,

I am using OC 10 (from OC repo) on CentOS 7.

I enabled local storage for a large folder with loads of files. Since then, my httpd uses a lot of CPU power. The server is now for days at around 75% of CPU useage. There are not many users on it (6 configured) so it does not appear to be user load:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
32423 apache    20   0  610304  38928   6800 R  25,6  1,0   0:38.21 httpd
32508 apache    20   0  613176  41316   6412 S  23,3  1,1   1:04.13 httpd
32422 apache    20   0  611308  39676   6408 S  18,6  1,0   1:05.22 httpd
32516 apache    20   0  692052 120452   6384 R  17,6  3,1   0:53.05 httpd
32517 apache    20   0  604388  32616   6384 S  12,6  0,8   0:32.11 httpd
32519 apache    20   0  609028  37788   6800 S  11,3  1,0   0:33.59 httpd
 1850 mysql     20   0 1496392 248732   8808 S   5,0  6,4   5:01.99 mysqld

As the mysqld as well has some load I assume this is OC as there is no other instance using mysql on this server.

So just a question:
Does OC index the local storage files? Is this why the load is so high? How long might it take to index the storage with approx 140,000 files and ~600GB?

Can I speed up the indexing somehow?

Thanks!

/KNEBB

What are your exact external storage settings? Does the load only go higher when there is a user in the webinterface?

On the little gear on the right hand side of the configured external storage mount point, you can configure whether ownCloud should check for changes on the filesystem. Either “once every direct access”, e.g when a user is browsing the files on the web interface or never.

I think there also exists a background job to scan the files on the external storage. But I’m not sure how to configure that. Perhaps @jvillafanez knows more about that? :wink:

I guess it’s indexing / scanning the external storage to get the metadata of the files there. I’m not sure about all the implementation details, so it’s difficult to evaluate if your situation is normal or not, mainly for the days you say it’s taking.
Note that there are quite a bunch of factors such as the CPU speed, FS access speed, number of files, memory… that will affect the overall performance of ownCloud, and most of them could vary depending on the workload.

In general, the process is kind of slow, but it should be just one time. Further scans shouldn’t take so much time.

I think you can take a peek at the DB, in the oc_filecache table, to check how many entries are there and get an idea of what the state is. Note that there are also additional information there (versions, thumbnails…) so it won’t be a 1:1 match.
Taking a sneak peek is fine, but changing things might break other things, so be careful.

The background job itself is always enabled. It should be smart enough to scan only what is marked as incomplete, but the first time it will need to scan the whole storage.

Anyway, I expect the load to eventually stabilize by itself once the scanning finishes.

1 Like