I enabled local storage for a large folder with loads of files. Since then, my httpd uses a lot of CPU power. The server is now for days at around 75% of CPU useage. There are not many users on it (6 configured) so it does not appear to be user load:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
32423 apache 20 0 610304 38928 6800 R 25,6 1,0 0:38.21 httpd
32508 apache 20 0 613176 41316 6412 S 23,3 1,1 1:04.13 httpd
32422 apache 20 0 611308 39676 6408 S 18,6 1,0 1:05.22 httpd
32516 apache 20 0 692052 120452 6384 R 17,6 3,1 0:53.05 httpd
32517 apache 20 0 604388 32616 6384 S 12,6 0,8 0:32.11 httpd
32519 apache 20 0 609028 37788 6800 S 11,3 1,0 0:33.59 httpd
1850 mysql 20 0 1496392 248732 8808 S 5,0 6,4 5:01.99 mysqld
As the mysqld as well has some load I assume this is OC as there is no other instance using mysql on this server.
So just a question:
Does OC index the local storage files? Is this why the load is so high? How long might it take to index the storage with approx 140,000 files and ~600GB?
What are your exact external storage settings? Does the load only go higher when there is a user in the webinterface?
On the little gear on the right hand side of the configured external storage mount point, you can configure whether ownCloud should check for changes on the filesystem. Either “once every direct access”, e.g when a user is browsing the files on the web interface or never.
I think there also exists a background job to scan the files on the external storage. But I’m not sure how to configure that. Perhaps @jvillafanez knows more about that?
I guess it’s indexing / scanning the external storage to get the metadata of the files there. I’m not sure about all the implementation details, so it’s difficult to evaluate if your situation is normal or not, mainly for the days you say it’s taking.
Note that there are quite a bunch of factors such as the CPU speed, FS access speed, number of files, memory… that will affect the overall performance of ownCloud, and most of them could vary depending on the workload.
In general, the process is kind of slow, but it should be just one time. Further scans shouldn’t take so much time.
I think you can take a peek at the DB, in the oc_filecache table, to check how many entries are there and get an idea of what the state is. Note that there are also additional information there (versions, thumbnails…) so it won’t be a 1:1 match.
Taking a sneak peek is fine, but changing things might break other things, so be careful.
The background job itself is always enabled. It should be smart enough to scan only what is marked as incomplete, but the first time it will need to scan the whole storage.
Anyway, I expect the load to eventually stabilize by itself once the scanning finishes.
