High CPU usage owncloud server

[thomsany]

Hi, I have a Vmware centos installation with 8 cores, 16GB of RAM and I have around 30 users connection and syncing with the owncloud server.
For some reason when client starts syncing, the CPU goes all the way up to 30.3 or 40.4 then down to around 15.1 to 20.1.
How can I improve CPU usage so that owncloud makes better use of CPU and memory?
I think with the current cores/memory should be ok? or do I need to add more CPU and memory?

How can I fix it?
Thanks in advance.
Teo

[jvillafanez]

While this is happening, enter the mysql DB and use show processlist to check what’s going on there. I suppose it’s something about the file locking, so there is likely heavy activity in the “oc_file_locks” table.
If this is the case, try to setup the file locking mechanism with redis. There should be documentation for this.

[thomsany]

Strangely not seeing the file locks stuff.
I am getting above 1.0 on the CPU and this is what I see:
MariaDB [(none)]> SHOW PROCESSLIST;
±---------±-------------±----------±-----------±--------±-----±-------------------------±-----------------±---------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
±---------±-------------±----------±-----------±--------±-----±-------------------------±-----------------±---------+
| 1 | system user | | NULL | Daemon | NULL | InnoDB purge coordinator | NULL | 0.000 |
| 4 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 |
| 2 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 |
| 3 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 |
| 5 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 |
| 16593977 | root | localhost | NULL | Query | 0 | Init | SHOW PROCESSLIST | 0.000 |
| 16594181 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
±---------±-------------±----------±-----------±--------±-----±-------------------------±-----------------±---------+
7 rows in set (0.000 sec)

[thomsany]

CPU over 30.0 and here is the process list. Anything suspicious?

MariaDB [(none)]> SHOW PROCESSLIST;
±---------±-------------±----------±-----------±--------±-----±-------------------------±-----------------------------------------------------------------------------------------------------±---------+
| Id | User | Host | db | Command | Time | State | Info | Progress |
±---------±-------------±----------±-----------±--------±-----±-------------------------±-----------------------------------------------------------------------------------------------------±---------+
| 1 | system user | | NULL | Daemon | NULL | InnoDB purge coordinator | NULL | 0.000 |
| 4 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 |
| 2 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 |
| 3 | system user | | NULL | Daemon | NULL | InnoDB purge worker | NULL | 0.000 |
| 5 | system user | | NULL | Daemon | NULL | InnoDB shutdown handler | NULL | 0.000 |
| 16616927 | root | localhost | NULL | Sleep | 4837 | | NULL | 0.000 |
| 16646311 | root | localhost | NULL | Query | 0 | Init | SHOW PROCESSLIST | 0.000 |
| 16646907 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646911 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646919 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646921 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646924 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646925 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646926 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646929 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646930 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646935 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646936 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646938 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646940 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646941 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646942 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646944 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646945 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646946 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646947 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646948 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
| 0.000 |
| 16646949 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646950 | ownclouduser | localhost | mydb | Sleep | 1 | | NULL | 0.000 |
| 16646951 | ownclouduser | localhost | mydb | Query | 0 | Statistics | SELECT fileid FROM oc_filecache WHERE storage = ‘44’ AND path_hash = 'd41d8cd98f00b204e98009 | 0.000 |
| 16646952 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646953 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646954 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646955 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646956 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646957 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646958 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646959 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
| 16646960 | ownclouduser | localhost | mydb | Sleep | 0 | | NULL | 0.000 |
±---------±-------------±----------±-----------±--------±-----±-------------------------±-----------------------------------------------------------------------------------------------------±---------+

[jvillafanez]

It seems that’s a “normal” file scan, so nothing really suspicious.

I’m not sure if ownCloud 10.8 will improve your situation, but I think it’s something to try.

Some other things to try:

• Don’t use the desktop client for a while. This is mainly to compare the load with and without the desktop client to give you an idea of the load the desktop client is generating in the system.
• Start the desktop clients with the environment variable OWNCLOUD_MAX_PARALLEL=1 (the default is 6). I’m not sure when the option was introduced so check the manual for your specific desktop version. It will make the desktop sync slower, but it should reduce the amount of load in the server.
• Move the DB to a different server. I’m not really sure if it’s possible or not, so take precautions before attempting.

You should also try to investigate what all those httpd processes are doing. With the little info we have I don’t know all those processes are required (I mean someone is doing something legit) or if there is something else going on (like an unwanted loop caused by a bug somewhere).
I guess the high amount of cpu usage by the mysqld process is due to the high number of httpd processes requesting data from the DB.

If there isn’t anything wrong, you might want to consider to split your system in multiple hosts, I’d say at least one host for the DB and two (planning for three) for the ownCloud server, with a load balancer in front ot them. The setup is more complex, but maybe your daily usage is too much for having everything in the same machine.

[thomsany]

We use around 30 owncloud Windows client to connect to the server everyday.
When they are not connected the load goes down all the way.
What is strange is that the server performs OK even when it’s at 30 or 40 in the CPU load.
Is that normal and safe?
Thanks!

[jvillafanez]

As said, it depends on how your clients are using ownCloud. It’s normal if there are constant changes that are needed to be updated in the server. Also note that there are potentially 30 connections scanning different parts of the FS at the same time.

If that’s you’re average load, you might need to split components into different servers, if possible (I’m not sure if it’s easy or maybe impossible in a production environment).

A different possibility is that either the client or the server has some bug that causes the client to keep scanning. If the system is at a stable point where everything is synced in all the clients, the clients should just ping the server for changes, which shouldn’t cause a high load.
In any case, it’s better to use the latest versions both for the client and server.

[aclemence]

I have found that file deletions add a lot to the server load figures. When I was running processes that required deleting hundreds or thousands of files each month, it would take between 6 hours and 2 days to do so with similarly large load figures and multiple httpd processes.

You could try turning off the trashbin and see how that goes, but I was never brave enough to do so as it’s almost a guarantee that the second I do so, someone would need it.

I never had any server stability issues, only the odd user complaint that file uploads would be slower as a result of the loading.

We are also using CentOS on a VMware ESXi host.

[thomsany]

After doing some troubleshooting I enabled the memcache using:
‘memcache.local’ => ‘\OC\Memcache\APCu’,

Also I enabled the Strict transport security feature that wasn’t turned on and looks like CPU use went down a lot. The only thing I am getting is that every 24 hours owncloud server goes down and I need to reboot it. How can I find what is causing the crash? It works perfectly and then suddnly clients stop syncing and when I go into the server I see mysql and httpd services not showing anymore.
Only fix is to reboot the server which is not the most efficient way and it wasn’t happening before the change.
Also installed Redis but haven’t been able to move transactional file locking yet since I haven’t found a proper tutorial for Centos based systems (most or all are Debian or Ubuntu examples)
What is the best way to find out what happened and why mysql and owncloud stop working? its not even a resource issue.

Thanks,
Teo

[aclemence]

I don’t think you need anything more than the owncloud configuration parameters for the transactional file locking, as long as redis is installed and working. My cache setup is as follows in config.php:
‘memcache.local’ => ‘\OC\Memcache\APCu’,
‘filelocking.enabled’ => true,
‘memcache.locking’ => ‘\OC\Memcache\Redis’,
‘redis’ =>
array (
‘host’ => ‘localhost’,
‘port’ => 6379,
‘timeout’ => 0,
‘password’ => ‘’,
),

From what I recall, as long as Redis is installed, then it will work on the above using the default port.

Check that Redis is running first, per the final install steps here:

How to Install and Configure Redis on CentOS 7

[jvillafanez]

You might want to monitor your memory usage globally. It might be possible that mysql is requesting more memory but the OS can’t give it (maybe ownCloud is using too much), so the DB crashes. That might explain why mysql stops working.

In addition, if the crash happens every 24 hours, check if you have a background job running daily. You might also want to monitor what processes are consuming too much memory.