Low but constant CPU usage triggers throttling by hosting company

Server
1

Steps to reproduce

  1. Not sure. My hosting company started throttling in June as apparently I’m abusing the service, and their analysis is OC using constant 10 to 20% CPU. Can’t recall making any changes (running 10.0.4), except maybe enabling/disabling Apps in the admin web interface. We don’t use any apps in practice, just the core OC functions from original install.
  2. I upgraded to 10.0.8 hoping that the issue would go away. Instead my hosting provider has limited my server access (CPU throttling for my domain) after one week, as OC has exceeded the CPU quota for the month

Expected behaviour

OC should just run normally with 3 active users and small file traffic (max 10 file updates per day), without using excessive CPU

Actual behaviour

I get this message in Cpanel ressource Usage Overview:

You have reached entry processes (number of simultaneously running php and cgi scripts, as well as cron jobs and shell sessions) limit 37 times. 
I/O usage resources were limited for your site

Also, Webalizer shows that the top URL “hit” on my box is :"/owncloud/ocs/v2.php/apps/notifications/api/v1/notifications" with 9,000++ hits in 22 days.
We are all running OC windows client. No one is accessing web browser client. This amount of hits makes no sense. This URL (and perhaps others, I am not a sysadmin and don’t know what to look for) causes a fairly constant aprox. 10-20% CPU usage throughout the day, which adds up at the end of the month. My hosting provider is not happy with my hosting setup and asking to upgrade to a higher hosting package. Makes no sense, we have very little traffic.

Server configuration

Operating system: Linux

Web server: Apache 2.4.33

Database: My SQL 10.1.34

PHP version: 5.6.36

ownCloud version: 10.0.8

Updated from an older ownCloud or fresh install: Updated from 10.0.4

Where did you install ownCloud from: Softaculous from hosting provider

Signing status (ownCloud 9.0 and above): Don’t understand the question. Pls clarify?
I have no problem to sign in to OC, with Windows and with web client, regular user and admin user

Login as admin user into your ownCloud and access 
http://example.com/
paste the results into https://gist.github.com/ and puth the link here.

The content of config/config.php:

https://gist.github.com/virgulesystems/7adfc00d799fe64bb43ce3424a2002ac

List of activated apps: I do not have command-line access

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

Client configuration

Browser: Chrome (updated)

Operating system: Win7/64

Logs

Web server error log:

I don’t think that I have access to web server log (and I don’t know where to find it…)
This is the file \public_html\error_log:

text/x-generic error_log ( ASCII text, with very long lines )
[15-May-2017 17:51:12 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:63
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(146): Doctrin in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 63
[15-May-2017 17:51:12 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:63
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(146): Doctrin in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 63
[15-May-2017 17:51:13 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:63
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(146): Doctrin in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 63
[15-May-2017 17:51:14 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:63
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(146): Doctrin in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 63

.
.
.
. same error goes on forever
.
.
.
[13-Mar-2018 06:03:26 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (11 "Resource temporarily unavailable")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:64
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 64

.
.
.
. same error goes on forever
.
.
.
[10-Jun-2018 17:26:25 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:64
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(147 in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 64
[10-Jun-2018 17:26:25 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:64
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(147 in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 64
[10-Jun-2018 17:26:26 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:64
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(147 in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 64
[10-Jun-2018 17:26:26 UTC] PHP Fatal error:  Uncaught exception 'Doctrine\DBAL\DBALException' with message 'Failed to connect to the database: An exception occured in driver: SQLSTATE[HY000] [2002] Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2 "No such file or directory")' in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php:64
Stack trace:
#0 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429): OC\DB\Connection->connect()
#1 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389): Doctrine\DBAL\Connection->getDatabasePlatformVersion()
#2 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(328): Doctrine\DBAL\Connection->detectDatabasePlatform()
#3 /home/MyUserRoot/public_html/owncloud/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(623): Doctrine\DBAL\Connection->getDatabasePlatform()
#4 /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php(147 in /home/MyUserRoot/public_html/owncloud/lib/private/DB/Connection.php on line 64
[25-Jun-2018 19:07:34 Asia/Hong_Kong] PHP Fatal error:  Cannot destroy active lambda function in [no active file] on line 0
[25-Jun-2018 19:11:01 Asia/Hong_Kong] PHP Fatal error:  Cannot destroy active lambda function in [no active file] on line 0
[25-Jun-2018 21:16:24 Asia/Hong_Kong] PHP Fatal error:  Cannot destroy active lambda function in [no active file] on line 0
[25-Jun-2018 23:18:34 Asia/Hong_Kong] PHP Fatal error:  Cannot destroy active lambda function in [no active file] on line 0
[26-Jun-2018 01:20:57 Asia/Hong_Kong] PHP Fatal error:  Cannot destroy active lambda function in [no active file] on line 0

ownCloud log (data/owncloud.log)

The ownCloud log  contains thousands of entries with user names and confidential file names that would be really difficult to edit out. Is there a way to extract a log that does not contain sensitive info?

Browser log

probably not relevant, the issue is not related to web browsing
2

Maybe the question is if this is just the “normal” usage of ownCloud and expected?

From what i know e.g. the sync client needs to check for changes on the ownCloud server in a specific interval. If there are three sync clients running then i think it could be possible that the clients keeps the CPU busy.

Additionally 10-20 % is probably quite vague and depends on the CPU power available from the hosting machine. If this is a low end vCPU 10-20% might sound high but actually isn’t that high.

3

I’ve thought about that, but it is not a reassuring assumption. It would mean 3 x 10 = 30 users exchanging just 10 files per day would totally saturate a (low-end) server. That would imply OC has terrible performance…

I would understand (maybe) if we were all on web clients and refreshing file lists all day long. But even then - a web server that can’t handle 30 users browsing text pages with zero multimedia?

Here is a download link the Cpanel Resource usage report if someone can assist me to understand if I have an issue:

PS: a thought for the developers who are familiar with the synch process:
if really the clients are sucking up CPU cycles refreshing file lists all day long, wouldn’t it make sense to implement some kind of file system transaction (journaling) log and just check the last transaction # when there is no change of state? That should keep background usage very low.

4

My ESXi host is reporting an Average of 400MHz CPU usage for my install. I’m running Version 9.1.8 though.

I have approx. 100 users synced via LDAP, up to 50 local clients setup with just over 2GB shared folder to sync (they aren’t all online at once) and a few users using local clients for document storage in the order of 4 to 200 GB (depending how much of a career they are dragging along behind them).

I don’t really have an answer for you, just helping to identify the expectations.

1 Like
5

Hi aclemence. Thanks for sharing your experience.

You obviously have a lot more load than me. I would like to suggest that a CPU frequency of 400MHz is highly unlikely these days (should be more like 2-3Ghz, unless your server is running a really low-power CPU??).
In any case, CPU frequency is not a useful measure of CPU load. It should be a % of total CPU bandwidth available (ie non-idle CPU cycles). Your hosting environment should include some stat software to tell you how much CPU bandwidth you are using up.

Thanks for trying to help.

6

Hi virgule,

The 400 MHz was the total ‘in-use’ frequency reported by VMWare ESXi. The actual CPU’s are E5-2630 @ 2.4GHz and I have assigned 4 vCPU to this host (since I have more than I need). After adding about 12 GB of data this afternoon (shared with 8 people, half of which would be syncing) I’m now idling along at about 1400 MHz (approx. 15% usage) with peaks up to 7700 MHz (80% usage). The app and CentOS base appear to handle spreading the load across all the cores quite well (unless that’s ESXi, who knows).

Percent of CPU alone as a metric is useless, unless you know the frequency of the CPU. You will want to understand from your provider how much CPU you are permitted to use. I provide my experience to help you estimate how much you need to ask for (my guest OS is purpose built so figures are accurate for ownCloud, it runs only ownCloud, the required stack and it’s MariaDB database, data stored on a NAS).

Regards,
aclemence

7

I don’t have access to similar data with my basic hosting plan, so I can’t compare. The only info I do have is the resource usage report I shared earlier.
Thanks for the follow up.

8

The desktop clients are also using the notifications endpoint to display notifications to the user of the desktop client.
If you want to reduce the load on your server - disable the notifications app.

9

Thanks DeepDiver. That was my first thought, and I went to Admin/Apps to disable the “notifications” app. The Webalizer report enclosed is 3 days after disabling the app (unless I’ve made a mistake? Should be clear in the various files I’ve enclosed)
Hosting company says no change to CPU usage. I will try now to disable “Update notification” app to see if there is an improvement.

There are also a lot of errors showing up in the log. I have no idea why. OC is working fine, and I have not tinkled with anything, apart from changing the login image once (overwritten by OC update it seems, back to default login). Could the errors trigger spurious CPU usage? And not show up in webalizer stats?

10

In addition to the permanent requests to the notifications endpoint there is also the cyclic checks of all desktop clients back to the server (20 secs afaik) to see if anything changed.

There is no way around this. Maybe the interval can be adopted but this will result in higher latency until updated are detected.

11

Ohhh, well that would go a long way to explain the number of hits! Obvious in hindsight, but I assumed there was a mechanism to say “no change”, without having to check anything.

I don’t mind playing around with latency if there is a hidden setting somewhere. If I could increase to 60 seconds, I’d divide the load by 2/3 and hosting co would be happy…as long as I don’t add more users.

12

@guruz is the etag ping time interval configurable? THX

13

Hey,

maybe there was a misunderstanding but i was referring to this interval directly in my initial post:

14

Hi Tom. No misunderstanding, just discussion going around in circle, my apologies :hugs:

I should have written “impact” rather than “totally saturate”. Hosting company is not saying I’m overloading the box, only that I’m abusing the shared service which is supposed to be a low-traffic website.
3 users x 10 files / day was slow traffic in my mind. I didn’t realize we’d be executing server code 13,000 times per day just checking that there is no change (3 users x 3 checks x 60 min x 24 hr)

15

https://doc.owncloud.org/desktop/2.4/advancedusage.html#configuration-file remotePollInterval

1 Like
16

That’s indeed not much at all.
Either your hoster is paranoid or your ownCloud for some reason is using more CPU than expected.
You’re still on PHP 5.6, do you have APC enabled?

17

Thanks for spending time on this thread Guruz. I will look up and test the “remotepollinterval”

My hoster would like me to upgrade to solve the “problem” :slight_smile: I can’t deny that the webalizer stats are not in my favor right now.

I do not believe that I have APC enabled and I do not know how to do that. I understand what is a cache, but I don’t know how to install/configure one on Linux. I do know that OC complains that “No memory cache has been configured…please configure a memcache if available”. My hoster says I do not have a memcache in my basic hosting package

I have a few PHP extensions that I could switch on, but I have no idea what the consequences will be:
apcu / memcache / memcached / xcache_3 (opcache is already = on).
I can also switch to PHP 7.0, 7.1 or 7.2 but again I have no idea what the consequences are for me.

I will speak with the hoster on this and revert. Thank you.

EDIT: might be 91,00 pings per day (13K x 7): we have 7 distinct Folder Sync connections. Old habit from Windows, of partitioning my disks / data to keep each index small.

18

I think this would be the one you’re looking for if i’m understanding this correctly:

https://doc.owncloud.org/server/latest/admin_manual/configuration/server/caching_configuration.html?highlight=memcache#small-private-home-server