Owncloud docker-compose: server unhealthy, cannot stop or even exec a shell inside container

Server
1

Steps to reproduce

  1. Debian stretch X64
  2. install owncloud (OWNCLOUD_VERSION=10.4.0) via docker-compose
  3. wait several weeks… (that’s the only thing I know for sure)

Expected behaviour

I should be able to connect the owncloud web gui and service should be up.

Actual behaviour

Owncloud after a while stops working, then it becomes unhealthy:

image
image1735×190 117 KB

I cannot seem to stop docker-compose or kill the affected container… docker-compose stop will timeout even after setting COMPOSE_HTTP_TIMEOUT=120

Server configuration

Operating system: Debian Stretch
Web server: provided by owncloud via docker, I’d say Apache
Database: owncloud docker (webhippie/mariadb)
PHP version: owncloud docker (no clue)
ownCloud version: 10.4.0

Updated from an older ownCloud or fresh install: updated from daily-stable10
Where did you install ownCloud from: Docker hub
Signing status (ownCloud 9.0 and above): ?

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results into https://gist.github.com/ and puth the link here.

cannot login!

The content of config/config.php:

Log in to the web-UI with an administrator account and click on
'admin' -> 'Generate Config Report' -> 'Download ownCloud config report'
This report includes the config.php settings, the list of activated apps
and other details in a well sanitized form.

or 

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

*cannot login!*

*ATTENTION:* Do not post your config.php file in public as is. Please use one of the above
methods whenever possible. Both, the generated reports from the web-ui and from occ config:list
consistently remove sensitive data. You still may want to review the report before sending.
If done manually then it is critical for your own privacy to dilligently
remove *all* host names, passwords, usernames, salts and other credentials before posting.
You should assume that attackers find such information and will use them against your systems.

List of activated apps:
cannot login!

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your ownCloud installation folder.

cannot login!
Are you using external storage, if yes which one: local/smb/sftp/…
local

Are you using encryption: yes/no
no

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…
no

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

Attaching to smowncloud_owncloud_1
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:47:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:48:15 +0000] "GET /status.php HTTP/1.1" 200 1066 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:48:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:48:45 +0000] "GET /status.php HTTP/1.1" 200 1062 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:48:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:49:16 +0000] "GET /status.php HTTP/1.1" 200 1058 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:49:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:49:46 +0000] "GET /status.php HTTP/1.1" 200 1058 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:49:47 +0000] "GET /ocs/v2.php/apps/notifications/api/v1/notifications?format=json HTTP/1.0" 200 1122 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:49:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:50:16 +0000] "GET /status.php HTTP/1.1" 200 1068 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:50:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:50:47 +0000] "GET /status.php HTTP/1.1" 200 1058 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:50:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:51:17 +0000] "GET /status.php HTTP/1.1" 200 1060 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:51:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:51:47 +0000] "GET /status.php HTTP/1.1" 200 1068 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:51:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:52:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:52:18 +0000] "GET /status.php HTTP/1.1" 200 1068 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:52:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:52:48 +0000] "GET /status.php HTTP/1.1" 200 1068 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:53:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:53:18 +0000] "GET /status.php HTTP/1.1" 200 1062 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:53:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:53:48 +0000] "GET /status.php HTTP/1.1" 200 1062 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:54:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:54:18 +0000] "GET /status.php HTTP/1.1" 200 1060 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:54:47 +0000] "GET /ocs/v2.php/apps/notifications/api/v1/notifications?format=json HTTP/1.0" 200 1122 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:54:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:54:49 +0000] "GET /status.php HTTP/1.1" 200 1058 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:55:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:55:19 +0000] "GET /status.php HTTP/1.1" 200 1064 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:55:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:55:49 +0000] "GET /status.php HTTP/1.1" 200 1060 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:56:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:56:19 +0000] "GET /status.php HTTP/1.1" 200 1068 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:56:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:56:50 +0000] "GET /status.php HTTP/1.1" 200 1070 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:57:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:57:20 +0000] "GET /status.php HTTP/1.1" 200 1064 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:57:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:57:50 +0000] "GET /status.php HTTP/1.1" 200 1066 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:58:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:58:20 +0000] "GET /status.php HTTP/1.1" 200 1060 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:58:47 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:58:51 +0000] "GET /status.php HTTP/1.1" 200 1060 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m w.x.y.z - - [17/May/2020:04:59:17 +0000] "PROPFIND /remote.php/dav/files/simmessa/ HTTP/1.0" 207 893 "-" "Mozilla/5.0 (Windows) mirall/2.6.1 (build 12992)"
e[36mowncloud_1  |e[0m 127.0.0.1 - - [17/May/2020:04:59:21 +0000] "GET /status.php HTTP/1.1" 200 1062 "-" "curl/7.65.3"
e[36mowncloud_1  |e[0m [Sun May 17 05:57:42.660181 2020] [mpm_prefork:error] [pid 198] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting
[...]
2
  1. You should make sure you have a proper backup.
  2. Have you checked system logs, for a possible oom event (Out of Memory)?
  3. Have you tried restarting the docker service, or restarting the machine?
1 Like
3

Hello there!

Thanks for your assistance.

  1. Well I will backup owncloud data before attempting anything too extreme, I guess.

  2. Docker daemon logs on file seem to be non-existent, but what I’ve been able to see via systemctl or journalctl -u docker do not show anything strange.

  3. I have tried restarting the docker service but the situation didn’t change, I’m a bit reluctant to restart the server since it’s handling some other things but I will try this and let you know.

I’ll let you know as soon as I have some updates.

Simone.

4

Well in the /var/log/daemon.log file I have found thousands of occurrences of this log:

May 31 06:26:22 blackbox dockerd[770]: time="2020-05-31T06:26:22.907861987+02:00" level=warning msg="Health check for container de1f45020ce3a4a8262f74918b5fc525fee044fb030ce31dc6620810f72cb87e error: context deadline exceeded"

That is my owncloud server container ID, but from what I can understand this is only a warning for the healthcheck failing, I don’t believe we can gather much detail from here.

Simone.

5

Nothing, absolutely nothing, changed after a reboot, does anybody know how I could better diagnose what’s wrong with my Owncloud install ? Thanks.

6

Try to disable the healthcheck for now to see if the situation improves.

You might want to check the “context deadline exceeded” error in docker, if you haven’t checked it yet. I don’t know how can you fix it.

You can also check with a newer docker version (if available), or with a different linux distribution.

3 Likes