Background job stuck - Notifications not send anymore

Steps to reproduce

1.sudo -u www-data php /var/www/owncloud/occ background:queue:status
2.sudo -u www-data php /var/www/owncloud/occ activity:send-emails
3.

Expected behaviour

Notification send

Actual behaviour

Notification not working anymore. The test message work fine. The occ background:queue:status show strange date , maybe the cause? The origin of the problem is i check because i don’t see any notification since a lot of time. When i tried to launch the cron manually with : sudo -u www-data php /var/www/owncloud/occ activity:send-emails

I received : sudo: unable to resolve host PTCDEB-OWNCLOUD
It’s because i changed the name of the server several weeks… I corrected the /etc/hosts with good name and ip, now i don’t have any errors when i use the command but no email was send…

| 11 | OCA\UpdateNotification\Notification\BackgroundJob | 2019-12-02T12:06:02+00:00 | |
| 12 | OC\Authentication\Token\DefaultTokenCleanupJob | 2019-12-02T21:00:02+00:00 | |
| 90 | OCA\Activity\BackgroundJob\EmailNotification | 2019-12-02T20:47:03+00:00 | |
| 91 | OCA\Activity\BackgroundJob\ExpireActivities | 2019-12-02T17:47:02+00:00 | |
| 97 | OCA\DAV\BackgroundJob\CleanProperties | 1970-01-01T00:00:00+00:00 | |
| 98 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Trashbin\Command\Expire”:1:{s:39:"\OCA\Files_Trashbin\Command\Expire\user";s:3:“jcs”;} |
| 99 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Versions\Command\Expire”:2:{s:43:"\OCA\Files_Versions\Command\Expire\fileName";s:63:“Indépendant Complémenaire/Factures de Sorties/2019/F_0139.pdf”;s:39:"\OCA\Files_Versions\Command\Expire\user";s:3:“jcs”;} |
| 100 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Versions\Command\Expire”:2:{s:43:"\OCA\Files_Versions\Command\Expire\fileName";s:58:“Indépendant Complémenaire/Factures de Sorties/2019/0.pdf”;s:39:"\OCA\Files_Versions\Command\Expire\user";s:3:“jcs”;} |
| 101 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Versions\Command\Expire”:2:{s:43:"\OCA\Files_Versions\Command\Expire\fileName";s:63:“Indépendant Complémenaire/Factures de Sorties/2019/F_0134.pdf”;s:39:"\OCA\Files_Versions\Command\Expire\user";s:3:“jcs”;} |
| 102 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Versions\Command\Expire”:2:{s:43:"\OCA\Files_Versions\Command\Expire\fileName";s:64:“Indépendant Complémenaire/Factures de Sorties/2019/F_0133.docx”;s:39:"\OCA\Files_Versions\Command\Expire\user";s:3:“jcs”;} |
| 103 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Versions\Command\Expire”:2:{s:43:"\OCA\Files_Versions\Command\Expire\fileName";s:63:“Indépendant Complémenaire/Factures de Sorties/2019/F_0133.pdf”;s:39:"\OCA\Files_Versions\Command\Expire\user";s:3:“jcs”;} |
| 104 | OC\Command\CommandJob | 1970-01-01T00:00:00+00:00 | O:33:“OCA\Files_Versions\Command\Expire”:2:{s:43:"\OCA\Files_Versions\Command\Expire\fileName";s:64:“Indépendant Complémenaire/Factures de Sorties/2019/F_0134.docx”;s:39:"\OCA\Files_Versions\Command\Expire\user";s:3:“jcs”;} |

Server configuration

Operating system:
Debian 9
Web server:
Apache 2.4.25
Database:
MySQL
PHP version:
7.0.33
ownCloud version: (see ownCloud admin page)
10.3.1
Updated from an older ownCloud or fresh install:
updated
Where did you install ownCloud from:
package download from owncloud.org
Signing status (ownCloud 9.0 and above):
?

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results into https://gist.github.com/ and puth the link here.
No errors have been found.


**The content of config/config.php:**
<?php
$CONFIG = array (
  'instanceid' => 'ocsnwt6bevmn',
  'passwordsalt' => 'Smbg6vh0AvwYI9OKEPHz1KUU0S+1zF',
  'secret' => 'nSQNoxU5B0lov96bRYOyer9yO6ZQ3klg73VlvLYvltn2HZ9H',
  'trusted_domains' =>
  array (
    0 => '192.168.0.122',
    1 => XXX',
    2 => 'XXX',
  ),
  'datadirectory' => '/var/www/owncloud/data',
  'overwrite.cli.url' => 'https://192.168.122/owncloud',
  'dbtype' => 'mysql',
  'version' => '10.3.1.1',
  'versions_retention_obligation' => 'auto',
  'dbname' => 'owncloud',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'owncloud',
  'dbpassword' => 'XFY6RvpL6zk2Fc',
  'logtimezone' => 'Europe/Brussels',
  'installed' => true,
  'force ssl' => true,
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'mail_domain' => 'XXXc.be',
  'mail_from_address' => 'noreply',
  'mail_smtpmode' => 'smtp',
  'filelocking.enabled' => true,
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' =>
  array (
    'host' => 'localhost',
    'port' => 6379,
    'timeout' => 0.0,
    'password' => '',
  ),
  'mail_smtpauthtype' => 'LOGIN',
  'mail_smtpport' => '587',
  'mail_smtphost' => 'ssl0.ovh.net',
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'noreply@XXX.be',
  'mail_smtppassword' => 'XXX',
  'loglevel' => 2,
  'maintenance' => false,
  'theme' => '',
  'updater.secret' => '$2y$10$wUHWqKzuXXXXXXXXXXXDYFuO5XoQ0gQcl8wkoLTX.',
);


Log in to the web-UI with an administrator account and click on
'admin' -> 'Generate Config Report' -> 'Download ownCloud config report'
This report includes the config.php settings, the list of activated apps
and other details in a well sanitized form.

or 

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

*ATTENTION:* Do not post your config.php file in public as is. Please use one of the above
methods whenever possible. Both, the generated reports from the web-ui and from occ config:list
consistently remove sensitive data. You still may want to review the report before sending.
If done manually then it is critical for your own privacy to dilligently
remove *all* host names, passwords, usernames, salts and other credentials before posting.
You should assume that attackers find such information and will use them against your systems.

List of activated apps:
Enabled:

  • activity: 2.5.2
  • comments: 0.3.0
  • configreport: 0.2.0
  • dav: 0.5.0
  • external: 1.4.0
  • federatedfilesharing: 0.5.0
  • federation: 0.1.0
  • files: 1.5.2
  • files_external: 0.7.1
  • files_external_dropbox: 1.2.0
  • files_mediaviewer: 1.0.1
  • files_pdfviewer: 0.11.1
  • files_sharing: 0.12.0
  • files_trashbin: 0.9.1
  • files_versions: 1.3.0
  • firstrunwizard: 1.2.0
  • guests: 0.8.2
  • market: 0.5.0
  • notifications: 0.5.0
  • provisioning_api: 0.5.0
  • systemtags: 0.3.0
  • updatenotification: 0.2.1
    Disabled:
  • encryption
  • files_external_ftp
  • user_external
If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your ownCloud installation folder.

Are you using external storage, if yes which one: local/smb/sftp/…
YES SMB
Are you using encryption: yes/no
NO
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…
NO

LDAP configuration (delete this part if not used)

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your ownCloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
Chrome
Operating system:
Windows 10

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

Insert your ownCloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log 
c) ...

Can you run the command with -vvv.

Also please paste the output of:

SELECT * FROM oc_jobs;
1 Like

Hello @eneubauer,

root@PTCDEB-OWNCLOUD:~# sudo -u www-data php /var/www/owncloud/occ activity:send-emails -vvv
    0 [>---------------------------] < 1 sec 16.0 MiBroot@PTCDEB-OWNCLOUD:~#  

Is it what you asked?

For the second command i don’t post all but it’s very long

| 223 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:61:\"Persos\/Documents\/Factures - Garantie\/Nouveau document 3_1.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                                |          0 |   1594144190 |           0 |                 -1 |
| 224 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:57:\"Persos\/Documents\/Factures - Garantie\/Gardena_devidoir.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                                    |          0 |   1594144190 |           0 |                 -1 |
| 225 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:74:\"Persos\/Documents\/Factures - Garantie\/Facture_Bricoplanit_Portearri\u00e8re.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                               |          0 |   1594144190 |           0 |                 -1 |
| 226 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:59:\"\/Persos\/Documents\/Factures - Garantie\/Samsung_S3_ticket.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                                 |          0 |   1594144191 |           0 |                 -1 |
| 227 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:60:\"\/Persos\/Documents\/Factures - Garantie\/Samsung_S3_facture.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                                |          0 |   1594144191 |           0 |                 -1 |
| 228 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:64:\"\/Persos\/Documents\/Factures - Garantie\/garantie_frigo_samsung.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                            |          0 |   1594144191 |           0 |                 -1 |
| 229 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:64:\"\/Persos\/Documents\/Factures - Garantie\/garantie_lavelinge_aeg.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                            |          0 |   1594144192 |           0 |                 -1 |
| 230 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:65:\"\/Persos\/Documents\/Factures - Garantie\/garantie_lave-linge_aeg.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                           |          0 |   1594144192 |           0 |                 -1 |
| 231 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:46:\"Protech-PC\/Factures de Sorties\/2020\/F-0155.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                                               |          0 |   1594213887 |           0 |                 -1 |
| 232 | OC\Command\CommandJob                             | "O:33:\"OCA\\Files_Versions\\Command\\Expire\":2:{s:43:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000fileName\";s:46:\"Protech-PC\/Factures de Sorties\/2020\/F-0154.pdf\";s:39:\"\u0000OCA\\Files_Versions\\Command\\Expire\u0000user\";s:3:\"jcs\";}"                                                               |          0 |   1594213959 |           0 |                 -1 |
+-----+---------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------+--------------+-------------+

Thank you for help

Edit : i exported the result in the file : result.txt (44.8 KB)

How is your cron job configured? Can you confirm it is actually working.

If you see a line in your system log similar to the following right after the cron execution line then there is error output that has been discarded.

[…] CROND […] no MTA installed output discarded.

MTA stands for Mail Transfer Agent. Cron is automatically configured to use an MTA to send cron output to the email address of the root user. If you have an MTA installed, you might also get the messages for root delivered locally. Then you would see a message in the CLI when logging in about the user root having a bunch of emails.

If you get this, you’ll have to debug the cron job, which can be quite difficult. In most cases you can find the error by running it manually or by redirecting all cron output to a temporary log file.

If manual execution or the cron log don’t help you could also export your cron environment and import it into a local session and then rerun the cron manually.

Your problem is most likely though that after the upgrade to 10.3.1 the cron job wasn’t adjusted from cron.php execution to occ system:cron execution.

2 Likes

Hello @eneubauer,

here the crontab www-data

*  *  *  *  * /usr/bin/php -f /var/www/owncloud/cron.php
*/5 * * * * /usr/bin/php /var/www/owncloud/occ files:scan --all
*/5 * * * * /usr/bin/php /var/www/owncloud/occ activity:send-emails 

The cron works fine until i see the problem… All health check passed…
Do you think if i make the upgrade to the last version, the problem could be fixed ? (10.3.2) ?

The latest version is 10.4.1, it might very well be fixed, make sure to change your cronjob, like I described.

1 Like

Hi @eneubauer,

i changed crontab to
* * * * * /usr/bin/php /var/www/owncloud/occ system:cron
*/5 * * * * /usr/bin/php /var/www/owncloud/occ files:scan --all
*/5 * * * * /usr/bin/php /var/www/owncloud/occ activity:send-emails

I executed manually
sudo -u www-data /usr/bin/php /var/www/owncloud/occ system:cron

I receive some emails and now the command
sudo -u www-data php /var/www/owncloud/occ background:queue:status

return
root@PTCDEB-OWNCLOUD:/home/linus# sudo -u www-data php /var/www/owncloud/occ background:queue:status
±-------±--------------------------------------------------±--------------------------±--------------+
| Job ID | Job | Last Run | Job Arguments |
±-------±--------------------------------------------------±--------------------------±--------------+
| 1 | OCA\Files\BackgroundJob\ScanFiles | 2020-07-21T09:11:02+00:00 | |
| 2 | OCA\Files\BackgroundJob\DeleteOrphanedItems | 2020-07-21T09:11:02+00:00 | |
| 3 | OCA\Files\BackgroundJob\CleanupFileLocks | 2020-07-21T09:11:02+00:00 | |
| 4 | OCA\DAV\CardDAV\SyncJob | 2020-07-21T09:11:03+00:00 | |
| 5 | OCA\Federation\SyncJob | 2020-07-21T09:11:03+00:00 | |
| 6 | OCA\Files_Sharing\DeleteOrphanedSharesJob | 2020-07-21T09:11:02+00:00 | |
| 7 | OCA\Files_Sharing\ExpireSharesJob | 2020-07-21T09:11:02+00:00 | |
| 8 | OCA\Files_Trashbin\BackgroundJob\ExpireTrash | 2020-07-21T09:11:02+00:00 | |
| 9 | OCA\Files_Versions\BackgroundJob\ExpireVersions | 2020-07-21T09:11:02+00:00 | |
| 10 | OCA\Market\CheckUpdateBackgroundJob | 2020-07-21T09:11:03+00:00 | |
| 11 | OCA\UpdateNotification\Notification\BackgroundJob | 2020-07-21T09:11:06+00:00 | |
| 12 | OC\Authentication\Token\DefaultTokenCleanupJob | 2020-07-21T09:12:02+00:00 | |
| 90 | OCA\Activity\BackgroundJob\EmailNotification | 2020-07-21T09:11:06+00:00 | |
| 91 | OCA\Activity\BackgroundJob\ExpireActivities | 2020-07-21T09:11:07+00:00 | |
| 97 | OCA\DAV\BackgroundJob\CleanProperties | 2020-07-21T09:11:07+00:00 | |
| 132 | OCA\UpdateNotification\ResetTokenBackgroundJob | 2020-07-21T09:11:10+00:00 | |
±-------±--------------------------------------------------±--------------------------±--------------+

Seems ok now… I don’t receive all old notification but i received update package notification.
Just a litle think the link inside this email notification show the local IP and not the external address, i will check why

 	Hello,
See https://192.168.122/owncloud/index.php/apps/market/#/app/activity on ownCloud for more information
 
 	--
ownCloud - A safe home for all your data
https://owncloud.org 

I will make a test , download a file and check if the notification is send after i downloaded it.
Thank you for help!

All works ! Thank you @eneubauer

2 Likes

Hey,

i think this is because you have ownCloud configured to do so:

I changed it, thank you !

1 Like