Most files deleted after files:scan command

Hi,

my issues seems a bit odd, as I can not really fill in the form below (I will try, anyway, later). But here is my general description of what just happened. Please excuse my not strictly following the form. By this description I hope to describe the history. Maybe one of you might get a better understanding of any background problem:

I run OwnCloud Server v9.1.6 on a RaspberryPi v3. With just a small USB Memory stick of 16GB size. Only 2 active users.

In the End of 2017, I have had some problems coming from the USB drive being full and thus blocking the entire communication. But this was solved by removing most of the files from the client computers and thus stopping data pressing onto the server. Only one oddity remained from there: the TOTAL FILE SIZE still shows about 15 GB of used space, although the real used space must be less than 7 GB (record from all client computers while no exclusion of folders for synchronization).

So, yesterday, I recognized that all my newly saved files (to be synchronized by the clients of different versions on different Windows computers) were not transferred to the server and thus didn't appear on the other connected client computers. I looked into the synchronization log of the clients and saw a "403 Forbidden" error, detailling something with "Sabre\DAV\Exception\Forbidden". Reading myself through the internet, I found a link to an external thread, which claimed to have solved a similar situation by issuing a specific command directly on the server:

sudo -uwww-data php occ files:scan --all

This was said to have the effect of re-indexing the file system, which sounded appropriate to me. Then, I saw a

Scanning files for 2 users...

After 1 hour and 25 minutes, the procedure was finished. On the server's command line it thrrew a couple of errors concerning not my but the other user's folders, which was ok for me at this time:

Exception during scan: opendir(/media/usb1/ownCloud/data/XXXXX/thumbnails/771): failed to open dir: Permission denied
#0 [internal function]: OCA\Files\Command\Scan->exceptionErrorHandler(2, 'opendir(/media/...', '/var/www/ownclo...', 116, Array)
#1 /var/www/owncloud/lib/private/Files/Storage/Local.php(116): opendir('/media/usb1/own...')
#2 /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php(96): OC\Files\Storage\Local->opendir('thumbnails/771')
#3 /var/www/owncloud/lib/private/Files/Storage/Wrapper/Wrapper.php(96): OC\Files\Storage\Wrapper\Wrapper->opendir('thumbnails/771')
#4 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(346): OC\Files\Storage\Wrapper\Wrapper->opendir('thumbnails/771')
#5 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(397): OC\Files\Cache\Scanner->getNewChildren('thumbnails/771')
#6 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(377): OC\Files\Cache\Scanner->handleChildren('thumbnails/771', true, 3, '37571', true, 0)
#7 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('thumbnails/771', true, 3, '37571', true)
#8 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(380): OC\Files\Cache\Scanner->scanChildren('thumbnails', true, 3, '37336', true)
#9 /var/www/owncloud/lib/private/Files/Cache/Scanner.php(311): OC\Files\Cache\Scanner->scanChildren('', true, 3, 2080, true)
#10 /var/www/owncloud/lib/private/Files/Utils/Scanner.php(234): OC\Files\Cache\Scanner->scan('', true, 3)
#11 /var/www/owncloud/apps/files/lib/Command/Scan.php(159): OC\Files\Utils\Scanner->scan('/Hanna')
#12 /var/www/owncloud/apps/files/lib/Command/Scan.php(228): OCA\Files\Command\Scan->scanFiles('Hanna', '/Hanna', false, Object(Symfony\Component\Console\Output\ConsoleOutput), false)
#13 /var/www/owncloud/3rdparty/symfony/console/Command/Command.php(259): OCA\Files\Command\Scan->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /var/www/owncloud/core/Command/Base.php(158): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /var/www/owncloud/3rdparty/symfony/console/Application.php(844): OC\Core\Command\Base->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#16 /var/www/owncloud/3rdparty/symfony/console/Application.php(192): Symfony\Component\Console\Application->doRunCommand(Object(OCA\Files\Command\Scan), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#17 /var/www/owncloud/3rdparty/symfony/console/Application.php(123): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /var/www/owncloud/lib/private/Console/Application.php(146): Symfony\Component\Console\Application->run(NULL, NULL)
#19 /var/www/owncloud/console.php(102): OC\Console\Application->run()
#20 /var/www/owncloud/occ(11): require_once('/var/www/ownclo...')
#21 {main}

and

+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 3218    | 25112 | 01:25:03     |
+---------+-------+--------------+

Now, I recognized client activity and I thought the synchronization would now transfer the files held back by the error message, previously. But the synchronization DELETED about half the files from the server. I checked: Via the server's WEB interface, most of the files seem to be gone without leaving any information. No file recovery possible. On the clients... the same picture many completely removed folders.

Is there a way to get back the lost files?

Is there a way to stabilize my installation?

Steps to reproduce

?

Expected behaviour

give me my files back! :wink:

Actual behaviour

now, everything works fine again. The clients show correct synchronization and the remaining files are synched successfully.

Server configuration

Operating system: Raspbian

Web server:

Database:

PHP version:

ownCloud version: 9.1.6

Updated from an older ownCloud or fresh install: -

Where did you install ownCloud from: -

Signing status (ownCloud 9.0 and above):

-- has been disabled --

The content of config/config.php:

<?php
$CONFIG = array (
  'instanceid' => 'ocuxl9zqninv',
  'passwordsalt' => 'XXXXXXXX',
  'secret' => 'YYYYYYYY',
  'trusted_domains' =>
  array (
    0 => '192.168.1.33',
    1 => '192.168.2.169',
  ),
  'datadirectory' => '/media/usb1/ownCloud/data',
  'overwrite.cli.url' => 'https://192.168.1.46',
  'dbtype' => 'sqlite3',
  'version' => '9.1.6.2',
  'installed' => true,
  'mail_from_address' => 'XXXXXXXX',
  'mail_smtpmode' => 'smtp',
  'mail_domain' => 'XXXXXXX',
  'mail_smtphost' => 'XXXXXXXX',
  'mail_smtpport' => '587',
  'mail_smtpsecure' => 'tls',
  'mail_smtpauthtype' => 'LOGIN',
  'maintenance' => false,
  'theme' => '',
  'loglevel' => 2,
  'trashbin_retention_obligation' => 'auto',
  'integrity.check.disabled' => true,
  'mail_smtpauth' => 1,
  'mail_smtpname' => 'XXXXXXX',
  'mail_smtppassword' => 'XXXXXXXX',
);

Log in to the web-UI with an administrator account and click on
'admin' -> 'Generate Config Report' -> 'Download ownCloud config report'
This report includes the config.php settings, the list of activated apps
and other details in a well sanitized form.

-- empty --

or

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

[16:06:40] $ sudo -u www-data php occ config:list system
{
    "system": {
        "instanceid": "ocuxl9zqninv",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "192.168.1.33",
            "192.168.2.169"
        ],
        "datadirectory": "\/media\/usb1\/ownCloud\/data",
        "overwrite.cli.url": "https:\/\/192.168.1.46",
        "dbtype": "sqlite3",
        "version": "9.1.6.2",
        "installed": true,
        "mail_from_address": "XXXXXXXX",
        "mail_smtpmode": "smtp",
        "mail_domain": "XXXXXXXX",
        "mail_smtphost": "XXXXXXXX",
        "mail_smtpport": "587",
        "mail_smtpsecure": "tls",
        "mail_smtpauthtype": "LOGIN",
        "maintenance": false,
        "theme": "",
        "loglevel": 2,
        "trashbin_retention_obligation": "auto",
        "integrity.check.disabled": true,
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***"
    }
}

List of activated apps: calendar

Enabled:
  - activity: 2.3.2
  - calendar: true
  - comments: 0.3.0
  - configreport: 0.1.1
  - dav: 0.2.7
  - federatedfilesharing: 0.3.0
  - federation: 0.1.0
  - files: 1.5.1
  - files_pdfviewer: 0.8.1
  - files_sharing: 0.10.0
  - files_texteditor: 2.1
  - files_trashbin: 0.9.0
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 15.0.0
  - notifications: 0.3.0
  - provisioning_api: 0.5.0
  - systemtags: 0.3.0
  - templateeditor: 0.1
  - updatenotification: 0.2.1
Disabled:
  - encryption
  - external
  - files_antivirus
  - files_external
  - user_external
  - user_ldap

Are you using external storage, if yes which one: local/smb/sftp/... USB

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

LDAP configuration (delete this part if not used)

  [Symfony\Component\Console\Exception\CommandNotFoundException]
  There are no commands defined in the "ldap" namespace.

Client configuration

Browser: Firefox

Operating system: Windows 7

Logs

Web server error log

2018/04/06 12:55:51 [error] 2237#0: *57501 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.34, server: 192.168.1.46, request: "MKCOL /remote.php/webdav/XXXXXXXX HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "192.168.1.33"
2018/04/06 13:33:03 [error] 2237#0: *57902 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 192.168.1.34, server: 192.168.1.46, request: "MKCOL /remote.php/webdav/XXXXXXXX HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "192.168.1.33"
2018/04/06 15:21:52 [error] 2237#0: *59208 directory index of "/var/www/owncloud/core/doc/user/_static/" is forbidden, client: 192.168.1.34, server: 192.168.1.46, request: "GET /core/doc/user/_static/ HTTP/1.1", host: "192.168.1.33", referrer: "https://192.168.1.33/core/doc/user/index.html"
2018/04/06 16:09:46 [error] 2237#0: *59488 open() "/var/www/owncloud404" failed (2: No such file or directory), client: 192.168.1.34, server: 192.168.1.46, request: "GET /favicon.ico HTTP/1.1", host: "192.168.1.33"
2018/04/06 16:09:46 [error] 2237#0: *59488 open() "/var/www/owncloud404" failed (2: No such file or directory), client: 192.168.1.34, server: 192.168.1.46, request: "GET /favicon.ico HTTP/1.1", host: "192.168.1.33"

ownCloud log (data/owncloud.log)

Insert your ownCloud log here

Browser log

too long...

{"reqId":"VCKFX2SStVxxMLSJnD2Z","remoteAddr":"192.168.1.34","app":"PHP","message":"Undefined index: You don\u2019t have permission to upload or create files here at \/var\/www\/owncloud\/lib\/private\/legacy\/l10n\/string.php#65","level":3,"time":"2018-04-06T05:12:08+00:00","method":"GET","url":"\/index.php\/apps\/files\/?dir=\/","user":"XXXXXXXX"}

{"reqId":"V+xvlFt4gEBXJpC4KslG","remoteAddr":"192.168.1.34","app":"PHP","message":"PHP Startup: apc.shm_size now uses M\/G suffixes, please update your ini files at Unknown#0","level":3,"time":"2018-04-06T07:30:09+00:00","method":"PROPFIND","url":"\/remote.php\/webdav\/","user":"XXXXXXXX"}

{"reqId":"\/t+rWXSUrmDMkddu9oC4","remoteAddr":"192.168.1.34","app":"PHP","message":"fopen(\/media\/usb1\/ownCloud\/data\/nnako\/thumbnails\/109918\/44-44.png): failed to open stream: No such file or directory at \/var\/www\/owncloud\/lib\/private\/Files\/Storage\/Local.php#278","level":3,"time":"2018-04-06T12:37:13+00:00","method":"GET","url":"\/index.php\/apps\/files_trashbin\/ajax\/preview.php?file=%2FXXXXXXXX.MP3.d1520882055&c=1520882055000","user":"XXXXXXXX"}

Maybe this version was too outdated for this job? At https://owncloud.org/changelog/server/v9/ version 9.1.8 is listed as the latest available version.

What do you see under "deleted files"? Check both users. If there is nothing I don't see an option.
In regards of total file size ... maybe you don't count the versions/deleted files in?