StorageNotAvailableException intermittent failures

Edorta · February 13, 2023, 1:16pm

Steps to reproduce

External storage SMB (Netgear ReadyNAS)
Add external storage access on owncloud (SMB/CIFS) using: host , share, username, password, remote subfolder. Link check green.
Create a Cron task to update share files: “*/10 * * * * occ files:scan --all”

Expected behaviour

Correct operation to shared folders of the NAS

Actual behaviour

I have installed two univention servers with the owncloud application. Client files are stored on a Netgear ReadyNAS.
I have detected that the owncloud windows clients (Client version 2.7.6) show errors: “error transfering https_://IP/owncloud/remote.php/dav/files/USER/FOLDER/ - server replied:Service unavailble”
On server logs show:
“Storage smb::USER@IP_NAS//FOLDER//DIRECTORY/ not available”}

When this failure occurs, owncloud clients do not access their folders but after a few seconds they reconnect. It is a temporary error.
The same thing happens from the web browser, access to the folder fails when clicking, but if you try again a little later it works.

Server configuration

Operating system:
Univention 4.4
Web server:
apache 2.4.25
Database:
MySQL
PHP version:
php 7.0.3
ownCloud version: (see ownCloud admin page)
10.4.1.3 and 10.6.0.5
Updated from an older ownCloud or fresh install:
Fresh install
Where did you install ownCloud from:
From Univention appcenter
Signing status (ownCloud 9.0 and above):

Login as admin user into your ownCloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results into https://gist.github.com/ and puth the link here.

No errors have been found.

The content of config/config.php:

0 => 'localhost',

),
‘datadirectory’ => ‘/var/lib/univention-appcenter/apps/owncloud/data/files’,
‘dbtype’ => ‘mysql’,
‘dbhost’ => ‘—’,
‘dbname’ => ‘----’,
‘dbuser’ => ‘----’,
‘dbpassword’ => ‘*REMOVED SENSITIVE VALUE’,
‘dbtableprefix’ => ‘oc_’,
‘log_type’ => ‘owncloud’,
‘supportedDatabases’ =>
array (
0 => ‘sqlite’,
1 => ‘mysql’,
2 => ‘pgsql’,
),
‘upgrade.disable-web’ => true,
‘default_language’ => ‘en’,
‘overwrite.cli.url’ => ‘http://*REMOVED SENSITIVE VALUE’,
‘htaccess.RewriteBase’ => ‘/owncloud’,
‘logfile’ => ‘/var/lib/univention-appcenter/apps/owncloud/data/files/owncloud.log’,
‘loglevel’ => ‘3’,
‘memcache.local’ => ‘\OC\Memcache\APCu’,
‘mysql.utf8mb4’ => true,
‘filelocking.enabled’ => true,
‘memcache.distributed’ => ‘\OC\Memcache\Redis’,
‘memcache.locking’ => ‘\OC\Memcache\Redis’,
‘redis’ =>
array (
‘host’ => ‘redis’,
‘port’ => ‘6379’,
),
‘passwordsalt’ => ‘*REMOVED SENSITIVE VALUE’,
‘secret’ => ‘*REMOVED SENSITIVE VALUE’,
‘version’ => ‘10.6.0.5’,
‘logtimezone’ => ‘UTC’,
‘installed’ => true,
‘instanceid’ => ‘*REMOVED SENSITIVE VALUE’,
‘ldapIgnoreNamingRules’ => false,
‘log_rotate_size’ => 104857600,
‘onlyoffice’ =>
array (
‘verify_peer_off’ => true,
),
‘maintenance’ => false,
‘log_rotate_size’ => 104857600,
‘onlyoffice’ => array (‘verify_peer_off’ => TRUE),

Log in to the web-UI with an administrator account and click on
‘admin’ → ‘Generate Config Report’ → ‘Download ownCloud config report’
This report includes the config.php settings, the list of activated apps
and other details in a well sanitized form.

or

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your ownCloud installation folder

ATTENTION: Do not post your config.php file in public as is. Please use one of the above
methods whenever possible. Both, the generated reports from the web-ui and from occ config:list
consistently remove sensitive data. You still may want to review the report before sending.
If done manually then it is critical for your own privacy to dilligently
remove all host names, passwords, usernames, salts and other credentials before posting.
You should assume that attackers find such information and will use them against your systems.


**List of activated apps:**

No apps

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your ownCloud installation folder.

Are you using external storage, if yes which one: local/smb/sftp/…
Yes
Are you using encryption: yes/no
NAS1: SMB3 and NAS2: no
Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…

LDAP configuration (delete this part if not used)

With access to your command line run e.g.:
sudo -u www-data php occ ldap:show-config
from within your ownCloud installation folder

Without access to your command line download the data/owncloud.db to your local
computer or access your SQL server remotely and run the select query:
SELECT * FROM `oc_appconfig` WHERE `appid` = 'user_ldap';


Eventually replace sensitive data as the name/IP-address of your LDAP server or groups.

Client configuration

Browser:
Chrome
Operating system:
windowsxp, windows 7, windows 10

Logs

Web server error log

Insert your webserver log here

ownCloud log (data/owncloud.log)

Insert your ownCloud log here
{"reqId":"tCmICnJbHfqaisdCXZdf","level":3,"time":"2023-02-13T08:34:29+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"--","url":"--","message":"Storage smb::user@LOCAL IP NAS\/\/shared_folder\/\/DATOS\/AURTENETXEA\/L01-AURTENETXEA\/ not available"}
{"reqId":"tCmICnJbHfqaisdCXZdf","level":3,"time":"2023-02-13T08:34:29+00:00","remoteAddr":"","user":"--","app":"no app in context","method":"--","url":"--","message":"Exception: {\"Exception\":\"OCP\\\\Files\\\\StorageNotAvailableException\",\"Message\":\"\",\"Code\":1,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Availability.php(457): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Availability->checkAvailability()\\n#1 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Encryption.php(176): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Availability->getMetaData()\\n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Checksum.php(207): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Encryption->getMetaData()\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Wrapper.php(587): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Checksum->getMetaData()\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(115): OC\\\\Files\\\\Storage\\\\Wrapper\\\\Wrapper->getMetaData()\\n#5 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(152): OC\\\\Files\\\\Cache\\\\Scanner->getData()\\n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Cache\\\/Scanner.php(329): OC\\\\Files\\\\Cache\\\\Scanner->scanFile()\\n#7 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Utils\\\/Scanner.php(246): OC\\\\Files\\\\Cache\\\\Scanner->scan()\\n#8 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files\\\/lib\\\/Command\\\/Scan.php(251): OC\\\\Files\\\\Utils\\\\Scanner->scan()\\n#9 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files\\\/lib\\\/Command\\\/Scan.php(400): OCA\\\\Files\\\\Command\\\\Scan->scanFiles()\\n#10 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files\\\/lib\\\/Command\\\/Scan.php(362): OCA\\\\Files\\\\Command\\\\Scan->userScan()\\n#11 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files\\\/lib\\\/Command\\\/Scan.php(328): OCA\\\\Files\\\\Command\\\\Scan->processUserChunks()\\n#12 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Command\\\/Command.php(255): OCA\\\\Files\\\\Command\\\\Scan->execute()\\n#13 \\\/var\\\/www\\\/owncloud\\\/core\\\/Command\\\/Base.php(159): Symfony\\\\Component\\\\Console\\\\Command\\\\Command->run()\\n#14 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Application.php(1009): OC\\\\Core\\\\Command\\\\Base->run()\\n#15 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Application.php(273): Symfony\\\\Component\\\\Console\\\\Application->doRunCommand()\\n#16 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/symfony\\\/console\\\/Application.php(149): Symfony\\\\Component\\\\Console\\\\Application->doRun()\\n#17 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Console\\\/Application.php(165): Symfony\\\\Component\\\\Console\\\\Application->run()\\n#18 \\\/var\\\/www\\\/owncloud\\\/console.php(116): OC\\\\Console\\\\Application->run()\\n#19 \\\/var\\\/www\\\/owncloud\\\/occ(11): require_once('\\\/var\\\/www\\\/ownclo...')\\n#20 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Storage\\\/Wrapper\\\/Availability.php\",\"Line\":75}"}


#### Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) …

Edorta · February 13, 2023, 3:25pm

Hi, I couldn’t include Owncloud Config Report because it was too long and I couldn’t upload it as a file.

jvillafanez · February 14, 2023, 9:23am

In addition to the usual “update your system to the latest version”, you should probably double-check the settings your Netgear server.

Unless you REALLY needed, you should remove the files:scan cron job. It might be causing a lot workload in your Netgear server, and it might be rejecting some connections. I know windows has a limit in the number of concurrent connections you can have to a share, so I guess your server also has something similar.

There have been some improvements regarding the sync mechanism that could help with this issue in the latest versions, so it’s recommended for you to upgrade if you can. Otherwise, you probably need to check the options your server has.

Edorta · February 14, 2023, 12:56pm

I have configured the cron task for file scanning because the files are hosted on the NAS, that is, in a location external to the owncloud servers. When I installed the owncloud servers I configured the access to the NAS folders but when I added files to a NAS folder the clients did not sync these files. Synchronization was only performed when accessing that folder from the owncloud website.

Going through the owncloud documentation I found this:

Detecting Files Added to External Storages

Rescanning S3 for manually added files is not possible
	The command occ files:scan or a background job containing files scan is only available for POSIX but not for S3 and compatible filesystems.

We recommend configuring the background job Webcron or Cron to enable ownCloud to automatically detect files added to your external storages.
	You cannot scan/detect changed files on external storage mounts when you select the Log-in credentials, save in session authentication mechanism. However, there is a workaround, and that is to use Ajax cron mode. See Password-based Mechanisms for more information.

ownCloud may not always be able to find out what has been changed remotely (files changed without going through ownCloud), especially when it’s very deep in the folder hierarchy of the external storage.

You might need to setup a cron job that runs

sudo -u www-data ./occ files:scan --all`

Alternatively, replace --all with the user name to trigger a rescan of the user’s files periodically, for example every 15 minutes, which includes the mounted external storage.

The task in cron is for that reason.

Thank you for your reply

jvillafanez · February 14, 2023, 2:59pm

Ok, that’s fair.
If you can’t ensure the files are always added to the NAS through ownCloud, there are very few options.

If you’re the only one adding the files, you could create a script to run the occ files:scan command in targeted folders (the ones you’re uploading into) after you finish the upload, so the workload should be less.

Another option could be to accept that the files won’t be accessible to the client immediately, so you could run the occ files:scan command once per day, at night when there is little activity. I expect the scan to finish in the morning.

Last option is to check if the NAS has any setting that could help you. As said, windows has a limit on the concurrent connections to a share, so maybe your NAS has a similar setting you can adjust (if this is what is causing the problem).

There is a paid option, but you’ll need an enterprise license, so I’ll skip it.

Edorta · February 15, 2023, 2:00pm

Hi jvillafanez,

I have checked the SAMBA server on the Netgear ReadyNAS. I have not found a limit on the concurrent connections. Run the following command: testparm -sv | grep “max connections” the value that is received is “max connections = 0” That is, there is no limit of concurrent connections.

Thank you

system · May 16, 2023, 2:00pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.