Problems to upload 2 GB+ files

Server
, ,
1

It seems that I'm unable to upload files around or larger than 2 GB. I've tested this via WebDav and syncing with the ownCloud client (Windows 10) and via the web uploaader. Since the most recent update was to the server, I assume it's a server issue.
I did configure my phpfpm to allow uploads of 4 GB.

Steps to reproduce
1. move a file with more than 2 GB in size into any owncloud folder
2. ownCloud Client starts syncing, seemingly uploads the file
3. When the upload should finish there is a "Bad Gateway" error message and it just restarts syncing the entire file from the start

Expected behaviour
File (2 GB+) uploads and is available via web interface

Actual behaviour
Getting a saberDAV error "Bad Gateway after upload finished
Syncing seems to restart seemingly indefintely

Server configuration
Operating system: Ubuntu 14.04
Web server: nginx1.10.1 / phpfpm5
Database:MySQL
PHP version: 5.5.9
ownCloud version (see ownCloud admin page): 9.1
Updated from an older ownCloud or fresh install: updated form 9.0.4
ownCloud log: (last 100 lines from log when it happened)

{"reqId":"SbiAbl1un5SuqH2c5DHN","remoteAddr":"88.150.24.183","app":"webdav","message":"Exception: {\"Message\":\"HTTP\\/1.1 423 \\"someotheruser's Hut\\/2016 Spain\\/somefile_2016.28.07_14.06_somefile.mp4\\" is locked\",\"Exception\":\"OCA\\DAV\\Connector\\Sabre\\Exception\\FileLocked\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(490): OCA\\DAV\\Connector\\Sabre\\File->convertToSabreException(Object(OCP\\Lock\\LockedException))\n#1 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(103): OCA\\DAV\\Connector\\Sabre\\File->createFileChunked(Resource id #234)\n#2 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/Directory.php(136): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #234)\n#3 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1036): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('somefile_2016...', Resource id #234)\n#4 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(523): Sabre\\DAV\\Server->createFile('someotheruser's Hut\\/2016...', Resource id #234, NULL)\n#5 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#6 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#7 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(459): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#8 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(248): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#9 \\/var\\/www\\/owncloud\\/apps\\/dav\\/appinfo\\/v1\\/webdav.php(56): Sabre\\DAV\\Server->exec()\n#10 \\/var\\/www\\/owncloud\\/remote.php(164): require_once('\\/var\\/www\\/ownclo...')\n#11 {main}\",\"File\":\"\\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php\",\"Line\":546,\"User\":\"someotheruser\"}","level":4,"time":"2016-07-29T07:36:48+00:00","method":"PUT","url":"\/remote.php\/webdav\/someotheruser's%20Hut\/2016%20Spain\/somefile_2016.28.07_14.06_somefile.mp4-chunking-2506917686-219-180","user":"someotheruser"}
{"reqId":"y7eeGuW9n38zZwwm8fFX","remoteAddr":"88.150.24.183","app":"webdav","message":"Exception: {\"Message\":\"HTTP\\/1.1 423 \\"someotheruser's Hut\\/2016 Spain\\/somefile_2016.28.07_14.06_somefile.mp4\\" is locked\",\"Exception\":\"OCA\\DAV\\Connector\\Sabre\\Exception\\FileLocked\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(490): OCA\\DAV\\Connector\\Sabre\\File->convertToSabreException(Object(OCP\\Lock\\LockedException))\n#1 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(103): OCA\\DAV\\Connector\\Sabre\\File->createFileChunked(Resource id #234)\n#2 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/Directory.php(136): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #234)\n#3 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1036): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('somefile_2016...', Resource id #234)\n#4 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(523): Sabre\\DAV\\Server->createFile('someotheruser's Hut\\/2016...', Resource id #234, NULL)\n#5 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#6 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#7 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(459): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#8 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(248): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#9 \\/var\\/www\\/owncloud\\/apps\\/dav\\/appinfo\\/v1\\/webdav.php(56): Sabre\\DAV\\Server->exec()\n#10 \\/var\\/www\\/owncloud\\/remote.php(164): require_once('\\/var\\/www\\/ownclo...')\n#11 {main}\",\"File\":\"\\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php\",\"Line\":546,\"User\":\"someotheruser\"}","level":4,"time":"2016-07-29T07:38:17+00:00","method":"PUT","url":"\/remote.php\/webdav\/someotheruser's%20Hut\/2016%20Spain\/somefile_2016.28.07_14.06_somefile.mp4-chunking-2506917686-219-181","user":"someotheruser"}
{"reqId":"xR8j9bSicVkRPzQ1lj0C","remoteAddr":"88.150.24.183","app":"webdav","message":"Exception: {\"Message\":\"HTTP\\/1.1 423 \\"someotheruser's Hut\\/2016 Spain\\/somefile_2016.28.07_14.06_somefile.mp4\\" is locked\",\"Exception\":\"OCA\\DAV\\Connector\\Sabre\\Exception\\FileLocked\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(490): OCA\\DAV\\Connector\\Sabre\\File->convertToSabreException(Object(OCP\\Lock\\LockedException))\n#1 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(103): OCA\\DAV\\Connector\\Sabre\\File->createFileChunked(Resource id #234)\n#2 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/Directory.php(136): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #234)\n#3 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1036): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('somefile_2016...', Resource id #234)\n#4 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(523): Sabre\\DAV\\Server->createFile('someotheruser's Hut\\/2016...', Resource id #234, NULL)\n#5 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#6 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#7 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(459): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#8 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(248): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#9 \\/var\\/www\\/owncloud\\/apps\\/dav\\/appinfo\\/v1\\/webdav.php(56): Sabre\\DAV\\Server->exec()\n#10 \\/var\\/www\\/owncloud\\/remote.php(164): require_once('\\/var\\/www\\/ownclo...')\n#11 {main}\",\"File\":\"\\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php\",\"Line\":546,\"User\":\"someotheruser\"}","level":4,"time":"2016-07-29T07:40:04+00:00","method":"PUT","url":"\/remote.php\/webdav\/someotheruser's%20Hut\/2016%20Spain\/somefile_2016.28.07_14.06_somefile.mp4-chunking-2506917686-219-179","user":"someotheruser"}

{"reqId":"4hEV8h7AsEOcdzFA0mOL","remoteAddr":"88.150.24.183","app":"files","message":"Exception: {\"Exception\":\"BadMethodCallException\",\"Message\":\"Not in batch\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Utils\\/Scanner.php(143): OC\\Files\\Cache\\Propagator->commitBatch()\n#1 \\/var\\/www\\/owncloud\\/apps\\/files\\/lib\\/BackgroundJob\\/ScanFiles.php(87): OC\\Files\\Utils\\Scanner->backgroundScan('')\n#2 \\/var\\/www\\/owncloud\\/apps\\/files\\/lib\\/BackgroundJob\\/ScanFiles.php(111): OCA\\Files\\BackgroundJob\\ScanFiles->runScanner(Object(OC\\User\\User))\n#3 \\/var\\/www\\/owncloud\\/lib\\/private\\/BackgroundJob\\/Job.php(52): OCA\\Files\\BackgroundJob\\ScanFiles->run(NULL)\n#4 \\/var\\/www\\/owncloud\\/lib\\/private\\/BackgroundJob\\/TimedJob.php(53): OC\\BackgroundJob\\Job->execute(Object(OC\\BackgroundJob\\JobList), Object(OC\\Log))\n#5 \\/var\\/www\\/owncloud\\/cron.php(144): OC\\BackgroundJob\\TimedJob->execute(Object(OC\\BackgroundJob\\JobList), Object(OC\\Log))\n#6 {main}\",\"File\":\"\\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Cache\\/Propagator.php\",\"Line\":145}","level":3,"time":"2016-07-29T11:24:43+00:00","method":"GET","url":"\/cron.php","user":"--"}

{"reqId":"FbCaEtzOWXZovU9nO6zq","remoteAddr":"88.150.24.183","app":"no app in context","message":"Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.","level":3,"time":"2016-07-30T00:34:34+00:00","method":"PUT","url":"\/remote.php\/webdav\/someotheruser's%20Hut\/2016%20Spain\/somefile_2016.28.07_14.06_somefile.mp4-chunking-2506918615-219-184","user":"someotheruser"}
{"reqId":"FbCaEtzOWXZovU9nO6zq","remoteAddr":"88.150.24.183","app":"no app in context","message":"Couldn't re-calculate unencrypted size for files_versions\/someotheruser's Hut\/2016 Spain\/somefile_2016.28.07_14.06_somefile.mp4.v1469820622","level":3,"time":"2016-07-30T00:34:34+00:00","method":"PUT","url":"\/remote.php\/webdav\/someotheruser's%20Hut\/2016%20Spain\/somefile_2016.28.07_14.06_somefile.mp4-chunking-2506918615-219-184","user":"someotheruser"}
{"reqId":"FbCaEtzOWXZovU9nO6zq","remoteAddr":"88.150.24.183","app":"no app in context","message":"Exception: {\"Exception\":\"OC\\Encryption\\Exceptions\\DecryptionFailedException\",\"Message\":\"Can not decrypt this file, probably this is a shared file. Please ask the file owner to reshare the file with you.\",\"Code\":0,\"Trace\":\"#0 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Storage\\/Wrapper\\/Encryption.php(573): OCA\\Encryption\\Crypto\\Encryption->decrypt(*** sensitive parameters replaced ***)\n#1 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Storage\\/Wrapper\\/Encryption.php(492): OC\\Files\\Storage\\Wrapper\\Encryption->fixUnencryptedSize('files_versions\\/...', 1777639424, 1777639424)\n#2 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Storage\\/Wrapper\\/Encryption.php(184): OC\\Files\\Storage\\Wrapper\\Encryption->verifyUnencryptedSize('files_versions\\/...', 1777639424)\n#3 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Storage\\/Wrapper\\/Wrapper.php(572): OC\\Files\\Storage\\Wrapper\\Encryption->getMetaData('files_versions\\/...')\n#4 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Cache\\/Scanner.php(113): OC\\Files\\Storage\\Wrapper\\Wrapper->getMetaData('files_versions\\/...')\n#5 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Cache\\/Scanner.php(145): OC\\Files\\Cache\\Scanner->getData('files_versions\\/...')\n#6 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Cache\\/Scanner.php(309): OC\\Files\\Cache\\Scanner->scanFile('files_versions\\/...', 3, -1, NULL, false)\n#7 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/Cache\\/Updater.php(123): OC\\Files\\Cache\\Scanner->scan('files_versions\\/...', false, -1, false)\n#8 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/View.php(313): OC\\Files\\Cache\\Updater->update('files_versions\\/...', 1469838874)\n#9 \\/var\\/www\\/owncloud\\/lib\\/private\\/Files\\/View.php(895): OC\\Files\\View->writeUpdate(Object(OCA\\Files_Trashbin\\Storage), 'files_versions\\/...')\n#10 \\/var\\/www\\/owncloud\\/apps\\/files_versions\\/lib\\/Storage.php(192): OC\\Files\\View->copy('files\\/\\/someotheruser's H...', 'files_versions\\/...')\n#11 \\/var\\/www\\/owncloud\\/apps\\/files_versions\\/lib\\/Hooks.php(60): OCA\\Files_Versions\\Storage::store('\\/someotheruser's Hut\\/201...')\n#12 [internal function]: OCA\\Files_Versions\\Hooks::write_hook(Array)\n#13 \\/var\\/www\\/owncloud\\/lib\\/private\\/legacy\\/hook.php(105): call_user_func(Array, Array)\n#14 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(268): OC_Hook::emit('OC_Filesystem', 'write', Array)\n#15 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(424): OCA\\DAV\\Connector\\Sabre\\File->emitPreHooks(true, '\\/someotheruser's Hut\\/201...')\n#16 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/File.php(103): OCA\\DAV\\Connector\\Sabre\\File->createFileChunked(Resource id #233)\n#17 \\/var\\/www\\/owncloud\\/apps\\/dav\\/lib\\/Connector\\/Sabre\\/Directory.php(136): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #233)\n#18 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(1036): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('somefile_2016...', Resource id #233)\n#19 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/CorePlugin.php(523): Sabre\\DAV\\Server->createFile('someotheruser's Hut\\/2016...', Resource id #233, NULL)\n#20 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#21 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/event\\/lib\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#22 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(459): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#23 \\/var\\/www\\/owncloud\\/3rdparty\\/sabre\\/dav\\/lib\\/DAV\\/Server.php(248): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#24 \\/var\\/www\\/owncloud\\/apps\\/dav\\/appinfo\\/v1\\/webdav.php(56): Sabre\\DAV\\Server->exec()\n#25 \\/var\\/www\\/owncloud\\/remote.php(164): require_once('\\/var\\/www\\/ownclo...')\n#26 {main}\",\"File\":\"\\/var\\/www\\/owncloud\\/apps\\/encryption\\/lib\\/Crypto\\/Encryption.php\",\"Line\":360}","level":3,"time":"2016-07-30T00:34:34+00:00","method":"PUT","url":"\/remote.php\/webdav\/someotheruser's%20Hut\/2016%20Spain\/somefile_2016.28.07_14.06_somefile.mp4-chunking-2506918615-219-184","user":"someotheruser"}

Special configuration (external storage, external authentication, reverse proxy, server-side-encryption):

server side encryption enabled

Integrity status for oC9+

No errors have been found.
2

Have a look at the docs available here:

https://doc.owncloud.org/server/9.1/admin_manual/configuration_files/big_file_upload_configuration.html

the faq here: https://central.owncloud.org/t/how-to-change-upload-limits-and-fix-upload-problems/851

as well as:

https://github.com/owncloud/documentation/wiki/Uploading-files-up-to-16GB

Such "gateway" issues are quite often caused by timeouts within your webserver.

3

Thanks for the hint, I didn't know the last document. I have changed the settings in nginx and php5-fpm like this:

php.ini

post_max_size = 4000M;

upload_max_filesize = 4000M;

max_input_time = 6000

output_buffering = Off

max_execution_time = 6000

upload_tmp_dir = /tmp

owncloud.conf for nginx

client_max_body_size 5G;
fastcgi_buffers 64 4k;
fastcgi_read_timeout 6000;

Then restarted services. Still I get Bad Gateway errors in the owncloud client after upload seemingly finished.
I have had a look at the actual filesystem on the server and it seems like actually the file (or part of it) is uploaded, as there usually are .part files in the exact places I put the files in. Also the Bad Gateway errors usually refer to "chunking" files, which I guess are the file chunks uploaded via the owncloud client. I don't understand how the client cannot find the chunks he just uploaded really. I figured it might be a timeout issue, but with the above setting even an upload that lasts about an hour should probably go through fine, or am I mistaken? And it was my understanding that the size limits shouldn't affect the owncloud client uploading. (?)

4

In your logfile, there are errors with file-locking and encryption, you should try to solve them. They are perhaps related to your problem.

In order to avoid problems specific to file-chunking and upload timeouts, you can try to upload a file directly from the server itself (the file should be outside the owncloud-folders and larger than 2G):
https://doc.owncloud.org/server/8.0/user_manual/files/access_webdav.html#accessing-files-using-curl

Also put a document on your webserver and check with phpinfo() the actual php settings. There are several logfiles and value can be overwritten.

5

thx tflidd, you think so? Well, I do believe they are connected, but I'm not sure if they are the cause of the upload problem or whether the upload problem causes the locking and encryption errors. Could you perhaps point me to anything that helps with this?

I'll try the uploading directly from the server though, thank!

Update: same error when uploading with cadaver on the server itself, Bad Gateway after upload finishes.

phpinfo()-check showed the settings exactly like I set them in the ini, so that seems to be fine.

6

You can unlock files: https://central.owncloud.org/t/file-is-locked-how-to-unlock/985?u=tflidd
However, if it happens very often and you use the upload function a lot, you should install the redis cache for file-locking (https://doc.owncloud.org/server/9.0/admin_manual/configuration_server/caching_configuration.html#id4).

I don't use external storage where encryption provides a benefit. I would avoid shared folder or large folder that might not be completely indexed or are subject to a lot of changes.

Your webserver or php logfile, do they show errors? I mean if there are real timeouts, you should be able to find some hints. And you can monitor the server (CPU usage, ...). But you are on a 64-bit system?

7

Thanks for the link! I should have mentioned earlier, but I do use the redis-server. But I found out I used an outdated version, so I upgraded now. This however does not fix the issue with uploading.

I do get errors in the webserver log file for each file that fails uploading:

example

recv() failed (104: Connection reset by peer) while reading response header from upstream, client: xxx.xxx.xxx.xxx [..] upstreamt: "fastcgi://unix:/var/rin/php-fpm-cloud.socket"

I don't really know what that refers to, seems like the php5-fpm connection times out. but why?

8

Hi,

but why?

i think thats a question perfectly fitting into a support channel of nginx:

http://nginx.org/en/support.html

They might have some additional background info for that.