Uploading large files causes Internal server error on 10.0.2

Server
1

Hi,

I'm stuck for a good while now trying to upload larger files (say 200MB+) either through a desktopclient (mac&linux) or through the web interface. At first I would just get 'Internal server error' without specifics, but I've managed to generate the following log entries:

.owncloudsync.log
SQLSTATE[HY000]: General error: 2006 MySQL server has gone away)|0|0|0|||INST_NONE|
||test/testfile|INST_ERROR|Up|1508786388||217973163||6|The item is not synced because of previous errors: Error transferring https://mysite.com/owncloud/remote.php/dav/uploads/willem/3160764148/00000001 - server replied: Internal Server Error (An exception occurred while executing 'SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
storage_mtime, encrypted, etag, permissions, checksum
FROM oc_filecache WHERE storage = ? AND path_hash = ?' with params [3, "3389a613f9c35a0fb1ec66fb147a92e4"]:

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away)|0|0|0|||INST_NONE|

owncloud.log
{"reqId":"JSpq5mAVnEnGwLs6KDdX","level":4,"time":"2017-11-19T16:00:25+00:00","remoteAddr":"62.108.3.41","user":"willem","app":"webdav","method":"PUT","url":"\/owncloud\/remote.php\/dav\/uploads\/willem\/3160764148\/00000004","message":"Exception: {\"Message\":\"HTTP\\\/1.1 500 An exception occurred while executing 'SELECTfileid,storage,path,parent,name,mimetype,mimepart,size,mtime,\\n\\t\\t\\t\\t\\tstorage_mtime,encrypted,etag,permissions,checksum\\n\\t\\t\\t\\tFROMoc_filecacheWHEREstorage= ? ANDpath_hash= ?' with params [3, \\\"68f64ae39132852c75e2b5f02641f246\\\"]:\\n\\nSQLSTATE[HY000]: General error: 2006 MySQL server has gone away\",\"Exception\":\"Sabre\\\\DAV\\\\Exception\",\"Code\":0,\"Trace\":\"#0 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php(164): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->convertToSabreException(Object(Doctrine\\\\DBAL\\\\Exception\\\\DriverException))\\n#1 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/Directory.php(158): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\File->put(Resource id #464)\\n#2 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Upload\\\/UploadFolder.php(38): OCA\\\\DAV\\\\Connector\\\\Sabre\\\\Directory->createFile('00000004', Resource id #464)\\n#3 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(1095): OCA\\\\DAV\\\\Upload\\\\UploadFolder->createFile('00000004', Resource id #464)\\n#4 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/CorePlugin.php(525): Sabre\\\\DAV\\\\Server->createFile('uploads\\\/willem\\\/...', Resource id #464, NULL)\\n#5 [internal function]: Sabre\\\\DAV\\\\CorePlugin->httpPut(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#6 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/event\\\/lib\\\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\\n#7 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(479): Sabre\\\\Event\\\\EventEmitter->emit('method:PUT', Array)\\n#8 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/lib\\\/composer\\\/sabre\\\/dav\\\/lib\\\/DAV\\\/Server.php(254): Sabre\\\\DAV\\\\Server->invokeMethod(Object(Sabre\\\\HTTP\\\\Request), Object(Sabre\\\\HTTP\\\\Response))\\n#9 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Server.php(229): Sabre\\\\DAV\\\\Server->exec()\\n#10 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/apps\\\/dav\\\/appinfo\\\/v2\\\/remote.php(31): OCA\\\\DAV\\\\Server->exec()\\n#11 \\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/remote.php(165): require_once('\\\/home\\\/wzwartho\\\/...')\\n#12 {main}\",\"File\":\"\\\/home\\\/wzwartho\\\/public_html\\\/owncloud\\\/apps\\\/dav\\\/lib\\\/Connector\\\/Sabre\\\/File.php\",\"Line\":604,\"User\":\"willem\"}"}

So far I've tried setting several values in php.ini and .user.ini:
display_errors = On
max_execution_time = 3600
max_input_time = 3600
max_input_vars = 1000
memory_limit = 2048M
post_max_size = 2048M
session.gc_maxlifetime = 3600
upload_max_filesize = 2048M
error_reporting = E_ALL & ~E_NOTICE
interactive_timeout = 3600
wait_timeout = 3600

But nothing's helping. I'm at a loss, can someone please help me out?

Thanks,

Willem

Server configuration

It's a shared hosting server
Operating system:
Linux
Web server:
Apache 2.4.29
Database:
MySQL 10.1.24-MariaDB-cll-lve
PHP version:
5.6.32
ownCloud version: (see ownCloud admin page)
10.0.2
Updated from an older ownCloud or fresh install:
fresh
Where did you install ownCloud from:
softaculous / cpanel

2 

SQLSTATE[HY000]: General error: 2006 MySQL server has gone away

It seems the server lost the connection with MySQL. You should investigate if the MySQL server is down or if the connection with MySQL is cut somehow during the operation.

3

This notification is clear: "General error: 2006 MySQL server has gone away"

Ask your hosting provider about the database connection and what is going wrong here. They have to offer a server with database connection for you.

4

Ok, thanks for confirming the issue is in "SQLSTATE[HY000]: General error: 2006 MySQL server has gone away". That would mean it is either a timeout or a reconnect issue, correct?

Now I need to figure out how to fix this :wink:
I don't have much experience with php and mysql. So far none of the variables I've tried setting have worked. I can change some MySQL parameters from php.ini, but not all. And because it is shared hosting I can't set MySQL variables directly.

I've been reading here: https://dev.mysql.com/doc/refman/5.7/en/gone-away.html
and trying to check and set values. But I can't find things like
- the reconnect flag in the MYSQL structure
- if MySQL is started with the --skip-networking

I'll keep searching. Any help is most welcome!

Thanks,

Willem

5

So, unfortunately I'm still stuck on this, with not a clue as to how to proceed.

At present the error show like this in .owncloudsync.log for files over 100M (Uploading a 100M file takes about 10 minutes):

||testfile_100mb|INST_NEW|Up|1514546452||103915520||2|Error transferring https://mysitecom/owncloud/remote.php/dav/uploads/willem/905447493/.file - server replied: Internal Server Error|500|0|0|||INST_NONE|

So no 'MySQL server has gone away' anymore. Also, owncloud.log doesn't show the error anymore.

The file uploads, then the client seems to hang for a while on 100%, then it fails.

I've tried a bunch of things, including:

max_execution_time=3600
post_max_size=16G
upload_max_filesize=16G
mysql.connect_timeout=28800
mysql.wait_timeout=28800

(these in .user.ini and .htaccess in my document root and in the owncloud folder and some of these are available through the cpanel/php selector)

'dbdriveroptions' => 
  array (
  	PDO::MYSQL_ATTR_INIT_COMMAND => 'SET wait_timeout = 28800',
  ),

(in config.php)

export OWNCLOUD_CHUNK_SIZE=1048576
(in client ~/.bashrc)

Also tried setting the tempdirectory, but phpinfo() shows this setting doesn't take, nor does using the php var upload_tmp_dir. The other settings show up as expected in phpinfo().

One thing that might be related is the memcache.locking which I can only set to APCu. I can enable the redis php module, but it doesn't work, which I read is expected with shared hosting.

So I'm afraid it has to do with some setting that's inaccessible to me due to shared hosting, but I don't know where to look, or what to ask my hosting provider to change.

Can anyone provide some insight? Other than switching to a host which offers more control...

Thanks,

Willem

6

I think you could ask for the logentry of the webserver or PHP matching the time at which the "Error transferring https://mysitecom/owncloud/remote.php/dav/uploads/willem/905447493/.file - server replied: Internal Server Error" happened. It could be possible that this gives you some more info.