Large uploads to Swift storage backfires

help

#1

Hey everyone. I have managed to get my OC up and running with LDAP authentication and Openstack Swift Object storage. According to this page we should be able to upload larger files. As we will be needing OC for research support, this is a key feature that is needed at our university.

From what I could gather so far, we should be able to upload these large files, but we aren't. The swift documentation as seen here states one needs to enable DLO (Dynamic Large Objects) which we have done. The reasoning behind this is that when you upload a larger object (swift hardcoded a 5GB limit), the client that does the upload needs to be able to chunk the upload. From OC documents it shows that the OC desktop client already does chunking. However, OC seemingly assembles the file again on server side, and to me it seems that this happens befor the upload to Swift Object Store starts, which means we are running into 5GB limit again. Could someone please tell me if I am correct?

Steps to reproduce
1. Install/upgrade to OC 9.1
2. Attach Openstack Swift Object storage as only storage
3. Upload files > 5GB

Expected behaviour
The file should successfully upload and return success

Actual behaviour
Owncloud comes back with a message from the swift node stating:

{"reqId":"V6Q4a4bcLdL48RWMAcK6sQAAAAk","remoteAddr":"CLIENT_IP","app":"objectstore","message":"Could not create object: Client error response\n[status code] 413\n[reason phrase] Request Entity Too Large\n[url] https:\/\/swift01.mydomain/\/owncloud2\/urn:oid:2107","level":3,"time":"2016-08-05T08:59:16+02:00","method":"POST","url":"\/index.php\/apps\/files\/ajax\/upload.php"

Note: The url's and IP's has been generified, so please note that they are actually working.

Server configuration
Operating system: CentOS 7
Web server: httpd-2.4.6-40.el7.centos.4.x86_64
Database: mariadb-5.5.47-1.el7_2.x86_64
PHP version: 5.5
ownCloud version: 9.1
Updated from an older ownCloud or fresh install: Updated from 9.0.2

Special configuration (external storage, external authentication, reverse proxy, server-side-encryption):
Openstack Swift Object storage
LDAP Authentication

Integrity status for oC9+

No errors have been found.

Additional info:

I have updated the file limits to 50GB and time-outs to 3600, as stated in OC documentation here

Here is my owncloud config file, I of course removed any privileged info.

<?php
$CONFIG = array (
'instanceid' => '',
'passwordsalt' => '',
'secret' => '',
'trusted_domains' =>
array (
0 => 'myserver.example.com',
),
'datadirectory' => '/var/www/html/owncloud/data',
'overwrite.cli.url' => 'https://myserver.example.com',
'dbtype' => 'mysql',
'version' => '9.1.0.15',
'dbname' => 'owncloud',
'dbhost' => 'localhost',
'dbtableprefix' => 'oc_',
'dbuser' => 'owncloud',
'dbpassword' => 'owncloud',
'logtimezone' => 'Africa/Johannesburg',
'installed' => true,
'memcache.local' => '\OC\Memcache\Memcached',
'memcache.distributed' => '\OC\Memcache\Memcached',
'memcached_servers' =>
array (
0 =>
array (
0 => 'localhost',
1 => 11211,
),
),
'objectstore' =>
array (
'class' => 'OC\Files\ObjectStore\Swift',
'arguments' =>
array (
'username' => 'owncloud',
'password' => 'owncloudpassword',
'container' => 'containername',
'autocreate' => true,
'region' => 'OurRegion',
'url' => 'http://myswiftserver.example.com/v2.0',
'tenantName' => 'admin',
'serviceName' => 'swift',
),
),
'ldapIgnoreNamingRules' => false,
'loglevel' => 0,
'maintenance' => false,
);

Any help in this regard would be greatly appreciated.


#2

As far as I'm aware of we do not yet support DLO in ownCloud.


#3

Oh no, this is bad news for us then. So in your opinion, should we try to move to the static objects then? And disable DLO totally? This is not preferred, as a timeout on a static object of 250GB can cause quite an upset with end-users.


#4

here we go - this is the proposed change to introduce DLO support

You can help testing and we can make this part of 9.2

I cannot judge on this because I have limited knowledge about SWIFT


#5

I will gladly help test this! How do I go about doing that?