Performance of SFTP external storage very bad, but only through ownCloud

ph.z · February 12, 2019, 12:58pm

Steps to reproduce

Connect ownCloud to an SFTP external storage (I use rsync net)
Upload a 100 MB file to the external storage
Download the file directly from the external storage and through ownCloud and compare speeds

Expected behaviour

I would expect the download to only be slightly slower than downloading directly from the external storage

Actual behaviour

Download through OwnCLoud WebInterface: 120 KB/s
Download from SFTP-Server directly (to client PC): 1.1 MB/s
Download from SFTP-Server directly (to ownCloud server): 700 KB/s

Server configuration

Operating system:
Debian 9.3.0

Web server:
Apache 2.4.25

Database:
MariaDB 10.1.37

PHP version: 7.2.15

ownCloud version: (see ownCloud admin page)
10.0.10

Updated from an older ownCloud or fresh install:
fresh install

Where did you install ownCloud from:
From owncloud.org

Signing status (ownCloud 9.0 and above):
No errors have been found.

The content of config/config.php:

gist.github.com

https://gist.github.com/Schnatterowski/cddca6611e9bcbfc900b0d2028703992

config.php

{
    "system": {
        "instanceid": "oc1y0qyax6s0",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "hellingerdoll.nesevo.com",
            "cloud.hellingerdoll.com"
        ],
        "datadirectory": "\/var\/owncloud\/data",

This file has been truncated. show original

List of activated apps:
Enabled:

comments: 0.3.0
configreport: 0.1.1
dav: 0.4.0
federatedfilesharing: 0.3.1
federation: 0.1.0
files: 1.5.1
files_external: 0.7.1
files_external_dropbox: 1.1.0
files_external_ftp: 0.2.1
files_sharing: 0.11.0
files_trashbin: 0.9.1
files_versions: 1.3.0
files_videoplayer: 0.9.8
firstrunwizard: 1.1
market: 0.2.5
notifications: 0.3.5
provisioning_api: 0.5.0
systemtags: 0.3.0
updatenotification: 0.2.1
Disabled:
encryption
external
user_external

Are you using external storage, if yes which one: local/smb/sftp/…
SFTP (rsync.net)

Are you using encryption: yes/no
Only SSL

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…
No
There are no commands defined in the “ldap” namespace.

Client configuration

Browser:
Firefox 65.0 64 bit

Operating system:
Linux Mint 18.2

Logs

Web server error log

[Tue Feb 12 06:25:01.472806 2019] [mpm_prefork:notice] [pid 29107] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2q configured – resuming normal operations
[Tue Feb 12 06:25:01.472822 2019] [core:notice] [pid 29107] AH00094: Command line: ‘/usr/sbin/apache2’

ownCloud log (data/owncloud.log)

gist.github.com

https://gist.github.com/Schnatterowski/b6e36675e1d15cb5a16059b6b2613826

owncloud.log

{"reqId":"FNuxDOPpz0cFcp4ujlaY","level":3,"time":"2019-01-05T19:34:44+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/index.php\/settings\/admin?sectionid=general","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}
{"reqId":"gaSxlwScifx15RKDHMvj","level":3,"time":"2019-01-05T19:39:55+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v1\/notifications?format=json","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}
{"reqId":"wEGrCtOKnLP6TPBs29zp","level":3,"time":"2019-01-05T19:44:55+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v1\/notifications?format=json","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}
{"reqId":"8HIICh8rDLG5TsHn1zxu","level":3,"time":"2019-01-05T19:49:55+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v1\/notifications?format=json","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}
{"reqId":"I2v7TjKz7pgVasyboF3T","level":3,"time":"2019-01-05T19:54:55+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v1\/notifications?format=json","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}
{"reqId":"1HNARu8yE5bvAsh9bIEa","level":3,"time":"2019-01-05T19:55:12+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"index","method":"GET","url":"\/index.php\/settings\/admin?sectionid=general","message":"Exception: {\"Exception\":\"RedisException\",\"Message\":\"Connection refused\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/RedisFactory.php(81): Redis->connect('localhost', 6379, 0)\\n#1 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/RedisFactory.php(101): OC\\\\RedisFactory->create()\\n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Memcache\\\/Redis.php(37): OC\\\\RedisFactory->getInstance()\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Memcache\\\/Factory.php(131): OC\\\\Memcache\\\\Redis->__construct('fa94c09e758bee1...')\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(721): OC\\\\Memcache\\\\Factory->createLocking('lock')\\n#5 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/pimple\\\/pimple\\\/src\\\/Pimple\\\/Container.php(113): OC\\\\Server->OC\\\\{closure}(Object(OC\\\\Server))\\n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php(108): Pimple\\\\Container->offsetGet('LockingProvider')\\n#7 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/ServerContainer.php(86): OC\\\\AppFramework\\\\Utility\\\\SimpleContainer->query('LockingProvider')\\n#8 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(1485): OC\\\\ServerContainer->query('LockingProvider')\\n#9 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/View.php(120): OC\\\\Server->getLockingProvider()\\n#10 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(172): OC\\\\Files\\\\View->__construct()\\n#11 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/pimple\\\/pimple\\\/src\\\/Pimple\\\/Container.php(113): OC\\\\Server->OC\\\\{closure}(Object(OC\\\\Server))\\n#12 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php(108): Pimple\\\\Container->offsetGet('EncryptionManag...')\\n#13 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/ServerContainer.php(86): OC\\\\AppFramework\\\\Utility\\\\SimpleContainer->query('EncryptionManag...')\\n#14 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(916): OC\\\\ServerContainer->query('EncryptionManag...')\\n#15 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(751): OC\\\\Server->getEncryptionManager()\\n#16 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(665): OC::registerEncryptionWrapper()\\n#17 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(994): OC::init()\\n#18 \\\/var\\\/www\\\/owncloud\\\/index.php(53): require_once('\\\/var\\\/www\\\/ownclo...')\\n#19 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/RedisFactory.php\",\"Line\":81}"}
{"reqId":"3IhkkV5VsTQ8n4gDSAC0","level":3,"time":"2019-01-05T19:55:23+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"index","method":"GET","url":"\/","message":"Exception: {\"Exception\":\"RedisException\",\"Message\":\"Connection refused\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/RedisFactory.php(81): Redis->connect('localhost', 6379, 0)\\n#1 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/RedisFactory.php(101): OC\\\\RedisFactory->create()\\n#2 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Memcache\\\/Redis.php(37): OC\\\\RedisFactory->getInstance()\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Memcache\\\/Factory.php(131): OC\\\\Memcache\\\\Redis->__construct('fa94c09e758bee1...')\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(721): OC\\\\Memcache\\\\Factory->createLocking('lock')\\n#5 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/pimple\\\/pimple\\\/src\\\/Pimple\\\/Container.php(113): OC\\\\Server->OC\\\\{closure}(Object(OC\\\\Server))\\n#6 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php(108): Pimple\\\\Container->offsetGet('LockingProvider')\\n#7 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/ServerContainer.php(86): OC\\\\AppFramework\\\\Utility\\\\SimpleContainer->query('LockingProvider')\\n#8 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(1485): OC\\\\ServerContainer->query('LockingProvider')\\n#9 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/View.php(120): OC\\\\Server->getLockingProvider()\\n#10 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(172): OC\\\\Files\\\\View->__construct()\\n#11 \\\/var\\\/www\\\/owncloud\\\/lib\\\/composer\\\/pimple\\\/pimple\\\/src\\\/Pimple\\\/Container.php(113): OC\\\\Server->OC\\\\{closure}(Object(OC\\\\Server))\\n#12 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/AppFramework\\\/Utility\\\/SimpleContainer.php(108): Pimple\\\\Container->offsetGet('EncryptionManag...')\\n#13 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/ServerContainer.php(86): OC\\\\AppFramework\\\\Utility\\\\SimpleContainer->query('EncryptionManag...')\\n#14 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Server.php(916): OC\\\\ServerContainer->query('EncryptionManag...')\\n#15 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(751): OC\\\\Server->getEncryptionManager()\\n#16 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(665): OC::registerEncryptionWrapper()\\n#17 \\\/var\\\/www\\\/owncloud\\\/lib\\\/base.php(994): OC::init()\\n#18 \\\/var\\\/www\\\/owncloud\\\/index.php(53): require_once('\\\/var\\\/www\\\/ownclo...')\\n#19 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/RedisFactory.php\",\"Line\":81}"}
{"reqId":"9GqQR4X4ufGQqUUuGHty","level":3,"time":"2019-01-05T20:00:09+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v1\/notifications?format=json","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}
{"reqId":"RMCHGdUyFlQOjOoMGbmI","level":3,"time":"2019-01-05T20:05:01+00:00","remoteAddr":"","user":"--","app":"files","method":"--","url":"--","message":"Exception: {\"Exception\":\"OC\\\\ForbiddenException\",\"Message\":\"\",\"Code\":0,\"Trace\":\"#0 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Utils\\\/Scanner.php(139): OC\\\\Files\\\\Utils\\\\Scanner->shouldScan(Object(OC\\\\Files\\\\Mount\\\\MountPoint))\\n#1 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files\\\/lib\\\/BackgroundJob\\\/ScanFiles.php(87): OC\\\\Files\\\\Utils\\\\Scanner->backgroundScan('')\\n#2 \\\/var\\\/www\\\/owncloud\\\/apps\\\/files\\\/lib\\\/BackgroundJob\\\/ScanFiles.php(111): OCA\\\\Files\\\\BackgroundJob\\\\ScanFiles->runScanner(Object(OC\\\\User\\\\User))\\n#3 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/BackgroundJob\\\/Job.php(57): OCA\\\\Files\\\\BackgroundJob\\\\ScanFiles->run(NULL)\\n#4 \\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/BackgroundJob\\\/TimedJob.php(53): OC\\\\BackgroundJob\\\\Job->execute(Object(OC\\\\BackgroundJob\\\\JobList), Object(OC\\\\Log))\\n#5 \\\/var\\\/www\\\/owncloud\\\/cron.php(120): OC\\\\BackgroundJob\\\\TimedJob->execute(Object(OC\\\\BackgroundJob\\\\JobList), Object(OC\\\\Log))\\n#6 {main}\",\"File\":\"\\\/var\\\/www\\\/owncloud\\\/lib\\\/private\\\/Files\\\/Utils\\\/Scanner.php\",\"Line\":192}"}
{"reqId":"24C3i0HsLapxmeoInDia","level":3,"time":"2019-01-05T20:05:36+00:00","remoteAddr":"213.47.214.178","user":"admin","app":"no app in context","method":"GET","url":"\/ocs\/v2.php\/apps\/notifications\/api\/v1\/notifications?format=json","message":"User backend OC\\User\\Database is returning home: \/var\/owncloud\/data\/admin for user: admin which differs from existing value: \/var\/www\/owncloud\/data\/admin"}

This file has been truncated. show original

tom42 · February 24, 2019, 10:29am

Hey,

i don’t think there is that much you can do here or where other users can give specific pointers.

From what i know a download of files placed on the external storages is first downloaded to the ownCloud server and the forwarded to the client (requesting the download) like:

external storage <-Network/Internet-> ownCloud server <-Network/Internet-> client

where you have a lot additional overhead for the network connection. I think there is also additional overhead from ownCloud side due to the use of PHP, database queries, file versions / checksums etc. which is probably also lowering the transfer speed.

Comparing the transfer speed of ownCloud to a protocol purely implemented for file transfer like SFTP seems to me not possible / comparable.