Hello Guy! :slight_smile:
Did you plan the multi file sync? Because when we have to upload 8000 files of 12kb each for example, it's very very slow (2-3hours) whereas it's only 30mb :stuck_out_tongue: .
May the force be with you!
We looked strongly into it, ownCloud Desktop Client 2.3 will again have some improvements. Going forward we are looking into http/2 versus bundling, etc.
Okay, thank you for your fast answer!
tflidd
February 22, 2017, 5:23pm
5
Database tuning and filecache locking via redis should help a lot:
opened 12:22AM - 06 Dec 15 UTC
closed 04:54PM - 06 Apr 17 UTC
enhancement
discussion
performance
Upload of many small files is very slow. I tested on a raspbery pi 2 which is kn… own to have a bad i/o-performance. During uploads, mysql-operation create an important io-wait on the system. It was worse with journaling of ext4, so I disabled it (it's a test system).
Uploading a folder with 1180 files (18 MB) via the oc-client takes about 20-30 minutes (pictures, a lot of text files, take any php code of a project), but the number of sql-queries is extremely high (>50 queries per uploaded file!):
| command | number |
| --- | --- |
| select | 49,681 |
| update | 11,523 |
| insert select | 2,487 |
| set option | 1,526 |
| delete | 511 |
| insert | 423 |
| begin | 17 |
| commit | 17 |
For upload, a smaller number of queries can be obtained when files are uploaded in an external folder via ftp (takes a few seconds) then this folder is included into owncloud, the scanning process takes a few minutes (it's hard to tell, 5 minutes?) and the number of queries (some queries are due to handling the web-interface):
| command | number |
| --- | --- |
| select | 11,054 |
| update | 3,040 |
| insert select | 3,004 |
| set option | 425 |
| delete | 138 |
I remember you worked on the deletion process for OC 8.0, this scales much better: 3000 files take 3500 update and 1400 select queries. Perhaps there is a similar improvement possible for upload operations ;-)
System: RPi2, Debian Jessie
Webserver: nginx/php-fpm
Database: mariadb
Owncloud 8.2.1
Apps enabled:
- activity: 2.1.3
- calendar: 0.8.1
- contacts: 0.5.0.0
- files: 1.2.0
- files_external: 0.3.0
- files_pdfviewer: 0.7
- files_sharing: 0.7.0
- files_texteditor: 2.0
- files_trashbin: 0.7.0
- files_versions: 1.1.0
- files_videoviewer: 0.1.3
- firstrunwizard: 1.1
- galleryplus: 14.3.0
- provisioning_api: 0.3.0
- templateeditor: 0.1
I haven't tested with a pure webdav-client yet.
1 Like
