Desktop Sync Client 2.2.3 - Failed to sync folder with a large amount of files

9.1.x
help

#1

Please help us by providing the following info.

Steps to reproduce
1. Folder with 15.000+ files in it
2. Try to sync it with Desktop client (Win10)
3. Error Message: Unknown Error

Expected behaviour
Files should get synced even with 15k+ files in it

Actual behaviour
Getting error message (Unknown Error) while client is "discovering" the folder with 15k+ files in it

Client configuration
Client version: 2.2.3
Client operating system: Windows 10 Pro

Server configuration
Operating system: Debian 7
Web server: apache
Database: MySQL
PHP version: 5.4
ownCloud version (see ownCloud admin page): 9.1
Updated from an older ownCloud or fresh install: Fresh installed
ownCloud log (data/owncloud.log): https://madgeeks.de/owncloud/index.php/s/1yhi9IE8sHfBFyU

Special configurations (external storage, external authentication, reverse proxy, server-side-encryption): -


#2

From the log:

08-10 02:09:04:274 0x9b94348 OCC::DiscoveryJob::remote_vio_opendir_hook: OCC::DiscoveryJob(0x14c39d30) Ahmet  Pascal Ippo Ordner YO!/Anime/Konachan Calling into main thread...
08-10 02:09:04:276 0x1e7908 OCC::AbstractNetworkJob::start: !!! OCC::LsColJob created for "https://madgeeks.de/owncloud" + "/Ahmet  Pascal Ippo Ordner YO!/Anime/Konachan" "OCC::DiscoverySingleDirectoryJob"
08-10 02:09:24:181 0x1e7908 OCC::LsColXMLParser::parse: Invalid href "" expected starting with "/owncloud/remote.php/webdav/Ahmet  Pascal Ippo Ordner YO!/Anime/Konachan"
08-10 02:09:24:183 0x1e7908 OCC::DiscoverySingleDirectoryJob::lsJobFinishedWithErrorSlot: void OCC::DiscoverySingleDirectoryJob::lsJobFinishedWithErrorSlot(QNetworkReply*) "Unbekannter Fehler" 207 0
08-10 02:09:24:184 0x1e7908 OCC::DiscoveryMainThread::singleDirectoryJobFinishedWithErrorSlot: void OCC::DiscoveryMainThread::singleDirectoryJobFinishedWithErrorSlot(int, const QString&) 5 "Unbekannter Fehler"
08-10 02:09:24:184 0x9b94348 OCC::DiscoveryJob::remote_vio_opendir_hook: OCC::DiscoveryJob(0x14c39d30) Ahmet  Pascal Ippo Ordner YO!/Anime/Konachan ...Returned from main thread
08-10 02:09:24:184 0x9b94348 OCC::DiscoveryJob::remote_vio_opendir_hook: 5 when opening Ahmet  Pascal Ippo Ordner YO!/Anime/Konachan msg= "Unbekannter Fehler"
08-10 02:09:24:184 0x9b94348 csync_ftw: opendir failed for ownclouds://madgeeks.de/owncloud/remote.php/webdav/Ahmet  Pascal Ippo Ordner YO!/Anime/Konachan - errno 5
08-10 02:09:24:184 0x9b94348 OCC::DiscoveryJob::remote_vio_closedir_hook: static void OCC::DiscoveryJob::remote_vio_closedir_hook(csync_vio_handle_t*, void*) OCC::DiscoveryJob(0x14c39d30) "/Ahmet  Pascal Ippo Ordner YO!/Anime"
08-10 02:09:24:184 0x9b94348 OCC::DiscoveryJob::remote_vio_closedir_hook: static void OCC::DiscoveryJob::remote_vio_closedir_hook(csync_vio_handle_t*, void*) OCC::DiscoveryJob(0x14c39d30) "/Ahmet  Pascal Ippo Ordner YO!"
08-10 02:09:24:184 0x9b94348 OCC::DiscoveryJob::remote_vio_closedir_hook: static void OCC::DiscoveryJob::remote_vio_closedir_hook(csync_vio_handle_t*, void*) OCC::DiscoveryJob(0x14c39d30) ""
08-10 02:09:24:184 0x9b94348 csync_statedb_close: sqlite3_close=0
08-10 02:09:24:189 0x1e7908 OCC::SyncEngine::handleSyncError:  #### ERROR during  csync_update :  "Beim Öffnen eines Ordners ist ein Fehler aufgetreten. Unbekannter Fehler"

I wonder why we get this "Invalid href" message. 20 sec time is also quite long..
@PVince81 ...


#3

@Towerplease Do you have anything in the server owncloud.log when this error happens on the client?


#4

Yeah weird, would be good to do a direct PROPFIND with curl from the CLI to get it:

% curl -X PROPFIND "https://madgeeks.de/owncloud/remote.php/webdav/Ahmet Pascal Ippo Ordner YO!/Anime/Konachan" | xmllint --format -

I wonder if there are some invalid chars or weird spaces in that folder name.


#5

(might need to add a -u USERNAME after "curl")


#6

Hey guruz,

thank you for the fast reply, i am actually not into that very much, but i here i have a link to the server log: https://madgeeks.de/owncloud/index.php/s/jkioOz4lwYxw0j2

i found an Error wich said "User supribo cant read a file because of public key"? i deleted him temporary (Friends account)

So i Hope you can find something in that Server log, if i look into that file, it says nothing to me :slight_smile:

Thank you very much

Greetings Towerplease


#7

Hey PVince81,

thank you for the fast reply,

i tried the command in the CLI of my VPS, i think there is a problem with the "!" in the foldername, after i write: % curl -X PROPFIND "https://madgeeks.de/owncloud/remote.php/webdav/Ahmet Pascal Ippo Ordner YO!/Anime/Konachan" | xmllint --format -

the output is:

-bash: !/Anime/Konachan": event not found

any ideas how to get that working?

Greetings Towerplease


#8

Ah sorry, try with single quotes instead of double quotes around the URL


#9

Ok, did it. Output:

curl -X PROPFIND 'https://madgeeks.de/owncloud/remote.php/webdav/Ahmet Pascal Ippo Ordner YO!/Anime/Konachan' | xmllint --format -

-bash: xmllint: command not found
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.


#10

Ok, let's change it to:

curl -k -X PROPFIND
'https://madgeeks.de/owncloud/remote.php/webdav/Ahmet Pascal Ippo Ordner
YO!/Anime/Konachan'


#11

So, i tried that out:
curl -u admin -k -X PROPFIND 'https://madgeeks.de/owncloud/remote.php/webdav/Ahmet Pascal Ippo Ordner YO!/Anime/Konachan'

Output was:

Enter host password for user 'admin':
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotFound</s:exception>
  <s:message>File with name Ahmet could not be located</s:message>
</d:error>

I think he has Problem with getting the Path, i tried to replace the ' with " but that throws another Error.

Sorry for bothering you again :slight_smile:

any ideas? :smiley:


#12

I suggest you copy-paste the path directly from your local folder. Or in the web UI, open the rename box and then copy the name and cancel rename. Then paste the folder name as is into this curl command. I suspect that you have some weird alien spaces in it.


#13

Hey,

i have some other Problem with this. So, i tried to copy&paste the URL but it still gives me that Error, so i renamed the folder from "Ahmet Pascal..." to "TESTNAME". While in Webinterface of OwnCloud, the folder is "TESTNAME" but when go into my sync client he still uses "Ahmet Pascal..." as foldername (clean sync).

I opened up a fresh install of 9.1 and the Problem is, i think, with the Foldername. I could delete the OC Server and install a fresh 9.1 but when i upload my stuff i could rename the folder before uploading, you think that could be a solution?

My CLI input was:

curl -u admin -k -X PROPFIND 'https://madgeeks.de/owncloud/remote.php/webdav/TESTNAME/Anime/Konachan'

and output was:

Enter host password for user 'admin':
<?xml version="1.0" encoding="utf-8"?>
<d:error xmlns:d="DAV:" xmlns:s="http://sabredav.org/ns">
  <s:exception>Sabre\DAV\Exception\NotFound</s:exception>
  <s:message>File with name TESTNAME/Anime/Konachan could not be located</s:message>
</d:error>

#14

Hello Guys,

so i completely installed a fresh Debian 8 on my VPS and OC 9.1 after some Troubles i renamed the folder from Pascal Ahmet Ippo Ordner Yo! to Ahmet_Pascal_Ippo_Ordner_YO.

I Uploaded the Files back to my OC but the windows client got an Error, but this time, its not Unknown Error:

08-22 17:17:32:946 0x4f282f0 OCC::Folder::slotSyncFinished:  - client version 2.2.3 (build 6307)  Qt 5.4.0  SSL  OpenSSL 1.0.2h  3 May 2016
08-22 17:17:32:946 0x4f282f0 OCC::Folder::slotSyncFinished: -> SyncEngine finished with ERROR, warn count is 0
08-22 17:17:32:946 0x4f282f0 OCC::Folder::bubbleUpSyncResult: Processing result list and logging took  0  Milliseconds.
08-22 17:17:32:946 0x4f282f0 OCC::Folder::bubbleUpSyncResult: OO folder slotSyncFinished: result:  2
08-22 17:17:32:946 0x4f282f0 OCC::Folder::slotSyncFinished:   ** error Strings:  ("Beim Öffnen eines Ordners ist ein Fehler aufgetreten. Error downloading https://madgeeks.de/owncloud/remote.php/webdav/Ahmet_Pascal_Ippo_Ordner_YO/Anime/Konachan - server replied: Internal Server Error")
08-22 17:17:32:947 0x4f282f0 OCC::Folder::slotSyncFinished:     * owncloud csync thread finished with error
08-22 17:17:32:947 0x4f282f0 OCC::Folder::slotSyncFinished: the last 4 syncs failed

server replied: Internal Server Error

Server Log:

https://mega.nz/#!EtAjTIzb!tpVpp9NHr1KTPFeEtWp7e6VCZ1UZoRbZiWbAVsqd6hE

Something is not right here :confused:

Thank you already guys! :slight_smile:


#15

Hey Guys,

i finally found the solution!

so, i had to edit the php.ini in
etc/php5/apache2/php.ini

i made these changes:

upload_tmp_dir = /var/www/owncloud/data/upload-tmp
post_max_size = 50G
upload_max_filesize = 25G
max_file_uploads = 40000
max_input_time = 3600
max_execution_time = 3600
session.gc_maxlifetime = 3600
memory_limit = 512M

the actual Problem was the "max_file_uploads" the default value was 200.

Thank you for all your help!

Greetings Towerplease