Desktop Client not syncing (650,000 files)

Desktop
1

Expected behaviour

Files should sync within a few minutes.

Actual behaviour

Files sometimes don’t appear to sync for days. The Desktop client is basically permanently ‘checking for changes’. We’ve left the clients to fully sync for days to ensure its been through the entire thing at least one.

The server itself is fine in terms of resource usage. No where near hitting its max resources.

Same for the desktop client, resource monitor isn’t showing anything being anywhere close to maxed out. (New Dell XPS laptops with SSDs).

If I update a file on the SMB share, its almost instantly reflected in the Web UI. And vice versa. The problem is only with the desktop clients.

File sizes vary, but the majority are small MS Office files.

I’m getting the impression that 650,000 files (603GB) is too much for the desktop client to keep up with.

I’m unable to include log files due to the filenames, but searching for any errors is fruitless.

Steps to reproduce

  1. Installed Owncloud on an Ubuntu Server.
  2. Sync 650000 files
  3. Use Windows 10 laptops with latest desktop client to try and sync files.

Server configuration

Operating system:

Web server: Apache2

Database: MySQL

PHP version: 7 (5.6 for Redis)

ownCloud version: Latest

Storage backend (external storage): SMB

Client configuration

Client version: Latest Available

Operating system: WIndows 10

OS language: English

Installation path of client: Default

Logs

Please use Gist (https://gist.github.com/) or a similar code paster for longer
logs.

Template for output < 10 lines

  1. Client logfile: Output of owncloud --logwindow or owncloud --logfile log.txt
    (On Windows using cmd.exe, you might need to first cd into the ownCloud directory)
    (See also http://doc.owncloud.org/desktop/2.2/troubleshooting.html#client-logfile )

Basically every entry from OCclient.log looks like this

05-22 14:42:13:567 [ info sync.csync.updater ]: file: DataShare/Directors/Clients/filename.file, instruction: INSTRUCTION_NONE <<=
05-22 14:42:13:567 [ info sync.csync.updater ]: Database entry found, compare: 1526986279 <-> 1517504605, etag: 5b03f8f537316 <-> 5ad87bd362c75, inode: 0 <-> 361884, size: 0 <-> 0, perms: 4fd <-> 4fd, ignore: 0
  1. Web server error log:

Nothing even from today.

  1. Server logfile: ownCloud log (data/owncloud.log):

Primarily ‘Trusted Domain Errors’ for random IPs. Nothing in relation to this it seems.

2

The issue is with the external SMB storage you seem to be using. You will need to do a complete file scan and wait for the results. Then the desktop client will kick in. In the Enterprise Version we then have a listener which keeps up with the changes on the SMB side and you will need to do a couple of additional tweaks and it will work.

So in any case you want to do a file scan once and take it from there or contact sales@owncloud.com for a project.

3

In addition to what @hodyroff have said, I guess that in case of a huge amount of files like this, the performance will depend on how the file tree is shaped.
If you have a all the files in the root directory (relative to the SMB connection), whenever a change is reported to the client, it will need to scan all the files to know which one has changed.
If you have half of the files in one folder and the rest in another, the client can know which folder has changed and ignore the other one (if it hasn’t changed in that interval). In this case the client checks 1+X files instead of 2X.

I’m not saying that you must have a perfect binary tree structure, but having all files in one folder will definitely hinder the performance of the desktop client.
You should check if this case affects you or not.

4

Many thanks for the answers.

@hodyroff - is what you are describing different from running occ files:scan --all on the server? (That takes roughly 17 minutes to do a full scan). As I mentioned, updating a file within the SMB drive,will be correct on the webui within seconds (and vice versa), it is purely the desktop application that seems to be struggling - or did you mean the Enterprise Desktop Client has the extra listener, or have I misunderstood your description?

@jvillafanez - Thanks for the insight. The files are stored in a typical ‘Office’ type way, different shares for different departments but some of these will certainly be top heavy. The worst offender is a raw copy of all their websites (that the client claims they HAVE to have locally synced with them at all times ) so tons of tiny html/js/css files etc.

5

Maybe you can limit the access to shares to specific people. For example, if you have a “sales” share for the sales department, you can configure ownCloud so only people from the “sales” group (or any other) access to the “sales” share. This way, people that isn’t in the sales department won’t sync that share.

While the configuration will be a bit more complex, this should also improve the performance a bit by not syncing some of the files.
It might not bring a lot of performance improvement for the affected people, but you’ll sync whatever you’re interested in.

6

Many thanks @jvillafanez. I’d love to! Unfortunately this company is mainly the two owners, who spend a lot of time out at sea. They claim they need access to all 650,000 files at all times.

Unfortunately they were using Dropbox beforehand which worked perfect for them! Might have to eat my words here…

I’ve used Owncloud for a long time and its always worked great for me, I think this huge amount of files may just cripple the desktop client :confused:

7

Please post your config.report. I am struggling to understand if you use SMB as primary storage or if you have it as external storage for those users. As you compare with Dropbox, there is no external storage, therefor I might not understand your setup yet.

8

When we took this client on they were using Dropbox as their main file share.

We proposed keeping all their files in house rather than Dropbox. - we began to bring them in line with best practices, purchased an onsite Windows server to do Active Directory, SMB share etc. Dropbox data was migrated to the server/SMB shares.
As they’re at sea with no internet access, they advised they needed to have ALL files ALL the time. A local sync was required. We initially tried using MS’s syncenter but the amount of files seems to cripple it it.

Next try was Owncloud - no files are hosted on the Owncloud server, it’s purely for connecting into the SMB Windows share. When we originally deployed it, we let them both fully sync for a couple days - initially it’d take about 2 minutes to sync a file from desktop > owncloud > other desktop. Now we’ve let the client loose with Owncloud and these continual ‘Checking for changes’ started happening.

I can’t seem to attach (images only) or paste (too long) the report.config - I’ve put it here https://pastebin.com/F37rRxXe

9

I’ve the same issues. Raspberry pi serving files from mounted smb shares. I’ve configured the system to access the local mount points. It is around 12Tb. I’ve shared smaller shares with other users on the same owncloud installation and it is working well. Is there any solution?