Use push mechanism instead of polling for syncing

DeepDiver1975 · August 12, 2016, 2:44pm

Current situation

The desktop client is polling the server every 30 seconds to see if there are changes which need to be synced.
Thinking about a big installation with thousands of clients this will put quite some load on the server for nothing (in case there are no changes to be synced)

Proposal

The desktop client can connect to a web socket. Web sockets have been designed for pushing events from the server to the client.
The server part can be designed as an optional component - in case the client cannot connect to the web socket (or the server is not advertising this via it’s capabilities) it will fallback to the regular polling.

The web socket server will be an isolated server process which needs to run continuously.
(We have to analyse how this can be deployed in a scalable manner)

This stand alone web socket server will receive etag propagations as done in the owncloud server via existing hook mechanisms.

Work packages

build web socket support into the client
use web socket if advertised by the client
server advertises web socket support via the capability api
add etag propagation hook into owncloud core
build a app which pushes the hook event information to the web socket server
implement the web socket server - e.g. using http://socketo.me/

To analyse

how to scale a web socket server?
is there something with in apache/nginx we can leverage instead of using a manually started process

anon30676603 · August 12, 2016, 3:05pm

Some thoughts where collected at the bugtracker:

github.com/owncloud/client

Optimization: Server to Client Notification System

opened 08:23PM - 04 Oct 13 UTC

Fmstrat

Enhancement Server Involved

Hi, I had started writing my own Dropbox replacement in Python until I ran acro…ss OwnCloud. First off, what a great find this was! One thing I've noticied though is that since the server is 100% web based, it requires polling every 30 seconds from the client to the server: ``` 10-04 15:35:51:695 * Polling "ownCloud" for changes. (time since next sync: 30 s) 10-04 15:35:51:696 Setting up host header: "***.***.com" 10-04 15:35:51:804 * Compare etag with previous etag: false 10-04 15:36:21:691 * Polling "ownCloud" for changes. (time since next sync: 60 s) 10-04 15:36:21:693 Setting up host header: "***.***.com" 10-04 15:36:21:802 * Compare etag with previous etag: false 10-04 15:36:51:687 * Polling "ownCloud" for changes. (time since next sync: 90 s) 10-04 15:36:51:689 Setting up host header: "***.***.com" 10-04 15:36:52:392 * Compare etag with previous etag: false 10-04 15:37:21:683 * Polling "ownCloud" for changes. (time since next sync: 120 s) 10-04 15:37:21:685 Setting up host header: "***.***.com" ``` While there appear to have been some great changes in efficiency in 1.4, I still wonder if this could be taken a step farther. When I started up, like OwnCloud, the version I was working on handled most of the backend functionality via PHP. However, I also created a service "helper" in Python that handled outbound notifications by keeping a live TCP connection with each client, allowing the helper service to notify clients when they needed to poll the server, vs. the "every 30 second poll." Here's how it could work for OwnCloud: - Client A starts, polls Web Server for changes - Client A completes initial sync - Client A opens a synchronous TCP connection to Helper Service validating as a connected client with username/password, then sets itself to receive - Client B makes a change - Web Server receives and completes change with Client B - Web Server notifies the Helper Service (running on the same machine) of a change for that user from Client B - Helper Service sends a notification via the constant TCP connection to all connected clients for that user (minus Client B) to poll the Web Server for changes - Client A receives notification, and polls for changes I should also note, it would be even more efficient to have the Helper Service send the file meta-data that needed to sync so the client knew exactly what to grab, but that may be more difficult given the current structure of OwnCloud. I had completed this programmatically, however I ended up using two TCP connections for communications instead of polling a PHP server because it was more efficient and I was using RDIFF to do comparisons on files at a binary level. Thanks, Ben <bountysource-plugin> --- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/998726-optimization-server-to-client-notification-system?utm_campaign=plugin&utm_content=tracker%2F216457&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F216457&utm_medium=issues&utm_source=github). </bountysource-plugin>