OC vs. NAS: benefits, cons and issues

[sebschlicht]

Dear OC-community,

I want to use OC on a Pi as my central data storage.
I’ve successfully installed OC and starting to think about the cons of OC which I believe to have found.
In particular, I’m wondering whether OC is worth the effort of switching from my current NAS-based setup.

Requirements

1. backup files on a daily basis, at least
2. support for unix-based computers and Android-based devices
3. make public (in a directory shared between users) multimedia files accessible to SmartTVs
4. run on a Pi (model 3)
5. store files on an external NTFS-formatted USB drive
6. play multimedia files on Android-based devices directly from server (remotely, without a permanent local copy)
7. have all bundled in a single application per client device
8. update meta data independent of the file content

Current NAS Setup

My current NAS-based setup is as follows:
I have an external USB drive mounted in a Pi which is accessible via SSH.
Several unix-based computers push their local files to the Pi on a daily basis via rsync and cron.
The Android app TotalCommander allows to connect to a Samba server, to sync files and play multimedia files remotely.
miniDLNA enables SmartTVs to access public multimedia files.
As you can see, this setup meets all requirements with a single application/script per device.

Experiences with OC

To test OC I’ve imported my music library (~40GB).
OC meets requirements 1-5 when combined with additional software such as miniDLNA (#3).

Unmet Requirements

6/7: The Android app which I’ve tested (Nextcloud) wasn’t able to play multimedia files remotely.
Technically OC could meet this requirement when you have the OC app Music installed on the server and use a browser on the Android device.
However, this wouldn’t comply with requirement #7.

8: This requirement seems to be not meet either.
If you replace a file with the same file but different meta data, the whole file is uploaded.

Additional Issues

Next to my hard requirements I’ve noticed additional issues:

Both the server as well as the Android app downloaded the music files at a whole.
They didn’t support on-demand streaming (c.f. YouTube) which would be very nice to have.

Though cloud services (c.f. DropBox) manage to sync incremental changes of a file, OC doesn’t.
If you marginally edit a large file it will be uploaded at whole.

OC puts some considerable traffic on my computer’s network device.
In my opinion this is partly inefficient, could be improved and mainly results from two issues:

1. sync init size (+ duration)
2. client-pulls

1: OC takes more than a minute just to check whether any of my music files has been changed remotely.
I can see that it synces 12MB of data multiple times during this check.
Again, I’ve just put my music files into OC and I fear what happens if I’d put all of my files in there.

2: The check for remote changes is done on a regular basis due to the client-pull concept.
There seems to be no option to make the server notify registered clients about changes yet.

Long Story Short

Are there possibilities which allow OC to meet all my requirements?
Any thoughts about the benefits and cons of OC compared to my current setup?
Any comments on these issues regarding how to avoid them or when they might get resolved?

[jvillafanez]

6/7: The Android app which I’ve tested ( Nextcloud ) wasn’t able to play multimedia files remotely.
Technically OC could meet this requirement when you have the OC app Music installed on the server and use a browser on the Android device.
However, this wouldn’t comply with requirement #7.

Having a player inside the app seems unlikely to happen, unless the code is trivial. Note that the ownCloud app is focused on accessing the files in the server, it isn’t a media player.

If you can contribute to the ownCloud app, you can check with the developer what options are available. (https://github.com/owncloud/android/issues)

8: This requirement seems to be not meet either.
If you replace a file with the same file but different meta data, the whole file is uploaded.

If the metadata is inside the file, changing the metadata implies changing the file, which implies bringing the new file.

Anyway, I’m not sure what is this requirement about. If you mean that the file in your local computer should be synced in the server using the desktop client, there are improvements incoming for ownCloud 11 (ownCloud’s zsync - no release date yet). I wouldn’t expect changes in the clients in this regard until ownCloud 11 is officially out.

1: OC takes more than a minute just to check whether any of my music files has been changed remotely.
I can see that it synces 12MB of data multiple times during this check.
Again, I’ve just put my music files into OC and I fear what happens if I’d put all of my files in there.

This should be ownCloud’s metadata, and only if some of the files changed. We need to look for the file that changed inside the server. Note that the amount of data will depend on the file structure you have in the server: if you have all the files in one folder we’ll need to get the metadata of all those files to check what have changed; if you have 2 folders with half of the files in each folder, we’ll have to check half of those files plus a bit extra (to know the folder containing the change)

[sebschlicht]

At first, thanks for your reply.

I feel like you feel attacked and I want to explicitly state that I did not intend that.
I don’t want to criticize OC or the hard work of its developers, I want to check what benefits OC may give me, compared to my current setup, and how to achieve that best.

Having a player inside the app seems unlikely to happen, unless the code is trivial. Note that the ownCloud app is focused on accessing the files in the server, it isn’t a media player.

If you can contribute to the ownCloud app, you can check with the developer what options are available. (https://github.com/owncloud/android/issues)

Thanks for this information! I can see that file sync is the primary use case for this app.

Personally I think that if you use your cloud for all your data, you won’t want to miss a multimedia player which can play files remotely from within the app, on the long term. Who has enough space on his device to sync all the files?

Unfortunately I’m neither willing nor able to contribute to OC or its app yet.

If the metadata is inside the file, changing the metadata implies changing the file, which implies bringing the new file.

Anyway, I’m not sure what is this requirement about. If you mean that the file in your local computer should be synced in the server using the desktop client, there are improvements incoming for ownCloud 11 (ownCloud’s zsync - no release date yet). I wouldn’t expect changes in the clients in this regard until ownCloud 11 is officially out.

As far as I can tell the only meta data that differed were the modified timestamp, no in-file meta data such as multimedia tags. Would different file permissions lead to a full file transfer?

Here is the process I’ve used to check for meta-data-only transfer:

1. have a local music file f, outside the OC folder
2. insert f into OC via SCP and occ file scan
3. sync local OC folder => downloads f from OC
4. paste original f into OC folder => differ in timestamps
5. sync transfers the whole file f to OC

Regarding in-file meta data: Of course, delta sync would be very nice to have as well but that’s a different issue.

This should be ownCloud’s metadata, and only if some of the files changed. We need to look for the file that changed inside the server. Note that the amount of data will depend on the file structure you have in the server: if you have all the files in one folder we’ll need to get the metadata of all those files to check what have changed; if you have 2 folders with half of the files in each folder, we’ll have to check half of those files plus a bit extra (to know the folder containing the change)

My music folder is a quite balanced tree with subfolders for interprets and a second level for albums.
I don’t see that this is solely an issue for unbalanced folders with many files.
Plus, I didn’t change a single file, be it local or remote.
What I can see is that OC checks for updates in each and every subfolder.

Don’t get me wrong, that’s okay. I’m just wondering about the speed of checking for changes because rsync compares the two parent directories and all its subfolders within seconds - regardless if files have been changed or not.

Again, it’s not like I want to talk bad about OC. I want to understand its limitations and where I may benefit from OC, as compared to my current setup. Thanks.

[jvillafanez]

Personally I think that if you use your cloud for all your data, you won’t want to miss a multimedia player which can play files remotely from within the app, on the long term. Who has enough space on his device to sync all the files?

The problem with this approach is that you don’t want to miss a media player, another one doesn’t want to miss a photo editor, another one a rich text editor, etc. What’s more, you might not like the simplest media player the app can bring, but a full fledge one. The same for the rest of the “sub” apps.
Meanwhile, there are still syncing problems the app has to deal with, which would also prevent the access to the file.

In addition, I think the app should keep a rather small file size. I don’t want to have a 500MB app doing everything while I’m using just a 10% of it, maybe because I don’t have a large music collection and I’m not interested in playing music on my phone (I’m exagerating, but it’s something that could happen).

This is why very likely you’ll have to rely on 3rd party apps, and ownCloud should have the focus on provide access to the files from those 3rd party apps.

I think streaming is what the app should allow, so your favorite player can reproduce music from the server without downloading the whole file. However, allowing streaming is something different than having your own media player within the app.

As far as I can tell the only meta data that differed were the modified timestamp, no in-file meta data such as multimedia tags. Would different file permissions lead to a full file transfer?

The modified timestamp is what the server uses to detect changes. This should be fine and fast on quite a wide range of scenarios. If the modified timestamp has changed, the file has changed and we need to get it again.
Changing the permissions might not change the timestamp of the file, so such change won’t be detected. Note that this will depend on the specific FS connector (for external storages), but I think all of them behave like that.

My music folder is a quite balanced tree with subfolders for interprets and a second level for albums.
I don’t see that this is solely an issue for unbalanced folders with many files.
Plus, I didn’t change a single file, be it local or remote.
What I can see is that OC checks for updates in each and every subfolder.

Maybe you should double-check if there is some app or service touching those files and modifying the timestamp, otherwise the changes shouldn’t reach the ownCloud’s root folder so the desktop client shouldn’t sync anything.
Just in case, update the desktop client to the latest version, and if it still happens, open a ticket in either here (new post) or in the client repository in github. Have client logs prepared.

Don’t get me wrong, that’s okay. I’m just wondering about the speed of checking for changes because rsync compares the two parent directories and all its subfolders within seconds - regardless if files have been changed or not.

There are always tricks that some apps can use and others not. rsync requires to be installed both in the client and in the server, and uses it own protocol. Just by removing the rsync in the server you want be able to sync data to that server:

bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(235) [sender=3.1.2]

The point here is that we’re bound by the limitations of the HTTP(S) / WebDAV protocol, as well as the client-server architecture. If you want to compare you should do it with other similar products with similar limitations.

I feel like you feel attacked and I want to explicitly state that I did not intend that.

I’m staying as neutral as I can. I’m just trying to explain what I think ownCloud won’t be able to do and what it might do. I might be wrong in both cases.