External Drive more issues - "umount and mount" - files go missing from UI

Server
#1

System Details

Device - Raspberry PI 4
OS - Stock Raspbian linux

Goal

  • Use Owncloud with Raspberry pi
  • Use multiple external drives
  • Unplugging/plugging back external drives should not cause files to be missing
  • Unplugging should not end up with uploaded files being written into the mountpoint directory in the sdcard

Steps

  1. Owncloud with RPi and external nvme 1tb ssd drive (no issues all good with mounting and permissions set. ).
  2. Uploading a file writes it into the external drive.
  3. Verified by unplugging from Rpi and checking on a mac
  4. Note: File system used “exfat”

Issue 1

  1. Unmounting the disk and remounting it makes the file disappear from the UI
  2. Can list the files in the shell and all the files are there

Issue 2

  1. When the device is unmounted uploading a file gets written to the local SD card in the mountpoint
  2. Tried protecting this with the ‘chattr +i /mountdir’
  3. Works as the file does not get written to the mountpoint
  4. However, no I end up with “do not have permissions” issue again

Anyone else seeing these issues?

Thanks in advance
Raj :pray: :slightly_smiling_face:

#2

Nothing wrong here, AFICS.

It’s up to the admin (you) to assert, that the external drive spec has to be removed BEFORE the volume is unplugged.

Not the best decision. wouldn’t do that without a very serious reason.

1 Like
#3

‘assert how?’

  • Not sure if I understand this correctly. I always unmounted the mountpoint and the device before re-mounting it. Never actually unplugged the device. Its still stuck in the ‘do not have permissions mode’. Easy to manually fix by trial and error. Am trying to find a repeatable way to I can script it as needed.
  • Also am still confused why owncloud does not read the files in the usb anymore. It only shows up files I write afterwards.
  • The files written before the unmount, are not to be seen anymore in the UI.

exfat vs ?

  • exfat - would like to store files up to 128Gigs.
  • And arbitrarily large drive sizes - fully expect 100TB drives to become available in the near future.
  • Any recommendations on alternative?
#4
Fixed Sync Issue
  1. Storage Sync on re-plugin: The external storage folder/mount has to be manually deleted from the admin>settings>storage. It has to be then readded from the owncloud UI, then and only then it works.
  2. How to read files that were already present: I tried moving a file from a different directory into the usb drive. I got a ‘file ownership’ related warning. So changed all the files to ‘www-data’. Then all the files that got added show up.

Note: File Resync - This is still a work around and I really do not like it not being automated. Owncloud should periodically check if a new drive got added. Something it had seen before. Anyway, acceptable limitation for now.

Open Issue
  • Will the files still belong to the specific users?
  • I’m hoping a hash of the file or something like that is what is in the mysql database. So when the drive is plugged back and remounted, and then added back into owncloud, it makes the appropriate associations of the user who is the file-owner.
#5

In my understanding: Any file under ownClouds control is owned by user www-data, remember the chown -R www-data…. :wink:
This is true until you download the file. At this moment , the ownership is transferred back to you.
So there is no need to “hashing” something.

1 Like
#6

Correct, for debian based distributions.

It’s generally a bad idea to use an external drive for hosting website files, that you are going to unplug and access regularly with another Linux system.

If you don’t want to constantly adjust file permissions and ownership, you absolutely need a dedicated external HDD for your RPi installation that you don’t unplug or access from other systems.

#7

Not using it for storing website files. Just home storage. Thats what Owncloud is for right? Or at least thats what I presumed. I’m running out of my google storage and have been thinking for a while it would be a good idea to move to an on-prem dependable storage solution that is low cost.

From a usage everyone in my family knows how to use google drive and to go to a URL to drag and drop, or to download. That ease of use is something I love about Owncloud. Its super easy to setup and configure so far. But still a tad geeky for anyone to use out of the box.

That said my goal is to try to make the experience a little more intuitive and robust. I’m just thinking of all the negative scenarios of what all can go wrong if I hang a bunch of USB drives to my RPi. We all have a ton of these USB drives and most of the times we forget where we kept. Or, if we lose those drives then theres no backup. Many little scenarios which a few scripts might solve (or may not). Not trying to trivialize it though.

Thanks for all the support here. I do not think would have figured out much otherwise. :pray: :slightly_smiling_face:

#8

The mount behaviour very much depends on hw exctly the file system gets mounted.
On my ubuntu box for example, when I plug in a USB stick and it gets automounted, all files and folders appear as owned by myself, the current desktop user. read permission for my default group is set, and read permission for othes. But my current user is the only user who can write.

So this type of mount would not work with ownCloud, when your apache server runs e.g. as user www-data, no write access.

That is just my example here. Not sure, if you do automount, if so, you can specify that your onCloud USB Stick should have different permissions. Find out the UUID of the stick with e.g. sudo blkid /dev/sdb1 (if it appears as sdb1) and then you can craft an entry in fstab to
assign UID=33 (if www-data is uid 33, that is what it is for me) or even umask=000 to make it writable by everybody.

Details are discussed e.g. in


I cannot give the exact config files you need to touch, that is probably raspian specific, soemwhere between fstab and hal/udev rules.

Take care and keep that stick safe. Data-corruption on USB sticks is such easy thing to do…

1 Like
#9

The ownCloud server application is a website, therefore I consider all files saved inside the server website files.

#10

Hello Erik,

IMHO :pray:

Agreed, the owncloud server application provides a web-based easy access. But truly, I would not diminish the value of calling it just a website :slightly_smiling_face:. As if we do, then everything on the internet is a website. I think the files themselves when stored in an external drive are become a powerful storage solution for homes/small-businesses etc.

With newer mobile devices, we will soon be creating 4K/8K content. Even 1 TB storage will not be enough in the cloud. Cloud hosted storage will not be able to keep up. Owncloud has a unique opportunity here with enhanced support for external storage devices.

One again, just my humble opinion. :pray:

thanks
Rajesh.

#11

Hello jnweiger,

Thanks for the detailed explanation and pointers. I will keep these in mind.

Rajesh

#12

@nrajesh,

I fully agree with @eneubauer here. OTOH I see your requirements for your storage system. But please remember, whenever you use ownCloud in a manner it was not created for, you cannot blame ownCloud for any of your issues.

Just my 2 cents.