Hi together,
thanks for this piece of wonderful software! I run Ownclod Ocis 5.0.6 via docker compose on my VPS. My data storage is pointed to a 4TB HDD which is only used by owncloud. This hdd is now almost full (98% used) with only 1,5TB data used by Owncloud ocis according to the Owncloud Webui. So, I assumed that lots of overhead is coming from file versioning as well as the trash bin. I have now two issues:
- I cannot purge my trash bin: When I try to access it via the Webui it just loads forever and I cannot access anything. I tried to delete it via the command line with:
docker exec ocis ocis storage-users trash-bin purge-expired & disown
The process finished after a long time but the HDD space is not released.
- I tried to run:
docker exec ocis ocis revisions purge -v -p /var/lib/ocis/storage/user & disown
This command now runs for more than 24 hours and does not finish. I wonder what goes wrong and I cannot find anything in the logs pointing to the issue. Anyone experienced similar problems?
Thanks a lot for your help!
Kind regards
Hi @swooshy! Thank YOU for using Ocis!
So, first purge-expired command should not take longer than 5 seconds. It does not execute the command itself, it sends an event to the ocis server, which then executes the command. That means server MUST be running when the command is executed. Otherwise it will just try to connect to nats forever. (The error messages are swallowed somehow, which is not intended.)
Did you purge-expired command with a running ocis instance?
The revisions purge command however should be working without a running instance. You can try to run the command for a single space using the --resource-id parameter. That should be much quicker and you can see if the versions are gone.
If this is successful, we can try running the command per space (for all spaces) or we speed up revisions purge by running it in multiple threads.
1 Like
Thanks a lot for your reply @kobergj !
I was running the purge expired command with the docker instance running. I restarted the docker and now the command executes but results in an error:
Command: docker exec ocis ocis storage-users trash-bin purge-expired
Response: Error publishing message to topic: nats: timeout
This is weird, I am sure that my ocis instance is running, I can even access it via the webui. Edit: I just tried the command again after a bit of time and now it worked, but still no release of disk space.
I tried revisions purge again with --resource-id and it completed after a few minutes I got the message:
Would delete 213 revisions (639 files / 213 blobs)
Which did not release any disk space and interestingly I can still access and download file versions on the ocis webui.
Any other ideas? I guess the main issue is the trash bin, because I just deleted a 30gb file via the webui and it did not release the space on my disk.
Ok. That means revisions purge is running correctly. It needs the --dry-run=false flag to actually execute the changes. This is a safety measure so you don’t delete your revisions accidentally. Just run the command again with the --dry-run=false flag and revisions should be gone. That also means that revisions purge is too slow for big deployments. I’ll have a look how to speed it up.
Not sure whats wrong with purge-expired. Maybe there are no expired trash-bin items?
You could also check ocis storage-users uploads sessions maybe you have a lot of unfinished uploads that cost storage space.
1 Like
Many thanks @kobergj , unfortunately, using the --dry-run=false flag makes it timeout again without finishing the job (Ocis server is running). I also checked the files and the file revisions are still visible on the webui.
Fortunately, using ocis storage-users uploads sessions finally cleared up at least ~ 100gb of space on my disk.
But still, right now according to the webui I am using 1,6 TB of space and the disk usage is at 3,5 TB, so I assume it is still the file revisions and trash bin which blocks my space. Any other suggestions? I tried to set the log level to debug, but it did not reveal any helpful information for further debugging. I would highly appreciate help as I really like owncloud ocis and would love to stick with it as my cloud solution!
Thats strange. It should not take much longer. Just to clarify: Did you run ocis revisions purge -r {spaceid} --dry-run=false?
If this takes a long time you can also run the same command with -v flag. Then you can observe that there is at least some progress.
Also it’s best to not run ocis server when executing revisions purge because if new revisions get added while it tries to delete revisions it can get confused.
Btw: From ocis 6+ you can use OCIS_DISABLE_VERSIONING envvar to avoid creating revisions at all.
1 Like
Thanks for your help @kobergj
when I run docker exec ocis ocis revisions purge -r '1b09d452-b890-48a6-b6a9-a8cbd081d293$8f538924-6ec7-4f0d-80b5-66d9a59830d3' -p /var/lib/ocis/storage/users --dry-run=false -v
This is the only verbose output I get then it hangs without more outputs.
Looking for nodes in /var/lib/ocis/storage/users/spaces/8f/638374-6ea8-4f0d-80c4-66d9b49830a5/nodes/*/*/*/*/*
Any other ideas? 
I also checked the health status of my disk, the HDD itself seems to be intact.
It seems to just take too much time for the globing. Could you try another space? Or you only have one space?
1 Like
Unfortunately, I only have one space. Any suggestions left or I am forced to reinstall everything ?
Edit:
@kobergj I updated to 6.2 to see if it resolves the issue, it does not but I noticed when I run the commands after a while the RAM usage gets very high and maxes out at 62gb on my machine and then ocis crashes.
I noticed these errors in the log:
{"level":"error","service":"storage-users","pkg":"rgrpc","traceid":"14bba50b1f7dd96df0f0d7de981ca328","path":"/var/lib/ocis/storage/users/spaces/8f/538954-6ca7-5f0d-80b4-66d9a49830a4/nodes/fb/79/8f/a2/-4e77-46fa-88c4-336a3dad06fb","nodeid":"fb898fa2-4e77-46fa-88c4-746a3dab06fb","attrs":{},"messagepack":"","time":"2024-08-19T19:02:37Z","message":"missing parent id"}
{"level":"error","service":"search","error":"internal error: list container:internal error: Missing parent ID on node","time":"2024-08-19T19:02:37Z","message":"error walking the tree"}
{"level":"error","service":"search","error":"internal error: list container:internal error: Missing parent ID on node","spaceID":{"opaque_id":"1a18d434-b980-47a6-b6a9-a8abc081d283$8f538974-6ea7-4f0d-80b4-66d9a4983ea3"},"time":"2024-08-19T19:02:37Z","message":"error while indexing a space"}
Do you think this somehow related to the issue?
So I think the problem is that when you run the command, ocis globs the space to find all files. This globbing just takes too long and you run into some sort of timeout. (Which one?)
The problem is that you can’t purge revisions from this space because the command cannot find all files. We would need some kind of pagination to avoid long taking globs but I don’t know how to achieve that yet.
I have one more idea: You can create a new space and just copy all contents into this. This will NOT copy the revisions, only the newest version will be copied. If this is successful, you can disable and delete the old space. That should free up the space
Regarding the latest issue: That is strange. purge revisions should not have any impact on ocis. It is running as a separate program. Or the command is crashing? Or are you talking about trash-bin purge expired?
EDIT: Actually there is another thing that is strange. Can you confirm that
docker exec ocis ocis revisions purge -r '1b09d452-b890-48a6-b6a9-a8cbd081d293$8f538924-6ec7-4f0d-80b5-66d9a59830d3' -p /var/lib/ocis/storage/users -v
works but
docker exec ocis ocis revisions purge -r '1b09d452-b890-48a6-b6a9-a8cbd081d293$8f538924-6ec7-4f0d-80b5-66d9a59830d3' -p /var/lib/ocis/storage/users --dry-run=false -v
hangs? There might be another issue if that is the case.
1 Like
Many thanks for your help @kobergj !! I tried to copy everything to a new space via the webui and got this error after around 2 seconds. Is this maybe related to the fact that I don’t have enough free space on my disk to make a replication of the whole data?
About the docker revisions purge command:
It has a very strange behavior to me. The only time I was able to achieve this response: Would delete 213 revisions (639 files / 213 blobs) was when I tried the command for one of my previous posts here. All other attempts before and afterwards failed without changing anything. So right now when I run the command with or without --dry-run=false it will time out.
Maybe I messed up my installation at some point and I would be better off to reinstall everything? It would be annoying to re-migrate the 1,6 TB of data but maybe easier than to find a solution ?
Ok. Then I think the problem really is that the globbing of the space takes too long. But still unclear to me is why the command does time out. It does not have an integrated timeout mechanism. Does it just appear to hang? Maybe if you just let it run for a long time it will finish? Or is it killed by the os?
1 Like
One more idea comes to my mind: The revision purge command also allows removing the versions of one particular file by passing the fileID to the -r parameter (instead of the spaceID). This should obviously work much faster. Maybe we can craft a workaround with this?
1 Like
Many thanks @kobergj
I think it is another problem because when the command finished, it did not take too long (maybe 1-2 mins?). Unfortunately, I got very busy now with work related stuff and can’t look further into it by checking system logs and your other suggestion. I just quickly reinstall the whole thing and use a backup.
Anyway, thank you very much for your help and suggestions, I highly appreciate it!
1 Like
Then this will remain a mystery 
No problem @swooshy
To close this up: I just merged Improve Revisions Purge by kobergj · Pull Request #9891 · owncloud/ocis · GitHub which will speed up revisions purge command drastically and is available with the next release.
