Decrypt-all too long. Do it with several steps?

encryption
9.1.x

#1

Hi

I want to deactive server side encryption because I have too many issues with and I do not use external storage.

I'am currently doing the decrypt-all, user by user.
For the moment, no major issues, except it is verrrryyyyy loooonnnng.

I have entered single user mode. So all my collaborators have to wait.... too long!
I have no exact idea how much time it will take to decrypt all the file?

1) is there a method to know how much files remain? Maybe a query in the database?

2) can I do the decrypt with several steps? i.e. do it at night, then come back to normal mode in the morning, even if decrypt-all is not finished?

3) If I restart decrypt-all, will occ start where it has stopped?

To come back in normal mode, and if decryp-all is still not finished, I have understood that I have to re-active encryption mode.

4) Is the user able to access both (already)decrypted and (still)encrypted files?

5) If the user updates/creates files during the day, I guess these files will be encrypted again, even if they have already been decrypted during last decrypt-all?

6) If I relaunch occ decryp-all, will it decrypt the new/modified files?

Thanks


#2

Very long is a vague description. Normally ownCloud does on the fly encryption and decryption, so I would expect that it can do a few GB per hour. This certainly depends a lot on your system and the amount of data you have to decrypt. If you have very little resources, you might improve the performance by optimizing some services. top and other tools might help you to find the bottle neck.

I would try to decrypt all at once. I'm not sure how well this intermediate states (partly decrypted) are tested.


#3

I have 3 Tb of data. Yes, you read "terra".

during decrypt-all, I can see the corresponding php process does consume between 0 and 30% of cpu, rarely more.
So there is place for other.

and another question is:

7) Can I launch several decrypt-all in paralel, one for each user?


#4

I answer myself to question 1:

the query I use:
SELECT * FROM oc_filecache WHERE storage = 1 AND mimetype != 2 AND size >= 1024 AND mtime <= 1490790000 ORDER BY fileid DESC

where:
- "storage" is the storage id associated to the user currently decrypted-all. You can find it in the oc_storages table.
- "mtime" is the timestamp when you have started the decrypt-all. You will find online calculator to compute mtime.

The quantity of results is an approximation of how many files remain to be decrypted.


#5

Not sure who from the developers is responsible for encryption. This list is not up to date: https://github.com/owncloud/core/wiki/Maintainers

@hodyroff can you help here?


#6

I had to dig deep into the decrypt all code recently. The occ encryption:decrypt-all command can be used to decrypt only a singe user. however, running it will deactivate the encryption completely and you have to re-enable it with occ encryption:enable before you can run occ encryption:decrypt-all again.

When encryption is disabled all encrypted files become read only. So, users will still be able to access all their files, but they won't be able to edit existing files. Keep in mind that the vast majority of files is only read, very few files are actually collaborated on. YMMV, so you will have to decide how long your user base can live with existing files being read only, before starting to eat you alive.

All you need to do to allow decrypt-all to run when decryption is disabled is to comment the return in https://github.com/owncloud/core/blob/stable9.1/core/Command/Encryption/DecryptAll.php#L128 (make the line read // return;). That should allow you to do the following:
1. disable encryption (remember, existing files are in effect read only, new files will already be stored decrypted)
2. run multiple occ encryption:decrypt-all <userid> commands in parallel

I haven't tested this, and would recommend you try this in your test environment with two users: one with many file one with fewer. Then first start decrypting the user with many files and while it is decrypting that user also start decrypting the second user so we can see if they interfere. AFAICT there should not be a problem.

Can you test this and report back? We might then create a PR that basically removes that return. Might need changes to the log messages or an additional 'do you really want to start decryption', along with a force option to make this nicer.


#7

Thank you @butonic

If I understand well:
- Your proposition allows to disable encryption, so users can continue to use owncloud. They will be able to add new files, and these files will be stored no encrypted, as they are.
- Users will be able to read existing (not encrypted) files and the new ones.
- But they won't be able to modifiy existing encrypted files. What happens if they modify an encrypted file? Is the file read-only on the user disk? What do you mean by "read-only"?
- In the mean time, I can launch decryp-all. By "in parallel", I guess you mean that users can still use owncloud, sync, ...?


#8

Unfortunatly, I don't have a test server. Only production. I can't take the risk to test.


#9

The decrypt-all procedure has just finished. 6 days!

I have unset single-user mode and the owncloud client has restarted, on all computer (my collaborators and I).

But what a bad surprise! the client seems to resync ALL the files :anguished:. For me and ALL my collaborators.
It represents 20 days for me (1,7Tb), and maybe more for some of my collaborators who have a bad connection.
During this time, we can't work on the files, or we might have version conflicts.

This behaviour is not documented anywhere. If I could be aware of that, I would have think twice before launching a decrypt-all.

:rage:


#10

Most likely because this is not the expected behavior. Please report a new bug to the developers of ownCloud reachable at https://github.com/owncloud/core/issues


#11

Ref to your bugreport below.

Not quite sure if the client issue tracker is the right place. If the issue is on the server side and the server is telling the client that the files have changed then the client is working as expected and the bug is caused by the server. But the devs at the tracker will probably tell you to re-open in core if this is an issue at server side.