It's definitely a latency/queue'ish condition problem. I ran some more tests.
Generally, the deletes take a very long time. I ran some deletes from a browser view of my account. I had 10-15 directories with thousands of files each (roughly). The "delete" of those folders took over an hour. This may because of my software raid and slow disk server setup (6TB drives at 5400prm on a software raid). I have the trash in enable as well so everything gets moved to the trash bin.
During the browser request, the browser had a spinning icon for the folders to delete. I could see the folders slowly getting deleted in batches on the owncloud server by looking at the actual data directory. A few directories at a time would be "removed" and periodically the browser would catch up and show those directories "gone."
During this time, I ran an owncloudcmd sync on that directory. A file that had been removed from the owncloud server disk but not from the web display started to appear again on my local laptop. I also cancelled the web spinning icons by refreshing the browser and noticed that it left some directories that had been deleted on the disk in the web display. Trying to "redelete" these directories resulting in an error in the browser but they there when removed from the display so the at the web browser listing was correct in the end.
So it appears that delete requests on "slow to delete" directories can create conditions where inconsistency can result. Also, this suggests that if the standard client "errors" out for any reason (say due to an inconsistency) it could result in more inconsistency. I never lost a file in all of this, but I did have inconsistent states of the client and server that could not be straightened out even after multiple sync attempts and required manual intervention. This also explained why some files would "reappear" even after I thought I had deleted them. It's also quite possible the sync client exits too early and does not allow the server operations to complete properly, much like refreshing the browser in the middle of a really long delete request. The client does seem to try and stay running for long operations but another operation from the web client or an error could interrupt it. Also, the web client (powered by the owncloud db) appears to run X minutes behind the actual file operations e.g. A folder I deleted on the laptop and sync' to the server did not show up as deleted in the browser UI for several minutes even though the owncloud filesystem on the server showed the folder gone.
The only real serious issue for me is that all of this could result in a file on my laptop that is not on the server and I could have a hard time tracking down the specifics. The client UI display fairly useless information to me when an error occurs although it is now clear that when I get the "cannot open dir" error in the UI client, it must because of an inconsistent state between my laptop, the owncloud database (which probably powers the web interface) and the owncloud filesystem files.
I reviewed a few other dirs that had large numbers of files and indeed, there were files that even after several months had not been synced to the server because of failures in the sync process earlier in the filesystem scan. The client probably needs some random noise injected into it so it does not follow the same "scan" path each time so even if there are errors greater coverage on the sync can be obtained.
The size and nature of the files that I sync may not be a good match for owncloud and my hardware/software configuration or usage of the web/laptop clients.
