Uploading large files - sometimes hangs - the last second or two

freebsd
nas
9.1.x
help

#1

through the admin GUI, I changed the MAX file size ( file handling ) and most of the time when files are being uploaded they hang at the end of the sequence. The upload window will state 1 second remaining and just sit there for a while. Eventually, it just tells you that there was a problem with the upload.

The issue appears to be mostly related to large file uploads.

I am using Freenas 9.10 and the OwnCloud Jail plug in 9.1

Thanks everyone


#2

You probably have to take a look into your logfiles and do some more debugging. Some ideas: could be something with the database (then you should see a higher load on the database at the end of the upload) or a bottleneck with you harddisk (if the uploaded files are moved from a upload folder to final destination, you can check with iotop). Not sure if the file is just moved or copied. If the files are moved it is much faster if the upload-folder and the final folder are on the same disk.


#3

Hello.

Thank you so much for the reply.
I am still having some issues with the uploading. I have done some further troubleshooting and have some more info that hopefully may assist with getting this issue resolved.

It appears that large files will SYNC correctly if I use the windows app ownCloud to transfer the files with. If I use the latest version of Firefox or IE 11, the issue persists. I can easily upload small files 20mb or so but anything larger than around 600 MB is hit or miss.

Owncloud resides in its own ZFS drive , in fact, that's all the server ( FreeNAs 9.10 ) is currently doing, nothing more.

If the Owncloud plug in appears to successfully sync the large files on the same computer I am troubleshooting with ( uploading via the web browsers ) it leads me to believe it might be an issue with some sort of configuration on the web server itself? apache ? or owncloud specific settings?

I have also disabled the server side encryption and anti virus as well.

I have checked the logs and I do not see anything that's worth mentioning, unless I am looking at the wrong logs?

I see warnings like :slight_smile:
\Files_Antivirus\BackgroundScanner::run, exception: The antivirus executable could not be found at \/usr\/bin\/clamscan","level":

which makes sort of sense as I have disabled the AV as well as encryption

thanks everyone


#4

Hi,

i doubt that this will help here. As @tflidd has pointed out that needs to be debugged from your side with the mentioned infos provided by @tflidd


#5

thanks for the reply @RealRancor

I will do my best to follow the roadmap layed out for me.

I am not the greatestlinux guru but definitively willing to learn. I will do further investigation.

From what I know so far:

  1. the files and upload are on the same disk.

  2. I am not sure where the upload folder is. owncloud is using the default Freenas jail plug in. I have not created any datastore or anything for it.

3.I am not sure if the disk are the bottle neck as there are no users except me doing the testing, the disks are 10,000 RPM disks. there's really no load at all. Just me uploading the files.


#6

Did you check your free memory? I think ZFS is supposed to use quite a lot.

If you monitor the server processes with top, you can see if a process uses extensively memory or cpu. Slow disk operations (or just too many) should be visible by an increased iowait.

If you can, use redis cache for file locking (it is not installed by default and normally your database is used which can slow down but that's normally more of an issue with a lot of smaller files).


#7

@tflidd

thanks for the the reply/post.

thanks for the tip with tops and memory! I think memory is playing a role. i see the physical amount of mem free is around 400 MB! I noticed it slowly dropping as the large was was being uploaded. Curiously, the system did not free up the memory after the file completed uploading to owncloud.

Furthermore, thanks to your tip, I took notice that while the system "appears" stuck at completing the last few seconds of the upload, CPU 0 spiked up at 100%.

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND

5164 www 1 52 0 683M 67484K lockf 0 2:06 100% httpd

Does owncloud compresses/uncompresses files being uploaded to it?


#8

No, owncloud doesn't use compression and normally mod_gzip should be disabled.
The definition of "free" memory on Linux is a bit different because Linux uses memory if it is not really needed for the processes:
http://www.linuxatemyram.com/


#9

Would be nice for this to get solved. I've been dealing with this for some time now. I have the same problem. Symptoms are identical.

I too run this as a FreeNAS jail.

Owncloud 9.0.4 running in jail of FreeNAS 9.10

experienced on older versions of both as well.


#10

Please update to the latest available release of the version before start debugging, some issues could already be resolved in the new version: https://owncloud.org/changelog/#latest9.0

Without more information from your side, it will be difficult to help you. As you haven't posted any error-logfiles I suppose there aren't any of them. So it could be some restriction of your system (do you use a real database and not sqlite?). Can you use and configure a cache such as redis? If you upload via FTP or other protocols, and you limit the bandwidth that the upload takes the same time, do the upload hang as well?


#11

I'll get to the latest version before proceeding. Until then, what is the best way of finding out if I use sqlite, redis, etc. (what process to look for)?
The FreeNAS owncloud jail comes prepackaged.
I am fairly experienced with POSIX based systems but not database unfortunately.

Also, I apologize if my initial statement sounded like a complaint. Not really critical for me, just annoying. Owncloud is by far one of my favorites! =).


#12

Just upgraded to 9.1.3, went through the procedure of increasing max upload size:
https://doc.owncloud.org/server/9.0/admin_manual/configuration_files/big_file_upload_configuration.html

Stops at 8 seconds.

During the upgrade procedure I discovered that I use sqlite3 (according to config.php)

No log event upon hang. Should I increase the log level or is there a certain limitation with sqlite3?


#13

Did you check the logfiles of other services (directly webserver log, enable logging off php). sqlite has serious performance limitations, so if you can use a real database (mysql or postgresql) you should use them.