Big files upload into Google Drive external storage

Expected behaviour

Sync big files into google drive external storage

Actual behaviour

ownCloud saying the big file (3Gb of size) are in the blacklist.

Steps to reproduce

  1. Fresh install of ownCloud Server
  2. Update php.ini props to big files
  3. Integrate with google drive
  4. Try upload some big file

Server configuration

Operating system: Ubuntu 18.04

Web server: Apache2

Database: mysql 8

PHP version: 7.4

ownCloud version: 10.6

Storage backend (external storage):Google drive

Client configuration

Client version:2.7.2

Operating system: Win 10 x64

OS language: windows

Logs

  1. Client logfile, Web server error log, Server logfile: https://gist.github.com/reberthkss/3619b4a37ea8ac67e9b7c40118d2c8fc

Hi,

Could you please paste your php.ini ?
And could you provide the message you are getting when trying to upload the big file? (screenshot)

Thanks!

1 Like

I’ve updated the gist with php.ini data!

Follow print below:

ps.: The file is successful uploaded to google drive account but ownCloud side is not recognizing.

Opened https://github.com/owncloud/core/issues/38328 . For now, I guess the workaround is to increase the allowed php memory to fit the whole file.

Side note: code follows the example provided by the google documentation. Note that ownCloud requires a data stream. Chunked / ranged downloads (which seems to be the natural alternative according to the google documentation) doesn’t fit here and will need to be adapted, which is quite complex to do.

3 Likes

Hi jv!

Thank you to reply. I’ll try increase php memory limit.
The unique parameter to change is memory_limit? I turn from 256MB to 4G
image

Increase memory_limit to 4G doesn’t solve the problem:

Some things to take into account:

  • Make sure you have plently of memory available. Other processes running in the server could also eat memory, so the php process might hit a hard limit before reaching the established 4GB memory for php.
  • Since this is a server problem, it’s way more interesting the server logs. We know the client will keep malfunction until the problem is solved in the server.
  • Double-check the memory limit is correctly applied. You’ll likely need to restart the web server. The easiest way to check is to download the file and check if there is an error log in apache; you’ll see the Allowed memory size of 536870912 bytes exhausted (tried to allocate 3410408792 bytes) error in the apache logs. There you can see the limit of ~512MB for php
2 Likes

Hi,

Did you restart your web server after making the change?

2 Likes

@Aki @jvillafanez i created a new gist: https://gist.github.com/reberthkss/9796fd05a8e6745ac23cf51a585bb2aa

also will try to update memory_limit value to 4095MB

[Fri Jan 22 16:07:12.014813 2021] [php7:error] [pid 2190] [client 187.75.157.120:56859] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 3410408792 bytes) in /var/www/html/cloud/apps/files_external/3rdparty/guzzlehttp/streams/src/Stream.php on line 135

It seems the php value memory_limit doesn’t is the reference for this process

Even I am facing the same issue with google drive transfers

there is a memory_limit directive in the .htaccess file inside the ownCloud’s directory. You might need to change that too.

2 Likes

Successfully uploaded big file! Thank you man!

Best regards,
Reberth.

1 Like

What about output_buffering?
https://doc.owncloud.com/server/10.6/admin_manual/configuration/files/big_file_upload_configuration.html#configuring-php

Output Buffering must be turned off in .htaccess or .user.ini or php.ini , or PHP will return memory-related errors:

Would it make a difference if they turned that off?

1 Like

Doubt. The problem is in the google library, that doesn’t stream the contents

2 Likes