Files:transfer-ownership - Allowed memory size exhausted

Server
,
1

Steps to reproduce

  1. sudo -u apache php /var/www/owncloud/occ files:transfer-ownership user1 user2

Expected behaviour

Tell us what should happen:
User2 receives user1 files. (This has worked for me 100% up until now). User1 has approx 22GB of files to be transferred.

Actual behaviour

“PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 72 bytes) in /var/www/owncloud/liv/composer/composer/ClassLoader.php on line 192”

It has copied some but not all, bit of a pain really with the user having so many files and folders to compare would be a nightmare!

Server configuration

Operating system:
Centos 7
Web server:
apache
Database:
mySQL
PHP version:
5.6
ownCloud version: (see ownCloud admin page)
9.1.8
Updated from an older ownCloud or fresh install:
Updated from 9.1.4
Where did you install ownCloud from:
tarball

The content of config/config.php:

{
    "system": {
        "instanceid": "oc47x9tjw1jc",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "\/mnt\/glusterfs",
        "htaccess.RewriteBase": "\/",
        "dbtype": "mysql",
        "version": "9.1.8.2",
        "dbname": "owncloud",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "enable_previews": false,
        "logtimezone": "UTC",
        "log_type": "owncloud",
        "logfile": "owncloud.log",
        "loglevel": 1,
        "installed": true,
        "mail_smtpmode": "smtp",
        "mail_from_address": "drive",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "ldapIgnoreNamingRules": false,
        "ldapUserCleanupInterval": 60,
        "appstore.experimental.enabled": true,
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "lost_password_link": "***REMOVED SENSITIVE VALUE***",
        "trashbin_retention_obligation": "auto, 30",
        "theme": "corvidtheme",
        "trusted_proxies": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "singleuser": false,
        "maintenance": false,
        "mail_smtpsecure": "tls"
    },
    "apps": {
        "activity": {
            "enabled": "yes",
            "installed_version": "2.3.2",
            "types": "filesystem"
        },
        "backgroundjob": {
            "lastjob": "3053"
        },
        "comments": {
            "enabled": "yes",
            "installed_version": "0.3.0",
            "types": "logging"
        },
        "configreport": {
            "enabled": "yes",
            "installed_version": "0.1.1",
            "types": "filesystem"
        },
        "core": {
            "OC_Channel": "production",
            "backgroundjobs_mode": "cron",
            "default_encryption_module": "OC_DEFAULT_MODULE",
            "encryption_enabled": "no",
            "incoming_server2server_share_enabled": "no",
            "installedat": "1489056684.1882",
            "lastcron": "1539940521",
            "lastupdateResult": "[]",
            "lastupdatedat": "1539939911",
            "oc.integritycheck.checker": "[]",
            "outgoing_server2server_share_enabled": "no",
            "public_documents": "documents\/public.php",
            "public_files": "files_sharing\/public.php",
            "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
            "shareapi_allow_resharing": "no",
            "shareapi_default_expire_date": "yes",
            "shareapi_enforce_expire_date": "no",
            "shareapi_enforce_links_password": "yes",
            "shareapi_exclude_groups": "no",
            "shareapi_exclude_groups_list": "[]",
            "shareapi_expire_after_n_days": "30",
            "shareapi_only_share_with_group_members": "no",
            "umgmt_send_email": "true",
            "umgmt_show_backend": "true",
            "umgmt_show_email": "true",
            "umgmt_show_last_login": "true",
            "umgmt_show_storage_location": "true"
        },
        "dav": {
            "enabled": "yes",
            "installed_version": "0.2.7",
            "types": "filesystem"
        },
        "documents": {
            "converter": "off",
            "converter_url": "http:\/\/localhost:16080",
            "enabled": "no",
            "installed_version": "0.13.1",
            "ocsid": "168711",
            "types": ""
        },
        "encryption": {
            "enabled": "no",
            "encryptHomeStorage": "1",
            "installed_version": "1.3.0",
            "masterKeyId": "***REMOVED SENSITIVE VALUE***",
            "publicShareKeyId": "***REMOVED SENSITIVE VALUE***",
            "recoveryAdminEnabled": "1",
            "recoveryKeyId": "***REMOVED SENSITIVE VALUE***",
            "types": "filesystem"
        },
        "external": {
            "enabled": "yes",
            "installed_version": "",
            "ocsid": "166046",
            "types": ""
        },
        "federatedfilesharing": {
            "enabled": "yes",
            "installed_version": "0.3.0",
            "types": ""
        },
        "federation": {
            "enabled": "no",
            "installed_version": "0.1.0",
            "types": "authentication"
        },
        "files": {
            "cronjob_scan_files": "500",
            "default_quota": "10 GB",
            "enabled": "yes",
            "installed_version": "1.5.1",
            "types": "filesystem"
        },
        "files_antivirus": {
            "av_cmd_options": "",
            "av_host": "",
            "av_infected_action": "delete",
            "av_max_file_size": "-1",
            "av_mode": "socket",
            "av_path": "\/usr\/bin\/clamscan",
            "av_port": "0",
            "av_socket": "\/var\/run\/clamd.scan\/clamd.sock",
            "av_stream_max_length": "26214400",
            "enabled": "yes",
            "installed_version": "0.9.0.1",
            "ocsid": "157439",
            "types": "filesystem"
        },
        "files_pdfviewer": {
            "enabled": "yes",
            "installed_version": "0.8.1",
            "ocsid": "166049",
            "types": ""
        },
        "files_sharing": {
            "enabled": "yes",
            "incoming_server2server_share_enabled": "no",
            "installed_version": "0.10.0",
            "outgoing_server2server_share_enabled": "no",
            "types": "filesystem"
        },
        "files_texteditor": {
            "enabled": "yes",
            "installed_version": "2.1",
            "ocsid": "166051",
            "types": ""
        },
        "files_trashbin": {
            "enabled": "yes",
            "installed_version": "0.9.0",
            "types": "filesystem"
        },
        "files_versions": {
            "enabled": "yes",
            "installed_version": "1.3.0",
            "types": "filesystem"
        },
        "files_videoplayer": {
            "enabled": "yes",
            "installed_version": "0.9.8",
            "types": ""
        },
        "firstrunwizard": {
            "enabled": "yes",
            "installed_version": "1.1",
            "ocsid": "166055",
            "types": ""
        },
        "gallery": {
            "enabled": "yes",
            "installed_version": "15.0.0",
            "types": ""
        },
        "notifications": {
            "enabled": "yes",
            "installed_version": "0.3.0",
            "types": "logging"
        },
        "provisioning_api": {
            "enabled": "yes",
            "installed_version": "0.5.0",
            "types": "prevent_group_restriction"
        },
        "systemtags": {
            "enabled": "yes",
            "installed_version": "0.3.0",
            "types": "logging"
        },
        "templateeditor": {
            "enabled": "yes",
            "installed_version": "0.1",
            "types": ""
        },
        "twofactor_totp": {
            "enabled": "[\"TOTP\"]",
            "installed_version": "0.4.2",
            "ocsid": "174726",
            "types": ""
        },
        "updatenotification": {
            "enabled": "yes",
            "installed_version": "0.2.1",
            "types": ""
        },
        "user_ldap": {
            "cleanUpJobOffset": "250",
            "enabled": "yes",
            "has_memberof_filter_support": "",
            "home_folder_naming_rule": "",
            "installed_version": "0.9.0",
            "last_jpegPhoto_lookup": "0",
            "ldap_agent_password": "***REMOVED SENSITIVE VALUE***",
            "ldap_attributes_for_group_search": "",
            "ldap_attributes_for_user_search": "",
            "ldap_backup_host": "***REMOVED SENSITIVE VALUE***",
            "ldap_backup_port": "389",
            "ldap_base": "***REMOVED SENSITIVE VALUE***",
            "ldap_base_groups": "***REMOVED SENSITIVE VALUE***",
            "ldap_base_users": "***REMOVED SENSITIVE VALUE***",
            "ldap_cache_ttl": "600",
            "ldap_configuration_active": "1",
            "ldap_display_name": "cn",
            "ldap_dn": "***REMOVED SENSITIVE VALUE***",
            "ldap_dynamic_group_member_url": "",
            "ldap_email_attr": "mail",
            "ldap_experienced_admin": "0",
            "ldap_expert_username_attr": "uid",
            "ldap_expert_uuid_group_attr": "",
            "ldap_expert_uuid_user_attr": "",
            "ldap_group_display_name": "cn",
            "ldap_group_filter": "",
            "ldap_group_filter_mode": "1",
            "ldap_group_member_assoc_attribute": "uniqueMember",
            "ldap_groupfilter_groups": "",
            "ldap_groupfilter_objectclass": "",
            "ldap_host": "***REMOVED SENSITIVE VALUE***",
            "ldap_login_filter": "(&(|(objectclass=posixAccount))(uid=%uid))",
            "ldap_login_filter_mode": "1",
            "ldap_loginfilter_attributes": "",
            "ldap_loginfilter_email": "0",
            "ldap_loginfilter_username": "1",
            "ldap_nested_groups": "0",
            "ldap_override_main_server": "0",
            "ldap_paging_size": "500",
            "ldap_port": "389",
            "ldap_quota_attr": "",
            "ldap_quota_def": "",
            "ldap_tls": "0",
            "ldap_turn_off_cert_check": "0",
            "ldap_user_display_name_2": "",
            "ldap_user_filter_mode": "1",
            "ldap_userfilter_groups": "",
            "ldap_userfilter_objectclass": "posixAccount",
            "ldap_userlist_filter": "(|(objectclass=posixAccount))",
            "types": "authentication",
            "use_memberof_to_detect_membership": "1"
        }
    }
}

List of activated apps:

Enabled:
  - activity: 2.3.2
  - comments: 0.3.0
  - configreport: 0.1.1
  - dav: 0.2.7
  - external: true
  - federatedfilesharing: 0.3.0
  - files: 1.5.1
  - files_antivirus: 0.9.0.1
  - files_pdfviewer: 0.8.1
  - files_sharing: 0.10.0
  - files_texteditor: 2.1
  - files_trashbin: 0.9.0
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 15.0.0
  - notifications: 0.3.0
  - provisioning_api: 0.5.0
  - systemtags: 0.3.0
  - templateeditor: 0.1
  - twofactor_totp: 0.4.2
  - updatenotification: 0.2.1
  - user_ldap: 0.9.0
Disabled:
  - encryption
  - federation
  - files_external
  - user_external

Are you using external storage, if yes which one: local/smb/sftp/…
GlusterFS

Are you using encryption: yes/no
No

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…
OpenLDAP

LDAP configuration (delete this part if not used)

+-----------------------------------------------+--------------------------------------------+
| Configuration                                   |                                                      |
+-----------------------------------------------+--------------------------------------------+
| hasMemberOfFilterSupport             |                                            |
| hasPagedResultSupport                 |                                            |
| homeFolderNamingRule                 |                                            |
| lastJpegPhotoLookup                     | 0                                          |
| ldapAgentName                              | ***REMOVED SENSITIVE VALUE***            |
| ldapAgentPassword                        | ***                                        |
| ldapAttributesForGroupSearch       |                                            |
| ldapAttributesForUserSearch         |                                            |
| ldapBackupHost                             | ***REMOVED SENSITIVE VALUE***    |
| ldapBackupPort                              | 389                                        |
| ldapBase                                        | ***REMOVED SENSITIVE VALUE***    |
| ldapBaseGroups                            | ***REMOVED SENSITIVE VALUE***  |
| ldapBaseUsers                              | ***REMOVED SENSITIVE VALUE***  |
| ldapCacheTTL                               | 600                                        |
| ldapConfigurationActive                | 1                                          |
| ldapDynamicGroupMemberURL   |                                            |
| ldapEmailAttribute                         | mail                                       |
| ldapExperiencedAdmin                 | 0                                          |
| ldapExpertUUIDGroupAttr            |                                            |
| ldapExpertUUIDUserAttr              |                                            |
| ldapExpertUsernameAttr              |  uid                                        |
| ldapGroupDisplayName               | cn                                         |
| ldapGroupFilter                            |                                            |
| ldapGroupFilterGroups                |                                            |
| ldapGroupFilterMode                   | 1                                          |
| ldapGroupFilterObjectclass         |                                            |
| ldapGroupMemberAssocAttr       | uniqueMember                               |
| ldapHost                                      | ***REMOVED SENSITIVE VALUE***    |
| ldapIgnoreNamingRules             |                                            |
| ldapLoginFilter                            | (&(|(objectclass=posixAccount))(uid=%uid)) |
| ldapLoginFilterAttributes             |                                            |
| ldapLoginFilterEmail                   | 0                                          |
| ldapLoginFilterMode                   | 1                                          |
| ldapLoginFilterUsername           | 1                                          |
| ldapNestedGroups                     | 0                                          |
| ldapOverrideMainServer            | 0                                          |
| ldapPagingSize                          | 500                                        |
| ldapPort                                      | 389                                        |
| ldapQuotaAttribute                     |                                            |
| ldapQuotaDefault                       |                                            |
| ldapTLS                                      | 0                                          |
| ldapUserDisplayName               | cn                                         |
| ldapUserDisplayName2             |                                            |
| ldapUserFilter                            | (|(objectclass=posixAccount))              |
| ldapUserFilterGroups                |                                            |
| ldapUserFilterMode                   | 1                                          |
| ldapUserFilterObjectclass         | posixAccount                               |
| ldapUuidGroupAttribute            | auto                                       |
| ldapUuidUserAttribute              | auto                                       |
| turnOffCertCheck                     | 0                                          |
| useMemberOfToDetectMembership | 1                                          |
+-----------------------------------------+--------------------------------------------+
2

~130MB memory setting in PHP config is a bit low.

Please review your php ini settings.

1 Like
3

Hi,

Thanks for the reply, php.ini Max script mem “memory_limit” is set to 128M.

What do Owncloud recommend for a limit? I couldn’t find any documentation on that one :frowning:

4

Hey,

i did the following search:

https://doc.owncloud.org/server/latest/admin_manual/search.html?q=memory

and found the following documentation on this topic:

https://doc.owncloud.org/server/latest/admin_manual/installation/system_requirements.html?highlight=memory#memory-requirements

5

Hi Tom42,

That refers to physical/ virtual server RAM requirements, not php memory_limit requirements.

6

Hey,

from what i know the “memory_limit” directive is exactly specifying this: It defines the upper limit of RAM PHP (and thus ownCloud) is allowed to use.

This seems to be confirmed if i’m checking my .htaccess file shipped by ownCloud:

php_value memory_limit 512M

which seems to match the recommended value from the posted documentation above.

7

Hi tom42,

Our installation has a application server RAM size of 16GB, are we suggesting I allow php memory_limit to use 16GB?

What i’m trying to find out is if OwnCloud recommend a size for this parameter, because it appears that their application (occ) is directly affected by this setting. I don’t want to be messing around with it until the command works, I just want to set it once!

8

I run with mem_limit -1 - but this is a developer setup. No idea if this is a valid recommendation for production.

9

I think that would allow ‘unlimited’ memory for that setting? I might just have to slowly increase the value until it all works.

10

Hey,

i think that an exact recommendation can’t be given. But the documentation above suggest a minimum of 128M for the memory_limit and a recommended value of 512M where you currently only have 128M configured as pointed out previously.

11

Hi tom42, I’m afraid you’re incorrect. The documentation you pointed me to is the recommended RAM for an OwnCloud server.
That’s the amount of physical RAM you put in a tin or the amount of virtual RAM you assign in a hypervisor, it is not the recommended php config memory_limit.

Sorry to be exact but others may read this and get the wrong advice.

12

Hey,

i think we’re in trouble now as we have two different opinions. :slightly_smiling_face:

If you have e.g. a look at the PHP documentation available here:

https://secure.php.net/manual/en/ini.core.php#ini.memory-limit

you will find the following documentation:

This sets the maximum amount of memory in bytes that a script is allowed to allocate. This helps prevent poorly written scripts for eating up all available memory on a server.

The script mentioned is probably “ownCloud” itself, and ownCloud is setting this in its own .htaccess to 512M as pointed out previously as well. You can verify this if you have a look at the .htaccess file in your ownCloud installation (see Files:transfer-ownership - Allowed memory size exhausted - Comment)

So why should ownCloud define the minimum amount of physical memory for the system to 128 MB (and recommends 512 MB) where it itself already assigning 512 MB for its own runtime? :slightly_smiling_face: (With that no other process like the web server running ownCloud or other processes would have any free memory left if ownCloud is taking the whole 512 MB).

I think the physical memory available for a server running ownCloud needs to be way higher then the 512MB as defined in the .htaccess to have resources left for other processed and services.

Everything else doesn’t make much sense IMHO.

13

So in short words: :slightly_smiling_face:

I think the documentation is the amount of memory which needs to be made available to ownCloud, 128 MB at a minimum and 512 MB recommended where the upper limit (e.g. the 512 MB) is limited by the memory_limit PHP setting which was recommended by @DeepDiver1975 to be raised in Files:transfer-ownership - Allowed memory size exhausted (Comment).

14

Here’s the problem; the memory_limit setting is the amount of memory a single script can run, a second script running at the same time would be able to use that much again.

And we have no idea what in the Owncloud installation uses a php script and how many potential instances of scripts could run at any one time.

Say 10 scripts run at once, at 512MB your suddenly allocating 5GB of RAM for scripts!

Could really do with an engineer response to this one, i.e. they probably know a sensible limit for their own scrips.

Thanks for the help with this, I like a debate!

15

Hey,

from my limited knowledge about PHP i think a “script” would be ownCloud itself and not all/different files of ownCloud. So if you have installed ownCloud and phpMyAdmin on the same Server both should be seen as “one script” and each can take the max amount of memory configured via memory_limit (e.g if set to 512 MB a max amout of 1024 MB could be used by both).

I don’t think that there are much engineers around here (even if there are ones around at all) :confused:. Most posts i had seen in this forums are from mainly home users running their own small ownCloud installation.

But if my previously posted link about the memory usage of ownCloud is written in a way that it allows so much speculations then maybe you could post a new issue to the people writing the documentation? From what i know the place for that would be https://github.com/owncloud/documentation/issues

16

I think you’ll find that very few scripts within ownCloud require memory over the 128MB limit. Scanning files is one (as I found out), it also appears now that file ownership transfer is another.

Now on the topic of running multiple scripts, there is no way in a modern operating system with so many different background tasks to know if other programs or other instances of programs will be chewing up all the RAM. This is why the operating system manages RAM requests and rejections (and swapping). Don’t under-allocate your own limits just because you want to do the job of the memory controller.

On the topic of just how much memory that ownCloud needs, I’ve not seen any evidence of anyone calculating this, but them recommending 512MB is a BIG HINT. If you box only has 512 MB, let it swap, the scripts needing this much shouldn’t be run that often anyway.

I set my own limit to about 2GB to guarantee no issues when I did a files:scan on a user with ~850GB of files, I see little need to dick around trying to find the lowest value that will work, because one day when you hit a new high for number of files, you’ll trigger the error again.

TL;DR:

  • Set your memory limit to 512MB or higher, chances of two scripts using this max limit is small
  • Don’t stress about setting it to same limit as your system RAM, the odd script that needs this much can swap out to disk and take a bit longer (if really worried, spend money on more RAM)

Disclaimer: I am a Computer Systems Engineer. I don’t manage IT systems full time.

1 Like
17

Hi Andrew,

Thanks for the response. It would seem that 512Mb is being used quite widely and doesn’t seem to have a negative impact on the server.

I did set the memory_limit to 512Mb, and have successfully transfer ownership on the files now.

So thank you all for your help.

1 Like