Migrating to Owncloud 10 and PHP7.1 results in severalproblems

Steps to reproduce

  1. Migrated from 9.1 to 10

Expected behaviour

files:scan should work

Actual behaviour

files:scan crashes with meomory exhausted

Server configuration

RHEL7.5

Apche2.4:

h-mariadb102-mariadb-10.2.8-5:

rh-php71-php-7.1.8-1:

**10.0.10 (stable))

Update from 9.1:

tar.gz:

index.php/settings/integrity/failed

The content of config/config.php:

[root@szhm58039 owncloud10.0.10]# sudo -u apache php occ config:list system
{
“system”: {
“instanceid”: “ocd5f9424070”,
“passwordsalt”: “REMOVED SENSITIVE VALUE”,
“trusted_domains”: [

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
],
“datadirectory”: “/var/www/html/owncloud10.0.10/data”,
“dbtype”: “mysql”,
“version”: “10.0.10.4”,
“dbname”: “zueribox”,
“dbhost”: “localhost”,
“dbtableprefix”: “oc_”,
“dbuser”: “REMOVED SENSITIVE VALUE”,
“dbpassword”: “REMOVED SENSITIVE VALUE”,
“installed”: true,
“maintenance”: false,
“has_internet_connection”: false,
“appstoreenabled”: false,
“check_for_working_webdav”: false,
“updatechecker”: false,
“ldapIgnoreNamingRules”: false,
“trashbin_retention_obligation”: “3, auto”,
“forcessl”: true,
“log_authfailip”: true,
“loglevel”: 2,
“theme”: “szh”,
“secret”: “REMOVED SENSITIVE VALUE”,
“share_folder”: “/Shared”,
“mail_smtpmode”: “php”,
“mail_smtphost”: “REMOVED SENSITIVE VALUE”,
“mail_smtpport”: “25”,
“mail_from_address”: “REMOVED SENSITIVE VALUE”,
“mail_domain”: “REMOVED SENSITIVE VALUE”,
“appstore.experimental.enabled”: true,
“log_rotate_size”: 524288000,
“overwrite.cli.url”: “/zueribox”,
“singleuser”: false,
“debug”: false,
“default_language”: “de”,
“knowledgebaseenabled”: true,
“enable_avatars”: true,
“enable_previews”: true,
“preview_max_x”: 2048,
“preview_max_y”: 2048,
“preview_max_filesize_image”: 25,
“enabledPreviewProviders”: [
“OC\Preview\PNG”,
“OC\Preview\JPEG”,
“OC\Preview\GIF”,
“OC\Preview\BMP”,
“OC\Preview\XBitmap”,
“OC\Preview\MP3”,
“OC\Preview\TXT”,
“OC\Preview\MarkDown”
],
“mail_smtpauthtype”: “LOGIN”,
“memcache.local”: “\OC\Memcache\APCu”
}
}

List of activated apps:

Enabled:

  • dav: 0.4.0
  • federatedfilesharing: 0.3.1
  • files: 1.5.1
  • files_external: 0.7.1
  • files_sharing: 0.11.0
  • files_trashbin: 0.9.1
  • impersonate: 0.2.0
  • provisioning_api: 0.5.0
  • user_ldap: 0.11.0
    Disabled:
  • announcementcenter
  • checksum
  • comments
  • configreport
  • diagnostics
  • encryption
  • external
  • federation
  • files_versions
  • files_videoplayer
  • firstrunwizard
  • market
  • notifications
  • systemtags
  • templateeditor
  • updatenotification
  • user_external

Are you using external storage, if yes which one: local/smb/sftp/…
/data is on NFS mount

Are you using encryption: yes/no
NO

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…

LDAP configuration (delete this part if not used)

YES

Client configuration

ANY:

Operating system:

memory_limit is 8G configured in php.ini and .htaccess and .user.ini File.


i am running Owncloud on RHEL7.5 with rh-php71 from RHSCL.

When i run the Scan, it fails with the following error message:

sudo -u apache /opt/rh/rh-php71/root/usr/bin/php occ files:scan --path=F1FE3E8F-22E9-4139-9174-EDAF236B01D9

Scanning files for 1 users
Starting scan for user 1 out of 1 (F1FE3E8F-22E9-4139-9174-EDAF236B01D9)

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/owncloud10.0.10/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 142
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1052672 bytes) in /var/www/html/owncloud10.0.10/lib/private/Lock/DBLockingProvider.php on line 266

– Note: PHP is installed under seperate Folder URL due using RHSCL.

Someone can help please?

Thank you in davance and best regards

Fernando

Hey,

could it be possible that you havn’t configured the correct php.ini which is used by the PHP version running the occ command?

Thanks for your reply.

i use to have the phpinfo page, and phpinfo is reacting if i do changes on PHP ini file.
also on console #php -i XX gives me the correct values.

There is only one php.ini file on the Filesystem. also i checked .htacces etc.

Thank you

Hey there

You could try the post and upload max attributes too. In php.ini, from my memory:

post_max_size=8G
upload_max_filesize=8G

Checked this too, thanks for your input

actually it seems there are other problems more important :slight_smile:

my DB is kind of stuck and the LOG is producing errors like the following, when deleting an File:

{“reqId”:“W74DVoJ7kquWH6D-E2uslgAAAAA”,“level”:4,“time”:“2018-10-10T13:49:10+00:00”,“remoteAddr”:“XXXXX”,“user”:“0F33A4E3-0B8F-4771-AEC5-1A105A256B73”,“app”:“webdav”,“method”:“DELETE”,“url”:"/zueribox/remote.php/webdav/diagnostic.log",“message”:“Exception: An exception occurred while executing ‘DELETE FROM oc_properties WHERE fileid = ?’ with params [2033781]:\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column ‘fileid’ in ‘where clause’: {“Exception”:“Doctrine\\DBAL\\Exception\\InvalidFieldNameException”,“Message”:“An exception occurred while executing ‘DELETE FROM oc_properties WHERE fileid = ?’ with params [2033781]:\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column ‘fileid’ in ‘where clause’”,“Code”:0,“Trace”:”#0 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/DBALException.php(128): Doctrine\\DBAL\\Driver\\AbstractMySQLDriver->convertException(‘An exception oc…’, Object(Doctrine\\DBAL\\Driver\\PDOException))\n#1 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Statement.php(177): Doctrine\\DBAL\\DBALException::driverExceptionDuringQuery(Object(Doctrine\\DBAL\\Driver\\PDOMySql\\Driver), Object(Doctrine\\DBAL\\Driver\\PDOException), ‘DELETE FROM `oc…’, Array)\n#2 \/var\/www\/html\/owncloud10.0.10\/apps\/dav\/lib\/DAV\/FileCustomPropertiesBackend.php(109): Doctrine\\DBAL\\Statement->execute(Array)\n#3 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/PropertyStorage\/Plugin.php(126): OCA\\DAV\\DAV\\FileCustomPropertiesBackend->delete(‘diagnostic.log’)\n#4 [internal function]: Sabre\\DAV\\PropertyStorage\\Plugin->afterUnbind(*** sensitive parameters replaced ***)\n#5 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#6 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/CorePlugin.php(288): Sabre\\Event\\EventEmitter->emit(‘afterUnbind’, Array)\n#7 [internal function]: Sabre\\DAV\\CorePlugin->httpDelete(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#8 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#9 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit(‘method:DELETE’, Array)\n#10 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#11 \/var\/www\/html\/owncloud10.0.10\/apps\/dav\/appinfo\/v1\/webdav.php(64): Sabre\\DAV\\Server->exec()\n#12 \/var\/www\/html\/owncloud10.0.10\/remote.php(165): require_once(’\/var\/www\/html\/o…’)\n#13 {main}",“File”:"\/var\/www\/html\/owncloud10.0.10\/lib\/composer\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Driver\/AbstractMySQLDriver.php",“Line”:71}"}

on mysql log this happens:

          445 Query     UPDATE `oc_filecache` SET `storage` =  '1114', `path` = 'files_trashbin/files/diagnostic.log.d1539180370', `path_hash` = '5559c5cd4988509109a33764d317d9ca', `name` = 'diagnostic.log.d1539180370', `parent` ='1937846' WHERE `fileid` = '2035021'
              445 Query     INSERT INTO `oc_files_trash` (`id`,`timestamp`,`location`,`user`) VALUES ('diagnostic.log','1539180370','FF','0F33A4E3-0B8F-4771-AEC5-1A105A256B73')

While i am running an scan on the data directory, the mysql log shows:

790 Query UPDATE oc_file_locks SET lock = lock + 1, ttl = ‘1539184213’ WHERE key = ‘files/4504b95a09e0de0dcae20201360192ef’ AND lock >= 0
790 Query SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
storage_mtime, encrypted, etag, permissions, checksum
FROM oc_filecache WHERE storage = ‘678’ AND path_hash = ‘6b4726f1a2755454491244d907731e78’
790 Query INSERT INTO oc_file_locks (key,lock,ttl) SELECT ‘files/5ebef0919a7c6985ccf10f152b615900’,‘1’,‘1539184213’ FROM oc_file_locks WHERE key = ‘files/5ebef0919a7c6985ccf10f152b615900’ HAVING COUNT() = 0
790 Query UPDATE oc_file_locks SET lock = lock + 1, ttl = ‘1539184213’ WHERE key = ‘files/5ebef0919a7c6985ccf10f152b615900’ AND lock >= 0
790 Query SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
storage_mtime, encrypted, etag, permissions, checksum
FROM oc_filecache WHERE storage = ‘678’ AND path_hash = ‘7cd34117977c2dc23705d0629bd0a6e0’
790 Query INSERT INTO oc_file_locks (key,lock,ttl) SELECT ‘files/24dc37e62edb6056af1a6c3f9212c688’,‘1’,‘1539184213’ FROM oc_file_locks WHERE key = ‘files/24dc37e62edb6056af1a6c3f9212c688’ HAVING COUNT(
) = 0
790 Query UPDATE oc_file_locks SET lock = lock + 1, ttl = ‘1539184213’ WHERE key = ‘files/24dc37e62edb6056af1a6c3f9212c688’ AND lock >= 0

any hint on this front would be appreciated too :slight_smile:

Thank you!

Hi again

I’m not an owncloud expert by any means, but missing DB columns might suggest a failed DB migration that should most likely run during upgrade. I’d suggest that the upgrade process failed and exited prior to completing the built in migrations.

Unfortunately I don’t know what to suggest to help you, but hopefully this is a hint of where to look. Maybe running occ upgrade -vvv would give you some idea of where it failed.

thanks for your input:

occ upgrade -vvv
ownCloud is already latest version

Seems the upgrade run smooth , i found the update in the console log:

sudo -u apache /opt/rh/rh-php71/root/usr/bin/php …/occ upgrade
ownCloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
2018-10-09T15:20:37+00:00 Set log level to debug
2018-10-09T15:20:37+00:00 Turned on maintenance mode
2018-10-09T15:20:37+00:00 Repair step: Repair MySQL database engine
2018-10-09T15:20:37+00:00 Repair step: Repair MySQL collation
2018-10-09T15:20:37+00:00 Repair info: All tables already have the correct collation -> nothing to do
2018-10-09T15:20:37+00:00 Repair step: Repair SQLite autoincrement
2018-10-09T15:20:37+00:00 Repair step: Repair orphaned reshare
2018-10-09T15:20:37+00:00 Repair step: Repair duplicate entries in oc_lucene_status
2018-10-09T15:20:37+00:00 Repair info: removing duplicate entries from lucene_status
2018-10-09T15:20:37+00:00 Repair step: Upgrade app code from the marketplace
2018-10-09T15:20:37+00:00 Repair info: No internet connection available - no app updates will be taken from the marketplace.
2018-10-09T15:20:37+00:00 Repair info: How to update apps in such situation please see https://doc.owncloud.org/server/10.0/go.php?to=admin-marketplace-apps
2018-10-09T15:20:37+00:00 Updating database schema
2018-10-09T15:20:37+00:00 Updated database
2018-10-09T15:20:37+00:00 Updating <user_ldap> …
2018-10-09T15:20:37+00:00 Updated <user_ldap> to 0.11.0
2018-10-09T15:20:37+00:00 Repair step: Repair mime types
2018-10-09T15:20:37+00:00 Repair step: Detect file cache entries with path that does not match parent-child relationships
2018-10-09T15:20:37+00:00 Repair step: Generate ETags for file where no ETag is present.
2018-10-09T15:20:38+00:00 Repair info: ETags have been fixed for 0 files/folders.
2018-10-09T15:20:38+00:00 Repair step: Clean tags and favorites
2018-10-09T15:20:38+00:00 Repair info: 50 tags of deleted users have been removed.
2018-10-09T15:20:38+00:00 Repair info: 0 tags for delete files have been removed.
2018-10-09T15:20:38+00:00 Repair info: 79 tag entries for deleted tags have been removed.
2018-10-09T15:20:38+00:00 Repair info: 0 tags with no entries have been removed.
2018-10-09T15:20:38+00:00 Repair step: Drop old database tables
2018-10-09T15:20:38+00:00 Drop old database tables
2018-10-09T15:20:38+00:00
2018-10-09T15:20:38+00:00 Done
28/28 [============================] 100%2018-10-09T15:20:38+00:00
2018-10-09T15:20:38+00:00 Repair step: Drop old background jobs
2018-10-09T15:20:38+00:00 Repair step: Remove getetag entries in properties table
2018-10-09T15:20:38+00:00 Repair info: Removed 0 unneeded “{DAV:}getetag” entries from properties table.
2018-10-09T15:20:38+00:00 Repair step: Repair outdated OCS IDs
2018-10-09T15:20:38+00:00 Repair step: Repair invalid shares
2018-10-09T15:20:38+00:00 Repair step: Remove old share propagation app entries
2018-10-09T15:20:38+00:00 Repair step: Move user avatars outside the homes to the new location
2018-10-09T15:20:38+00:00 Repair step: Remove shares of a users root folder
2018-10-09T15:20:38+00:00 Repair step: Repair unmerged shares
2018-10-09T15:20:38+00:00 Repair step: Disable extra themes
2018-10-09T15:20:38+00:00 Repair step: Repair sub shares
2018-10-09T15:20:38+00:00 Starting code integrity check…
2018-10-09T15:20:42+00:00 Finished code integrity check
2018-10-09T15:20:42+00:00 Update successful
2018-10-09T15:20:42+00:00 Turned off maintenance mode
2018-10-09T15:20:42+00:00 Reset log level

Thank you

In that case I’m out of ideas, sorry.

Best of luck!

Hey,

i did the following search: https://github.com/owncloud/core/search?q="Unknown+column"+"fileid"&type=Issues

and found a reported and closed issues there. It seems one might provide a possible workaround: https://github.com/owncloud/core/issues/30007#issuecomment-427789626

Hello Tom

Ok wow, thanks for that, i will check if this helps here with my issue.

Thank you very much!

Best regards
Fernando

Hello fernando,

I’m facing the same kind of error… Did the suggestion from tom42 work for you ?
I tried the few php occ migrations:execute dav, but unfortunately it didn’t solved the issue.

Alain

FYI, after comparing my owncloud database with a fresh new database 10.0.10 installed on a Virtual Machine, I got my owncloud database repairedand working by using the following MySQL instructions:

ALTER TABLE oc_properties ADD fileid bigint(20) AFTER id;
ALTER TABLE oc_properties DROP COLUMN userid;
ALTER TABLE oc_properties DROP COLUMN propertypath;
ALTER TABLE oc_properties MODIFY id BIGINT(20) UNSIGNED;

CREATE INDEX fileid_index ON oc_properties (fileid);

ALTER TABLE oc_jobs ADD execution_duration int(11) DEFAULT -1 AFTER reserved_at;

ALTER TABLE oc_mounts MODIFY root_id BIGINT(20) UNSIGNED;
1 Like