Migrating to Owncloud 10 and PHP7.1 results in severalproblems

fernando · October 10, 2018, 10:19am

Steps to reproduce

Migrated from 9.1 to 10

Expected behaviour

files:scan should work

Actual behaviour

files:scan crashes with meomory exhausted

Server configuration

RHEL7.5

Apche2.4:

h-mariadb102-mariadb-10.2.8-5:

rh-php71-php-7.1.8-1:

**10.0.10 (stable))

Update from 9.1:

tar.gz:

index.php/settings/integrity/failed

The content of config/config.php:

[root@szhm58039 owncloud10.0.10]# sudo -u apache php occ config:list system
{
“system”: {
“instanceid”: “ocd5f9424070”,
“passwordsalt”: “REMOVED SENSITIVE VALUE”,
“trusted_domains”: [

XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
],
“datadirectory”: “/var/www/html/owncloud10.0.10/data”,
“dbtype”: “mysql”,
“version”: “10.0.10.4”,
“dbname”: “zueribox”,
“dbhost”: “localhost”,
“dbtableprefix”: “oc_”,
“dbuser”: “REMOVED SENSITIVE VALUE”,
“dbpassword”: “REMOVED SENSITIVE VALUE”,
“installed”: true,
“maintenance”: false,
“has_internet_connection”: false,
“appstoreenabled”: false,
“check_for_working_webdav”: false,
“updatechecker”: false,
“ldapIgnoreNamingRules”: false,
“trashbin_retention_obligation”: “3, auto”,
“forcessl”: true,
“log_authfailip”: true,
“loglevel”: 2,
“theme”: “szh”,
“secret”: “REMOVED SENSITIVE VALUE”,
“share_folder”: “/Shared”,
“mail_smtpmode”: “php”,
“mail_smtphost”: “REMOVED SENSITIVE VALUE”,
“mail_smtpport”: “25”,
“mail_from_address”: “REMOVED SENSITIVE VALUE”,
“mail_domain”: “REMOVED SENSITIVE VALUE”,
“appstore.experimental.enabled”: true,
“log_rotate_size”: 524288000,
“overwrite.cli.url”: “/zueribox”,
“singleuser”: false,
“debug”: false,
“default_language”: “de”,
“knowledgebaseenabled”: true,
“enable_avatars”: true,
“enable_previews”: true,
“preview_max_x”: 2048,
“preview_max_y”: 2048,
“preview_max_filesize_image”: 25,
“enabledPreviewProviders”: [
“OC\Preview\PNG”,
“OC\Preview\JPEG”,
“OC\Preview\GIF”,
“OC\Preview\BMP”,
“OC\Preview\XBitmap”,
“OC\Preview\MP3”,
“OC\Preview\TXT”,
“OC\Preview\MarkDown”
],
“mail_smtpauthtype”: “LOGIN”,
“memcache.local”: “\OC\Memcache\APCu”
}
}

List of activated apps:

Enabled:

dav: 0.4.0
federatedfilesharing: 0.3.1
files: 1.5.1
files_external: 0.7.1
files_sharing: 0.11.0
files_trashbin: 0.9.1
impersonate: 0.2.0
provisioning_api: 0.5.0
user_ldap: 0.11.0
Disabled:
announcementcenter
checksum
comments
configreport
diagnostics
encryption
external
federation
files_versions
files_videoplayer
firstrunwizard
market
notifications
systemtags
templateeditor
updatenotification
user_external

Are you using external storage, if yes which one: local/smb/sftp/…
/data is on NFS mount

Are you using encryption: yes/no
NO

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/…

LDAP configuration (delete this part if not used)

YES

Client configuration

ANY:

Operating system:

memory_limit is 8G configured in php.ini and .htaccess and .user.ini File.

i am running Owncloud on RHEL7.5 with rh-php71 from RHSCL.

When i run the Scan, it fails with the following error message:

sudo -u apache /opt/rh/rh-php71/root/usr/bin/php occ files:scan --path=F1FE3E8F-22E9-4139-9174-EDAF236B01D9

Scanning files for 1 users
Starting scan for user 1 out of 1 (F1FE3E8F-22E9-4139-9174-EDAF236B01D9)

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/owncloud10.0.10/lib/composer/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOStatement.php on line 142
PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1052672 bytes) in /var/www/html/owncloud10.0.10/lib/private/Lock/DBLockingProvider.php on line 266

– Note: PHP is installed under seperate Folder URL due using RHSCL.

Someone can help please?

Thank you in davance and best regards

Fernando

tom42 · October 10, 2018, 10:55am

Hey,

could it be possible that you havn’t configured the correct php.ini which is used by the PHP version running the occ command?

fernando · October 10, 2018, 12:05pm

Thanks for your reply.

i use to have the phpinfo page, and phpinfo is reacting if i do changes on PHP ini file.
also on console #php -i XX gives me the correct values.

There is only one php.ini file on the Filesystem. also i checked .htacces etc.

Thank you

WISPa · October 10, 2018, 12:18pm

Hey there

You could try the post and upload max attributes too. In php.ini, from my memory:

post_max_size=8G
upload_max_filesize=8G

fernando · October 10, 2018, 1:49pm

Checked this too, thanks for your input

fernando · October 10, 2018, 1:50pm

actually it seems there are other problems more important

my DB is kind of stuck and the LOG is producing errors like the following, when deleting an File:

{“reqId”:“W74DVoJ7kquWH6D-E2uslgAAAAA”,“level”:4,“time”:“2018-10-10T13:49:10+00:00”,“remoteAddr”:“XXXXX”,“user”:“0F33A4E3-0B8F-4771-AEC5-1A105A256B73”,“app”:“webdav”,“method”:“DELETE”,“url”:"/zueribox/remote.php/webdav/diagnostic.log",“message”:“Exception: An exception occurred while executing ‘DELETE FROM oc_properties WHERE fileid = ?’ with params [2033781]:\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column ‘fileid’ in ‘where clause’: {“Exception”:“Doctrine\\DBAL\\Exception\\InvalidFieldNameException”,“Message”:“An exception occurred while executing ‘DELETE FROM oc_properties WHERE fileid = ?’ with params [2033781]:\n\nSQLSTATE[42S22]: Column not found: 1054 Unknown column ‘fileid’ in ‘where clause’”,“Code”:0,“Trace”:”#0 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/DBALException.php(128): Doctrine\\DBAL\\Driver\\AbstractMySQLDriver->convertException(‘An exception oc…’, Object(Doctrine\\DBAL\\Driver\\PDOException))\n#1 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Statement.php(177): Doctrine\\DBAL\\DBALException::driverExceptionDuringQuery(Object(Doctrine\\DBAL\\Driver\\PDOMySql\\Driver), Object(Doctrine\\DBAL\\Driver\\PDOException), ‘DELETE FROM `oc…’, Array)\n#2 \/var\/www\/html\/owncloud10.0.10\/apps\/dav\/lib\/DAV\/FileCustomPropertiesBackend.php(109): Doctrine\\DBAL\\Statement->execute(Array)\n#3 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/PropertyStorage\/Plugin.php(126): OCA\\DAV\\DAV\\FileCustomPropertiesBackend->delete(‘diagnostic.log’)\n#4 [internal function]: Sabre\\DAV\\PropertyStorage\\Plugin->afterUnbind(*** sensitive parameters replaced ***)\n#5 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#6 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/CorePlugin.php(288): Sabre\\Event\\EventEmitter->emit(‘afterUnbind’, Array)\n#7 [internal function]: Sabre\\DAV\\CorePlugin->httpDelete(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#8 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#9 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit(‘method:DELETE’, Array)\n#10 \/var\/www\/html\/owncloud10.0.10\/lib\/composer\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#11 \/var\/www\/html\/owncloud10.0.10\/apps\/dav\/appinfo\/v1\/webdav.php(64): Sabre\\DAV\\Server->exec()\n#12 \/var\/www\/html\/owncloud10.0.10\/remote.php(165): require_once(’\/var\/www\/html\/o…’)\n#13 {main}",“File”:"\/var\/www\/html\/owncloud10.0.10\/lib\/composer\/doctrine\/dbal\/lib\/Doctrine\/DBAL\/Driver\/AbstractMySQLDriver.php",“Line”:71}"}

on mysql log this happens:

          445 Query     UPDATE `oc_filecache` SET `storage` =  '1114', `path` = 'files_trashbin/files/diagnostic.log.d1539180370', `path_hash` = '5559c5cd4988509109a33764d317d9ca', `name` = 'diagnostic.log.d1539180370', `parent` ='1937846' WHERE `fileid` = '2035021'
              445 Query     INSERT INTO `oc_files_trash` (`id`,`timestamp`,`location`,`user`) VALUES ('diagnostic.log','1539180370','FF','0F33A4E3-0B8F-4771-AEC5-1A105A256B73')

While i am running an scan on the data directory, the mysql log shows:

790 Query UPDATE oc_file_locks SET lock = lock + 1, ttl = ‘1539184213’ WHERE key = ‘files/4504b95a09e0de0dcae20201360192ef’ AND lock >= 0
790 Query SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
storage_mtime, encrypted, etag, permissions, checksum
FROM oc_filecache WHERE storage = ‘678’ AND path_hash = ‘6b4726f1a2755454491244d907731e78’
790 Query INSERT INTO oc_file_locks (key,lock,ttl) SELECT ‘files/5ebef0919a7c6985ccf10f152b615900’,‘1’,‘1539184213’ FROM oc_file_locks WHERE key = ‘files/5ebef0919a7c6985ccf10f152b615900’ HAVING COUNT() = 0
790 Query UPDATE oc_file_locks SET lock = lock + 1, ttl = ‘1539184213’ WHERE key = ‘files/5ebef0919a7c6985ccf10f152b615900’ AND lock >= 0
790 Query SELECT fileid, storage, path, parent, name, mimetype, mimepart, size, mtime,
storage_mtime, encrypted, etag, permissions, checksum
FROM oc_filecache WHERE storage = ‘678’ AND path_hash = ‘7cd34117977c2dc23705d0629bd0a6e0’
790 Query INSERT INTO oc_file_locks (key,lock,ttl) SELECT ‘files/24dc37e62edb6056af1a6c3f9212c688’,‘1’,‘1539184213’ FROM oc_file_locks WHERE key = ‘files/24dc37e62edb6056af1a6c3f9212c688’ HAVING COUNT() = 0
790 Query UPDATE oc_file_locks SET lock = lock + 1, ttl = ‘1539184213’ WHERE key = ‘files/24dc37e62edb6056af1a6c3f9212c688’ AND lock >= 0

any hint on this front would be appreciated too

Thank you!

WISPa · October 10, 2018, 2:07pm

Hi again

I’m not an owncloud expert by any means, but missing DB columns might suggest a failed DB migration that should most likely run during upgrade. I’d suggest that the upgrade process failed and exited prior to completing the built in migrations.

Unfortunately I don’t know what to suggest to help you, but hopefully this is a hint of where to look. Maybe running occ upgrade -vvv would give you some idea of where it failed.

fernando · October 10, 2018, 2:09pm

thanks for your input:

occ upgrade -vvv
ownCloud is already latest version

Seems the upgrade run smooth , i found the update in the console log:

sudo -u apache /opt/rh/rh-php71/root/usr/bin/php …/occ upgrade
ownCloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
2018-10-09T15:20:37+00:00 Set log level to debug
2018-10-09T15:20:37+00:00 Turned on maintenance mode
2018-10-09T15:20:37+00:00 Repair step: Repair MySQL database engine
2018-10-09T15:20:37+00:00 Repair step: Repair MySQL collation
2018-10-09T15:20:37+00:00 Repair info: All tables already have the correct collation -> nothing to do
2018-10-09T15:20:37+00:00 Repair step: Repair SQLite autoincrement
2018-10-09T15:20:37+00:00 Repair step: Repair orphaned reshare
2018-10-09T15:20:37+00:00 Repair step: Repair duplicate entries in oc_lucene_status
2018-10-09T15:20:37+00:00 Repair info: removing duplicate entries from lucene_status
2018-10-09T15:20:37+00:00 Repair step: Upgrade app code from the marketplace
2018-10-09T15:20:37+00:00 Repair info: No internet connection available - no app updates will be taken from the marketplace.
2018-10-09T15:20:37+00:00 Repair info: How to update apps in such situation please see https://doc.owncloud.org/server/10.0/go.php?to=admin-marketplace-apps
2018-10-09T15:20:37+00:00 Updating database schema
2018-10-09T15:20:37+00:00 Updated database
2018-10-09T15:20:37+00:00 Updating <user_ldap> …
2018-10-09T15:20:37+00:00 Updated <user_ldap> to 0.11.0
2018-10-09T15:20:37+00:00 Repair step: Repair mime types
2018-10-09T15:20:37+00:00 Repair step: Detect file cache entries with path that does not match parent-child relationships
2018-10-09T15:20:37+00:00 Repair step: Generate ETags for file where no ETag is present.
2018-10-09T15:20:38+00:00 Repair info: ETags have been fixed for 0 files/folders.
2018-10-09T15:20:38+00:00 Repair step: Clean tags and favorites
2018-10-09T15:20:38+00:00 Repair info: 50 tags of deleted users have been removed.
2018-10-09T15:20:38+00:00 Repair info: 0 tags for delete files have been removed.
2018-10-09T15:20:38+00:00 Repair info: 79 tag entries for deleted tags have been removed.
2018-10-09T15:20:38+00:00 Repair info: 0 tags with no entries have been removed.
2018-10-09T15:20:38+00:00 Repair step: Drop old database tables
2018-10-09T15:20:38+00:00 Drop old database tables
2018-10-09T15:20:38+00:00
2018-10-09T15:20:38+00:00 Done
28/28 [============================] 100%2018-10-09T15:20:38+00:00
2018-10-09T15:20:38+00:00 Repair step: Drop old background jobs
2018-10-09T15:20:38+00:00 Repair step: Remove getetag entries in properties table
2018-10-09T15:20:38+00:00 Repair info: Removed 0 unneeded “{DAV:}getetag” entries from properties table.
2018-10-09T15:20:38+00:00 Repair step: Repair outdated OCS IDs
2018-10-09T15:20:38+00:00 Repair step: Repair invalid shares
2018-10-09T15:20:38+00:00 Repair step: Remove old share propagation app entries
2018-10-09T15:20:38+00:00 Repair step: Move user avatars outside the homes to the new location
2018-10-09T15:20:38+00:00 Repair step: Remove shares of a users root folder
2018-10-09T15:20:38+00:00 Repair step: Repair unmerged shares
2018-10-09T15:20:38+00:00 Repair step: Disable extra themes
2018-10-09T15:20:38+00:00 Repair step: Repair sub shares
2018-10-09T15:20:38+00:00 Starting code integrity check…
2018-10-09T15:20:42+00:00 Finished code integrity check
2018-10-09T15:20:42+00:00 Update successful
2018-10-09T15:20:42+00:00 Turned off maintenance mode
2018-10-09T15:20:42+00:00 Reset log level

Thank you

WISPa · October 10, 2018, 2:19pm

In that case I’m out of ideas, sorry.

Best of luck!

tom42 · October 10, 2018, 3:18pm

Hey,

i did the following search: https://github.com/owncloud/core/search?q="Unknown+column"+"fileid"&type=Issues

and found a reported and closed issues there. It seems one might provide a possible workaround: https://github.com/owncloud/core/issues/30007#issuecomment-427789626

fernando · October 10, 2018, 3:27pm

Hello Tom

Ok wow, thanks for that, i will check if this helps here with my issue.

Thank you very much!

Best regards
Fernando

acordier · December 14, 2018, 12:46pm

Hello fernando,

I’m facing the same kind of error… Did the suggestion from tom42 work for you ?
I tried the few php occ migrations:execute dav, but unfortunately it didn’t solved the issue.

Alain

acordier · January 3, 2019, 2:54pm

FYI, after comparing my owncloud database with a fresh new database 10.0.10 installed on a Virtual Machine, I got my owncloud database repairedand working by using the following MySQL instructions:

ALTER TABLE oc_properties ADD fileid bigint(20) AFTER id;
ALTER TABLE oc_properties DROP COLUMN userid;
ALTER TABLE oc_properties DROP COLUMN propertypath;
ALTER TABLE oc_properties MODIFY id BIGINT(20) UNSIGNED;

CREATE INDEX fileid_index ON oc_properties (fileid);

ALTER TABLE oc_jobs ADD execution_duration int(11) DEFAULT -1 AFTER reserved_at;

ALTER TABLE oc_mounts MODIFY root_id BIGINT(20) UNSIGNED;