NFS mount w/ DB size issues

modifications

#1

Hi,

I actually have an ownCloud instance in order to share outside of my LAN everything that is on my cluster storage (NFS mounts).

Despite the fact that i disabled versioning, trash, preview, etc my DB have been "thrown" from 113Mb to 270Mb after 30', and to 444Mb after an hour (metadatas i guess).

The problem is...the storage in question is quite big (1Pb and soon 3 others will come) so what ? i'll neet one Petabyte just for the DB ? :slight_smile:

No, joke aside...are there any options that i missed while configuring ownCloud ?

Regards,
Laurent Fournier.


#2

You can check the amount of data for each DB table in order to have an idea of what tables are taking more space. I never checked this information so if you know some tools for this, I'd be happy to know :slight_smile:

Note that if you have a large amount of files in the external storage, we'll need to get the metadata for each of them. The more files you have, the larger the metadata we'll need to store will be. In any case, the size of each file won't matter for the size of the metadata we need to store. The DB size should be stable once we have all the metadata for all the files in the external storage.

Additionally, it should be possible for ownCloud to connect to a DB in another machine in case you want to install a mysql (or other) DB in a bigger harddrive.


#3

Quite a good thing to know.. Even more with the fact that i'm working on a way to reduce inodes on a large scale (through some mountables 'RW' containers) :smile:

I am speaking with our technical support atm (hope that they didn't already erase our test VM hahah).

As soon as i get some stats i come back to you then :slight_smile:


#4

There we go.. It stabilized (less than 1Gb is totally acceptable).

laurent@serv677:~$ sudo ls -lh /var/lib/mysql/ocdb/ | sort -n
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_calendarchanges.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_calendarobjects.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_calendarsubscriptions.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_comments_read_markers.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_credentials.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_dav_properties.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_dav_shares.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_group_admin.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_group_user.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_properties.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_systemtag.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_trusted_servers.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 11:24 oc_vcategory_to_object.ibd
-rw-rw---- 1 mysql mysql 112K Jan 30 12:42 oc_mimetypes.ibd
-rw-rw---- 1 mysql mysql 112K Jan 31 10:41 oc_addressbookchanges.ibd
-rw-rw---- 1 mysql mysql 112K Jan 31 10:41 oc_addressbooks.ibd
-rw-rw---- 1 mysql mysql 112K Jan 31 10:41 oc_calendars.ibd
-rw-rw---- 1 mysql mysql 112K Jan 31 10:41 oc_cards.ibd
-rw-rw---- 1 mysql mysql 112K Jan 31 11:10 oc_jobs.ibd
-rw-rw---- 1 mysql mysql 112K Jan 31 11:19 oc_storages.ibd
-rw-rw---- 1 mysql mysql 1.1K Jan 30 11:24 oc_groups.frm
-rw-rw---- 1 mysql mysql 128K Jan 30 11:24 oc_account_terms.ibd
-rw-rw---- 1 mysql mysql 128K Jan 30 11:24 oc_share_external.ibd
-rw-rw---- 1 mysql mysql 128K Jan 31 11:08 oc_external_options.ibd
-rw-rw---- 1 mysql mysql 128K Jan 31 11:10 oc_appconfig.ibd
-rw-rw---- 1 mysql mysql 128K Jan 31 11:10 oc_external_config.ibd
-rw-rw---- 1 mysql mysql 128K Jan 31 11:23 oc_authtoken.ibd
-rw-rw---- 1 mysql mysql 144K Jan 30 11:24 oc_files_trash.ibd
-rw-rw---- 1 mysql mysql 144K Jan 30 11:24 oc_vcategory.ibd
-rw-rw---- 1 mysql mysql 144K Jan 31 10:41 oc_cards_properties.ibd
-rw-rw---- 1 mysql mysql 144K Jan 31 11:03 oc_external_applicable.ibd
-rw-rw---- 1 mysql mysql 1.4K Jan 30 11:24 oc_systemtag_object_mapping.frm
-rw-rw---- 1 mysql mysql 15K Jan 30 11:24 oc_external_config.frm
-rw-rw---- 1 mysql mysql 15K Jan 30 11:24 oc_jobs.frm
-rw-rw---- 1 mysql mysql 1.5K Jan 30 11:24 oc_migrations.frm
-rw-rw---- 1 mysql mysql 15K Jan 30 11:24 oc_mounts.frm
-rw-rw---- 1 mysql mysql 1.5K Jan 30 11:24 oc_preferences.frm
-rw-rw---- 1 mysql mysql 160K Jan 30 11:24 oc_comments.ibd
-rw-rw---- 1 mysql mysql 160K Jan 30 11:24 oc_notifications.ibd
-rw-rw---- 1 mysql mysql 160K Jan 31 10:42 oc_accounts.ibd
-rw-rw---- 1 mysql mysql 160K Jan 31 11:10 oc_mounts.ibd
-rw-rw---- 1 mysql mysql 160K Jan 31 11:23 oc_share.ibd
-rw-rw---- 1 mysql mysql 17K Jan 30 11:24 oc_share_external.frm
-rw-rw---- 1 mysql mysql 1.7K Jan 30 11:24 oc_storages.frm
-rw-rw---- 1 mysql mysql 1.7K Jan 30 11:24 oc_systemtag.frm
-rw-rw---- 1 mysql mysql 1.7K Jan 30 11:24 oc_systemtag_group.frm
-rw-rw---- 1 mysql mysql 1.7K Jan 30 11:24 oc_vcategory_to_object.frm
-rw-rw---- 1 mysql mysql 18K Jan 30 11:24 oc_filecache.frm
-rw-rw---- 1 mysql mysql 1.8K Jan 30 11:24 oc_group_admin.frm
-rw-rw---- 1 mysql mysql 1.8K Jan 30 11:24 oc_group_user.frm
-rw-rw---- 1 mysql mysql 1.8K Jan 30 11:50 oc_users_external.frm
-rw-rw---- 1 mysql mysql 1.9K Jan 30 11:24 oc_credentials.frm
-rw-rw---- 1 mysql mysql 1.9K Jan 30 11:24 oc_external_mounts.frm
-rw-rw---- 1 mysql mysql 208M Jan 31 11:23 oc_file_locks.ibd
-rw-rw---- 1 mysql mysql 2.1K Jan 30 11:24 oc_comments_read_markers.frm
-rw-rw---- 1 mysql mysql 2.1K Jan 30 11:24 oc_users.frm
-rw-rw---- 1 mysql mysql 2.2K Jan 30 11:24 oc_file_locks.frm
-rw-rw---- 1 mysql mysql 2.2K Jan 30 11:24 oc_mimetypes.frm
-rw-rw---- 1 mysql mysql 2.3K Jan 30 11:24 oc_addressbookchanges.frm
-rw-rw---- 1 mysql mysql 2.3K Jan 30 11:24 oc_appconfig.frm
-rw-rw---- 1 mysql mysql 2.3K Jan 30 11:24 oc_calendarchanges.frm
-rw-rw---- 1 mysql mysql 2.5K Jan 30 11:24 oc_account_terms.frm
-rw-rw---- 1 mysql mysql 2.5K Jan 30 11:24 oc_cards.frm
-rw-rw---- 1 mysql mysql 2.7K Jan 30 11:24 oc_external_applicable.frm
-rw-rw---- 1 mysql mysql 2.7K Jan 30 11:24 oc_schedulingobjects.frm
-rw-rw---- 1 mysql mysql 28K Jan 30 11:24 oc_notifications.frm
-rw-rw---- 1 mysql mysql 2.9K Jan 30 11:24 oc_external_options.frm
-rw-rw---- 1 mysql mysql 3.0K Jan 30 11:24 oc_properties.frm
-rw-rw---- 1 mysql mysql 3.1K Jan 30 11:24 oc_authtoken.frm
-rw-rw---- 1 mysql mysql 3.4K Jan 30 11:24 oc_calendarobjects.frm
-rw-rw---- 1 mysql mysql 3.5K Jan 30 11:24 oc_cards_properties.frm
-rw-rw---- 1 mysql mysql 3.5K Jan 30 11:24 oc_privatedata.frm
-rw-rw---- 1 mysql mysql 3.6K Jan 30 11:24 oc_vcategory.frm
-rw-rw---- 1 mysql mysql 3.8K Jan 30 11:24 oc_dav_properties.frm
-rw-rw---- 1 mysql mysql 3.8K Jan 30 11:24 oc_dav_shares.frm
-rw-rw---- 1 mysql mysql 4.2K Jan 30 11:24 oc_comments.frm
-rw-rw---- 1 mysql mysql 4.6K Jan 30 11:24 oc_addressbooks.frm
-rw-rw---- 1 mysql mysql 5.1K Jan 30 11:24 oc_calendarsubscriptions.frm
-rw-rw---- 1 mysql mysql 54 Jan 30 11:24 db.opt
-rw-rw---- 1 mysql mysql 5.5K Jan 30 11:24 oc_calendars.frm
-rw-rw---- 1 mysql mysql 5.8K Jan 30 11:24 oc_files_trash.frm
-rw-rw---- 1 mysql mysql 6.6K Jan 30 11:24 oc_trusted_servers.frm
-rw-rw---- 1 mysql mysql 712M Jan 31 11:22 oc_filecache.ibd
-rw-rw---- 1 mysql mysql 7.9K Jan 30 11:24 oc_share.frm
-rw-rw---- 1 mysql mysql 9.4K Jan 30 11:24 oc_accounts.frm
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_federated_reshares.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_groups.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_migrations.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_privatedata.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_schedulingobjects.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_systemtag_group.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:24 oc_systemtag_object_mapping.ibd
-rw-rw---- 1 mysql mysql 96K Jan 30 11:50 oc_users_external.ibd
-rw-rw---- 1 mysql mysql 96K Jan 31 10:41 oc_preferences.ibd
-rw-rw---- 1 mysql mysql 96K Jan 31 10:41 oc_users.ibd
-rw-rw---- 1 mysql mysql 96K Jan 31 11:03 oc_external_mounts.ibd
-rw-rw---- 1 mysql mysql 996 Jan 30 11:24 oc_federated_reshares.frm
total 926M

I asked myself on an other question which is : Those metadatas, are they shared between users (each of those have his mount point in his home from where symbolic links send him to different partitions).

In that matter, i created/mounted a second user in oC but so far we are still at 926M.

Thanks for your advices ! Without 'em we'd just move to other solutions :wink:


#5

I see 712MB for the file cache, which for a 1PB of data seems fine. The other big table is the file locks, which should shrink after a while. You can also try to use redis for file locking, which should move that lock information to a redis server. (There should be documentation about this)

The metadata is stored per ownCloud storage. Depending on how the connection is configured that ownCloud storage can be shared among several users or not.
For example, if the admin sets the external storage connection with username + password, that ownCloud storage will be shared among the user that have access to it. If the same connection uses login credentials, each user will have a different storage because the credentials to access to the storage will be different. In this second case, the size might be duplicated (depending on what files are being access by who)


#6

Anyway, thanks..you saved my day :wink: