Low speed processing of files when uploading to external storage Samba

external_storage

#1

Hello.
I needed to create a cloud storage with folders that were protected from deletion. I implemented this on the external storage of SAMBA (with ACL) and sticky-bit "t".

This solution works fine, except for speed. If I upload one large file to the external storage of Samba via the web interface Owncloud, then everything is normal - the speed is about 5-7 Mb/sec. But if you try to upload a folder with 50 files at 10kb, then it will be uploaded during 3 minutes!

The network log of the browser shows that the query processing 3.5 seconds for each file.

I thought the problem was in Samba, but connecting to it directly from Windows, the test folder was uploaded in 3 seconds.

The desktop client Owncloud uploads the folder for about a minute, but this is also very long.

The query log of MySQL contains more than 9,500 queries, of which 6,000 are to the "oc_filecache" table. In my opinion, this is a too lot for the operation of uploading 50 files.

Are there any solutions?

PS: I use parameter: "part_file_in_storage": false

Server configuration

Operating system: Ubuntu 16.04.3

Web server: Apache 2.4.18

Database: MySQL 5.7

PHP version: PHP 7

ownCloud version: 10.0.4.4

Updated from an older ownCloud or fresh install:
Updated from 10.0.3
Where did you install ownCloud from:
Repository

The content of config/config.php:

{
    "basic": {
        "license key": "***REMOVED SENSITIVE VALUE***",
        "date": "Thu, 01 Feb 2018 12:16:48 +0000",
        "ownCloud version": "10.0.4.4",
        "ownCloud version string": "10.0.4",
        "ownCloud edition": "Community",
        "server OS": "Linux",
        "server OS version": "Linux cloud-buro 4.13.0-32-generic #35~16.04.1-Ubuntu SMP Thu Jan 25 10:13:43 UTC 2018 x86_64",
        "server SAPI": "apache2handler",
        "webserver version": "Apache\/2.4.18 (Ubuntu)",
        "hostname": "disk.mydomain.ru",
        "user count": 4,
        "user directories": 4,
        "logged-in user": "stasya"
    },
    "config": {
        "instanceid": "oc5iv7qc9tq0",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "disk.mydomain.ru"
        ],
        "datadirectory": "\/home\/data\/cloud",
        "overwrite.cli.url": "https:\/\/disk.mydomain.ru",
        "dbtype": "mysql",
        "version": "10.0.4.4",
        "dbname": "owncloud",
        "dbhost": "localhost",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "part_file_in_storage": false,
        "tempdirectory": "\/tmp\/owncloudtemp",
        "logfile": "\/var\/log\/owncloud.log",
        "loglevel": 3,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "localhost",
            "port": 6379
        },
        "theme": "",
        "maintenance": false,
        "mail_from_address": "noreply-cloud",
        "mail_smtpmode": "php",
        "mail_domain": "mydomain.ru",
        "mail_smtpsecure": "ssl"
    },
    "integritychecker": {
        "passing": true,
        "enabled": true,
        "result": []
    },
    "core": {
        "backgroundjobs_mode": "cron",
        "enable_external_storage": "yes",
        "installedat": "1512128647.9152",
        "lastcron": "1512728245",
        "lastupdateResult": "[]",
        "lastupdatedat": "1517486437",
        "public_files": "files_sharing\/public.php",
        "public_webdav": "dav\/appinfo\/v1\/publicwebdav.php",
        "vendor": "owncloud"
    },
    
    "phpinfo": {
        "apache2handler": {
            "Apache Version": "Apache\/2.4.18 (Ubuntu)",
            "Apache API Version": "20120211",
            "Server Administrator": "[no address given]",
            "Hostname:Port": "disk.mydomain.ru:0",
            "User\/Group": "www-data(33)\/33",
            "Max Requests": "Per Child: 0 - Keep Alive: on - Max Per Connection: 100",
            "Timeouts": "Connection: 300 - Keep-Alive: 5",
            "Virtual Server": "Yes",
            "Server Root": "\/etc\/apache2",
            "Loaded Modules": "core mod_so mod_watchdog http_core mod_log_config mod_logio mod_version mod_unixd mod_access_compat mod_alias mod_auth_basic mod_authn_core mod_authn_file mod_authz_core mod_authz_host mod_authz_user mod_autoindex mod_deflate mod_dir mod_env mod_filter mod_headers mod_mime prefork mod_negotiation mod_php7 mod_rewrite mod_setenvif mod_socache_shmcb mod_ssl mod_status",
            "engine": "1",
            "last_modified": "0",
            "xbithack": "0"
        },
        "Apache Environment": {
            "HTTP_AUTHORIZATION": "no value",
            "modHeadersAvailable": "true",
            "htaccessWorking": "true",
            "HTTPS": "on",
            "SSL_TLS_SNI": "disk.mydomain.ru",
            "HTTP_HOST": "disk.mydomain.ru",
            "HTTP_CONNECTION": "keep-alive",
            "HTTP_UPGRADE_INSECURE_REQUESTS": "1",
            "HTTP_USER_AGENT": "Mozilla\/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/63.0.3239.132 Safari\/537.36",
            "HTTP_ACCEPT": "text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/webp,image\/apng,*\/*;q=0.8",
            "HTTP_ACCEPT_ENCODING": "gzip, deflate, br",
            "HTTP_ACCEPT_LANGUAGE": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7",
            "HTTP_X_COMPRESS": "null",
            "SERVER_SIGNATURE": "<address>Apache\/2.4.18 (Ubuntu) Server at disk.mydomain.ru Port 443<\/address>",
            "SERVER_SOFTWARE": "Apache\/2.4.18 (Ubuntu)",
            "SERVER_NAME": "disk.mydomain.ru",
            "SERVER_ADDR": "172.16.18.222",
            "SERVER_PORT": "443",
            "REMOTE_ADDR": "172.16.18.127",
            "DOCUMENT_ROOT": "\/var\/www\/owncloud\/",
            "REQUEST_SCHEME": "https",
            "CONTEXT_PREFIX": "no value",
            "CONTEXT_DOCUMENT_ROOT": "\/var\/www\/owncloud\/",
            "SERVER_ADMIN": "[no address given]",
            "SCRIPT_FILENAME": "\/var\/www\/owncloud\/index.php",
            "REMOTE_PORT": "32457",
            "GATEWAY_INTERFACE": "CGI\/1.1",
            "SERVER_PROTOCOL": "HTTP\/1.1",
            "REQUEST_METHOD": "GET",
            "QUERY_STRING": "no value",
            "REQUEST_URI": "\/index.php\/apps\/configreport\/report",
            "SCRIPT_NAME": "\/index.php",
            "PATH_INFO": "\/apps\/configreport\/report",
            "PATH_TRANSLATED": "\/var\/www\/owncloud\/apps\/configreport\/report"
        },
        "HTTP Headers Information": {
            "HTTP Request": "GET \/index.php\/apps\/configreport\/report HTTP\/1.1",
            "Host": "disk.mydomain.ru",
            "Connection": "keep-alive",
            "Upgrade-Insecure-Requests": "1",
            "User-Agent": "Mozilla\/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/63.0.3239.132 Safari\/537.36",
            "Accept": "text\/html,application\/xhtml+xml,application\/xml;q=0.9,image\/webp,image\/apng,*\/*;q=0.8",
            "Accept-Encoding": "gzip, deflate, br",
            "Accept-Language": "ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7",
            "X-Compress": "null",
            "Expires": "Thu, 19 Nov 1981 08:52:00 GMT",
            "Cache-Control": "no-store, no-cache, must-revalidate",
            "Pragma": "no-cache",
            "Content-Security-Policy": "default-src self ; script-src self unsafe-eval ; style-src self unsafe-inline ; frame-src *; img-src * data: blob:; font-src self data:; media-src *; connect-src *"
        },
        "apcu": {
            "Version": "5.1.3",
            "APCu Debugging": "Disabled",
            "MMAP Support": "Enabled",
            "MMAP File Mask": "no value",
            "Serialization Support": "php",
            "Build Date": "Apr 5 2016 02:16:47",
            "apc.coredump_unmap": "Off",
            "apc.enable_cli": "Off",
            "apc.enabled": "On",
            "apc.entries_hint": "4096",
            "apc.gc_ttl": "3600",
            "apc.mmap_file_mask": "no value",
            "apc.preload_path": "no value",
            "apc.serializer": "php",
            "apc.shm_segments": "1",
            "apc.shm_size": "32M",
            "apc.slam_defense": "On",
            "apc.smart": "0",
            "apc.ttl": "0",
            "apc.use_request_time": "On",
            "apc.writable": "\/tmp"
        },
        "calendar": {
            "Calendar support": "enabled"
        },
        "Core": {
            "PHP Version": "7.0.22-0ubuntu0.16.04.1",
            "allow_url_fopen": "On",
            "allow_url_include": "Off",
            "arg_separator.input": "&",
            "arg_separator.output": "&",
            "auto_append_file": "no value",
            "auto_globals_jit": "On",
            "auto_prepend_file": "no value",
            "browscap": "no value",
            "default_charset": "UTF-8",
            "default_mimetype": "text\/html",
            "disable_classes": "no value",
            "disable_functions": "pcntl_alarm,pcntl_fork,pcntl_waitpid,pcntl_wait,pcntl_wifexited,pcntl_wifstopped,pcntl_wifsignaled,pcntl_wifcontinued,pcntl_wexitstatus,pcntl_wtermsig,pcntl_wstopsig,pcntl_signal,pcntl_signal_dispatch,pcntl_get_last_error,pcntl_strerror,pcntl_sigprocmask,pcntl_sigwaitinfo,pcntl_sigtimedwait,pcntl_exec,pcntl_getpriority,pcntl_setpriority,",
            "display_errors": "Off",
            "display_startup_errors": "Off",
            "doc_root": "no value",
            "docref_ext": "no value",
            "docref_root": "no value",
            "enable_dl": "Off",
            "enable_post_data_reading": "On",
            "error_append_string": "no value",
            "error_log": "no value",
            "error_prepend_string": "no value",
            "error_reporting": [
                "32767",
                "22527"
            ],
            "exit_on_timeout": "Off",
            "expose_php": "Off",
            "extension_dir": "\/usr\/lib\/php\/20151012",
            "file_uploads": "On",
            "highlight.comment": "#FF8000",
            "highlight.default": "#0000BB",
            "highlight.html": "#000000",
            "highlight.keyword": "#007700",
            "highlight.string": "#DD0000",
            "html_errors": "On",
            "ignore_repeated_errors": "Off",
            "ignore_repeated_source": "Off",
            "ignore_user_abort": "Off",
            "implicit_flush": "Off",
            "input_encoding": "no value",
            "internal_encoding": "no value",
            "log_errors": "On",
            "log_errors_max_len": "1024",
            "mail.add_x_header": "On",
            "mail.force_extra_parameters": "no value",
            "mail.log": "no value",
            "max_execution_time": [
                "3600",
                "30"
            ],
            "max_file_uploads": "20",
            "max_input_nesting_level": "64",
            "max_input_time": "60",
            "max_input_vars": "1000",
            "memory_limit": [
                "512M",
                "128M"
            ],
            "open_basedir": "no value",
            "output_buffering": [
                "0",
                "4096"
            ],
            "output_encoding": "no value",
            "output_handler": "no value",
            "post_max_size": [
                "513M",
                "8M"
            ],
            "precision": "14",
            "realpath_cache_size": "4096K",
            "realpath_cache_ttl": "120",
            "register_argc_argv": "Off",
            "report_memleaks": "On",
            "report_zend_debug": "On",
            "request_order": "GP",
            "sendmail_from": "no value",
            "sendmail_path": "\/usr\/sbin\/sendmail -t -i ",
            "serialize_precision": "17",
            "short_open_tag": "Off",
            "SMTP": "localhost",
            "smtp_port": "25",
            "sql.safe_mode": "Off",
            "sys_temp_dir": "no value",
            "track_errors": "Off",
            "unserialize_callback_func": "no value",
            "upload_max_filesize": [
                "513M",
                "2M"
            ],
            "upload_tmp_dir": "no value",
            "user_dir": "no value",
            "user_ini.cache_ttl": "300",
            "user_ini.filename": ".user.ini",
            "variables_order": "GPCS",
            "xmlrpc_error_number": "0",
            "xmlrpc_errors": "Off",
            "zend.assertions": "-1",
            "zend.detect_unicode": "On",
            "zend.enable_gc": "On",
            "zend.multibyte": "Off",
            "zend.script_encoding": "no value"
        },
        "mysqli": {
            "Client API library version": "mysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $",
            "Active Persistent Links": "0",
            "Inactive Persistent Links": "0",
            "Active Links": "0",
            "mysqli.allow_local_infile": "On",
            "mysqli.allow_persistent": "On",
            "mysqli.default_host": "no value",
            "mysqli.default_port": "3306",
            "mysqli.default_pw": "no value",
            "mysqli.default_socket": "no value",
            "mysqli.default_user": "no value",
            "mysqli.max_links": "Unlimited",
            "mysqli.max_persistent": "Unlimited",
            "mysqli.reconnect": "Off",
            "mysqli.rollback_on_cached_plink": "Off"
        },
        "mysqlnd": {
            "Version": "mysqlnd 5.0.12-dev - 20150407 - $Id: b5c5906d452ec590732a93b051f3827e02749b83 $",
            "Compression": "supported",
            "core SSL": "supported",
            "extended SSL": "supported",
            "Command buffer size": "4096",
            "Read buffer size": "32768",
            "Read timeout": "31536000",
            "Collecting statistics": "Yes",
            "Collecting memory statistics": "No",
            "Tracing": "n\/a",
            "Loaded plugins": "mysqlnd,debug_trace,auth_plugin_mysql_native_password,auth_plugin_mysql_clear_password,auth_plugin_sha256_password",
            "API Extensions": "mysqli,pdo_mysql",
            "bytes_sent": "353544",
            "bytes_received": "1397850",
            "packets_sent": "4084",
            "packets_received": "28109",
            "protocol_overhead_in": "112436",
            "protocol_overhead_out": "16336",
            "bytes_received_ok_packet": "0",
            "bytes_received_eof_packet": "0",
            "bytes_received_rset_header_packet": "15687",
            "bytes_received_rset_field_meta_packet": "0",
            "bytes_received_rset_row_packet": "15281",
            "bytes_received_prepare_response_packet": "923012",
            "bytes_received_change_user_packet": "431075",
            "packets_sent_command": "2004",
            "packets_received_ok": "0",
            "packets_received_eof": "0",
            "packets_received_rset_header": "1743",
            "packets_received_rset_field_meta": "0",
            "packets_received_rset_row": "1967",
            "packets_received_prepare_response": "12775",
            "packets_received_change_user": "11548",
            "result_set_queries": "1743",
            "non_result_set_queries": "224",
            "no_index_used": "247",
            "bad_index_used": "0",
            "slow_queries": "0",
            "buffered_sets": "1743",
            "unbuffered_sets": "0",
            "ps_buffered_sets": "0",
            "ps_unbuffered_sets": "0",
            "flushed_normal_sets": "0",
            "flushed_ps_sets": "0",
            "ps_prepared_never_executed": "0",
            "ps_prepared_once_executed": "0",
            "rows_fetched_from_server_normal": "9805",
            "rows_fetched_from_server_ps": "0",
            "rows_buffered_from_client_normal": "9805",
            "rows_buffered_from_client_ps": "0",
            "rows_fetched_from_client_normal_buffered": "9805",
            "rows_fetched_from_client_normal_unbuffered": "0",
            "rows_fetched_from_client_ps_buffered": "0",
            "rows_fetched_from_client_ps_unbuffered": "0",
            "rows_fetched_from_client_ps_cursor": "0",
            "rows_affected_normal": "64",
            "rows_affected_ps": "0",
            "rows_skipped_normal": "9805",
            "rows_skipped_ps": "0",
            "copy_on_write_saved": "0",
            "copy_on_write_performed": "0",
            "command_buffer_too_small": "0",
            "connect_success": "38",
            "connect_failure": "14",
            "connection_reused": "0",
            "reconnect": "0",
            "pconnect_success": "0",
            "active_connections": "18446744073709551566",
            "active_persistent_connections": "0",
            "explicit_close": "37",
            "implicit_close": "0",
            "disconnect_close": "0",
            "in_middle_of_command_close": "0",
            "explicit_free_result": "1743",
            "implicit_free_result": "0",
            "explicit_stmt_close": "0",
            "implicit_stmt_close": "0",
            "mem_emalloc_count": "0",
            "mem_emalloc_amount": "0",
            "mem_ecalloc_count": "0",
            "mem_ecalloc_amount": "0",
            "mem_erealloc_count": "0",
            "mem_erealloc_amount": "0",
            "mem_efree_count": "0",
            "mem_efree_amount": "0",
            "mem_malloc_count": "0",
            "mem_malloc_amount": "0",
            "mem_calloc_count": "0",
            "mem_calloc_amount": "0",
            "mem_realloc_count": "0",
            "mem_realloc_amount": "0",
            "mem_free_count": "0",
            "mem_free_amount": "0",
            "mem_estrndup_count": "0",
            "mem_strndup_count": "0",
            "mem_estndup_count": "0",
            "mem_strdup_count": "0",
            "proto_text_fetched_null": "0",
            "proto_text_fetched_bit": "0",
            "proto_text_fetched_tinyint": "0",
            "proto_text_fetched_short": "161",
            "proto_text_fetched_int24": "0",
            "proto_text_fetched_int": "4969",
            "proto_text_fetched_bigint": "7569",
            "proto_text_fetched_decimal": "10",
            "proto_text_fetched_float": "0",
            "proto_text_fetched_double": "0",
            "proto_text_fetched_date": "0",
            "proto_text_fetched_year": "0",
            "proto_text_fetched_time": "0",
            "proto_text_fetched_datetime": "0",
            "proto_text_fetched_timestamp": "0",
            "proto_text_fetched_string": "19733",
            "proto_text_fetched_blob": "0",
            "proto_text_fetched_enum": "4118",
            "proto_text_fetched_set": "0",
            "proto_text_fetched_geometry": "0",
            "proto_text_fetched_other": "0",
            "proto_binary_fetched_null": "0",
            "proto_binary_fetched_bit": "0",
            "proto_binary_fetched_tinyint": "0",
            "proto_binary_fetched_short": "0",
            "proto_binary_fetched_int24": "0",
            "proto_binary_fetched_int": "0",
            "proto_binary_fetched_bigint": "0",
            "proto_binary_fetched_decimal": "0",
            "proto_binary_fetched_float": "0",
            "proto_binary_fetched_double": "0",
            "proto_binary_fetched_date": "0",
            "proto_binary_fetched_year": "0",
            "proto_binary_fetched_time": "0",
            "proto_binary_fetched_datetime": "0",
            "proto_binary_fetched_timestamp": "0",
            "proto_binary_fetched_string": "0",
            "proto_binary_fetched_json": "0",
            "proto_binary_fetched_blob": "0",
            "proto_binary_fetched_enum": "0",
            "proto_binary_fetched_set": "0",
            "proto_binary_fetched_geometry": "0",
            "proto_binary_fetched_other": "0",
            "init_command_executed_count": "0",
            "init_command_failed_count": "0",
            "com_quit": "37",
            "com_init_db": "0",
            "com_query": "1967",
            "com_field_list": "0",
            "com_create_db": "0",
            "com_drop_db": "0",
            "com_refresh": "0",
            "com_shutdown": "0",
            "com_statistics": "0",
            "com_process_info": "0",
            "com_connect": "0",
            "com_process_kill": "0",
            "com_debug": "0",
            "com_ping": "0",
            "com_time": "0",
            "com_delayed_insert": "0",
            "com_change_user": "0",
            "com_binlog_dump": "0",
            "com_table_dump": "0",
            "com_connect_out": "0",
            "com_register_slave": "0",
            "com_stmt_prepare": "0",
            "com_stmt_execute": "0",
            "com_stmt_send_long_data": "0",
            "com_stmt_close": "0",
            "com_stmt_reset": "0",
            "com_stmt_set_option": "0",
            "com_stmt_fetch": "0",
            "com_deamon": "0",
            "bytes_received_real_data_normal": "348199",
            "bytes_received_real_data_ps": "0"
        },
        "openssl": {
            "OpenSSL support": "enabled",
            "OpenSSL Library Version": "OpenSSL 1.0.2g 1 Mar 2016",
            "OpenSSL Header Version": "OpenSSL 1.0.2g 1 Mar 2016",
            "Openssl default config": "\/usr\/lib\/ssl\/openssl.cnf",
            "openssl.cafile": "\/etc\/apache2\/ssl\/server.crt",
            "openssl.capath": "\/etc\/apache2\/ssl"
        },
        "Zend OPcache": {
            "Opcode Caching": "Up and Running",
            "Optimization": "Enabled",
            "SHM Cache": "Enabled",
            "File Cache": "Disabled",
            "Startup": "OK",
            "Shared memory model": "mmap",
            "Cache hits": "248764",
            "Cache misses": "996",
            "Used memory": "24646880",
            "Free memory": "42461984",
            "Wasted memory": "0",
            "Interned Strings Used memory": "2458448",
            "Interned Strings Free memory": "1735856",
            "Cached scripts": "996",
            "Cached keys": "1918",
            "Max keys": "3907",
            "OOM restarts": "0",
            "Hash keys restarts": "0",
            "Manual restarts": "0",
            "opcache.blacklist_filename": "no value",
            "opcache.consistency_checks": "0",
            "opcache.dups_fix": "Off",
            "opcache.enable": "On",
            "opcache.enable_cli": "Off",
            "opcache.enable_file_override": "Off",
            "opcache.error_log": "no value",
            "opcache.fast_shutdown": "0",
            "opcache.file_cache": "no value",
            "opcache.file_cache_consistency_checks": "1",
            "opcache.file_cache_only": "0",
            "opcache.file_update_protection": "2",
            "opcache.force_restart_timeout": "180",
            "opcache.huge_code_pages": "Off",
            "opcache.inherited_hack": "On",
            "opcache.interned_strings_buffer": "4",
            "opcache.lockfile_path": "\/tmp",
            "opcache.log_verbosity_level": "1",
            "opcache.max_accelerated_files": "2000",
            "opcache.max_file_size": "0",
            "opcache.max_wasted_percentage": "5",
            "opcache.memory_consumption": "64",
            "opcache.optimization_level": "0x7FFFBFFF",
            "opcache.preferred_memory_model": "no value",
            "opcache.protect_memory": "0",
            "opcache.restrict_api": "no value",
            "opcache.revalidate_freq": "2",
            "opcache.revalidate_path": "Off",
            "opcache.save_comments": "1",
            "opcache.use_cwd": "On",
            "opcache.validate_permission": "Off",
            "opcache.validate_root": "Off",
            "opcache.validate_timestamps": "On"
        }
		.....
    }
}

List of activated apps:

Enabled:
  - activity: 2.3.6
  - comments: 0.3.0
  - configreport: 0.1.1
  - dav: 0.3.2
  - federatedfilesharing: 0.3.1
  - federation: 0.1.0
  - files: 1.5.1
  - files_external: 0.7.1
  - files_pdfviewer: 0.8.2
  - files_sharing: 0.10.1
  - files_trashbin: 0.9.1
  - files_versions: 1.3.0
  - files_videoplayer: 0.9.8
  - firstrunwizard: 1.1
  - gallery: 16.0.2
  - impersonate: 0.1.2
  - market: 0.2.3
  - notifications: 0.3.2
  - onlyoffice: 1.2.0
  - provisioning_api: 0.5.0
  - systemtags: 0.3.0
  - updatenotification: 0.2.1
Disabled:
  - encryption
  - external
  - templateeditor
  - theme-example
  - user_external

Are you using external storage, if yes which one:
SAMBA 4

Are you using encryption:
Only HTTPS

Are you using an external user-backend, if yes which one:
No

Client configuration

Browser: Chrome (final)

Operating system: Windows 7 and 10

Logs

Web server error log

No error

ownCloud log (data/owncloud.log)

No error

#2

Trying uploading a file via smbclient from the ownCloud server and check the speed. This is a better test than uploading from another machine. If the speed is the same, then the problem won't be in ownCloud but somewhere else.


#3

Via smbclient from the OwnCloud server the files are uploaded instantly - 1 second. OwnCloud and Samba are installed the same server.


Every ".txt" file have size 10 Kb.

Uploading files via Web-interface to the ownCloud folder takes a 5-6 seconds (100-150ms per file)... it's a good.

When I upload test files into the OwnCloud folder, the query log MySQL also contains 9700 entries, so Mysql is not the source of the problem. Although the number of requests is still huge.

Apparently, the source of the problem is in the "gateway" (External Storage module) between the ownCloud and the Samba.


#4

I did not have the php-smbclient module installed. After its installation, the upload time to the Samba was slightly reduced - 2.3 sec, instead of 3.5 sec. But it's still long.


#5

The files_version app might affect if the files aren't new because we need to download the files to store the version (plus more processing). Something similar for files_trashbin in case the file is deleted.
You might want to disable those apps if you don't plan to use them and check again.

I'm not familiar with the onlyoffice app, but I doubt it influences the upload / download speed.


#6

I disabled the Versions, Deleted Files, Notification, Onlyoffice and Activity apps, but this did not solve the problem ((


#7

If you can manage PHP code I can give you a small code snippet to use the libsmbclient-php library directly in order to upload the files, so we can discard problems with the library.

I expect a slight decrease due to the reduced overhead. If the timing matches the on from the smbclient then we'll need to find the reason in additional actions not directly involved with the data transfer between ownCloud and the server (reasons such as the huge amount of MySQL queries).


#8

I am very grateful to you for your help!

I can manage PHP code.


#9

<?php
$fileList = [
	'/tmp/fooo',
	'/tmp/owncloud_install',
];
$remoteLocationDir = 'smb://10.0.2.8/opt/008/';

$username = 'root';
$password = 'root';
$workgroup = 'workgroup';

foreach ($fileList as $file) {
	$t1 = microtime(true);

	$smb = smbclient_state_new();
	smbclient_state_init($smb, $workgroup, $username, $password);

	$filename = basename($file);
	$remote = "$remoteLocationDir$filename";
	$srcFile = fopen($file, "rb");
	$dstFile = smbclient_open($smb, $remote, "w");
	while (!feof($srcFile)) {
		smbclient_write($smb, $dstFile, fread($srcFile, 8096));
	}
	smbclient_close($smb, $dstFile);
	smbclient_state_free($smb);

	$timeTaken = microtime(true) - $t1;
	echo "$filename -> $timeTaken\n";
}

There you have it. You'll need to adjust the parameters at the beginning of the file with the ones you're using. After adjusting the parameters just run the script from the ownCloud server (as long as the file in the ownCloud server it doesn't matter where is placed).

The reason that I'm creating a new state as well as reinitializing it with the credentials is because for ownCloud each file will be uploaded using different requests. This means that the states won't be reused.

If it shows acceptable values, then the problem might be in the DB, so you can check the diagnostics app to gather data. I haven't checked the app but it should help to detect performance problems with the DB.


#10

So, I modified the script and got such results.

    <?php
    $fileList = [
    '/var/test/1.txt',
    '/var/test/2.txt',
    '/var/test/3.txt',
    '/var/test/4.txt',
    '/var/test/5.txt'
    ];
    $remoteLocationDir = 'smb://localhost/home/texts/test2/';

    $username = 'smbadmin';
    $password = '123456789';
    $workgroup = null;

    foreach ($fileList as $file) {
            $filename = basename($file);
            echo "======== $filename  ========<br><br>";
            $t1 = microtime(true);

            $smb = smbclient_state_new();
            $t2 = microtime(true);
            echo 'smbclient_state_new: '.($t2-$t1).'<br><br>';

            $smbinit = smbclient_state_init($smb, $workgroup, $username, $password);
            $t3 = microtime(true);
            echo "smbInit -> ";
            var_dump($smbinit);
            echo '<br>';
            echo 'smbclient_state_init: '.($t3-$t2).'<br><br>';

            $remote = "$remoteLocationDir$filename";
            $srcFile = fopen($file, "r");
            echo "srcFile -> ";
            var_dump($srcFile);
            echo '<br>';
            $t4 = microtime(true);
            echo 'fopen: '.($t4-$t3).'<br><br>';

            $dstFile = smbclient_open($smb, $remote, "w");
            echo "dstFile -> ";
            var_dump($dstFile);
            echo '<br>';
            $t5 = microtime(true);
            echo 'smbclient_open: '.($t5-$t4).'<br><br>';

            while (!feof($srcFile)) {
                    smbclient_write($smb, $dstFile, fread($srcFile, filesize($file)));
            }
            $t6 = microtime(true);
            echo 'copyFile: '.($t6-$t5).'<br><br>';

            smbclient_close($smb, $dstFile);
            smbclient_state_free($smb);
            fclose($srcFile);

            $timeTaken = microtime(true) - $t1;
            echo "$filename -> $timeTaken<br><br><br>";
    }
    ?>

Result (each file is 10Kb):

======== 1.txt ========

smbclient_state_new: 0.00070905685424805

smbInit -> bool(true) 
smbclient_state_init: 2.0980834960938E-5

srcFile -> resource(3) of type (stream) 
fopen: 5.6982040405273E-5

dstFile -> resource(4) of type (smbclient file) 
smbclient_open: 1.022125005722

copyFile: 0.0003058910369873

1.txt -> 1.0236411094666


======== 2.txt ========

smbclient_state_new: 0.00029897689819336

smbInit -> bool(true) 
smbclient_state_init: 7.1525573730469E-6

srcFile -> resource(6) of type (stream) 
fopen: 1.6927719116211E-5

dstFile -> resource(7) of type (smbclient file) 
smbclient_open: 1.0138759613037

copyFile: 0.00028800964355469

2.txt -> 1.0147459506989


======== 3.txt ========

smbclient_state_new: 0.00023984909057617

smbInit -> bool(true) 
smbclient_state_init: 6.1988830566406E-6

srcFile -> resource(9) of type (stream) 
fopen: 1.4781951904297E-5

dstFile -> resource(10) of type (smbclient file) 
smbclient_open: 1.0141611099243

copyFile: 0.00028109550476074

3.txt -> 1.014967918396


======== 4.txt ========

smbclient_state_new: 0.00023007392883301

smbInit -> bool(true) 
smbclient_state_init: 5.0067901611328E-6

srcFile -> resource(12) of type (stream) 
fopen: 1.5020370483398E-5

dstFile -> resource(13) of type (smbclient file) 
smbclient_open: 1.0155980587006

copyFile: 0.0002598762512207

4.txt -> 1.0164270401001


======== 5.txt ========

smbclient_state_new: 0.00025200843811035

smbInit -> bool(true) 
smbclient_state_init: 5.0067901611328E-6

srcFile -> resource(15) of type (stream) 
fopen: 1.5974044799805E-5

dstFile -> resource(16) of type (smbclient file) 
smbclient_open: 1.0152590274811

copyFile: 0.0002439022064209

5.txt -> 1.0160698890686

The total time to copy 5 files of 10KB each is about 6 seconds... a very long time (((
The longest function is "smbclient_open" about 1 second.


#11

If just opening the file takes 1 sec, that could explain at least a part of the problem. I have to check the data in my environment, but unless you're connecting to an external server, it's weird. In any case, the data transfer shouldn't be so fast.

I think there are a couple of things for you to check:

  • The libsmbclient version (the native C/C++ one) you're using is kind of buggy in this regard
  • The SMB server has problems to get the file. Maybe an antivirus program analizying the file or the SMB server overloaded.
  • SMB authentication might also slow down the system if it's doing weird things

You can move the smbclient_state_new() and smbclient_state_init() as well as the smbclient_state_free() out of the loop.
I guess the authentication against the SMB server will happen during the first connection (the first open), so if it's a problem with the authentication the rest of the operations should happen faster in the script.

If that doesn't give successful data, you might want to check against another server in order to keep discarding more things.


#12

The libsmbclient is installed in the latest version with all dependencies. It was installed from Ubuntu repositories. Antivirus is not installed. Loads on the server are not present - I one test this machine.

I shared the folder on my Windows computer on the same network as the ownCloud/Samba server, and tried to copy the files uses this script. The files uploaded instantly. Apparently, the problem is in the Samba server ((

======== 1.txt ========

smbclient_state_new: 0.0016660690307617

smbInit -> bool(true) 
smbclient_state_init: 6.103515625E-5

srcFile -> resource(3) of type (stream) 
fopen: 6.103515625E-5

dstFile -> resource(4) of type (smbclient file) 
smbclient_open: 0.016686916351318

copyFile: 0.0030510425567627

1.txt -> 0.02332592010498


======== 2.txt ========

smbclient_state_new: 0.00069212913513184

smbInit -> bool(true) 
smbclient_state_init: 1.6927719116211E-5

srcFile -> resource(6) of type (stream) 
fopen: 4.4107437133789E-5

dstFile -> resource(7) of type (smbclient file) 
smbclient_open: 0.0080299377441406

copyFile: 0.0030829906463623

2.txt -> 0.013449192047119


======== 3.txt ========

smbclient_state_new: 0.00066494941711426

smbInit -> bool(true) 
smbclient_state_init: 1.5020370483398E-5

srcFile -> resource(9) of type (stream) 
fopen: 4.1007995605469E-5

dstFile -> resource(10) of type (smbclient file) 
smbclient_open: 0.0082519054412842

copyFile: 0.0029699802398682

3.txt -> 0.013675928115845


======== 4.txt ========

smbclient_state_new: 0.00065994262695312

smbInit -> bool(true) 
smbclient_state_init: 1.8835067749023E-5

srcFile -> resource(12) of type (stream) 
fopen: 6.0081481933594E-5

dstFile -> resource(13) of type (smbclient file) 
smbclient_open: 0.0082099437713623

copyFile: 0.017964124679565

4.txt -> 0.028661966323853


======== 5.txt ========

smbclient_state_new: 0.00066399574279785

smbInit -> bool(true) 
smbclient_state_init: 3.6001205444336E-5

srcFile -> resource(15) of type (stream) 
fopen: 4.2200088500977E-5

dstFile -> resource(16) of type (smbclient file) 
smbclient_open: 0.0082039833068848

copyFile: 0.0030269622802734

5.txt -> 0.013666152954102

#13

I lied that the latest version of the libsmbclient was installed. A version was installed 0.8.0-RC1, but the github have 0.9.0. I compile and install the new version, but the problem not resolve.

I did another test. How did you recommend, I moved the smbclient_state_new(), smbclient_state_init() and the smbclient_state_free() out of the loop.

smbclient_state_new: 0.001723051071167

smbInit -> bool(true) 
smbclient_state_init: 7.2956085205078E-5

======== 1.txt ========

srcFile -> resource(4) of type (stream) 
fopen: 5.5789947509766E-5

dstFile -> resource(5) of type (smbclient file) 
smbclient_open: 1.0218710899353

copyFile: 0.00027298927307129

1.txt -> 1.0223808288574


======== 2.txt ========

srcFile -> resource(6) of type (stream) 
fopen: 2.3126602172852E-5

dstFile -> resource(7) of type (smbclient file) 
smbclient_open: 0.0003969669342041

copyFile: 0.00017595291137695

2.txt -> 0.00074100494384766


======== 3.txt ========

srcFile -> resource(8) of type (stream) 
fopen: 1.5020370483398E-5

dstFile -> resource(9) of type (smbclient file) 
smbclient_open: 0.00027894973754883

copyFile: 0.0001678466796875

3.txt -> 0.00060081481933594


======== 4.txt ========

srcFile -> resource(10) of type (stream) 
fopen: 1.3113021850586E-5

dstFile -> resource(11) of type (smbclient file) 
smbclient_open: 0.0002739429473877

copyFile: 0.00016498565673828

4.txt -> 0.00059199333190918


======== 5.txt ========

srcFile -> resource(12) of type (stream) 
fopen: 1.4066696166992E-5

dstFile -> resource(13) of type (smbclient file) 
smbclient_open: 0.00028085708618164

copyFile: 0.00016403198242188

5.txt -> 0.00061202049255371

The first call of smbclient_open was executed for about 1 second, but then the files were copied instantly.


#14

I bet it's something related to the authentication process in your samba server. At least that could explain those results.
Unless you have some kind of external authentication in the samba server, I don't really know what could be slowing down the authentication process; maybe a buggy samba server version, but I don't have proof at the moment.