OCIS slower than OC10 - how to accelerate?

walt · March 9, 2023, 4:49am

We’re starting to test deployment of OCIS and have found it to be slower than ownCloud 10 when there are hundreds of files.

We have what we’ll call “folder A” with 944 files in 45 folders totaling 5.72GB. On our existing OC10 deployment (pretty vanilla setup with mariadb) this takes 4 minutes 2 seconds to sync from the server to Windows using ownCloud Desktop 3.2. The same on OCIS (also vanilla, but running on server with faster CPUs and SSDs that are more than 2x the speed of the server we have OC10 on) takes 4 minutes 40 seconds to sync from the server. Both the OC10 and OCIS deployments are using AWS S3 for primary storage, OCIS is keeping system and metadata storage on ocis.

Conversely, what we’ll call “folder B” with 53 files in a single folder totaling 5.72GB syncs from the server in 2 minutes 5 seconds on OC10 and 1 minute 30 seconds on OCIS.

Seems the per-file overhead on OCIS is bogging things down.

What can we do to accelerate OCIS so it’s at least as fast as OC10 and ideally faster? Speed of file syncs is of the utmost importance for us.

walt · March 9, 2023, 5:45am

Downloads of the same “folder A” direct from the OCIS web frontend via the archiver service complete in about 2 minutes start to finish, so the slowness appears to be specific to ownCloud Desktop sync’ing from an OCIS server.

hodyroff · March 10, 2023, 9:04am

which desktop client version and which kind of authentication are you using? We definetly expect performance to be better. For authentication OIDC works best.

michaelstingl · March 10, 2023, 9:13am

Desktop clients prints duration in the HTTP log lines:
Appendix Troubleshooting :: ownCloud Documentation

What requests take longer with oCIS than oC10 ?

Also with the X-REQUEST-ID’s from the desktop log you could check details what takes that long in the backend.

walt · March 11, 2023, 3:20am

Desktop client version 3.2.1.10355 Windows

We’re using OIDC for auth on OCIS and OC10 with the same IdP

walt · March 11, 2023, 4:17am

There’s a ton of stuff in the logs and the details appear to be a bit different between OC10 and OCIS. The one thing that did jump out comparing the two was that the same file(s) GET instructions report taking way longer on OCIS vs OC10. For example:

OC10:

23-03-11 09:01:32:411 [ info sync.httplogger ]:	"7f106951-9289-4ef2-b6cf-2c25609baa41: Response: GET 200 (298ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Andariel Demon - From Diablo 2/mat4_s.jpg Header: { Date: Sat, 11 Mar 2023 03:31:32 GMT, Server: Apache/2.4.52 (Ubuntu), X-Content-Type-Options: nosniff, X-XSS-Protection: 0, X-Robots-Tag: none, X-Frame-Options: SAMEORIGIN, X-Download-Options: noopen, X-Permitted-Cross-Domain-Policies: none, Vary: Authorization, Expires: Thu, 19 Nov 1981 08:52:00 GMT, Cache-Control: no-store, no-cache, must-revalidate, Pragma: no-cache, Set-Cookie: oc_sessionPassphrase=LEwfbAyjfZwCzA0bMHVH5r05bOV1nbNWMDXjLaHqXzAQ0r9OCjoIRjtbnkdkvbAe%2FuHPbp29mqOfFIZGCve9rDppo6GdyyjM9H%2BoYbvtio96QM8LMddkmU5864JozjvE; expires=Sat, 11-Mar-2023 03:51:32 GMT; Max-Age=1200; path=/; secure; HttpOnly; SameSite=None, Content-Security-Policy: default-src 'none';, Last-Modified: Thu, 22 Sep 2016 20:22:42 GMT, ETag: \"6402feef1c9af\", Content-Length: 109406, OC-ETag: \"6402feef1c9af\", Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", OC-Checksum: SHA1:fd8a102667234795d0760b520861a653e844075c, X-Accel-Buffering: no, Keep-Alive: timeout=5, max=91, Connection: Keep-Alive, Content-Type: image/jpeg, } Data: []"

23-03-11 09:01:35:441 [ info sync.httplogger ]:	"9b5b7116-7205-43fd-b976-df9e1a24eea8: Response: GET 200 (361ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Baal Demon Base - From Diablo 3/mat4_s.jpg Header: { Date: Sat, 11 Mar 2023 03:31:35 GMT, Server: Apache/2.4.52 (Ubuntu), X-Content-Type-Options: nosniff, X-XSS-Protection: 0, X-Robots-Tag: none, X-Frame-Options: SAMEORIGIN, X-Download-Options: noopen, X-Permitted-Cross-Domain-Policies: none, Vary: Authorization, Expires: Thu, 19 Nov 1981 08:52:00 GMT, Cache-Control: no-store, no-cache, must-revalidate, Pragma: no-cache, Set-Cookie: oc_sessionPassphrase=LEwfbAyjfZwCzA0bMHVH5r05bOV1nbNWMDXjLaHqXzAQ0r9OCjoIRjtbnkdkvbAe%2FuHPbp29mqOfFIZGCve9rDppo6GdyyjM9H%2BoYbvtio96QM8LMddkmU5864JozjvE; expires=Sat, 11-Mar-2023 03:51:35 GMT; Max-Age=1200; path=/; secure; HttpOnly; SameSite=None, Content-Security-Policy: default-src 'none';, Last-Modified: Thu, 22 Sep 2016 20:12:38 GMT, ETag: \"6402fefd3fcaa\", Content-Length: 341648, OC-ETag: \"6402fefd3fcaa\", Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", OC-Checksum: SHA1:a4b989ba247308f39fb03c9dcc3af89800237ed1, X-Accel-Buffering: no, Keep-Alive: timeout=5, max=87, Connection: Keep-Alive, Content-Type: image/jpeg, } Data: []"

23-03-11 09:01:46:239 [ info sync.httplogger ]:	"b3f25ec2-a468-47de-86f2-a68e05cf243a: Response: GET 200 (276ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Behemoth Demon - Fan Art Concept/mat4_s.jpg Header: { Date: Sat, 11 Mar 2023 03:31:46 GMT, Server: Apache/2.4.52 (Ubuntu), X-Content-Type-Options: nosniff, X-XSS-Protection: 0, X-Robots-Tag: none, X-Frame-Options: SAMEORIGIN, X-Download-Options: noopen, X-Permitted-Cross-Domain-Policies: none, Vary: Authorization, Expires: Thu, 19 Nov 1981 08:52:00 GMT, Cache-Control: no-store, no-cache, must-revalidate, Pragma: no-cache, Set-Cookie: oc_sessionPassphrase=LEwfbAyjfZwCzA0bMHVH5r05bOV1nbNWMDXjLaHqXzAQ0r9OCjoIRjtbnkdkvbAe%2FuHPbp29mqOfFIZGCve9rDppo6GdyyjM9H%2BoYbvtio96QM8LMddkmU5864JozjvE; expires=Sat, 11-Mar-2023 03:51:46 GMT; Max-Age=1200; path=/; secure; HttpOnly; SameSite=None, Content-Security-Policy: default-src 'none';, Last-Modified: Thu, 22 Sep 2016 22:36:32 GMT, ETag: \"6402ff1eb0706\", Content-Length: 27221, OC-ETag: \"6402ff1eb0706\", Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", OC-Checksum: SHA1:08a911896d09ccef878f06be11816e0e05a17d90, X-Accel-Buffering: no, Keep-Alive: timeout=5, max=68, Connection: Keep-Alive, Content-Type: image/jpeg, } Data: []"

23-03-11 09:01:53:434 [ info sync.httplogger ]:	"7a84c02e-0b4a-4d4f-b34f-7ebe26369b47: Response: GET 200 (536ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Crossfire - Character Design/mat4_s.jpg Header: { Date: Sat, 11 Mar 2023 03:31:53 GMT, Server: Apache/2.4.52 (Ubuntu), X-Content-Type-Options: nosniff, X-XSS-Protection: 0, X-Robots-Tag: none, X-Frame-Options: SAMEORIGIN, X-Download-Options: noopen, X-Permitted-Cross-Domain-Policies: none, Vary: Authorization, Expires: Thu, 19 Nov 1981 08:52:00 GMT, Cache-Control: no-store, no-cache, must-revalidate, Pragma: no-cache, Set-Cookie: oc_sessionPassphrase=LEwfbAyjfZwCzA0bMHVH5r05bOV1nbNWMDXjLaHqXzAQ0r9OCjoIRjtbnkdkvbAe%2FuHPbp29mqOfFIZGCve9rDppo6GdyyjM9H%2BoYbvtio96QM8LMddkmU5864JozjvE; expires=Sat, 11-Mar-2023 03:51:53 GMT; Max-Age=1200; path=/; secure; HttpOnly; SameSite=None, Content-Security-Policy: default-src 'none';, Last-Modified: Mon, 26 Sep 2016 01:21:04 GMT, ETag: \"6402ff3b500f3\", Content-Length: 146022, OC-ETag: \"6402ff3b500f3\", Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", OC-Checksum: SHA1:8de9348d4d43ff6fdfcbd2bd4db0434de6ccc088, X-Accel-Buffering: no, Keep-Alive: timeout=5, max=26, Connection: Keep-Alive, Content-Type: image/jpeg, } Data: []"

OCIS:

23-03-11 08:55:59:026 [ info sync.httplogger ]:	"a25b67ed-b762-4d80-b5f8-01c1e9a17743: Response: GET 200 (Piplined,2187ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Andariel Demon - From Diablo 2/mat4_s.jpg Header: { Accept-Ranges: bytes, Access-Control-Allow-Origin: *, Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", Content-Length: 109406, Content-Security-Policy: default-src 'none';, Content-Type: image/jpeg, Date: Sat, 11 Mar 2023 03:25:59 GMT, Etag: \"dc93e7c1e529b8a52160a0b8abae8673\", Last-Modified: Wed, 08 Mar 2023 05:48:43 +0000, Oc-Checksum: SHA1:fd8a102667234795d0760b520861a653e844075c, Oc-Etag: \"dc93e7c1e529b8a52160a0b8abae8673\", Oc-Fileid: b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad!e7188893-3343-4f00-80a3-cceca9c4239d, Vary: Origin, X-Content-Type-Options: nosniff, X-Download-Options: noopen, X-Frame-Options: SAMEORIGIN, X-Permitted-Cross-Domain-Policies: none, X-Robots-Tag: none, X-Xss-Protection: 1; mode=block, } Data: []"

23-03-11 08:56:03:430 [ info sync.httplogger ]:	"2d18bb97-ead4-4098-8e26-dc232f303ee4: Response: GET 200 (718ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Baal Demon Base - From Diablo 3/mat4_s.jpg Header: { Accept-Ranges: bytes, Access-Control-Allow-Origin: *, Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", Content-Length: 341648, Content-Security-Policy: default-src 'none';, Content-Type: image/jpeg, Date: Sat, 11 Mar 2023 03:26:03 GMT, Etag: \"14b34eb4a9f099a79bc1089a1f492849\", Last-Modified: Wed, 08 Mar 2023 05:48:48 +0000, Oc-Checksum: SHA1:a4b989ba247308f39fb03c9dcc3af89800237ed1, Oc-Etag: \"14b34eb4a9f099a79bc1089a1f492849\", Oc-Fileid: b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad!dddfe17e-f1a8-4575-8a35-65eae5bfc45e, Vary: Origin, X-Content-Type-Options: nosniff, X-Download-Options: noopen, X-Frame-Options: SAMEORIGIN, X-Permitted-Cross-Domain-Policies: none, X-Robots-Tag: none, X-Xss-Protection: 1; mode=block, } Data: []"

23-03-11 08:56:25:986 [ info sync.httplogger ]:	"de72c548-5d45-4fd0-a4fd-5e5351736384: Response: GET 200 (675ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Behemoth Demon - Fan Art Concept/mat4_s.jpg Header: { Accept-Ranges: bytes, Access-Control-Allow-Origin: *, Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", Content-Length: 27221, Content-Security-Policy: default-src 'none';, Content-Type: image/jpeg, Date: Sat, 11 Mar 2023 03:26:26 GMT, Etag: \"55efab42657788a89d3ad54d4b558541\", Last-Modified: Wed, 08 Mar 2023 05:49:02 +0000, Oc-Checksum: SHA1:08a911896d09ccef878f06be11816e0e05a17d90, Oc-Etag: \"55efab42657788a89d3ad54d4b558541\", Oc-Fileid: b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad!293da62c-ca67-430f-ad72-7e581789ae0c, Vary: Origin, X-Content-Type-Options: nosniff, X-Download-Options: noopen, X-Frame-Options: SAMEORIGIN, X-Permitted-Cross-Domain-Policies: none, X-Robots-Tag: none, X-Xss-Protection: 1; mode=block, } Data: []"

23-03-11 08:56:40:694 [ info sync.httplogger ]:	"7553d1d7-3e1a-45b6-9e99-79af6d81cfc5: Response: GET 200 (721ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Crossfire - Character Design/mat4_s.jpg Header: { Accept-Ranges: bytes, Access-Control-Allow-Origin: *, Content-Disposition: attachment; filename*=UTF-8''mat4_s.jpg; filename=\"mat4_s.jpg\", Content-Length: 146022, Content-Security-Policy: default-src 'none';, Content-Type: image/jpeg, Date: Sat, 11 Mar 2023 03:26:41 GMT, Etag: \"a4af8513e999924e8e62de19f52b069e\", Last-Modified: Wed, 08 Mar 2023 05:49:16 +0000, Oc-Checksum: SHA1:8de9348d4d43ff6fdfcbd2bd4db0434de6ccc088, Oc-Etag: \"a4af8513e999924e8e62de19f52b069e\", Oc-Fileid: b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad!f4c4cf28-8c8b-455c-bd43-d374bf2310c8, Vary: Origin, X-Content-Type-Options: nosniff, X-Download-Options: noopen, X-Frame-Options: SAMEORIGIN, X-Permitted-Cross-Domain-Policies: none, X-Robots-Tag: none, X-Xss-Protection: 1; mode=block, } Data: []"

I’m happy to share the full logs if there’s other data in here that could be useful in finding the cause of the OCIS slowness.

jvillafanez · March 13, 2023, 8:15am

Any chance to rule out network issues? I’m not sure how we can get data only from the server. There seem to be some monitoring planned in Monitoring & Tracing | ownCloud but I don’t know how difficult is to setup.

michaelstingl · March 13, 2023, 8:52pm

Bigger difference in small or big files?

oC10

23-03-11 09:01:32:411 [ info sync.httplogger ]:	"7f106951-9289-4ef2-b6cf-2c25609baa41: 
Response: GET 200 (298ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Andariel Demon - From Diablo 2/mat4_s.jpg 
Header: { Content-Length: 109406, } 
Data: []"

23-03-11 09:01:35:441 [ info sync.httplogger ]:	"9b5b7116-7205-43fd-b976-df9e1a24eea8: 
Response: GET 200 (361ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Baal Demon Base - From Diablo 3/mat4_s.jpg 
Header: { Content-Length: 341648, } 
Data: []"

23-03-11 09:01:46:239 [ info sync.httplogger ]:	"b3f25ec2-a468-47de-86f2-a68e05cf243a: 
Response: GET 200 (276ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Behemoth Demon - Fan Art Concept/mat4_s.jpg 
Header: { Content-Length: 27221, } 
Data: []"

23-03-11 09:01:53:434 [ info sync.httplogger ]:	"7a84c02e-0b4a-4d4f-b34f-7ebe26369b47: 
Response: GET 200 (536ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/3D Model Characters/3D Model Characters/Crossfire - Character Design/mat4_s.jpg 
Header: { Content-Length: 146022, } 
Data: []"

oCIS

23-03-11 08:55:59:026 [ info sync.httplogger ]:	"a25b67ed-b762-4d80-b5f8-01c1e9a17743: 
Response: GET 200 (Piplined,2187ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Andariel Demon - From Diablo 2/mat4_s.jpg 
Header: { Content-Length: 109406, } 
Data: []"

23-03-11 08:56:03:430 [ info sync.httplogger ]:	"2d18bb97-ead4-4098-8e26-dc232f303ee4: 
Response: GET 200 (718ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Baal Demon Base - From Diablo 3/mat4_s.jpg 
Header: { Content-Length: 341648, } 
Data: []"

23-03-11 08:56:25:986 [ info sync.httplogger ]:	"de72c548-5d45-4fd0-a4fd-5e5351736384: 
Response: GET 200 (675ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Behemoth Demon - Fan Art Concept/mat4_s.jpg 
Header: { Content-Length: 27221, } 
Data: []"

23-03-11 08:56:40:694 [ info sync.httplogger ]:	"7553d1d7-3e1a-45b6-9e99-79af6d81cfc5: 
Response: GET 200 (721ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/3D Model Characters/3D Model Characters/Crossfire - Character Design/mat4_s.jpg 
Header: { Content-Length: 146022, } 
Data: []"

Could you upload the full log files?
ownCloud Enterprise Edition
(Anonymous upload)

walt · March 14, 2023, 4:48am

Everything’s on the same isolated subnet and tests were run at the same time. Pretty safe to say there’s no network issues influencing the speed of things, particularly since downloads from the OCIS webUI to the same client machine take less than half the time as the Desktop app.

walt · March 14, 2023, 5:04am

Large files in the same sync appear to be way faster on OCIS vs OC10 in the logs:

OC10:

23-03-11 09:05:27:275 [ info sync.httplogger ]:	"04c1ef0c-3cfa-4eaf-a9ca-b124bc86d18c: Response: GET 200 (Piplined,60650ms) https://oc10-redacted/remote.php/dav/files/oidc-user-609b5b8e0119f3c22eb35a3d05298bcf/model/KB3D_LunarBase_FBXOBJ/KB3D_LunarBase-Native.obj Header: { Date: Sat, 11 Mar 2023 03:34:36 GMT, Server: Apache/2.4.52 (Ubuntu), X-Content-Type-Options: nosniff, X-XSS-Protection: 0, X-Robots-Tag: none, X-Frame-Options: SAMEORIGIN, X-Download-Options: noopen, X-Permitted-Cross-Domain-Policies: none, Vary: Authorization,Accept-Encoding, Expires: Thu, 19 Nov 1981 08:52:00 GMT, Cache-Control: no-store, no-cache, must-revalidate, Pragma: no-cache, Set-Cookie: oc_sessionPassphrase=LEwfbAyjfZwCzA0bMHVH5r05bOV1nbNWMDXjLaHqXzAQ0r9OCjoIRjtbnkdkvbAe%2FuHPbp29mqOfFIZGCve9rDppo6GdyyjM9H%2BoYbvtio96QM8LMddkmU5864JozjvE; expires=Sat, 11-Mar-2023 03:54:36 GMT; Max-Age=1200; path=/; secure; HttpOnly; SameSite=None, Content-Security-Policy: default-src 'none';, Last-Modified: Wed, 03 Feb 2021 05:46:40 GMT, ETag: \"6402f9cd2dee0-gzip\", OC-ETag: \"6402f9cd2dee0\", Content-Disposition: attachment; filename*=UTF-8''KB3D_LunarBase-Native.obj; filename=\"KB3D_LunarBase-Native.obj\", OC-Checksum: SHA1:ed61a77f1700cc1cef7c8ad0cf17dd089c4c8b81, X-Accel-Buffering: no, Content-Encoding: gzip, Keep-Alive: timeout=5, max=69, Connection: Keep-Alive, Transfer-Encoding: chunked, Content-Type: text/plain;charset=UTF-8, } Data: []"

OCIS:

23-03-11 09:00:25:888 [ info sync.httplogger ]:	"fdeecd15-cb6d-4227-9af5-d899594ca561: Response: GET 200 (19140ms) https://ocis-redacted/dav/spaces/b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad/model/KB3D_LunarBase_FBXOBJ/KB3D_LunarBase-Native.obj Header: { Accept-Ranges: bytes, Access-Control-Allow-Origin: *, Content-Disposition: attachment; filename*=UTF-8''KB3D_LunarBase-Native.obj; filename=\"KB3D_LunarBase-Native.obj\", Content-Length: 611472498, Content-Security-Policy: default-src 'none';, Content-Type: application/x-tgif, Date: Sat, 11 Mar 2023 03:30:08 GMT, Etag: \"91944e8b78f8ba4ec7ee2e7788df303b\", Last-Modified: Wed, 08 Mar 2023 05:51:06 +0000, Oc-Checksum: SHA1:ed61a77f1700cc1cef7c8ad0cf17dd089c4c8b81, Oc-Etag: \"91944e8b78f8ba4ec7ee2e7788df303b\", Oc-Fileid: b657a09b-c5ca-4a36-af62-fdb5a2eb06f6$aa2b1147-6c7f-4086-9a25-d1b3d16053ad!f88a12ec-62e4-4ee5-be08-7fa5003ddcd7, Vary: Origin, X-Content-Type-Options: nosniff, X-Download-Options: noopen, X-Frame-Options: SAMEORIGIN, X-Permitted-Cross-Domain-Policies: none, X-Robots-Tag: none, X-Xss-Protection: 1; mode=block, } Data: []"

I’ve uploaded logs for Desktop syncs with both OC10 and OCIS:
Screenshot 2023-03-14 at 13.03.25

michaelstingl · March 14, 2023, 8:54am

Quite some entries from oC10 where the Content-Length header is missing:

% cat oc10_4_07-redacted.log | grep "Response\:\ GET\ 200" | grep "\/dav\/" | grep -v "Content-Length" | wc -l
      64

Maybe check the X-Request-ID in the desktop client log and grep for it in the owncloud.log.

michaelstingl · March 14, 2023, 4:58pm

I made this graph in Excel:

Excel file is here:
ownCloud Enterprise Edition
(really not an expert)

walt · March 15, 2023, 3:22am

The server logs don’t appear to contain anything for Content-Length unless there was an error. What level of logging would we need to enable to capture these?

walt · March 15, 2023, 3:23am

Very cool - makes it quite clear that OCIS is running slower than OC10. Any thoughts on how to get it running faster?

wkloucek · March 15, 2023, 7:37am

@walt Are you using a reverse Proxy for oCIS?

walt · March 15, 2023, 7:51am

No - everything’s a direct connection

butonic · March 20, 2023, 11:03am

Thats interesting: we tried to replicate this with ocis and owncloud 10 in docker compose, but we see completely opposite results.

Using a bucket in AWS eu central uploading 1k small files took <1min with ocis web but >5min with oc10 (we did not bother measuring exactly but we already perceived ocis to be way faster).
Downloading the 1k files with the desktop client took ~17sec on ocis and ~90sec on oc10.

@walt can you elaborate on you config? We were using the oc10 dockerfile with default settings, so it uses redis and mariadb with reasonable defaults. Any config options you tuned?

Also, we were using withe ocis master, but in comparison to ocis 2.0 storing metadata would only be slow if you mounted the user storage on an nfs, paying the price of additional latency.

Last but not least upload performance can significantly be increased by setting STORAGE_USERS_OCIS_ASYNC_UPLOADS=true … but since you reported download slowness that should not affect your results.

Finally, in ocis v2.0 we were writing the uploaded file blob twice on uploads, which obviously is slow. But again, that does not affect downloads.

So @walt, you made me curious. Something is bogging down the downloads. Can you paste redacted snippets of your config?

butonic · March 20, 2023, 11:13am

Ohhh, I have another possible reason: are you using OpenID Connect or basic auth? With the build in oCIS IDP or an external service like keycloak?

oCIS dos not support basic auth by design. We rely on OpenID Connect. For development purposes it is possible to enable basic auth, but since oCIS does not use sessions, the desktop client has to always send the Basic auth header. Which is expensive to verify, also by design, because our internal LDAP implementation uses argon2 hashes to protect passwords against brute force attacks.

The password check actually introduces a noticeable lag in requests … 200ms seems possible, but too much. It depends on the available hardware and CPU performance.

walt · March 20, 2023, 11:40am

Happy to post the configs… they’re both pretty straight forward. No dockers or containers or anything - these both are running natively on Ubuntu itself. Both OC10 and OCIS were default installs.

We’re using OCIS 2.0 storing metadata on the local disk which is a faster SSD than the OC10 server.

You mentioned writing the blobs twice on OCIS 2.0… should we test a different version to avoid that particular issue? Or is STORAGE_USERS_OCIS_ASYNC_UPLOADS=true enough?

OC10 config:

<?php
$CONFIG = array (
  'passwordsalt' => 'redacted',
  'secret' => 'redacted',
  'trusted_domains' => 
  array (
    0 => 'localhost',
    1 => '10.0.0.1',
    2 => 'oc10-redacted',
  ),
  'datadirectory' => '/var/www/owncloud/data',
  'overwrite.cli.url' => 'http://localhost',
  'dbtype' => 'mysql',
  'version' => '10.11.0.6',
  'dbname' => 'owncloud',
  'dbhost' => 'localhost',
  'dbtableprefix' => 'oc_',
  'mysql.utf8mb4' => true,
  'dbuser' => 'owncloud',
  'dbpassword' => 'redacted',
  'allow_user_to_change_mail_address' => '',
  'logtimezone' => 'UTC',
  'apps_paths' => 
  array (
    0 => 
    array (
      'path' => '/var/www/owncloud/apps',
      'url' => '/apps',
      'writable' => false,
    ),
    1 => 
    array (
      'path' => '/var/www/owncloud/apps-external',
      'url' => '/apps-external',
      'writable' => true,
    ),
  ),
  'installed' => true,
  'instanceid' => 'ocu1r2rh9ed0',
  'memcache.local' => '\\OC\\Memcache\\APCu',
  'memcache.locking' => '\\OC\\Memcache\\Redis',
  'redis' => 
  array (
    'host' => '127.0.0.1',
    'port' => '6379',
  ),
  'objectstore' => 
  array (
    'class' => 'OCA\Files_Primary_S3\S3Storage',
    'arguments' => 
    array (
      // replace with your bucket
      'bucket' => 'redacted-oc10',
      // uncomment to enable server side encryption
      //'serversideencryption' => 'AES256',
      'options' => 
      array (
        // version and region are required
        'version' => '2006-03-01',
        // change to your region
        'region'  => 'ap-south-1',
        'credentials' => 
        array (
          // replace key and secret with your credentials
          'key' => 'redacted',
          'secret' => 'redacted',
        ),
	'use_path_style_endpoint' => true,
        'endpoint' => 'https://bucket.vpce-redacted-arp.s3.ap-south-1.vpce.amazonaws.com',
      ),
    ),
  ),
  'http.cookie.samesite' => 'None',
  'openid-connect' => 
  array (
    'auto-provision' => 
    array (
      'enabled' => true,
      'email-claim' => 'email',
      'display-name-claim' => 'full_name',
      'username-claim' => 'preferred_username',
      'auto-provision' => 
      array (
        'update' => 
        array (
          'enabled' => false,
        ),
      ),
    ),
    'provider-url' => 'https://sso-redacted/realms/master',
    'client-id' => 'owncloud-web',
    'client-secret' => 'redacted',
    'loginButtonName' => 'OpenId Connect',
    'autoRedirectOnLoginPage' => true,
    'mode' => 'email',
  ),
  'token_auth_enforced' => true,
);

OCIS config:

OCIS_CONFIG_DIR=/etc/ocis/config
OCIS_INSECURE=false
PROXY_HTTP_ADDR=0.0.0.0:443
OCIS_URL=https://ocis-redacted:443
PROXY_TRANSPORT_TLS_KEY=/var/www/certs/redacted.key
PROXY_TRANSPORT_TLS_CERT=/var/www/certs/redacted.crt
OCIS_LOG_LEVEL=error

# Activate s3ng storage driver
STORAGE_USERS_DRIVER=s3ng
STORAGE_USERS_S3NG_ROOT=/home/ocis/.ocis/storage/users
STORAGE_SYSTEM_DRIVER=ocis

# s3ng config
STORAGE_USERS_S3NG_ENDPOINT=https://bucket.vpce-redacted-arp.s3.ap-south-1.vpce.amazonaws.com
STORAGE_USERS_S3NG_REGION=ap-south-1
STORAGE_USERS_S3NG_ACCESS_KEY=redacted
STORAGE_USERS_S3NG_SECRET_KEY=redacted
STORAGE_USERS_S3NG_BUCKET=redacted-ocis

# OpenID
OCIS_OIDC_ISSUER=https://sso-redacted/realms/master
WEB_OIDC_CLIENT_ID=ocis-web
PROXY_OIDC_REWRITE_WELLKNOWN=true
PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=none
PROXY_AUTOPROVISION_ACCOUNTS=true
PROXY_USER_OIDC_CLAIM=email

# Archiver limits for downloading zip/tar files
FRONTEND_ARCHIVER_MAX_NUM_FILES=10000
FRONTEND_ARCHIVER_MAX_SIZE=107374182400

walt · March 20, 2023, 11:45am

We’re using KeyCloak for everything on the auth side. Both the OC10 and OCIS deployments are set up for OpenID Connect.