@butonic@michaelstingl The slowness is entirely due to OpenID Connect with Keycloak. We ran a test commenting out OCIS_OIDC_ISSUER and downloads of our test with 944 files went from 4:31 to 1:46.
Our Keycloak server is on a different network that’s definitely not the fastest to access from the subnet on which OC10 and OCIS are running. Normally that’s not an issue for us as services authenticate once and if that takes an extra 500ms no one cares. If ownCloud Desktop syncs are running auth to the Keycloak server for every file, it’s absolutely going to crawl.
We ran a test with a temporary Keycloak server on the same subnet as OCIS and syncs of those 944 files ran around 2:45. Definitely better than 4:31, but nowhere near as fast as the 1:46 we get when using OCIS’ built-in IdP.
It seems one way or another we need to greatly reduce OCIS’ reliance on Keycloak for authentication. Is there a way to have OCIS auth against Keycloak only at login and not on every file request? Or can we have OCIS use Keycloak purely as an external IdP to provision accounts on the internal IdP?
OICD with the desktop client relies on access_token and refresh_token. Client connects with the access_token to oCIS, and oCIS has the access_token cached for a while (5 Minutes in our deployment examples). After access_token expiration, client sends the refresh_token in a POST request to the Keycloak token endpoint, to get a new access_token.
Hmmm… definitely strange if that’s how things are set up.
FWIW, our Keycloak is configured with an access_token lifespan of 30 days. Refresh tokens should also have a lifespan of 30 days since our SSO and client session idles and session maxes are all 30 days.
@walt in oCIS v2.0 PROXY_OIDC_USERINFO_CACHE_TTL defaults to 10 … which is documented as Max TTL in seconds for the OIDC user info cache. This default is used as a fallback when the token does not have an exp claim or when PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD is set to none.
Unfortunately, there seems to be a bug in how the TTL of 10 is interpreted, as I cannot find a place that turns it into seconds. AFAICT it is interpretet in nanoeconds. Could you try PROXY_OIDC_USERINFO_CACHE_TTL=10000000000 to get the 10s or maybe even PROXY_OIDC_USERINFO_CACHE_TTL=1800000000000 to get 30min?
Why don’t you use PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=jwt?
In any case I am literally working on the proxy cache right now and the env var will change to be parsed as a duration, which will allow you to use an h, m or h suffix, so the numbers will become better understandable. … at the cost of some variable renaming …
produces similar results as without it set: average 2:45 to sync those 944 files via the Desktop app.
It wasn’t used in the example setup we followed from Helge Klein and everything seemed to work fine without it. If you believe this would be of benefit in v2.0.0 we’re happy to give it a try.
keycloak uses a JWT access token, so you can use the default PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=jwt. The proxy will then cache the token until the contained expiry date time is reached. It should no longer make any subsequent requests to keycloak for that access token.
That being said, PROXY_OIDC_USERINFO_CACHE_TTL=1800000000000 should have had the same effect for you. Now … leaving OCIS_OIDC_ISSUER unconfigured will tell the clients to talk to the oCIS built in IDP (you may want to disable it with OCIS_EXCLUDE_RUN_SERVICES=idp since you are using keycloak anyway). The proxy also only needs to talk to another service on the same host, so that reduces latency in comparison to keycloak on another machine (which not only adds the network latency but also the actual access token verification).
Still, that userinfo lookup request from the proxy should only be made in the first request. For subsequent requests the cache should be hit. No more latency …
Could you set PROXY_LOG_LEVEL=debug? It should log the cache hits, misses and inserts.
That did the trick! With both PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=jwt and OCIS_EXCLUDE_RUN_SERVICES=idp we’re now matching the speed of transfers with the internal IdP - 1:46 for 944 files down, 1:34 for 944 files up - via the Desktop app.
Absolutely will. This was a tough one to solve, even if the ultimate answer is pretty simple. Couldn’t have done it without you and @michaelstingl both poking at this from every possible angle.
This was an interesting read. Nicely investigated.
I have a related question; I am wondering if you have any insights on how to successfully use JWTs to verify tokens when using Authelia as IAM?
I am running OCIS 3.0.0 with Authelia, and when PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=jwt is set, then authentication fails with the following message:
2023-07-22T13:49:58Z ERR failed to authenticate the request error=\"failed to verify access token: token contains an invalid number of segments\" authenticator=oidc path=/ocs/v1.php/cloud/user service=proxy
Authentication works fine when PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=none
AFAIK for validating the access tokens issued by authelia, we’d need to query the introspection endpoind. Unfortunately ocis does not support that currently. So PROXY_OIDC_ACCESS_TOKEN_VERIFY_METHOD=none is the correct setting for authelia currently.
We can be sure that we have a valid access_token as we use it to query the IDPs userinfo endpoint. Doing that with an invalid token would fail.
Authelia actually does technically support the JWT Profile for Access Tokens in the latest beta.
However I feel like there is not a use case for validating the Access Token in this way as it’s a stateless validation (i.e. if it were revoked the token would still be considered valid using this method). The introspection endpoint provides accurate stateful information about the token being valid, whereas the JWT method does not. The JWT validation in addition if applied to the ID Token would yield similar results. I am curious about what is intended to be validated in regards to the access token?
It should also be noted that the introspection flow is very simple and light.
It depends a bit on the configuration. When using Keycloak IIRC we just extracting the exp claim and use that as a ttl for our internal userinfo cache. As we support backchannel logout with keycloak, we’ll get informed via the backchannel as soon a session is revoked. In other configurations we can also extract stuff like role-assignments from the access token.
True. We just don’t have an implementation for that yet. Also it obviously needs another roundtrip to the IDP.