Is it possible to filter files and hide config.json?

[yfzhao2021]

Hi,

I’m trying to pass some security check for my infinite scale server to apply for a domain name from my university. In this security check, they find two problems that I have to fix before a domain name can be issued:

1/ Infinite scale does not filter files. If a piece of xss code is inserted in a pdf file, the web pdf viewer can still run the code, which would be unsafe.
2/ The config.json is accessible through a url link like $server$/config.json . (config.json in the official demo is also accessible in the same way.) I tried to block this url link in a nginx proxy, but the web UI stopped working because it could not find config.json.

So I’m wondering if it is possible to implement a file filter in the server and hide the access to config.json? I’m very unfamiliar with web development, so I would be appreciated if anyone can give me some suggestions. Thank you !

Server configuration

Operating system: Debian 12

Web server: OwnCloud web 7.1.0

ownCloud version: Infinite Scale 4.0.2

Updated from an older ownCloud or fresh install: Fresh install

Where did you install ownCloud from: The binary file from OwnCloud official website

List of activated apps: from ocis list :

±-------------------+
| SERVICE |
±-------------------+
| app-provider |
| app-registry |
| auth-basic |
| auth-machine |
| eventhistory |
| frontend |
| gateway |
| graph |
| groups |
| idm |
| idp |
| invitations |
| nats |
| notifications |
| ocdav |
| ocs |
| postprocessing |
| proxy |
| search |
| settings |
| sharing |
| storage-publiclink |
| storage-shares |
| storage-system |
| storage-users |
| store |
| thumbnails |
| userlog |
| users |
| web |
| webdav |
| webfinger |
±-------------------+

Are you using external storage, if yes which one: no

Are you using encryption: no

Are you using an external user-backend, if yes which one: no

[kobergj]

Hi @yfzhao2021,

1. What exactly do you mean by file filter?
   If you want to find malware in files, it is probably best to use the antivirus service (Antivirus Service Configuration). It will scan uploads and can be configured to delete files if they are infected.
   If you want to generally block uploading pdf files you can use the policies service (Policies Service Configuration) to blacklist (or whitelist) specific file types.

2. This is problematic. The config.json file contains the configuration for the web UI. It cannot work without it. But it does not contain any secrets or other sensitive data. Just theming and other configuration values the webui needs. All sensitive data can only be retrieved when logged in.

[yfzhao2021]

Hi,

Thank you for your response. Here I try to make my questions more clear.

1/ If I did not misunderstand, what the network administration in the university told me is: I should scan the content of the files to ensure they do not contain harmful codes and certain texts. They insisted that it is not antivirus. They showed me an example pdf as I attached. This pdf contains some xss codes. If I upload it in my ocis server and open it in the web pdf viewer, I will get a popup because a code is executed. I did not observe it in an old 8.x OwnCloud server.

2/ I understand that this configuration file is necessary for the web UI, and it does not contains any important information. But the network administration in the university considered it as a (low-risk) security threat. I have told them that this file is necessary and I may try to hide it by renaming, but they did not accept my explanation. :-/

I understand it may be impossible to have this domain name issued…

Best,
Yongfeng

(Attachment xss.pdf is missing)

[kobergj]

It depends on which antivirus scanner you use. They should be able to also find malware like xss in pdf. If you want to use custom file scanning you could still do this through policies service. But I would recommend relying on existing solutions.

Regarding the config.json: As far as I know there is currently no possibility for the webui to work without it.

[yfzhao2021]

Thank you a lot for the information ! I will check it.

[yfzhao2021]

UPDATE: I have set up ClamAV for antivirus, and it works well. But ClamAV cannot detect xss (Cross-Site Scripting) malware.

I did some search, and it seems ClamAV and other antivirus scanner are usually not considered responsible for protecting xss attack. It seems people usually mitigate the attack in front-end, and maybe it is the webUI or the pdf-viewer that is responsible. I’m trying to add a policy to do the job, but no luck so far. I don’t know which is the correct API for getting the uploaded data.

[fschade]

you can use the policies for that regard (rego), we have a ocis.resource.downloadstep, once you have the date you could scan if a certain string is part of the content.

Policies Service Configuration

[yfzhao2021]

Thank you. I have read that in the official example. But unfortunately it does not work even for the example provided in the github:

granted = false if {
    not utils.is_extension_allowed(input.resource.name)
}

granted = false if {
    bytes := ocis.resource.download(input.resource.url)
    mimetype := ocis.mimetype.detect(bytes)

    not utils.is_mimetype_allowed(mimetype)
}

I have set up the variables as in the document and included the utils.rego file. But all files are blocked, and in the log file I saw:

{"level":"error","service":"policies","request-id":"","error":"eval_cancel_error: caller cancelled query execution","time":"2023-11-06T17:42:53.888743946+08:00","message":"unable evaluate policy"}

The policy works as expected by deleting this one:

granted = false if {
    bytes := ocis.resource.download(input.resource.url)
    mimetype := ocis.mimetype.detect(bytes)

    not utils.is_mimetype_allowed(mimetype)
}

So I was suspecting if this API has been changed.

[kobergj]

No, the API hasn’t changed. There seems to be something else wrong. @fschade Have you seen something like this before?

[fschade]

hey yfzhao2021,
thanks for providing your feedback, i’ve tested the policies my self on a fresh environment moments ago, it works for me… i hope i can help anyway.

i have the following env set

OCIS_ADD_RUN_SERVICES=policies
POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted
OCIS_ASYNC_UPLOADS=true
POSTPROCESSING_STEPS=policies

and have these rego policies in use:

• https://github.com/owncloud/ocis/blob/master/deployments/examples/service_policies/policies/utils.rego
• https://github.com/owncloud/ocis/blob/master/deployments/examples/service_policies/policies/postprocessing.rego

its important to restart the server after environment changes have have been done.

you can find the allowed extensions in the rego file, uploading a allowed filetype worked for me, upload a forbidden one like some-forbidden-file.extension was declined as expected.

i also had a look in my debugger to be sure that the necessary code sections got called, same, worked.

can you please tell us more about your environment (ocis version, server version, docker or binary, …).

would it help if i create a example docker-compose.yml for you?

[yfzhao2021]

Thank you for the help. I’m using binary with systemd managing the service. The system is Debian 12, and the ocis is version 4.0.2. I’m now using the same utils.rego and postprocessing.rego as you posted.

My full ocis.env is

OCIS_URL=https://<domain_name>
PROXY_HTTP_ADDR=0.0.0.0:9200
PROXY_TRANSPORT_TLS_KEY=/etc/ocis/<domain_name>.key
PROXY_TRANSPORT_TLS_CERT=/etc/ocis/<domain_name>_bundle.crt
PROXY_HTTPS_CACERT=/etc/ocis/<domain_name>_root.crt

OCIS_GRPC_CLIENT_TLS_CACERT=/etc/ocis/<domain_name>_root.crt
OCIS_GRPC_TLS_KEY=/etc/ocis/<domain_name>.key
OCIS_GRPC_TLS_CERTIFICATE=/etc/ocis/<domain_name>_bundle.crt

OCIS_HTTP_TLS_CERTIFICATE=/etc/ocis/<domain_name>_bundle.crt
OCIS_HTTP_TLS_KEY=/etc/ocis/<domain_name>.key

NATS_TLS_CERTIFICATE=/etc/ocis/<domain_name>_bundle.crt
NATS_TLS_KEY=/etc/ocis/<domain_name>.key

OCIS_INSECURE=false

PROXY_POLICIES_QUERY=data.proxy.granted

OCIS_LOG_LEVEL=warn
OCIS_EVENTS_ENABLE_TLS=false

OCIS_CONFIG_DIR=/etc/ocis
OCIS_BASE_DATA_PATH=<ownCloudData_location>

OCIS_ASYNC_UPLOADS=true
STORAGE_USERS_OCIS_ASYNC_UPLOADS=true
OCIS_EXCLUDE_RUN_SERVICES=
OCIS_ADD_RUN_SERVICES="antivirus,policies"

And I have a policies.yaml:

grpc:
  addr: 127.0.0.1:9125
  tls: null
debug:
  addr: 127.0.0.1:9129
  token: ""
  pprof: false
  zpages: false
token_manager:
  jwt_secret: <my_jwt_secret>
events:
  endpoint: 127.0.0.1:9233
  cluster: ocis-cluster
  tls_insecure: false
  tls_root_ca_certificate: ""
  enable_tls: false
reva:
  address: com.owncloud.api.gateway
  tls:
    mode: ""
    cacert: ""
grpc_client_tls: null
log:
  level: ""
  pretty: false
  color: false
  file: ""
engine:
  timeout: 10
  policies: 
    - /etc/ocis/postprocessing.rego
    - /etc/ocis/proxy.rego
    - /etc/ocis/utils.rego
  mimes: ""
postprocessing:
  query: data.postprocessing.granted
tracing:
  enabled: false
  type: ""
  endpoint: ""
  collector: ""

Here is postprocessing.yaml:

tracing:
  enabled: false
  type: ""
  endpoint: ""
  collector: ""
log:
  level: ""
  pretty: false
  color: false
  file: ""
debug:
  addr: 127.0.0.1:9255
  token: ""
  pprof: false
  zpages: false
store:
  store: memory
  nodes: []
  database: postprocessing
  table: postprocessing
  ttl: 0
  size: 0
postprocessing:
  events:
    endpoint: 127.0.0.1:9233
    cluster: ocis-cluster
    tls_insecure: false
    tls_root_ca_certificate: ""
    enable_tls: true
  steps: 
    - virusscan
    - policies
  delayprocessing: 0

proxy.yaml:

tracing:
  enabled: false
  type: ""
  endpoint: ""
  collector: ""
log:
  level: ""
  pretty: false
  color: false
  file: ""
debug:
  addr: 127.0.0.1:9205
  token: ""
  pprof: false
  zpages: false
http:
  addr: 0.0.0.0:9200
  root: /
  tls_cert: ~/.ocis/proxy/server.crt
  tls_key: ~/.ocis/proxy/server.key
  tls: true
reva:
  address: com.owncloud.api.gateway
  tls:
    mode: ""
    cacert: /etc/ocis/<domain_name>_root.crt
grpc_client_tls: null
role_quotas: {}
policies:
- name: ocis
  routes:
  - endpoint: /
    service: com.owncloud.web.web
    unprotected: true
  - endpoint: /.well-known/webfinger
    service: com.owncloud.web.webfinger
    unprotected: true
  - endpoint: /.well-known/openid-configuration
    service: com.owncloud.web.idp
    unprotected: true
  - endpoint: /branding/logo
    service: com.owncloud.web.web
  - endpoint: /konnect/
    service: com.owncloud.web.idp
    unprotected: true
  - endpoint: /signin/
    service: com.owncloud.web.idp
    unprotected: true
  - endpoint: /archiver
    service: com.owncloud.web.frontend
  - endpoint: /ocs/v2.php/apps/notifications/api/v1/notifications/sse
    service: com.owncloud.sse.sse
  - endpoint: /ocs/v2.php/apps/notifications/api/v1/notifications
    service: com.owncloud.userlog.userlog
  - type: regex
    endpoint: /ocs/v[12].php/cloud/user/signing-key
    service: com.owncloud.web.ocs
  - type: regex
    endpoint: /ocs/v[12].php/config
    service: com.owncloud.web.frontend
    unprotected: true
  - endpoint: /sciencemesh/
    service: com.owncloud.web.ocm
  - endpoint: /ocm/
    service: com.owncloud.web.ocm
  - endpoint: /ocs/
    service: com.owncloud.web.frontend
  - type: query
    endpoint: /remote.php/?preview=1
    service: com.owncloud.web.webdav
  - type: regex
    method: REPORT
    endpoint: (/remote.php)?/(web)?dav
    service: com.owncloud.web.webdav
  - type: query
    endpoint: /dav/?preview=1
    service: com.owncloud.web.webdav
  - type: query
    endpoint: /webdav/?preview=1
    service: com.owncloud.web.webdav
  - endpoint: /remote.php/
    service: com.owncloud.web.ocdav
  - endpoint: /dav/
    service: com.owncloud.web.ocdav
  - endpoint: /webdav/
    service: com.owncloud.web.ocdav
  - endpoint: /status
    service: com.owncloud.web.ocdav
    unprotected: true
  - endpoint: /status.php
    service: com.owncloud.web.ocdav
    unprotected: true
  - endpoint: /index.php/
    service: com.owncloud.web.ocdav
  - endpoint: /apps/
    service: com.owncloud.web.ocdav
  - endpoint: /data
    service: com.owncloud.web.frontend
    unprotected: true
  - endpoint: /app/list
    service: com.owncloud.web.frontend
    unprotected: true
  - endpoint: /app/
    service: com.owncloud.web.frontend
  - endpoint: /graph/v1.0/invitations
    service: com.owncloud.graph.invitations
  - endpoint: /graph/
    service: com.owncloud.graph.graph
  - endpoint: /api/v0/settings
    service: com.owncloud.web.settings
oidc:
  issuer: https://localhost:9200
  insecure: false
  access_token_verify_method: jwt
  skip_user_info: false
  user_info_cache:
    store: memory
    addresses: []
    database: ocis
    table: userinfo
    ttl: 10
    size: 0
  jwks:
    refresh_interval: 60
    refresh_timeout: 10
    refresh_limit: 60
    refresh_unknown_kid: true
  rewrite_well_known: false
service_account:
  service_account_id: ""
  service_account_secret: ""
role_assignment:
  driver: default
  oidc_role_mapper:
    role_claim: roles
    role_mapping:
    - role_name: admin
      claim_value: ocisAdmin
    - role_name: spaceadmin
      claim_value: ocisSpaceAdmin
    - role_name: user
      claim_value: ocisUser
    - role_name: guest
      claim_value: ocisGuest
policy_selector:
  static:
    policy: ocis
  claims: null
  regex: null
pre_signed_url:
  allowed_http_methods:
  - GET
  enabled: true
account_backend: cs3
user_oidc_claim: preferred_username
user_cs3_claim: username
machine_auth_api_key: <some_key>
auto_provision_accounts: false
enable_basic_auth: false
insecure_backends: false
backend_https_cacert: /etc/ocis/<domain_name>_root.crt
auth_middleware:
  credentials_by_user_agent: {}

proxy.rego is a constant true:

package proxy

default granted := true

In finding the potential config problems, I have yaml config files for many services, but they are not different from the examples in doc.owncloud.com. If you think some of them are suspicious, I can provide as well.

The reverse proxy is through nginx by the network administration, which is mostly out of my control. According to their message, the setting is like

server { 
	listen 80 ; 
	listen [::]:80 ; 
	server_name <domain_name>; 
	access_log /var/log/nginx/<domain_name>.access.log main; 
	location / { 
		if ( $http_user_agent ~ "(Mozilla/5.0)" ) { 
			return 301 https://$server_name$request_uri;
 		} 
		proxy_pass https://<IP_address>:9200; 
	}

	location /.well-known/ { 
		root /etc/nginx/ssl/web/; 
	} 
} 

server { 
	listen 443 ssl http2; 
	listen [::]:443 ssl http2; 
	server_name <domain_name>; 
	ssl_certificate /etc/nginx/ssl/<domain_name>.crt; 
	ssl_certificate_key /etc/nginx/ssl/<domain_name>.key; 
	add_header Strict-Transport-Security $hsts_header; 
	add_header Content-Security-Policy upgrade-insecure-requests; 
	access_log /var/log/nginx/<domain_name>.access.log main; 
	location / { 
		proxy_buffers 4 256k; 
		proxy_buffer_size 128k; 
		proxy_busy_buffers_size 256k;

		client_max_body_size 0; 
		proxy_pass https://<IP_address>:9200; 
		proxy_set_header Host $host; 
	} 

	location /favicon.ico { 
		root /etc/nginx/conf.d; 
	} 
}

[kobergj]

As far as I can see you are missing some envvars Could you try setting

POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted
POSTPROCESSING_STEPS=virusscan,policies

POLICIES_POSTPROCESSING_QUERY sets the query for the policies service.
POSTPROCESSING_STEPS sets the steps that should be evaluated in postprocessing.

[yfzhao2021]

The result is the same: File .pdf was deleted because it violates the policies.

I thought these variables are set by the .yaml files if not doing so in envvar, is it not true ?

[kobergj]

Yes, sorry. Missed that you set it through the yaml file.

[yfzhao2021]

I spent some time to setup a minimal docker installation to debug, but the problem persists. Here is the docker command:

docker run --rm -it --mount type=bind,source=/home/yfzhao/.ocis,target=/etc/ocis owncloud/ocis init

> Do you want to configure Infinite Scale with certificate checking disabled? yes

docker run --name ocis_runtime --rm -it -p 9200:9200 \
--mount type=bind,source=/home/yfzhao/.ocis,target=/etc/ocis \
--mount type=bind,source=/home/yfzhao/Downloads/test_ocis/data,target=/var/lib/ocis \
-e OCIS_INSECURE=true \
-e PROXY_HTTP_ADDR=0.0.0.0:9200\
-e OCIS_URL=https://localhost:9200 \
-e POLICIES_POSTPROCESSING_QUERY=data.postprocessing.granted \
-e POSTPROCESSING_STEPS=policies \
-e OCIS_LOG_LEVEL=warn \ 
-e OCIS_EVENTS_ENABLE_TLS=false \
-e OCIS_ASYNC_UPLOAD=true \ 
-e OCIS_ADD_RUN_SERVICES=policies owncloud/ocis

Here is the full log:

{"level":"warn","service":"storage-system","pkg":"rhttp","time":"2023-12-13T08:17:07Z","message":"missing or incomplete nats configuration. Events will not be published."}
{"level":"warn","service":"proxy","time":"2023-12-13T08:17:09Z","message":"No tls certificate provided, using a generated one"}
{"level":"warn","service":"idp","kid":"private-key","path":"/var/lib/ocis/idp/private-key.pem","time":"2023-12-13T08:17:09Z","message":"skipped as signer with same kid already loaded"}
{"level":"warn","service":"frontend","pkg":"rhttp","traceid":"00000000000000000000000000000000","time":"2023-12-13T08:17:16Z","message":"core access token not set"}
{"level":"warn","service":"frontend","pkg":"rhttp","traceid":"00000000000000000000000000000000","time":"2023-12-13T08:17:16Z","message":"core access token not set"}
{"level":"warn","service":"thumbnails","method":"Thumbnails.GetThumbnail","duration":9.139832,"error":"{\"id\":\"com.owncloud.api.thumbnails\",\"code\":425,\"detail\":\"File Processing\",\"status\":\"Too Early\"}","time":"2023-12-13T08:17:16Z","message":"Failed to execute"}
{"level":"warn","service":"thumbnails","method":"Thumbnails.GetThumbnail","duration":1.461754,"error":"{\"id\":\"com.owncloud.api.thumbnails\",\"code\":425,\"detail\":\"File Processing\",\"status\":\"Too Early\"}","time":"2023-12-13T08:17:17Z","message":"Failed to execute"}
{"level":"warn","service":"frontend","pkg":"rhttp","traceid":"00000000000000000000000000000000","time":"2023-12-13T08:17:22Z","message":"core access token not set"}
{"level":"warn","service":"storage-users","pkg":"rhttp","traceid":"00000000000000000000000000000000","time":"2023-12-13T08:17:22Z","message":"core access token not set"}
{"level":"error","service":"policies","request-id":"","error":"eval_cancel_error: caller cancelled query execution","time":"2023-12-13T08:17:22Z","message":"unable evaluate policy"}
{"level":"error","service":"clientlog","error":"unexpected status code while getting resource: CODE_NOT_FOUND","event":{"Type":"events.UploadReady","ID":"aa2f02e2-a65d-474c-b1b7-cfae8e156ee3","TraceParent":"","Event":{"UploadID":"6a5d3ccb-ddca-4e2b-b66a-78b20bd22719","Filename":"00000.png","SpaceOwner":{"idp":"https://localhost:9200","opaque_id":"a586808a-72ec-4040-8559-901c965f73cf","type":1},"ExecutingUser":{"id":{"idp":"https://localhost:9200","opaque_id":"a586808a-72ec-4040-8559-901c965f73cf","type":1},"username":"admin"},"FileRef":{"resource_id":{"storage_id":"ee1588e8-8013-4b5a-b336-1397cc12da19","opaque_id":"a586808a-72ec-4040-8559-901c965f73cf","space_id":"a586808a-72ec-4040-8559-901c965f73cf"},"path":"./00000.png"},"Failed":true,"Timestamp":{"seconds":1702455442,"nanos":740407619}}},"time":"2023-12-13T08:17:22Z","message":"error getting resource"}

Docker version is 24.0.5 installed in an Ubuntu 22.04 system. Image version is latest. Upload works fine without policies.

[fschade]

sorry for the late response @yfzhao2021, i prepared a repository for you to showcase how policies work, please have a look to the rego files.

By default everything is allowed, but you can:

• disallow pdfs at the proxy stage (proxy.rego)
• disallow pdfs at the postprocessing stage (postprocessing.rego)

to do so, please open the individual file and uncomment the rule. I highly recommend reading the OPA docs.

Policy Language

i hope this works for you, its not easy to recommend solutions without knowing or have access to the environment.

the example repo can be found here:

GitHub - fschade/ocis-sample-policies

please check the docker compose file, my OCIS_URL is set to https://host.docker.internal:9200 in the example, you have to tweak your hosts file to make it work.

[yfzhao2021]

Thank you very much for this example. This works for me. By comparing the configs between this example and my configs, I suspect that the problem was in the tls_insecure settings. I got the TLS certificates for my domain, but I don’t quite understand how they are used. Setting all tls_insecure to true and deleting unnecessary configs seem to work.

Now I can screen some suspicious JavaScript keywords. But I cannot fully prevent xss attack because the code can be obfuscated. How to identify these codes is beyond my knowledge. If the online PDF viewer can be configured not to run any JavaScript codes, I think it would be very helpful.

Then the only mystery left for me is that login with my desktop client shows 403 code in the HTTP Proxy, but the web system looks fine… I described this issue in another thread: Desktop Client 403 Forbidden - Web and mobile are working - #9 by raphaelben55