I am experiencing an issue where large files (Ex:20 GB text file) fail to upload to my OCIS instance, which is configured with an S3NG backend. Below is the setup and the steps I have tried:
Setup Details:
- OCIS Version: 5.0.0
- Backend Storage: S3-compatible object storage (Ceph S3 = bucket size 100GB)
- Deployment: Docker (command included below)
- Relevant Configuration:
-
- STORAGE_USERS_S3NG_MULTIPART_MIN_PART_SIZE=10485760 (10 MB)
-
- STORAGE_USERS_S3NG_MULTIPART_CONCURRENCY=10
-
- STORAGE_USERS_S3NG_METADATA_BACKEND=messagepack
Problem:
-
Small files upload successfully via the OCIS Web UI.
-
When uploading larger files, the operation fails. The OCIS logs show errors such as:
"missing parent id" / "internal error: Missing parent ID on node" -
HTTP status code: 507
Directly uploading large files to the S3 bucket via tools like rclone or s3cmd works, but these files are not visible in the OCIS UI.
Steps Taken to Troubleshoot:
- Increased multipart upload part size (STORAGE_USERS_S3NG_MULTIPART_MIN_PART_SIZE) to 10 MB.
- Enabled debug logs for both OCIS_LOG_LEVEL and REVA_LOG_LEVEL.
- Verified S3 bucket has sufficient storage and supports multipart uploads.
Expected Behavior:
Large file uploads should succeed through the OCIS interface, with proper metadata updates to reflect the files in the UI.
Questions:
- Is there a recommended approach to configure OCIS for large file uploads with an S3NG backend?
- Are there known limitations or additional parameters I should set for multipart uploads?
Any guidance, suggestions, or troubleshooting steps would be greatly appreciated!
Thank you