"panic: unaligned 64-bit atomic operation" crash on oCIS 6.2.0

Infinite Scale
1

We’re seeing constant crashes on oCIS 6.2.0 on Ubuntu 24 with the following message:

panic: unaligned 64-bit atomic operation

This seems to always happen when we’re uploading files or creating new folders. Not sure how to debug this further as there’s no further information available on what’s causing the crash. Here are the last few debug lines immediately before the panic message:

unix-directory","mtime":{"seconds":1724654558,"nanos":343397049},"path":".","permission_set":{"add_grant":true,"create_container":true,"delete":true,"get_path":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_recycle":true,"remove_grant":true,"purge_recycle":true,"restore_recycle_item":true,"stat":true},"size":626864,"owner":{"idp":"https://auth.example.org/realms/example","opaque_id":"0d4da3c4-fef9-4d16-b02b-35b954375043","type":1},"arbitrary_metadata":{"metadata":{"SAR:/lastmodified":"Mon, 26 Aug 2024 06:22:20 UTC","SAR:/lastmodified_server":"Mon, 26 Aug 2024 06:31:31 UTC","http://owncloud.org/ns/favorite":""}},"parent_id":{"space_id":"0d4da3c4-fef9-4d16-b02b-35b954375043"},"name":"Test User"},"time":"2024-08-26T06:42:39Z","line":"/home/ubuntu/ocis/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node/node.go:824","message":"AsResourceInfo"}
Aug 26 06:42:39 ip-10-10-10-247 ocis[50798]: {"level":"debug","service":"storage-users","pkg":"rgrpc","traceid":"4ece1695f6441388e38f6a9e6b580352","permissions":{},"spaceid":"0d4da3c4-fef9-4d16-b02b-35b954375043","nodeid":"0d4da3c4-fef9-4d16-b02b-35b954375043","user":{"id":{"idp":"none","opaque_id":"9b8d978f-692c-4415-a85e-9f1d2bcb025b","type":3}},"time":"2024-08-26T06:42:39Z","line":"/home/ubuntu/ocis/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node/node.go:1075","message":"returning aggregated permissions"}
Aug 26 06:42:39 ip-10-10-10-247 ocis[50798]: {"level":"debug","service":"storage-users","pkg":"rgrpc","traceid":"4ece1695f6441388e38f6a9e6b580352","spaceid":"0d4da3c4-fef9-4d16-b02b-35b954375043","nodeid":"0d4da3c4-fef9-4d16-b02b-35b954375043","nodepath":"/home/ocis/.ocis/storage/users/spaces/0d/4da3c4-fef9-4d16-b02b-35b954375043/nodes/0d/4d/a3/c4/-fef9-4d16-b02b-35b954375043","time":"2024-08-26T06:42:39Z","line":"/home/ubuntu/ocis/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node/node.go:888","message":"quota not set"}
Aug 26 06:42:39 ip-10-10-10-247 ocis[50798]: {"level":"debug","service":"storage-users","pkg":"rgrpc","traceid":"4ece1695f6441388e38f6a9e6b580352","spaceid":"0d4da3c4-fef9-4d16-b02b-35b954375043","nodeid":"0d4da3c4-fef9-4d16-b02b-35b954375043","ri":{"type":2,"id":{"opaque_id":"0d4da3c4-fef9-4d16-b02b-35b954375043","space_id":"0d4da3c4-fef9-4d16-b02b-35b954375043"},"etag":"\"b1881bf9419ebde793f79747969d49cd\"","mime_type":"httpd/unix-directory","mtime":{"seconds":1724654558,"nanos":343397049},"path":".","permission_set":{},"size":626864,"owner":{"idp":"https://auth.example.org/realms/example","opaque_id":"0d4da3c4-fef9-4d16-b02b-35b954375043","type":1},"arbitrary_metadata":{},"parent_id":{"space_id":"0d4da3c4-fef9-4d16-b02b-35b954375043"},"name":"Test User"},"time":"2024-08-26T06:42:39Z","line":"/home/ubuntu/ocis/vendor/github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node/node.go:824","message":"AsResourceInfo"}
Aug 26 06:42:39 ip-10-10-10-247 ocis[50798]: {"level":"debug","service":"storage-users","pkg":"rgrpc","traceid":"4ece1695f6441388e38f6a9e6b580352","user-agent":"grpc-go/1.65.0","from":"tcp://127.0.0.1:56820","uri":"/cs3.storage.provider.v1beta1.ProviderAPI/Stat","start":"26/Aug/2024:06:42:39 +0000","end":"26/Aug/2024:06:42:39 +0000","time_ns":235297,"code":"OK","time":"2024-08-26T06:42:39Z","line":"/home/ubuntu/ocis/vendor/github.com/cs3org/reva/v2/internal/grpc/interceptors/log/log.go:69","message":"unary"}
Aug 26 06:42:39 ip-10-10-10-247 ocis[50798]: {"level":"debug","service":"gateway","pkg":"rgrpc","traceid":"4ece1695f6441388e38f6a9e6b580352","user-agent":"grpc-go/1.65.0","from":"tcp://127.0.0.1:48318","uri":"/cs3.gateway.v1beta1.GatewayAPI/Stat","start":"26/Aug/2024:06:42:39 +0000","end":"26/Aug/2024:06:42:39 +0000","time_ns":938948,"code":"OK","time":"2024-08-26T06:42:39Z","line":"/home/ubuntu/ocis/vendor/github.com/cs3org/reva/v2/internal/grpc/interceptors/log/log.go:69","message":"unary"}

Any ideas on what’s going wrong or how we can find a solution to keep this from happening?

2

Looks like this is documented as [2.0] panic: unaligned 64-bit atomic operation on aarch64/arm64 · Issue #6104 · owncloud/ocis · GitHub, but supposedly fixed?

1 Like
3

Even though the error message is the same you might be hitting some completely different. As you noted, the zapx issue in bleve has been fixed quite some time ago.

Are there any more messages after the panic message? Usually there should be some stacktrace that would help to isolate the issue. If there is, please open a new issue on github with those messages attached. Also some more info about the system you’re running this on (e.g. which architecture) would be helpful.

1 Like
4

Thanks, Ralf. There’s no stack trace or anything after the panic message, just systemd saying the ocis.service failed. We’re running on Ubuntu 24 on x64 AMD EPYC 7R13 4 cores with 8GB RAM.

5

Pulled fresh 6.3.0 code and rebuilt ocis using the same procedure as before. Error has gone away.

1 Like