ownCloud log analysis with jq

[eneubauer]

Hi there,
I was just analyzing some (way too big) ownCloud log file. Standard ownCloud log files are JSON, so you can just parse them with the CLI tool jq.

jq is available in the standard repositories of all the main Linux distributions.

So I built the following filter, which I think could be very helpful for other people too:

jq 'select(.time | .[:-6] + "Z" | fromdateiso8601 > (now - (60*60*24*14)) ) | select(.message |  contains("NotFoundException") // contains("LockedException")  | not)' owncloud.log

Let’s break it down:

jq # calling the binary
'# beginning of filter
select # using the builtin select function 
(.time | .[:-6] + "Z" # making the value of the key "time" compatible with the fromdate function
| fromdate #fromdate converts date string to unix epoch
> # value of time needs to be bigger than (this is a comparison operator)
(now - (60*60*24*14)) ) # current unix epoch - seconds of 14 days
| # this filtered result we want to further filter down so you can just use the pipe to pass it to the next filter
select # we know this one already
(.message # this time we want to check what's in the message field
| contains("NotFoundException") # return true if the string is in the message field
// # logical OR operator
contains("LockedException") # also return true for this other string
| # here we are using the pipe like a logical AND operator
not) # negate the result, so we DON'T see the strings above in the output
' # end of the filter
owncloud.log # path to the owncloud log file to analyse

There are a lot more possibilities and I am also just scratching the surface. For more options see the offical jq docs:
https://stedolan.github.io/jq/manual/

[eneubauer]

I just found out that sometimes the time value is different:

 "time": "2021-02-16 07:46:30.316400",

With the select from the post above you will get an error message like this one:

date "2021-02-16 07:46:30.Z" does not match format "%Y-%m-%dT%H:%M:%SZ"

In that case you will have to adjust the time select like so:

jq 'select(.time | .[0:10] + "T" + .[-15:-7] + "Z" | fromdateiso8601 > (now - (60*60*24*14)) ) ' owncloud.log