-
-
Notifications
You must be signed in to change notification settings - Fork 287
Replies: 1 comment · 6 replies
-
It looks like your log file is in JSON-lines form, which is directly supported by lnav. You shouldn't have to use a regex to parse it out. The log format reference covers the details of how to write a log format file for a JSON-lines log. Unfortunately, the docs don't have an actual example, but there are some tests you can look at. For this log file: Lines 1 to 13 in 8d17561
The log format file looks like this: lnav/test/formats/jsontest/format.json Lines 1 to 36 in 8d17561
The rendered output then looks like this: lnav/test/expected/test_json_format.sh_469f005b0708d629bc95f0c48a5e390f440c1fef.out Lines 1 to 29 in 8d17561
(Note that the extra spacing is due to the leading Let me know if you need additional help in creating your format file. |
Beta Was this translation helpful? Give feedback.
All reactions
-
Ah, sorry. Yes, JSON-lines logs need to be in a file at the moment.
This behavior should work already. Any fields found that are not mentioned in {"ts": "2013-09-06T22:01:49.124817Z", "lvl": "FATAL", "msg": "shoot", "obj": { "field1" : "hi", "field2": 2 }, "arr" : ["hi", {"sub1": true}]} The There is a flag in the format definition, |
Beta Was this translation helpful? Give feedback.
All reactions
-
Totally new user (found lnav yesterday) - I'm trying to do something sort of similar, but with a few differences.
Is my final approach the right way to handle this scenario? By example, here are a few sample log entries:
With the custom format;
When running lnav against the file, I see the following when typing
The ultimate goal here (in this example) is to query the number of 500's by path so that a user can identify which path is having issues. so the field Oddly, I just scrolled through the lines and sometimes I get a hit for that as a field and sometimes not;
This is interesting, but even that message doesn't seem to return a result when querying, so I guess the easiest path is to just grep / pre-filter the file? |
Beta Was this translation helpful? Give feedback.
All reactions
-
Hmm, I'm not sure what you mean by this. Can you give an example?
With your format, I am able to run the following query and it returns a reasonable result for the sample log messages you provided: ;SELECT count(*), "http.req.path", "http.resp.status" FROM http_logrus_custom GROUP BY "http.req.path", "http.resp.status" To limit the query to cases where the status is 500, you can do: ;SELECT count(*), "http.req.path", "http.resp.status" from http_logrus_custom WHERE "http.resp.status" = 500 GROUP BY "http.req.path" I think that is doing what you want or are you looking for something else?
The display is only showing the fields actually in the message. All fields are still available as columns in the Your format had some errors/warnings (pass
Adding the |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1 -
❤️ 1
-
Oh wow, so I was really just getting caught up in poor SQL statement issue - I just had a look at yours and those examples are super helpful this removes the entire pre-filter requirement. I'll definitely add the schema and changes you've provided - I'm just now moving it from vim to vscode as we build up a repository of formats for use. @tstack you're amazing, this is a huge help - I appreciate the guidance, pointers, and patience :) |
Beta Was this translation helpful? Give feedback.
All reactions
-
Note that the column names need to be in double-quotes (e.g. |
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
I am using the following regex:
to capture messages in the following format:
That works fine, except some fields that show up that are not in the regex, like for example the
@errorObjectName
field above, it gets captured together as<message>
rather than being ignored.So I tried the following regex to achieve that desired behaviour which uses OR to capture the different fields that I want to parse out.
The problem is with the above when I run
lnav
I get the following error:So lnav won't allow me to load this formatter unless it fully matches everything......
Is there a way to achieve the behavior that I want? Or is it impossible with
lnav
?Beta Was this translation helpful? Give feedback.
All reactions