Parsing log line which could be 1 of N types (Apache Combined or Err or Custom, etc)

[ipsi]

I've had a look through the discussions here and the docs, and I'm not sure what the best approach here is. Basically, I'm running a bunch of services (that I don't own - self-hosting binge), and the logs are frequently intermingled, and could be any of, say,

• Apache Combined
• Apache Error
• PHP Composer logs
• A log format I don't know

As far as I can tell, I need to do something like this:

[sources.source_firefly_iii_core]
type = "docker_logs"
include_containers = ["firefly_iii_core"]

[transforms.transform_firefly_iii_core]
type = "route"
inputs = [ "source_firefly_iii_core" ]

[transforms.transform_firefly_iii_core.route]
apache = '''
    _, err = parse_apache_log(.message, "combined")
    err == null
'''
apache_error = '''
    _, err = parse_apache_log(.message, "error")
    err == null
'''
application = '''
    _, err = parse_groks(
        .message,
        patterns: [
            "\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{FIREFLY_LOG_LEVEL}: %{GREEDYDATA:message}"
        ],
        aliases: {
            "FIREFLY_LOG_LEVEL": "(?:%{DATA:host}\\.%{LOGLEVEL:level})"
        }
    )
    err == null
'''

[transforms.transform_firefly_iii_core_apache_combined_remap]
type = "remap"
inputs = [ "transform_firefly_iii_core.apache" ]
source = '''
    . |= parse_apache_log!(.message, "combined")
'''
[transforms.transform_firefly_iii_core_apache_err_remap]
type = "remap"
inputs = [ "transform_firefly_iii_core.apache_error" ]
source = '''
    . |= parse_apache_log!(.message, "error")
'''
[transforms.transform_firefly_iii_core_application_remap]
type = "remap"
inputs = [ "transform_firefly_iii_core.application" ]
source = '''
    . |= parse_groks!(
        .message,
        patterns: [
            "\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{FIREFLY_LOG_LEVEL}: %{GREEDYDATA:message}"
        ],
        aliases: {
            "FIREFLY_LOG_LEVEL": "(?:%{DATA:host}\\.%{LOGLEVEL:level})"
        }
    )
'''

[sinks.console_test_out]
type = "console"
inputs = [ "transform_firefly_iii_core_*_remap" ]
encoding.codec = "json"

This isn't really ideal, since I have to parse the log twice in the best-case scenario, or four times in the worst-case, plus the necessary parsing code has to be duplicated. Is there a better way of handling this, given that I can't reasonably go and make a PR for every single service I've got deployed to get them to use JSON?

I suppose I could also write some horribly nested VRL, along the lines of

  logs, err = parse_apache_log(.message, "combined")
  if err == null {
    . |= logs
  } else {
    logs, err = parse_apache_log(.message, "error")
    if err == null {
      . |= logs
    } else {
      logs, err = parse_groks(
          .message,
          patterns: [
              "\\[%{TIMESTAMP_ISO8601:timestamp}\\] %{FIREFLY_LOG_LEVEL}: %{GREEDYDATA:message}"
          ],
          aliases: {
              "FIREFLY_LOG_LEVEL": "(?:%{DATA:host}\\.%{LOGLEVEL:level})"
          }
      )
      if err == null {
        . |= logs
      }
    }
  }

But don't think that's any better, particularly since I'm not sure I can route it to a separate destination like that (e.g., I want Apache logs to go somewhere else, separate from application logs).

[jszwedko]

For mixed formats, I'd probably suggest just relying on parse_groks which can take a list of patterns to match against rather than involving the other parse_* functions like parse_apache_log.

[ipsi]

Although match doesn't accept Grok patterns, right (or at least is not documented as doing so)? So I'd need either craft a separate regex, or manually expand the Grok pattern and use that for matching.

I think a couple of features that would solve this here would be something along the lines of:

• Can add custom Grok patterns in the global config for easy re-use across transformers
• Can use Grok patterns with match (or a new grok_match function/etc) - could then use that directly in a route condition, and might be slightly more performant as it doesn't need to extract match groups.

A more complex feature would be a "combined" remap-and-route transformer, so it wouldn't have to run the parser twice for whichever format matches. Not sure how broadly applicable this would be, as having to run multiple parsers to figure out which format a message is in is already probably quite bad for performance.

If you'd be open to adding the first two, I'd be happy to add a couple of issues for those.

[jszwedko]

Although match doesn't accept Grok patterns, right (or at least is not documented as doing so)? So I'd need either craft a separate regex, or manually expand the Grok pattern and use that for matching.

Yeah, that is correct, match just takes regular expressions. I could see it being useful to have a match_grok or something, but, to your point, that would mean duplicate processing.

Maybe it'd be useful to have parse_groks be able to tag the pattern that matched and that could be used for subsequent processing?

There was a semi-recent addition to VRL that allows for loading grok aliases from a file that might be helpful to you: vectordotdev/vrl#194

[ipsi]

Yeah, that is correct, match just takes regular expressions. I could see it being useful to have a match_grok or something, but, to your point, that would mean duplicate processing.

Yeah, I'm sure there's a few use-cases for it. For me, personally, I'm not dealing with a large volume of logs, so while it hurts my soul that there'd be some duplicate processing going on, I don't see it having any practical impact.

Maybe it'd be useful to have parse_groks be able to tag the pattern that matched and that could be used for subsequent processing?

Yeah, that would also solve my problem - I think this would solve it slightly more neatly, but is a bit less flexible.

There was a semi-recent addition to VRL that allows for loading grok aliases from a file that might be helpful to you: vectordotdev/vrl#194

Ah yes, that looks like exactly what I want, but doesn't appear to be listed in the docs - is it available in the latest version of Vector?

[jszwedko]

Yeah, that is correct, match just takes regular expressions. I could see it being useful to have a match_grok or something, but, to your point, that would mean duplicate processing.

Yeah, I'm sure there's a few use-cases for it. For me, personally, I'm not dealing with a large volume of logs, so while it hurts my soul that there'd be some duplicate processing going on, I don't see it having any practical impact.

Maybe it'd be useful to have parse_groks be able to tag the pattern that matched and that could be used for subsequent processing?

Yeah, that would also solve my problem - I think this would solve it slightly more neatly, but is a bit less flexible.

Thinking about it more, I think I'd lean towards having parse_groks be able to tag which pattern matched. It seems unlikely to me that users will want to match a grok pattern, but not also parse the input using it. Feel free to open a feature request for it if you like over in https://github.com/vectordotdev/vrl/issues

There was a semi-recent addition to VRL that allows for loading grok aliases from a file that might be helpful to you: vectordotdev/vrl#194

Ah yes, that looks like exactly what I want, but doesn't appear to be listed in the docs - is it available in the latest version of Vector?

Good catch, it looks like this was never documented. I left a comment to prompt the contributor: vectordotdev/vrl#194 (comment). It is available in the latest versions of Vector though.

[ipsi]

Created a feature request for grok_patterns.