Parsing stops when hitting UTF-8 replacement char

[matejzero]

We are using mtail to parse some specific logs where sometimes there are replacement chars(�) in strings. It seems like when mtail hits a log line with this char, it stops parsing logs after this log line.

mtail version: mtail version 3.0.0-rc50

mtail program:

counter response_total by code
counter count_lines
const ONELINER /oneliner/

ONELINER {
  /<result code=\"(?P<code>1000|1300|1301|1500|2002|2200|2201|2202|2302|2303|2306|2308|greeting)\">/ {
  response_total[$code]++
  }
}

/$/ {
  count_lines++
}

Sample logs - non-working:

Nov 29 04:02:02.242 host.example.tld service INFO  [18636C07-1:foobar:1.2.3.4] service.ioLoop 123 : Transaction took: 0.500 seconds
Nov 29 04:02:02.242 host.example.tld service INFO  Loop(responseData -> oneliner) <result code="1000"> <contact:name>FOO�BAR LOG</contact:name>
Nov 29 04:02:02.242 host.example.tld service INFO  [18636C07-1:foobar:1.2.3.4] service.ioLoop 123 : Transaction took: 0.500 seconds

Sample logs - working:

Nov 29 04:02:02.242 host.example.tld service INFO  [18636C07-1:foobar:1.2.3.4] service.ioLoop 123 : Transaction took: 0.500 seconds
Nov 29 04:02:02.242 host.example.tld service INFO  Loop(responseData -> oneliner) <result code="1000"> <contact:name>FOOSBAR LOG</contact:name>
Nov 29 04:02:02.242 host.example.tld service INFO  [18636C07-1:foobar:1.2.3.4] service.ioLoop 123 : Transaction took: 0.500 seconds

Running with -one_shot, with replacement char, I get count_lines{prog="test"} 2and without it,count_lines{prog="test"} 3`

Is this a known problem and how can I get more info on why is it failing?

cc: @smrekarm12

[jaqx0r]

This is definitely a bug, thanks for finding it!

The problem is probably with how the file reader handles UTF-8 runes, and probably it's getting stuck by not discarding this after failing to convert the rune.

Can you find the INFO logs for mtail when this happens? https://google.github.io/mtail/Troubleshooting.html#deployment-problems has the location of the INFO log.

[matejzero]

I ran mtail with -logtostderr, -one_shot and -v=2 and the output is attached in log files.

Non-working log file: mtail_debug_nonworking.log

Working log file: mtail_debug_working.log

From the logs, I can only see the working one having one more decode.go:52] sendline log line and the non-working log is 2 bytes bigger, probably due to utf-8 rune.

[matejzero]

Any news regarding this issue? Just so we know if we should find another solution or wait?