Multiline events from a file using Vector FILE source

[atibdialpad]

I am using the "file" source to read log lines and I need to use the multi-line feature to group >1 lines together. In the most general case, I want something like: "every new log line starts with a DATE and all lines that follows and doesn't starts with a DATE is part of the same event"
The config that I am using is :

start_pattern = "^(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2})"
condition_pattern = "^(?P<year>[0-9]{4})-(?P<month>[0-9]{2})-(?P<day>[0-9]{2})"
mode = "halt_before"

with this context I have couple of questions :

1. Does this introduce a ( significant ) performance hit with having to check start and condition pattern on every line even before the event goes to the transform stage (which btw has its own GROKing)
2. Is their any better way to capture such multi-line events.

Sample Multi-line event :

event 1 starts here--->2021-08-04 06:06:37.877096 [INFO] switch_core.c:2515
FreeSWITCH Version 1.10.5-release+git~20210729T051935Z~90077fa1af~64bit (git 90077fa 2021-07-29 05:19:35Z 64bit)
FreeSWITCH Started
Max Sessions [4000]
Session Rate [100]
SQL [Enabled]
event2 starts here --->2021-08-04 06:38:27.336864 [NOTICE] switch_channel.c:1118 New Channel sofia/internal/[email protected]:5064 [prober-bfea31a7-75d4-4c45-a706-3833a4bb14e9]

[spencergilbert]

1. Does this introduce a ( significant ) performance hit with having to check start and condition pattern on every line even before the event goes to the transform stage (which btw has its own GROKing)

I don't think we have specific benchmarks comparing the multiline feature in the file source to not enabling it, but it's currently faster than the reduce transformation.

2. Is their any better way to capture such multi-line events.

It seems like your start_pattern could just be .*, and with your current mode of halt_before you should continue to aggregate multiple lines (up until it sees the date lines).