Syslog Replacement

[tastyfrankfurt]

Hey Guys,

We mainly deal with centos/ubuntu/redhat/rocky. These come with a default installation of rsyslog. Currently trying to work out a suitable configuration to replace the defaults outputs of these. So far i have struggled to get my head around the vrl implementation. Is someone able to help? Filtering the log is not an issue its the outputing in a format that is usable to the log files, ie data host processname proc id then the message. We are using journald as the source.

Thanks in advanced, i have also attached the outputs i am looking at.
.info;mail.none;authpriv.none;cron.none /var/log/messages
authpriv. /var/log/secure
mail.* -/var/log/maillog
cron.* /var/log/cron
.emerg :omusrmsg:
uucp,news.crit /var/log/spooler
local7.* /var/log/boot.log

[spencergilbert]

If you're just looking to parse the messages that are syslog formatted you can do something like:

parsed, err = parse_syslog(.message)
if err != null {
  log("Message not parsed: " + string!(.message), level: "error")
}
. = merge(., parsed)

[tastyfrankfurt]

Hey @spencergilbert got it done, will write up a config but this is what i went with
.message = format_timestamp!(now(),format: "%b %e %H:%M:%S") + " " + string!(.host) + " " + string!(.SYSLOG_IDENTIFIER) + "[" + string!(._PID) + "]: " + string!(.message)

[tastyfrankfurt]

Okay have written up a configuration that we will use for ubuntu and centos, its important that the vector user is a member of the syslog group on ubuntu, haven't tested redhat/centos yet. But this work, thanks for getting me the info i needed @spencergilbert, working copy for us to replace rsyslog completely with vector.

---

sources:
s_journald:
type: journald
batch_size: {{ vector_client_journald_batchsize }}
current_boot_only: true
exclude_units: {{ vector_client_journald_units_exclude }}
journalctl_path: /usr/bin/journalctl
transforms:
t_remap_message:
type: remap
inputs: [s_journald]
source: |
if (!is_null(.host) && !is_null(.SYSLOG_IDENTIFIER) && !is_null(._PID) && !is_null(.message))
{
.message = format_timestamp!(now(),format: "%b %e %H:%M:%S") + " " + string!(.host) + " " + string!(.SYSLOG_IDENTIFIER) + "[" + string!(._PID) + "]: " + string!(.message)
.drop = "fields not null"
}
t_local_filter:
type: route
inputs: [t_remap_message]
route:
"t_messages": includes(["6"], .PRIORITY) && !includes(["2","4","10","9"], .SYSLOG_FACILITY) && exists(.drop)
"t_secure": includes(["10"], .SYSLOG_FACILITY) && exists(.drop)
"t_mail": includes(["2"], .SYSLOG_FACILITY) && exists(.drop)
"t_cron": includes(["9"], .SYSLOG_FACILITY) && exists(.drop)
"t_emerg": includes(["8"], .SYSLOG_FACILITY) || (includes(["58"], .PRIORITY) && !includes(["7"], .SYSLOG_FACILITY)) && exists(.drop)
"t_local": includes(["23"], .SYSLOG_FACILITY) && exists(.drop)
sinks:
d_messages:
type: file
inputs: [ t_local_filter.t_messages ]
compression: none
path: "/var/log/messages"
encoding:
codec: "text"
d_secure:
type: file
inputs: [ t_local_filter.t_secure ]
compression: none
path: "/var/log/secure"
encoding:
codec: "text"
d_mail:
type: file
inputs: [ t_local_filter.t_mail ]
compression: none
path: "/var/log/maillog"
encoding:
codec: "text"
d_cron:
type: file
inputs: [ t_local_filter.t_cron ]
compression: none
path: "/var/log/cron"
encoding:
codec: "text"
d_emerg:
type: file
inputs: [ t_local_filter.t_emerg ]
compression: none
path: "/var/log/spooler"
encoding:
codec: "text"
d_local:
type: file
inputs: [ t_local_filter.t_local ]
compression: none
path: "/var/log/boot.log"
encoding:
codec: "text"

[spencergilbert]

Neat! If you wanted to make the remap more robust you could replace the ! (raising the error), with a result, err pattern and better inform you if one of those fields isn't in the expected format - though coming off of journald I expect you should be pretty safe.

[spencergilbert]

Related - #7117 would make it easier/cleaner to set up that message string you want

[tastyfrankfurt]

Yes indeed, but happy for now, my next challenge is to work out log rotate, dont suppose a simply nohup will support this otherwise could be looking at a full service restart eeeeeek