teler and custom logs / haproxy

[dwisiswant0]

From: Christian Ruppert <[REDACTED]>
To: <[email protected]>
Date: Fri, 11 Dec 2020 04:27:13 +0700
Subject: teler and custom logs / haproxy


Hey,

I'm just curious, is there a reference for all those log keywords /
variables?
Like how do I configure it to parse HAProxy logs?
https://cbonte.github.io/haproxy-dconv/2.3/configuration.html#8.2.3

The fields described there, from 1 to 16. captured header are listed
within the {} and separated by a |
Example as logged to a syslog-ng including timestamps and host:
Dec 10 22:01:10 localhost haproxy[32214]: someipv4_or_ipv6:client_port
[10/Dec/2020:22:01:10.707] frontend~ backend1/server1 0/0/0/29/29 304
178 - - ---- 5/3/0/0/0 0/0 {example.com|Mozilla/5.0 (Windows NT 10.0;
Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0|} "GET
https://example.com/ HTTP/2.0"

How can I skip some strings? Like the very specific termination states
or just something I don't need?

--
Regards,
Christian Ruppert

[dwisiswant0]

As we don't have clear documentation on how to define log_format variables for each type of server access log, which is different (related #85). If you refer to my slides (https://dw1.io/files/teler%20-%20Protect%20Your%20WebApp.pdf) on page 30/50, the log_format required (to analysis or alerting needs) on teler is correct for all variables of the Nginx server's log_format example.

In your case (HAProxy access log), to skip (some of) log string, you also have to specify a variable name for that (will be skipped) string in log_format (which will not be analyzed by teler because it's not needed). For example:

Your HAProxy access log-line is:
Feb 6 12:14:14 localhost haproxy[14389]: 10.0.1.2:33317 [06/Feb/2009:12:14:14.655] http-in static/srv1 10/0/30/69/109 400 2750 - - ---- 1/1/1/1/0 0/0 {1wt.eu} {} "GET /.env HTTP/1.1"

Your log_format on teler configuration file should be:
$x $x $x $x $x[$x]: $remote_addr:$x [$time_local] $x $x $x $status $body_bytes_sent $x $x $x $x $x {$x} {$x} "$request_method $request_uri $request_protocol"

teler.haproxy.yaml

log_format: |
  $x $x $x $x $x[$x]: $remote_addr:$x [$time_local] 
  $x $x $x $status $body_bytes_sent $x $x $x $x $x {$x} {$x} 
  "$request_method $request_uri $request_protocol"

If we breakdown it will look like:

Variable	Log
x	Feb
x	6
x	12:14:14
x	localhost
x	haproxy
x	14389
remote_addr	10.0.1.2
x	33317
time_local	06/Feb/2009:12:14:14.655
x	http-in
x	static/srv1
x	10/0/30/69/109
status	400
body_bytes_sent	2750
x	-
x	-
x	----
x	1/1/1/1/0
x	0/0
x	1wt.eu
x
request_method	GET
request_uri	/.env
request_protocol	HTTP/1.1

The x variables are string values that are NOT required/will be skipped by teler to analyze. 😺

[idl0r]

Thanks!
I ended up with

log_format: |
     $x $x $x $x $x[$x]: $remote_addr:$x [$time_local] $x $x $x $status $body_bytes_sent $x $x $x $x $x {$x} "$request_method $request_uri $request_protocol"

Although it seems the $remote_addr doesn't like IPv6. It often reports stuff like [2a01] [Bad IP Address] 2a01 which doesn't seem right.

Also it's somewhat difficult to read, when you have a global log for multiple (v)hosts. So e.g. [1.2.3.4] [Directory Bruteforce] /Search?s=c494c72d7f0180314d818fe4d87ecb27c7d02741 could need the host that was used so you know where it happened. Could the output be tweaked like this, to use a variable, if defined which represents the "Host" header value?

[dwisiswant0]

Aight, you got the point, @idl0r! When we refer to HAProxy configurations, looks like it (from example, 10.0.1.2) belongs to client_ip, not remote IP address, my bad.

We only parse the required log format, if you have multiple virtual hosts that only consume one access log file, maybe you can work around saving the Host header value (which you mentioned) as remote_addr variable. For note, that Bad IP Address detection may not work properly if you implement that. :)