[Q&A] fluent-bit 3.0.3 unable to integrate with openlens 2.13.0 in k8s

[amey-greenbone]

What is a problem?

opensearch: 2.13.0 [customized image with repository-s3 & prometheus plugin included]
opensearch dashboard: opensearchproject/opensearch:2.13.0
fluent-bit: fluent/fluent-bit:3.0.3-debug
Platform: k8s / 1.28.8 [hosted cluster]
Base OS: Ubuntu 22.04.4 LTS
Ubuntu Kernel version:5.15.0-102-generic
Container runtime: containerd://1.7.15
Kubelet version: v1.28.8


- All pods are up and running
- I am able to login to opensearch dashboard from web browser [port forwarding method]
- "Multi ElasticSearch Head" plug-in for Chrome browser is unable to show indices from fluent-bit 
- No logs inside Fluent-bit debug pod
- If I curl Cluster_IP_openlens_pod:9200 from the fluent-bit pod = Getting expected response [reference = https://github.com/fluent/fluent-bit/issues/8207]
- In OpenLens Dashboard I am unable to create a new index pattern with logstash-*

Describe the configuration of Fluentd

config.yaml =

`apiVersion: v1
kind: ConfigMap
metadata:
name: logging-fluent-bit-es-configmap
namespace: gcs-logging-dev
data:
fluent-bit.conf: |
[SERVICE]
Flush 5
Log_Level info
Daemon off
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
storage.path /var/log/flb-storage/
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M

@INCLUDE input-kubernetes.conf

@INCLUDE filters.conf

@INCLUDE output-opensearch.conf

input-kubernetes.conf: |
[INPUT]
Name tail
Path /var/log/containers/.log
Exclude_Path /var/log/containers/fluent-bit-.log, /var/log/containers/gmsp-es-logging-dashboards-.log
Path_Key filepath
Tag kubernetes.
DB /var/log/flb.containers.db
Mem_Buf_Limit 30MB
Skip_Long_Lines On
Refresh_Interval 10
read_from_head true
multiline.parser multiline
storage.type filesystem

filters.conf: |
[FILTER]
Name parser
Match kubernetes.*
Key_Name log
Reserve_Data On
Parser cri

[FILTER]
  Name                parser
  Match               kubernetes.*
  Key_Name            log
  Reserve_Data        On
  Parser              json-with-time-parser-remove-time

[FILTER]
  Name                modify
  Match               kubernetes.*
  Rename              log message

[FILTER]
  Name                parser
  Match               kubernetes.*
  Key_Name            filepath
  Reserve_Data        On
  Parser              kube-file-path-parser

[FILTER]
  Name                record_modifier
  Match               kubernetes.*
  Record              node_name ${NODE_NAME}
  Record              fluentbit_pod_id ${POD_ID}

[FILTER]
  Name                grep
  Match               kubernetes.* 
  Regex               namespace ^(gcs-app-dev|gcs-auth-dev)$

[FILTER]
  Name                      multiline
  Match                     kubernetes.*
  multiline.key_content     message
  multiline.parser          multiline

[FILTER]
  Name      modify
  Match     kubernetes.*
  Copy      message log_message

[FILTER]
  Name         parser
  Match        kubernetes.*
  Key_Name     log_message
  Reserve_Data On
  Parser       java_app_general

[FILTER]
  Name      modify
  Match     kubernetes.*
  Copy      message log_message

[FILTER]
  Name         parser
  Match        kubernetes.*
  Key_Name     log_message
  Reserve_Data On
  Parser       java_app_log_message  

[FILTER]
  Name        lua
  Match       kubernetes.*
  script      filters.lua
  call        cb_set_default_fields

output-opensearch.conf: |
[OUTPUT]
Name es
Match *
Host ${FLUENT_OPENSEARCH_HOST}
Port ${FLUENT_OPENSEARCH_PORT}
Logstash_Format On
Replace_Dots Off
Retry_Limit 3
Buffer_Size False
Include_Tag_Key On
Workers 1
storage.total_limit_size 50M

parsers.conf: |
[PARSER]
Name cri
Format regex
Regex ^(?[^ ]+) (?stdout|stderr) (?[^ ]) (?.)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
  Name            json-with-time-parser-remove-time
  Format          json
  Time_Key        time
  Time_Format     %Y-%m-%dT%H:%M:%S.%LZ

[PARSER]
  Name            kube-file-path-parser
  Format          regex
  Regex           ^\/var\/log\/containers\/(?<pod_id>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64})\.log$

[PARSER]
  Name        java_app_general
  Format      regex
  Regex       ^(?<log_time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3} [+\-]\d{4}) \[(?<image_version>\S+), (?<thread_name>.+?), ,\] (?<log_level>\S+)\s+(?<class_name>[^\s-]+) [^\n]+$
  Time_Key    log_time
  Time_Format %Y-%m-%dT%H:%M:%S.%L %z

[PARSER]
  Name        java_app_log_message
  Format      regex
  Regex       ^.*?\n(?<log_message>.+)$

[MULTILINE_PARSER]
  name            multiline
  type            regex
  flush_timeout   2000
  rule            "start_state"   "^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}"  "cont"
  rule            "cont"          "^[^0-9]" "cont"

filters.lua: |
function cb_set_default_fields(tag, timestamp, record)
if record["log_message"] == nil then
record["log_message"] = "-"
end
if record["image_version"] == nil then
record["image_version"] = "-"
end
if record["thread_name"] == nil then
record["thread_name"] = "-"
end
if record["class_name"] == nil then
record["class_name"] = "-"
end
if record["log_level"] == nil then
record["log_level"] = "-"
end
return 1, timestamp, record
end`

daemonset.yaml =

apiVersion: apps/v1 kind: DaemonSet metadata: name: fluent-bit namespace: gcs-logging-dev labels: app: fluent-bit spec: selector: matchLabels: app: fluent-bit template: metadata: labels: app: fluent-bit spec: serviceAccountName: fluent-bit containers: - name: fluent-bit image: fluent/fluent-bit:3.0.3-debug imagePullPolicy: Always resources: requests: memory: "64Mi" cpu: "100m" limits: memory: "128Mi" cpu: "200m" ports: - containerPort: 2020 env: - name: FLUENT_OPENSEARCH_HOST value: "gmsp-es-logging-headless" - name: FLUENT_OPENSEARCH_PORT value: "9200" - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: POD_ID valueFrom: fieldRef: fieldPath: metadata.name - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: fluent-bit-config mountPath: /fluent-bit/etc/ terminationGracePeriodSeconds: 10 volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: fluent-bit-config configMap: name: logging-fluent-bit-es-configmap

roles.yaml =

`apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluent-bit
namespace: gcs-logging-dev
labels:
app: fluent-bit
rules:

• apiGroups:
  • ""
    resources:
  • pods
  • namespaces
    verbs:
  • get
  • list
  • watch

---

apiVersion: v1
kind: ServiceAccount
metadata:
name: fluent-bit
namespace: gcs-logging-dev
labels:
app: fluent-bit

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluent-bit
roleRef:
kind: ClusterRole
name: fluent-bit
apiGroup: rbac.authorization.k8s.io
subjects:

• kind: ServiceAccount
  name: fluent-bit
  namespace: gcs-logging-dev`

Describe the logs of Fluentd

No response

Environment

- Fluentd version: 
- TD Agent version:
- Fluent Package version:
- Docker image (tag): fluent/fluent-bit:3.0.3-debug
- Operating system: Ubuntu 22.04.4 LTS
- Kernel version: 5.15.0-102-generic