Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\\n\" ws: false; Accept: */*; Connection: close; User-Agent: kube-probe/1.28; "

[zeyneprumeysayorulmaz]

My promtail.yaml file is as follows. I'm trying to read /var/log/nginx/access.log from an nginx pod running on a specific namespace. But I am getting the following error.

---

"level=info ts=2024-05-23T11:06:18.238787509Z caller=promtail.go:133 msg="Reloading configuration file" md5sum=acca7e29eb0375b3ecb9a74122b66759
level=info ts=2024-05-23T11:06:18.240036686Z caller=kubernetes.go:327 component=discovery discovery=kubernetes config=kubernetes-pods-zeynep-loki-test-app msg="Using pod service account via in-cluster config"
level=info ts=2024-05-23T11:06:18.241713464Z caller=server.go:322 http=[::]:3101 grpc=[::]:9095 msg="server listening on addresses"
level=info ts=2024-05-23T11:06:18.24187037Z caller=main.go:174 msg="Starting Promtail" version="(version=2.9.3, branch=HEAD, revision=2535f9bede)"
level=warn ts=2024-05-23T11:06:18.241954721Z caller=promtail.go:263 msg="enable watchConfig"
level=warn ts=2024-05-23T11:06:37.659338151Z caller=logging.go:123 msg="GET /ready (500) 55.257µs Response: "Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\n" ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.28; "
level=warn ts=2024-05-23T11:06:47.656961884Z caller=logging.go:123 msg="GET /ready (500) 47.257µs Response: "Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\n" ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.28; "
level=warn ts=2024-05-23T11:06:57.656534552Z caller=logging.go:123 msg="GET /ready (500) 67.024µs Response: "Not ready: Unable to find any logs to tail. Please verify permissions, volumes, scrape_config, etc.\n" ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.28; ""

promtail.yaml

server:
log_level: info
log_format: logfmt
http_listen_port: 3101

clients:

• tenant_id: admins
  url: http://loki-gateway/loki/api/v1/push

positions:
filename: /run/promtail/positions.yaml

scrape_configs:

• job_name: kubernetes-pods-zeynep-loki-test-app
  pipeline_stages:
  • cri: {}
  • match:
    selector: '{namespace="zeynep-loki-test-app"}'
    stages:
    - tenant:
    value: zeynep-loki-test-app
    kubernetes_sd_configs:
  • role: pod
    relabel_configs:
  • source_labels:
    • __meta_kubernetes_namespace
      regex: zeynep-loki-test-app
      action: keep
  • source_labels:
    • __meta_kubernetes_pod_label_app
      regex: nginx
      action: keep
  • source_labels:
    • __meta_kubernetes_pod_container_name
      target_label: container
  • source_labels:
    • __meta_kubernetes_pod_name
      target_label: pod
  • source_labels:
    • __meta_kubernetes_namespace
      target_label: namespace
  • source_labels:
    • __meta_kubernetes_pod_node_name
      target_label: node
  • source_labels:
    • __meta_kubernetes_pod_ip
      target_label: instance
  • action: labelmap
    regex: _meta_kubernetes_pod_label(.+)
  • source_labels:
    • __meta_kubernetes_pod_name
    • __meta_kubernetes_namespace
      target_label: path
      replacement: /var/log/pods/*$1/$2/nginx/access.log

limits_config:

tracing:
enabled: false

[dk03051996]

I am also facing same issue if I am trying to append log. It used to work 6 months back. I also tried with old version but no luck.

[dk03051996]

hi @zeyneprumeysayorulmaz , did you find any solution for this. I am also facing the same issue.