Unable to see dag_id and task_id tagging for mentioned metrics

[EMT-Swaroop]

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

2.9.0

What happened?

According to airflow documentation, from 2.8.2 airflow version the metrics ti_failures, ti_successes, zombie_killed metrics need to have tagging task_id and dag_id

Airflow documentation:https://airflow.apache.org/docs/apache-airflow/2.9.1/administration-and-deployment/logging-monitoring/metrics.html

In our airflow environment we have upgraded airflow 2.9.1 and helm chart 1.13.1. Still we are not able to see the tagging for setting up alerts and monitoring at task level.

What you think should happen instead?

The tagging of task-id and dag_id for metrics like ti_failures, ti_successes, zombie_killed need to be available so that we can locate the tasks accordingly to debug and we can have the alerting according to that.

How to reproduce

If we use 2.8.1 or later version tagging need to be there for mentioned metrics. But those are not coming, only the counter values are getting produced by the airflow statsd.

Operating System

Debian Bullseye

Versions of Apache Airflow Providers

recently we have upgraded to 2.9.1.

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[RNHTTR]

@dirrao can you take a look?

Also, it would probably be a good idea to add to the documentation what metrics tagging even is

[EMT-Swaroop]

Hey @RNHTTR, Below are the list of few metrics that are missing tagging which were mentioned in the airflow documentation which I have mentioned in the issue description.

ti_failures
ti_successes
zombie_killed

[dirrao]

@RNHTTR , @EMT-Swaroop
I will look into this issue.

[dirrao]

@EMT-Swaroop
If you are using statsd with influx backend store, you need to enable the influxdb_tags_enabled flag to propagate the tags. If not, Can you share the details about the metrics exporter?

[EMT-Swaroop]

Hey @dirrao , We are not using influxdb, we are using prometheus as backend store and didn't customize any configuration in the statsd deployment, we are using the default stasd exporter which is coming from the parent airflow chart (1.13.1)

[EMT-Swaroop]

@dirrao any update?