Ever-growing WAL folder

[vini-intenseye]

What did you do?

I have prometheus version 2.46 installed with the follwoing configuration..

storage.tsdb.min-block-duration : 1h
storage.tsdb.max-block-duration : 1h
--storage.tsdb.retention | 0s
--storage.tsdb.retention.size | 0B
--storage.tsdb.retention.time | 1w
--storage.tsdb.samples-per-chunk | 120
--storage.tsdb.wal-compression | true
--storage.tsdb.wal-compression-type | snappy
--storage.tsdb.wal-segment-size | 0B

I deleted the WAL folder as it took more than 7 hours for a replay to be compelted

ts=2024-04-26T11:40:10.556Z caller=head.go:792 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=4m27.932583409s wal_replay_duration=7h15m17.527240298s wbl_replay_duration=205ns total_replay_duration=7h23m9.09340072s

There is enough mem and CPU on the machine where the prometheus is running.

This is the stats from prometheus

Number of Series | Number of Chunks | Number of Label Pairs | Current Min Time | Current Max Time
-- | -- | -- | -- | --
39772186 | 170739354 | 712874 | 2024-04-11T07:00:00.000Z (1712818800000) | 2024-04-29T15:24:44.917Z (1714404284917)

I am unable to find a right configuration for Prometheus to stop it from growing WAL forever. When I deleted the WAL folder it had files as old 20 days.

Is there any confiuration that I have wrong?

The current WAL size in 4 hours is 140GB.

Upgrade is planned, but would it be the solution for the problem here?

Thanks

What did you expect to see?

The WAL is only retained for 2 hours or so.

What did you see instead? Under which circumstances?

The WAL folder not cleaned up.

System information

Linux 6.1.58+ x86_64

Prometheus version

Build Information
Version	2.46.0
Revision	cbb69e51423565ec40f46e74f4ff2dbb3b7fb4f0
Branch	HEAD
BuildUser	root@42454fc0f41e
BuildDate	20230725-12:31:24
GoVersion	go1.20.6

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response