PromQL: Multiplication by 1 creates additional floating point errors (but should be a no-op)

[gabrieljones]

What did you do?

grafana PromQL:

sum by (le) (increase(ingress_timer_millis_bucket[$__interval]) * on (job, instance) group_left target_info)

What did you expect to see?

The same PromQL response as

sum by (le) (increase(ingress_timer_millis_bucket[$__interval]))

What did you see instead? Under which circumstances?

with the group_left join in place tiny non zeros are sprinkled throughout the buckets at random differing with each repeated request of the same query

System information

No response

Prometheus version

prometheus_build_info{
  branch="HEAD", 
  goarch="amd64", 
  goos="linux", 
  goversion="go1.21.6", 
  instance="0.0.0.0:8080", 
  revision="43e14844a33b65e2a396e3944272af8b3a494071", 
  tags="netgo,builtinassets,stringlabels", 
  version="2.49.1"
}

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response

[gabrieljones]

compare (prometheus_http_request_duration_seconds_bucket)

• with join
• without join

[beorn7]

It's not so much the grouping or the join that creates these artifacts, but the multiplication. At least I can create the same "noisy" heatmap by multiplying with a 1 literal.

I'm confused why this is happening, as multiplication by 1 should be a no-op (and with the query linked above, it should be guaranteed that an actual 1.0 float arrives in the engine).

[beorn7]

Fundamentally, I would say there must be a different code path in the PromQL engine for "sum the result of a function call" and "sum the result of a bin-op", but from the top of my head, I cannot imagine how that would happen.

[gabrieljones]

One possible workaround is to use something like round(_, 0.00001). Does this give the same result as the unmultiplied query?

compare:

• round(sum by (le) (increase(prometheus_http_request_duration_seconds_bucket[1m]) * 1), 0.00001) link
• sum by (le) (increase(prometheus_http_request_duration_seconds_bucket[1m])) link

[beorn7]

We discussed on the chat-system-that-shall-not-be-named. I guess the most likely reason for this is what @bboreham said there: "At a high level, it’s likely that each new expression changes the order of evaluation, which will give different results since addition is non-commutative."

[bboreham]

Just wanted to highlight #14074, which I think could improve the optics, at some CPU cost.
I didn't understand this issue enough to make a test case.