Streaming PromQL engine: native histograms #8121

jhesketh · 2024-05-13T10:50:15Z

Add native histogram support to instant vectors.

Checklist

Tests updated.
Documentation added.
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
about-versioning.md updated with experimental features.

Add native histogram support to instant vectors.

pkg/streamingpromql/operator/pool.go

pkg/streamingpromql/query.go

pkg/streamingpromql/testdata/ours/native_histograms.test

jhesketh · 2024-05-14T12:32:38Z

I've marked this as ready-for-review if we want to put this in in its current state. I'm also happy to keep adding commits to this branch as we support more histogram operations.

charleskorn

Overall looks good, thanks for working on this!

Only other thing I'd like to see is some benchmarking:

It'd be good to add a benchmark scenario or two that include native histograms
It'd be good to compare the performance of the existing non-native histogram benchmarks before and after this change to confirm it doesn't have a significant impact on performance or memory utilisation - I can't imagine it would, but it'd be good to confirm

pkg/streamingpromql/query.go

pkg/streamingpromql/testdata/ours/native_histograms.test

pkg/streamingpromql/operator/instant_vector_selector.go

pkg/streamingpromql/testdata/ours/native_histograms.test

pkg/streamingpromql/query.go

pkg/streamingpromql/testdata/ours/native_histograms.test

We can't use OOO samples with native histograms, so instead of batching by series we'll batch by samples This reverts commit 91ba5c1.

jhesketh · 2024-05-28T04:08:29Z

Benchmark of this branch in details:

goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
cpu: 12th Gen Intel(R) Core(TM) i7-1270P
                                                                       │      Prometheus      │              streaming              │
                                                                       │        sec/op        │    sec/op     vs base               │
Query/a_1,_instant_query                                                         148.0µ ±  4%   133.4µ ±  6%   -9.88% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                            158.1µ ±  3%   146.0µ ±  1%   -7.66% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                           260.2µ ±  1%   242.8µ ±  4%   -6.66% (p=0.002 n=6)
Query/a_100,_instant_query                                                       1.071m ±  2%   1.047m ±  1%   -2.22% (p=0.015 n=6)
Query/a_100,_range_query_with_100_steps                                          2.301m ± 22%   1.798m ±  2%  -21.86% (p=0.009 n=6)
Query/a_100,_range_query_with_1000_steps                                         9.551m ±  1%   9.691m ±  3%   +1.47% (p=0.026 n=6)
Query/a_2000,_instant_query                                                      18.54m ±  1%   18.31m ±  3%        ~ (p=0.132 n=6)
Query/a_2000,_range_query_with_100_steps                                         32.44m ±  1%   32.01m ±  1%        ~ (p=0.065 n=6)
Query/a_2000,_range_query_with_1000_steps                                        156.2m ±  2%   155.4m ±  2%   -0.51% (p=0.041 n=6)
Query/nh_1,_instant_query                                                        246.6µ ±  7%   226.4µ ±  4%   -8.16% (p=0.002 n=6)
Query/nh_1,_range_query_with_100_steps                                           328.0µ ±  2%   319.7µ ±  5%        ~ (p=0.240 n=6)
Query/nh_1,_range_query_with_1000_steps                                          775.3µ ±  1%   953.6µ ± 20%        ~ (p=0.065 n=6)
Query/nh_100,_instant_query                                                      2.404m ±  2%   2.369m ±  2%        ~ (p=0.065 n=6)
Query/nh_100,_range_query_with_100_steps                                         9.274m ± 19%   7.308m ±  2%  -21.19% (p=0.002 n=6)
Query/nh_100,_range_query_with_1000_steps                                        57.76m ±  3%   58.53m ±  4%        ~ (p=0.589 n=6)
Query/nh_2000,_instant_query                                                     46.69m ±  2%   46.17m ±  3%        ~ (p=0.310 n=6)
Query/nh_2000,_range_query_with_100_steps                                        146.2m ±  3%   147.8m ±  2%        ~ (p=0.818 n=6)
Query/nh_2000,_range_query_with_1000_steps                                        1.309 ±  2%    1.289 ±  2%   -1.56% (p=0.026 n=6)
Query/a_1[1m],_instant_query                                                     203.8µ ±  3%   180.5µ ±  6%  -11.41% (p=0.002 n=6)
Query/a_100[1m],_instant_query                                                   583.6µ ±  2%   572.6µ ±  1%   -1.89% (p=0.002 n=6)
Query/a_2000[1m],_instant_query                                                  6.755m ±  3%   5.729m ±  1%  -15.19% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                               205.1µ ±  3%   184.2µ ±  4%  -10.16% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                  233.1µ ±  5%   200.7µ ±  5%  -13.91% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                 426.0µ ±  3%   338.0µ ±  3%  -20.66% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                             765.8µ ±  2%   579.7µ ±  3%  -24.30% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                2.519m ±  3%   1.764m ±  2%  -29.97% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                               16.34m ±  3%   10.33m ±  2%  -36.77% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                            9.358m ±  2%   6.639m ±  1%  -29.05% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                               34.88m ± 13%   25.84m ±  2%  -25.92% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                              253.8m ±  1%   157.4m ±  3%  -37.99% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                1.963m ±  2%   1.421m ±  1%  -27.60% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                             128.81m ±  2%   81.30m ±  1%  -36.89% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                              2.726 ±  1%    1.630 ±  2%  -40.21% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                               1.441m ±  3%   1.099m ±  2%  -23.75% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                  2.120m ±  2%   1.598m ±  2%  -24.65% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                 7.155m ±  2%   5.781m ±  2%  -19.20% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                             64.16m ±  2%   54.78m ±  2%  -14.62% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                118.5m ±  1%   100.0m ±  2%  -15.61% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                               577.0m ±  2%   504.2m ±  2%  -12.61% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                           1133.0m ±  3%   959.7m ±  1%  -15.30% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                                2.197 ±  1%    1.853 ±  2%  -15.67% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                              11.433 ±  1%    9.840 ±  1%  -13.93% (p=0.002 n=6)
Query/a_1_-_b_1,_instant_query                                                   357.2µ ±  3%   326.6µ ±  6%   -8.55% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                      407.2µ ±  1%   349.0µ ±  2%  -14.29% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                     891.0µ ±  3%   559.4µ ±  2%  -37.21% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                               2.338m ±  1%   2.323m ±  2%        ~ (p=0.180 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                  6.159m ±  2%   3.826m ±  2%  -37.89% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                 36.81m ±  1%   16.17m ±  4%  -56.09% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                             33.00m ±  2%   32.07m ±  2%   -2.81% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                               106.58m ±  1%   56.55m ±  1%  -46.94% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                               818.0m ±  1%   276.8m ±  2%  -66.16% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    4.875m ±  2%   2.415m ±  1%  -50.46% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                                338.5m ±  2%   134.8m ±  1%  -60.17% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                               8.693 ±  0%    2.675 ±  1%  -69.23% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                2.219m ±  2%   2.204m ±  3%        ~ (p=0.394 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                   4.818m ±  3%   3.377m ±  1%  -29.90% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                  26.86m ±  3%   14.42m ±  2%  -46.32% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                  25.30m ±  2%   21.53m ±  1%  -14.90% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                     72.59m ±  4%   36.64m ±  1%  -49.53% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                    517.3m ±  2%   173.2m ±  2%  -66.52% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                    181.9µ ±  2%   162.6µ ±  1%  -10.63% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                       202.3µ ±  1%   177.8µ ±  2%  -12.10% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                      345.1µ ±  3%   295.1µ ±  2%  -14.50% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                  1.193m ±  2%   1.158m ±  1%   -2.98% (p=0.004 n=6)
Query/sum(a_100),_range_query_with_100_steps                                     1.989m ±  2%   1.925m ±  2%   -3.18% (p=0.004 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                    8.729m ±  5%   8.543m ±  2%   -2.14% (p=0.009 n=6)
Query/sum(a_2000),_instant_query                                                 16.12m ±  3%   15.73m ±  3%   -2.42% (p=0.041 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                    29.80m ±  2%   28.41m ±  2%   -4.68% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                   153.5m ±  2%   143.7m ±  3%   -6.41% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                             249.1µ ±  4%   236.5µ ±  4%   -5.07% (p=0.009 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                316.3µ ±  2%   292.5µ ±  3%   -7.51% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                               874.3µ ±  2%   813.3µ ±  2%   -6.98% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                           5.650m ±  1%   5.559m ±  1%   -1.62% (p=0.015 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                             10.083m ±  2%   9.740m ±  1%   -3.40% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                             51.21m ±  2%   46.22m ±  2%   -9.75% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                          96.32m ±  2%   95.69m ±  3%        ~ (p=0.180 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                             177.6m ±  2%   169.4m ±  1%   -4.63% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                           1154.6m ±  2%   870.4m ±  1%  -24.61% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                            250.0µ ±  3%   231.1µ ±  2%   -7.55% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                               319.4µ ±  3%   301.1µ ±  1%   -5.73% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                              900.7µ ±  1%   840.5µ ±  1%   -6.67% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                          5.448m ±  1%   5.335m ±  2%   -2.06% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                             9.671m ±  1%   9.334m ±  2%   -3.48% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                            48.98m ±  2%   46.28m ±  1%   -5.51% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                         95.01m ±  1%   91.96m ±  1%   -3.21% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                            177.5m ±  2%   166.3m ±  2%   -6.28% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                          1106.8m ±  1%   857.5m ±  4%  -22.52% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                               355.9µ ±  2%   316.2µ ±  3%  -11.16% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                  433.8µ ±  1%   349.7µ ±  2%  -19.39% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                1002.2µ ±  2%   600.3µ ±  1%  -40.11% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                           1.460m ±  2%   1.134m ±  1%  -22.31% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps              6.636m ±  1%   3.205m ±  2%  -51.71% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps             48.90m ±  1%   18.59m ±  1%  -61.99% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                         17.70m ±  1%   13.02m ±  3%  -26.45% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps           118.20m ±  2%   45.63m ±  5%  -61.40% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps          1083.7m ±  2%   317.8m ±  2%  -70.67% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                              366.7µ ±  2%   336.7µ ±  3%   -8.19% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                 422.4µ ±  2%   356.8µ ±  2%  -15.52% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                                945.3µ ±  2%   581.9µ ±  2%  -38.44% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                          2.347m ±  1%   2.296m ±  1%   -2.16% (p=0.009 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                             6.101m ±  1%   3.793m ±  3%  -37.83% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                            37.57m ±  2%   16.63m ±  2%  -55.74% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                        33.04m ±  2%   31.80m ±  2%   -3.76% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                          107.76m ±  1%   57.14m ±  1%  -46.97% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                          840.0m ±  1%   281.6m ±  1%  -66.48% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                  224.8µ ±  2%   191.8µ ±  6%  -14.70% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                     358.3µ ±  2%   274.4µ ±  1%  -23.43% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                   1322.5µ ±  2%   925.7µ ±  1%  -30.00% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                3.075m ±  1%   2.219m ±  1%  -27.83% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                  11.788m ±  1%   7.754m ±  2%  -34.22% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                  85.07m ±  2%   52.19m ±  2%  -38.65% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                               49.76m ±  2%   35.74m ±  2%  -28.18% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                  214.0m ±  1%   137.1m ±  3%  -35.92% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                  1.812 ±  1%    1.000 ±  2%  -44.81% (p=0.002 n=6)
geomean                                                                          10.33m         7.892m        -23.59%

                                                                       │      Prometheus      │               streaming               │
                                                                       │         B/op         │      B/op       vs base               │
Query/a_1,_instant_query                                                         20.68Ki ± 0%    17.75Ki ±  0%  -14.16% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                            21.60Ki ± 0%    18.32Ki ±  0%  -15.19% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                           26.19Ki ± 0%    22.88Ki ±  0%  -12.65% (p=0.002 n=6)
Query/a_100,_instant_query                                                       169.6Ki ± 0%    151.8Ki ±  0%  -10.53% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                          222.4Ki ± 0%    208.6Ki ±  0%   -6.20% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                         658.3Ki ± 0%    645.4Ki ±  0%   -1.96% (p=0.002 n=6)
Query/a_2000,_instant_query                                                      3.019Mi ± 0%    2.690Mi ±  0%  -10.89% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                         3.965Mi ± 0%    3.771Mi ±  1%   -4.90% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                        12.34Mi ± 0%    12.14Mi ±  0%   -1.59% (p=0.002 n=6)
Query/nh_1,_instant_query                                                        37.10Ki ± 0%    34.09Ki ±  0%   -8.10% (p=0.002 n=6)
Query/nh_1,_range_query_with_100_steps                                           88.56Ki ± 0%    85.46Ki ±  0%   -3.50% (p=0.002 n=6)
Query/nh_1,_range_query_with_1000_steps                                          520.0Ki ± 0%    517.4Ki ±  0%   -0.48% (p=0.002 n=6)
Query/nh_100,_instant_query                                                      940.2Ki ± 0%    922.1Ki ±  0%   -1.93% (p=0.002 n=6)
Query/nh_100,_range_query_with_100_steps                                         5.568Mi ± 0%    5.563Mi ±  0%   -0.10% (p=0.002 n=6)
Query/nh_100,_range_query_with_1000_steps                                        47.51Mi ± 0%    47.52Mi ±  0%        ~ (p=0.818 n=6)
Query/nh_2000,_instant_query                                                     17.76Mi ± 0%    17.46Mi ±  0%   -1.68% (p=0.002 n=6)
Query/nh_2000,_range_query_with_100_steps                                        111.6Mi ± 0%    111.5Mi ±  0%        ~ (p=0.240 n=6)
Query/nh_2000,_range_query_with_1000_steps                                       977.2Mi ± 0%    976.8Mi ±  0%   -0.05% (p=0.004 n=6)
Query/a_1[1m],_instant_query                                                     21.19Ki ± 0%    17.38Ki ±  0%  -17.98% (p=0.002 n=6)
Query/a_100[1m],_instant_query                                                   132.7Ki ± 0%    109.2Ki ±  0%  -17.69% (p=0.002 n=6)
Query/a_2000[1m],_instant_query                                                  2.291Mi ± 0%    1.903Mi ±  0%  -16.92% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                               22.47Ki ± 0%    17.82Ki ±  0%  -20.70% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                  23.40Ki ± 1%    18.25Ki ±  0%  -21.98% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                 28.21Ki ± 1%    22.90Ki ±  0%  -18.82% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                             151.8Ki ± 0%    111.5Ki ±  0%  -26.51% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                192.2Ki ± 0%    155.6Ki ±  0%  -19.05% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                               651.5Ki ± 0%    613.3Ki ±  0%   -5.85% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                            2.616Mi ± 0%    1.878Mi ±  0%  -28.21% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                               3.381Mi ± 0%    2.794Mi ±  1%  -17.36% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                              12.08Mi ± 0%    11.51Mi ±  1%   -4.75% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                88.98Ki ± 6%    71.83Ki ±  1%  -19.26% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                              5.204Mi ± 3%    5.021Mi ±  2%   -3.53% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                            1069.4Mi ± 0%    600.0Mi ±  0%  -43.89% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                              669.59Ki ± 1%    71.99Ki ±  2%  -89.25% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                 671.67Ki ± 2%    71.70Ki ±  6%  -89.33% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                630.42Ki ± 4%    73.71Ki ± 11%  -88.31% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                             5.267Mi ± 2%    4.723Mi ±  2%  -10.32% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                5.311Mi ± 3%    4.719Mi ±  4%  -11.15% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                               5.767Mi ± 8%    5.374Mi ±  8%   -6.83% (p=0.026 n=6)
Query/rate(a_2000[1d]),_instant_query                                            90.78Mi ± 1%    88.04Mi ±  1%   -3.02% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                               94.53Mi ± 1%    94.14Mi ±  1%        ~ (p=0.180 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                              128.8Mi ± 0%    128.1Mi ±  0%   -0.57% (p=0.009 n=6)
Query/a_1_-_b_1,_instant_query                                                   42.60Ki ± 0%    36.46Ki ±  0%  -14.40% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                      48.87Ki ± 0%    37.57Ki ±  0%  -23.11% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                    100.78Ki ± 0%    46.61Ki ±  0%  -53.75% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                               473.9Ki ± 0%    338.6Ki ±  0%  -28.55% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                  585.8Ki ± 0%    452.7Ki ±  0%  -22.72% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                 1.472Mi ± 0%    1.296Mi ±  0%  -11.96% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                             8.178Mi ± 0%    5.940Mi ±  0%  -27.37% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                               10.218Mi ± 0%    8.094Mi ±  0%  -20.78% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                               27.94Mi ± 1%    24.82Mi ±  1%  -11.18% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    690.2Ki ± 3%    143.5Ki ±  4%  -79.21% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                               10.689Mi ± 2%    9.966Mi ±  1%   -6.76% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                             3108.2Mi ± 0%    700.4Mi ±  0%  -77.47% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                373.2Ki ± 0%    256.7Ki ±  0%  -31.21% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                   455.5Ki ± 0%    339.3Ki ±  0%  -25.52% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                 1159.1Ki ± 0%    999.3Ki ±  0%  -13.79% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                  6.160Mi ± 0%    3.930Mi ±  0%  -36.20% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                     7.703Mi ± 0%    5.339Mi ±  1%  -30.68% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                    20.74Mi ± 1%    16.40Mi ±  0%  -20.90% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                    21.80Ki ± 0%    18.33Ki ±  0%  -15.90% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                       22.79Ki ± 1%    20.00Ki ±  0%  -12.24% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                      27.42Ki ± 0%    32.59Ki ±  0%  +18.88% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                  174.2Ki ± 0%    153.4Ki ±  0%  -11.96% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                                     230.7Ki ± 0%    210.9Ki ±  0%   -8.57% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                    665.7Ki ± 0%    658.4Ki ±  0%   -1.10% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                                 3.082Mi ± 0%    2.677Mi ±  0%  -13.16% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                    4.109Mi ± 0%    3.711Mi ±  0%   -9.69% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                   12.48Mi ± 0%    12.24Mi ±  1%   -1.92% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                             29.75Ki ± 1%    25.34Ki ±  0%  -14.82% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                33.43Ki ± 1%    29.69Ki ±  0%  -11.19% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                               60.69Ki ± 0%    64.84Ki ±  0%   +6.83% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                          1120.5Ki ± 0%    977.8Ki ±  0%  -12.73% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                              1.399Mi ± 0%    1.374Mi ±  0%   -1.79% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                             3.926Mi ± 1%    4.722Mi ±  0%  +20.29% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                          22.49Mi ± 0%    19.57Mi ±  0%  -12.98% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                             28.32Mi ± 0%    29.07Mi ±  1%   +2.62% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                            298.8Mi ± 0%    105.6Mi ±  4%  -64.66% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                            31.20Ki ± 0%    25.50Ki ±  0%  -18.28% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                               34.77Ki ± 1%    35.28Ki ±  0%   +1.46% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                              62.03Ki ± 0%   110.46Ki ±  0%  +78.09% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                         1079.9Ki ± 0%    949.4Ki ±  0%  -12.08% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                             1.365Mi ± 0%    1.248Mi ±  0%   -8.54% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                            3.891Mi ± 1%    3.852Mi ±  0%   -1.00% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                         21.67Mi ± 0%    19.03Mi ±  0%  -12.20% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                            27.63Mi ± 0%    25.26Mi ±  0%   -8.58% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                          266.90Mi ± 0%    76.10Mi ±  0%  -71.49% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                               46.26Ki ± 0%    36.54Ki ±  0%  -21.01% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                  52.31Ki ± 0%    37.42Ki ±  0%  -28.47% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                104.59Ki ± 0%    46.63Ki ±  0%  -55.41% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                           431.9Ki ± 0%    253.0Ki ±  0%  -41.43% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps              519.6Ki ± 1%    341.5Ki ±  0%  -34.28% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps             1.448Mi ± 0%    1.229Mi ±  0%  -15.09% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                         7.275Mi ± 0%    4.233Mi ±  0%  -41.81% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps            8.953Mi ± 1%    6.031Mi ±  1%  -32.64% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps          121.77Mi ± 0%    23.45Mi ±  2%  -80.74% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                              43.92Ki ± 0%    37.09Ki ±  0%  -15.56% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                 50.17Ki ± 1%    39.29Ki ±  0%  -21.68% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                               102.10Ki ± 1%    56.42Ki ±  0%  -44.74% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                          478.6Ki ± 0%    340.4Ki ±  0%  -28.88% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                             595.3Ki ± 0%    453.9Ki ±  0%  -23.76% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                            1.479Mi ± 0%    1.309Mi ±  0%  -11.48% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                        8.249Mi ± 0%    5.909Mi ±  0%  -28.37% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                          10.358Mi ± 0%    8.020Mi ±  1%  -22.57% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                          28.06Mi ± 0%    25.05Mi ±  1%  -10.74% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                  32.43Ki ± 0%    23.42Ki ±  0%  -27.76% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                     34.89Ki ± 1%    32.49Ki ±  0%   -6.88% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                    62.62Ki ± 0%   108.41Ki ±  0%  +73.13% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                953.3Ki ± 0%    707.4Ki ±  0%  -25.79% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                  1216.0Ki ± 0%    977.7Ki ±  0%  -19.60% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                  3.859Mi ± 0%    3.654Mi ±  0%   -5.32% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                               19.17Mi ± 0%    14.16Mi ±  0%  -26.09% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                  24.05Mi ± 0%    19.41Mi ±  1%  -19.29% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                265.37Mi ± 0%    73.03Mi ±  1%  -72.48% (p=0.002 n=6)
geomean                                                                          1.242Mi         962.2Ki        -24.34%

                                                                       │      Prometheus      │             streaming              │
                                                                       │      allocs/op       │  allocs/op   vs base               │
Query/a_1,_instant_query                                                           370.0 ± 0%    313.0 ± 0%  -15.41% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                              380.0 ± 0%    317.0 ± 0%  -16.58% (p=0.002 n=6)
Query/a_1,_range_query_with_1000_steps                                             412.0 ± 0%    349.0 ± 0%  -15.29% (p=0.002 n=6)
Query/a_100,_instant_query                                                        2.287k ± 0%   2.222k ± 0%   -2.84% (p=0.002 n=6)
Query/a_100,_range_query_with_100_steps                                           2.700k ± 0%   2.630k ± 0%   -2.59% (p=0.002 n=6)
Query/a_100,_range_query_with_1000_steps                                          5.907k ± 0%   5.838k ± 0%   -1.16% (p=0.002 n=6)
Query/a_2000,_instant_query                                                       38.89k ± 0%   38.80k ± 0%   -0.24% (p=0.002 n=6)
Query/a_2000,_range_query_with_100_steps                                          46.92k ± 0%   46.84k ± 0%   -0.16% (p=0.002 n=6)
Query/a_2000,_range_query_with_1000_steps                                         110.9k ± 0%   110.9k ± 0%   -0.07% (p=0.002 n=6)
Query/nh_1,_instant_query                                                          638.0 ± 0%    581.0 ± 0%   -8.93% (p=0.002 n=6)
Query/nh_1,_range_query_with_100_steps                                            1.716k ± 0%   1.653k ± 0%   -3.67% (p=0.002 n=6)
Query/nh_1,_range_query_with_1000_steps                                           10.79k ± 0%   10.73k ± 0%   -0.57% (p=0.002 n=6)
Query/nh_100,_instant_query                                                       19.20k ± 0%   19.14k ± 0%   -0.36% (p=0.002 n=6)
Query/nh_100,_range_query_with_100_steps                                          120.1k ± 0%   120.0k ± 0%   -0.05% (p=0.002 n=6)
Query/nh_100,_range_query_with_1000_steps                                         1.027M ± 0%   1.027M ± 0%   -0.01% (p=0.002 n=6)
Query/nh_2000,_instant_query                                                      375.7k ± 0%   375.0k ± 0%   -0.17% (p=0.002 n=6)
Query/nh_2000,_range_query_with_100_steps                                         2.393M ± 0%   2.393M ± 0%   -0.01% (p=0.015 n=6)
Query/nh_2000,_range_query_with_1000_steps                                        20.54M ± 0%   20.54M ± 0%   -0.00% (p=0.002 n=6)
Query/a_1[1m],_instant_query                                                       380.0 ± 0%    310.0 ± 0%  -18.42% (p=0.002 n=6)
Query/a_100[1m],_instant_query                                                    1.903k ± 0%   1.819k ± 0%   -4.41% (p=0.002 n=6)
Query/a_2000[1m],_instant_query                                                   30.85k ± 0%   30.76k ± 0%   -0.30% (p=0.002 n=6)
Query/rate(a_1[1m]),_instant_query                                                 409.0 ± 0%    320.0 ± 0%  -21.76% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                    419.0 ± 0%    324.0 ± 0%  -22.67% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                   452.0 ± 0%    357.0 ± 0%  -21.02% (p=0.002 n=6)
Query/rate(a_100[1m]),_instant_query                                              2.324k ± 0%   1.927k ± 0%  -17.08% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                 2.736k ± 0%   2.334k ± 0%  -14.69% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                                6.051k ± 0%   5.650k ± 0%   -6.63% (p=0.002 n=6)
Query/rate(a_2000[1m]),_instant_query                                             38.88k ± 0%   32.75k ± 0%  -15.75% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                                46.95k ± 0%   40.84k ± 0%  -13.01% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                               113.0k ± 0%   106.9k ± 0%   -5.42% (p=0.002 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                  761.0 ± 0%    665.0 ± 0%  -12.61% (p=0.002 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                               36.41k ± 0%   36.01k ± 0%   -1.11% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                              732.3k ± 0%   720.0k ± 0%   -1.68% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                                 724.0 ± 0%    618.0 ± 0%  -14.64% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                    732.5 ± 0%    621.5 ± 0%  -15.15% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                   760.0 ± 0%    653.0 ± 0%  -14.08% (p=0.002 n=6)
Query/rate(a_100[1d]),_instant_query                                              32.26k ± 0%   31.83k ± 0%   -1.33% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                 32.56k ± 0%   32.15k ± 0%   -1.26% (p=0.002 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                                35.59k ± 0%   35.21k ± 0%   -1.05% (p=0.002 n=6)
Query/rate(a_2000[1d]),_instant_query                                             641.9k ± 0%   629.9k ± 0%   -1.86% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                                648.0k ± 0%   641.8k ± 0%   -0.95% (p=0.002 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                               708.2k ± 0%   702.1k ± 0%   -0.86% (p=0.002 n=6)
Query/a_1_-_b_1,_instant_query                                                     740.0 ± 0%    645.0 ± 0%  -12.84% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                        952.0 ± 0%    653.0 ± 0%  -31.41% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                      2817.0 ± 0%    718.0 ± 0%  -74.51% (p=0.002 n=6)
Query/a_100_-_b_100,_instant_query                                                5.198k ± 0%   5.270k ± 0%   +1.39% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                   6.016k ± 0%   6.085k ± 0%   +1.16% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                  14.24k ± 0%   12.50k ± 0%  -12.17% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                              89.80k ± 0%   93.61k ± 0%   +4.25% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                                 102.1k ± 0%   109.7k ± 0%   +7.48% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                                232.0k ± 0%   237.8k ± 0%   +2.48% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    21.431k ± 0%   1.328k ± 0%  -93.80% (p=0.002 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                                 92.34k ± 0%   72.61k ± 0%  -21.37% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                               1.484M ± 0%   1.446M ± 0%   -2.58% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                 4.044k ± 0%   4.015k ± 0%   -0.72% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                    4.711k ± 0%   4.629k ± 0%   -1.74% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                  11.329k ± 0%   9.447k ± 0%  -16.61% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                   66.59k ± 0%   58.17k ± 0%  -12.64% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                      75.81k ± 0%   67.35k ± 0%  -11.16% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                     173.7k ± 0%   136.4k ± 0%  -21.47% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                      394.0 ± 0%    322.0 ± 0%  -18.27% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_100_steps                                         404.0 ± 0%    330.0 ± 0%  -18.32% (p=0.002 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                        436.0 ± 0%    362.0 ± 0%  -16.97% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                   2.312k ± 0%   2.232k ± 0%   -3.46% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_100_steps                                      2.724k ± 0%   2.643k ± 0%   -2.97% (p=0.002 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                     5.931k ± 0%   5.849k ± 0%   -1.37% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                                  38.92k ± 0%   38.82k ± 0%   -0.26% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                     46.94k ± 0%   46.84k ± 0%   -0.21% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                    111.0k ± 0%   110.9k ± 0%   -0.04% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                               507.0 ± 0%    429.0 ± 0%  -15.38% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                  537.0 ± 0%    457.0 ± 0%  -14.90% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                                 729.0 ± 0%    650.0 ± 0%  -10.84% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                            12.69k ± 0%   12.59k ± 0%   -0.77% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                               15.10k ± 0%   15.41k ± 0%   +2.09% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                              34.32k ± 0%   34.66k ± 0%   +0.99% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                           260.1k ± 1%   246.4k ± 0%   -5.25% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                              293.5k ± 0%   303.8k ± 0%   +3.53% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                             707.6k ± 0%   691.2k ± 0%   -2.33% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                              515.0 ± 0%    434.0 ± 0%  -15.73% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                                 545.0 ± 0%    483.0 ± 0%  -11.38% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                                737.0 ± 0%    675.0 ± 0%   -8.41% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                           12.57k ± 0%   12.48k ± 0%   -0.73% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                              14.98k ± 0%   14.91k ± 0%   -0.48% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                             34.21k ± 0%   34.15k ± 0%   -0.17% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                          255.3k ± 1%   243.3k ± 0%   -4.68% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                             291.4k ± 0%   291.4k ± 0%        ~ (p=0.390 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                            701.6k ± 0%   675.7k ± 0%   -3.69% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                                 810.0 ± 0%    657.0 ± 0%  -18.89% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                   1023.0 ± 0%    665.0 ± 0%  -35.00% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                  2889.0 ± 0%    731.0 ± 0%  -74.70% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                            5.268k ± 0%   4.681k ± 0%  -11.14% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps               6.084k ± 0%   5.494k ± 0%   -9.70% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps              14.52k ± 0%   12.13k ± 0%  -16.47% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                          89.80k ± 0%   81.57k ± 0%   -9.17% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps            102.07k ± 0%   97.66k ± 0%   -4.32% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps            250.0k ± 0%   229.8k ± 0%   -8.08% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                                764.0 ± 0%    655.0 ± 0%  -14.27% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                   976.0 ± 0%    667.0 ± 0%  -31.66% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                                 2841.0 ± 0%    731.0 ± 0%  -74.27% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                           5.223k ± 0%   5.281k ± 0%   +1.12% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                              6.040k ± 0%   6.098k ± 0%   +0.96% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                             14.26k ± 0%   12.52k ± 0%  -12.21% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                         89.83k ± 0%   93.63k ± 0%   +4.24% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                            102.1k ± 0%   109.7k ± 0%   +7.45% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                           232.1k ± 0%   237.9k ± 0%   +2.49% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                    555.0 ± 0%    420.0 ± 0%  -24.32% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                       577.0 ± 0%    469.0 ± 0%  -18.72% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                      776.0 ± 0%    668.0 ± 0%  -13.92% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                 12.61k ± 0%   10.67k ± 0%  -15.34% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                    15.02k ± 0%   13.12k ± 0%  -12.69% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                   34.87k ± 0%   32.97k ± 0%   -5.46% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                                252.4k ± 1%   207.0k ± 0%  -17.97% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                   291.5k ± 0%   255.4k ± 0%  -12.40% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                  713.7k ± 0%   652.6k ± 0%   -8.56% (p=0.002 n=6)
geomean                                                                           13.41k        11.62k       -13.35%

                                                                       │      Prometheus      │              streaming               │
                                                                       │          B           │       B        vs base               │
Query/a_1,_instant_query                                                         74.50Ki ± 1%   73.19Ki ±  1%   -1.76% (p=0.002 n=6)
Query/a_1,_range_query_with_100_steps                                            74.31Ki ± 1%   72.93Ki ±  2%   -1.86% (p=0.039 n=6)
Query/a_1,_range_query_with_1000_steps                                           72.49Ki ± 2%   71.19Ki ±  2%   -1.81% (p=0.026 n=6)
Query/a_100,_instant_query                                                       67.75Ki ± 2%   67.16Ki ±  1%        ~ (p=0.143 n=6)
Query/a_100,_range_query_with_100_steps                                          67.39Ki ± 2%   67.35Ki ±  2%        ~ (p=0.937 n=6)
Query/a_100,_range_query_with_1000_steps                                         69.89Ki ± 1%   69.30Ki ±  1%        ~ (p=0.394 n=6)
Query/a_2000,_instant_query                                                      68.68Ki ± 1%   68.36Ki ±  1%        ~ (p=0.180 n=6)
Query/a_2000,_range_query_with_100_steps                                         75.65Ki ± 2%   76.92Ki ±  1%   +1.68% (p=0.004 n=6)
Query/a_2000,_range_query_with_1000_steps                                        135.6Ki ± 1%   136.0Ki ±  1%        ~ (p=0.167 n=6)
Query/nh_1,_instant_query                                                        82.00Ki ± 3%   82.31Ki ±  1%        ~ (p=0.574 n=6)
Query/nh_1,_range_query_with_100_steps                                           74.45Ki ± 2%   74.19Ki ±  2%        ~ (p=0.485 n=6)
Query/nh_1,_range_query_with_1000_steps                                          72.19Ki ± 2%   71.07Ki ±  1%        ~ (p=0.065 n=6)
Query/nh_100,_instant_query                                                      68.68Ki ± 2%   68.63Ki ±  1%        ~ (p=0.818 n=6)
Query/nh_100,_range_query_with_100_steps                                         73.36Ki ± 1%   73.59Ki ±  1%        ~ (p=0.725 n=6)
Query/nh_100,_range_query_with_1000_steps                                        118.4Ki ± 0%   118.2Ki ±  0%        ~ (p=0.699 n=6)
Query/nh_2000,_instant_query                                                     70.72Ki ± 2%   70.27Ki ±  1%        ~ (p=0.485 n=6)
Query/nh_2000,_range_query_with_100_steps                                        172.6Ki ± 1%   174.6Ki ±  1%   +1.16% (p=0.002 n=6)
Query/nh_2000,_range_query_with_1000_steps                                       706.8Ki ± 3%   715.1Ki ±  4%        ~ (p=1.000 n=6)
Query/a_1[1m],_instant_query                                                     73.56Ki ± 2%   72.56Ki ±  1%        ~ (p=0.331 n=6)
Query/a_100[1m],_instant_query                                                   67.68Ki ± 2%   67.46Ki ±  1%        ~ (p=0.589 n=6)
Query/a_2000[1m],_instant_query                                                  70.67Ki ± 1%   70.34Ki ±  1%        ~ (p=0.093 n=6)
Query/rate(a_1[1m]),_instant_query                                               72.84Ki ± 1%   72.88Ki ±  3%        ~ (p=0.851 n=6)
Query/rate(a_1[1m]),_range_query_with_100_steps                                  72.69Ki ± 1%   73.07Ki ±  2%        ~ (p=0.485 n=6)
Query/rate(a_1[1m]),_range_query_with_1000_steps                                 70.75Ki ± 1%   70.72Ki ±  2%        ~ (p=0.788 n=6)
Query/rate(a_100[1m]),_instant_query                                             67.58Ki ± 1%   67.64Ki ±  2%        ~ (p=0.818 n=6)
Query/rate(a_100[1m]),_range_query_with_100_steps                                67.50Ki ± 2%   66.71Ki ±  2%        ~ (p=0.065 n=6)
Query/rate(a_100[1m]),_range_query_with_1000_steps                               68.62Ki ± 1%   69.48Ki ±  1%   +1.25% (p=0.004 n=6)
Query/rate(a_2000[1m]),_instant_query                                            69.27Ki ± 2%   69.42Ki ±  1%        ~ (p=0.699 n=6)
Query/rate(a_2000[1m]),_range_query_with_100_steps                               75.45Ki ± 1%   76.88Ki ±  1%   +1.89% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_1000_steps                              130.6Ki ± 1%   131.8Ki ±  1%   +0.99% (p=0.004 n=6)
Query/rate(a_1[1m]),_range_query_with_10000_steps                                69.11Ki ± 2%   69.33Ki ±  1%        ~ (p=0.675 n=6)
Query/rate(a_100[1m]),_range_query_with_10000_steps                              100.5Ki ± 0%   108.8Ki ±  1%   +8.24% (p=0.002 n=6)
Query/rate(a_2000[1m]),_range_query_with_10000_steps                             547.8Ki ± 5%   426.8Ki ±  2%  -22.08% (p=0.002 n=6)
Query/rate(a_1[1d]),_instant_query                                               68.60Ki ± 1%   73.50Ki ±  1%   +7.14% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_100_steps                                  68.46Ki ± 1%   71.72Ki ±  2%   +4.76% (p=0.002 n=6)
Query/rate(a_1[1d]),_range_query_with_1000_steps                                 67.25Ki ± 1%   68.94Ki ±  2%   +2.50% (p=0.017 n=6)
Query/rate(a_100[1d]),_instant_query                                             71.87Ki ± 2%   73.06Ki ±  2%        ~ (p=0.126 n=6)
Query/rate(a_100[1d]),_range_query_with_100_steps                                70.94Ki ± 2%   73.03Ki ±  5%        ~ (p=0.104 n=6)
Query/rate(a_100[1d]),_range_query_with_1000_steps                               66.31Ki ± 2%   65.94Ki ±  7%        ~ (p=0.931 n=6)
Query/rate(a_2000[1d]),_instant_query                                            78.38Ki ± 3%   79.86Ki ±  2%        ~ (p=0.093 n=6)
Query/rate(a_2000[1d]),_range_query_with_100_steps                               80.65Ki ± 2%   81.81Ki ±  2%        ~ (p=0.180 n=6)
Query/rate(a_2000[1d]),_range_query_with_1000_steps                              113.8Ki ± 3%   113.3Ki ±  3%        ~ (p=0.461 n=6)
Query/a_1_-_b_1,_instant_query                                                   72.62Ki ± 1%   73.19Ki ±  2%        ~ (p=0.667 n=6)
Query/a_1_-_b_1,_range_query_with_100_steps                                      71.94Ki ± 1%   72.00Ki ±  2%        ~ (p=0.519 n=6)
Query/a_1_-_b_1,_range_query_with_1000_steps                                     69.89Ki ± 1%   70.29Ki ±  1%        ~ (p=0.310 n=6)
Query/a_100_-_b_100,_instant_query                                               67.86Ki ± 1%   67.59Ki ±  1%        ~ (p=0.132 n=6)
Query/a_100_-_b_100,_range_query_with_100_steps                                  68.24Ki ± 1%   68.00Ki ±  1%        ~ (p=0.394 n=6)
Query/a_100_-_b_100,_range_query_with_1000_steps                                 75.61Ki ± 2%   71.00Ki ±  1%   -6.10% (p=0.002 n=6)
Query/a_2000_-_b_2000,_instant_query                                             70.00Ki ± 1%   71.12Ki ±  1%   +1.60% (p=0.026 n=6)
Query/a_2000_-_b_2000,_range_query_with_100_steps                                93.87Ki ± 1%   80.00Ki ±  1%  -14.77% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps                               207.9Ki ± 1%   142.2Ki ±  2%  -31.62% (p=0.002 n=6)
Query/a_1_-_b_1,_range_query_with_10000_steps                                    69.82Ki ± 1%   69.91Ki ±  1%        ~ (p=0.818 n=6)
Query/a_100_-_b_100,_range_query_with_10000_steps                                141.7Ki ± 2%   119.8Ki ±  3%  -15.50% (p=0.002 n=6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps                             1615.4Ki ± 2%   459.0Ki ±  3%  -71.59% (p=0.002 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_instant_query                                67.64Ki ± 2%   67.81Ki ±  2%        ~ (p=0.851 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_100_steps                   67.87Ki ± 2%   67.42Ki ±  1%        ~ (p=0.240 n=6)
Query/a_100{l=~"[13579]."}_-_b_100,_range_query_with_1000_steps                  71.93Ki ± 2%   68.05Ki ±  1%   -5.40% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query                                  70.06Ki ± 2%   97.06Ki ±  3%  +38.54% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps                     84.11Ki ± 1%   99.19Ki ± 32%  +17.93% (p=0.002 n=6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps                    160.4Ki ± 0%   124.4Ki ±  8%  -22.45% (p=0.002 n=6)
Query/sum(a_1),_instant_query                                                    73.88Ki ± 2%   73.06Ki ±  3%        ~ (p=0.071 n=6)
Query/sum(a_1),_range_query_with_100_steps                                       72.75Ki ± 1%   74.10Ki ±  2%   +1.86% (p=0.009 n=6)
Query/sum(a_1),_range_query_with_1000_steps                                      71.67Ki ± 1%   78.88Ki ±  2%  +10.06% (p=0.002 n=6)
Query/sum(a_100),_instant_query                                                  66.99Ki ± 1%   67.00Ki ±  1%        ~ (p=0.937 n=6)
Query/sum(a_100),_range_query_with_100_steps                                     68.25Ki ± 1%   66.89Ki ±  1%   -1.99% (p=0.026 n=6)
Query/sum(a_100),_range_query_with_1000_steps                                    69.79Ki ± 2%   67.47Ki ±  0%   -3.33% (p=0.002 n=6)
Query/sum(a_2000),_instant_query                                                 68.79Ki ± 2%   67.79Ki ±  1%        ~ (p=0.093 n=6)
Query/sum(a_2000),_range_query_with_100_steps                                    76.46Ki ± 3%   67.95Ki ±  2%  -11.12% (p=0.002 n=6)
Query/sum(a_2000),_range_query_with_1000_steps                                  135.94Ki ± 1%   70.82Ki ±  1%  -47.90% (p=0.002 n=6)
Query/sum_by_(l)(h_1),_instant_query                                             70.93Ki ± 2%   70.50Ki ±  1%        ~ (p=0.284 n=6)
Query/sum_by_(l)(h_1),_range_query_with_100_steps                                70.99Ki ± 2%   70.92Ki ±  1%        ~ (p=0.799 n=6)
Query/sum_by_(l)(h_1),_range_query_with_1000_steps                               68.18Ki ± 1%   71.19Ki ±  1%   +4.41% (p=0.002 n=6)
Query/sum_by_(l)(h_100),_instant_query                                           67.54Ki ± 1%   66.95Ki ±  1%        ~ (p=0.180 n=6)
Query/sum_by_(l)(h_100),_range_query_with_100_steps                              70.45Ki ± 1%   69.50Ki ±  1%   -1.36% (p=0.009 n=6)
Query/sum_by_(l)(h_100),_range_query_with_1000_steps                             91.50Ki ± 2%   77.39Ki ±  1%  -15.43% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_instant_query                                          73.78Ki ± 2%   72.58Ki ±  1%        ~ (p=0.065 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_100_steps                            122.00Ki ± 0%   83.06Ki ±  2%  -31.92% (p=0.002 n=6)
Query/sum_by_(l)(h_2000),_range_query_with_1000_steps                            299.5Ki ± 0%   153.2Ki ±  4%  -48.84% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_instant_query                                            70.87Ki ± 2%   70.69Ki ±  2%        ~ (p=0.818 n=6)
Query/sum_by_(le)(h_1),_range_query_with_100_steps                               70.33Ki ± 1%   75.17Ki ±  2%   +6.87% (p=0.002 n=6)
Query/sum_by_(le)(h_1),_range_query_with_1000_steps                              68.62Ki ± 2%   89.20Ki ±  1%  +30.00% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_instant_query                                          67.83Ki ± 1%   66.63Ki ±  2%        ~ (p=0.093 n=6)
Query/sum_by_(le)(h_100),_range_query_with_100_steps                             70.74Ki ± 1%   67.58Ki ±  1%   -4.47% (p=0.002 n=6)
Query/sum_by_(le)(h_100),_range_query_with_1000_steps                            88.97Ki ± 1%   69.43Ki ±  1%  -21.96% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_instant_query                                         73.74Ki ± 2%   71.68Ki ±  1%   -2.79% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_100_steps                           114.60Ki ± 1%   71.63Ki ±  1%  -37.49% (p=0.002 n=6)
Query/sum_by_(le)(h_2000),_range_query_with_1000_steps                          267.00Ki ± 1%   75.02Ki ±  1%  -71.90% (p=0.002 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_instant_query                               72.24Ki ± 1%   72.37Ki ±  2%        ~ (p=0.699 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_100_steps                  71.42Ki ± 1%   71.75Ki ±  1%        ~ (p=0.310 n=6)
Query/rate(a_1[1m])_+_rate(b_1[1m]),_range_query_with_1000_steps                 69.40Ki ± 1%   70.75Ki ±  1%   +1.95% (p=0.002 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_instant_query                           67.72Ki ± 1%   68.03Ki ±  1%        ~ (p=0.394 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_100_steps              68.01Ki ± 1%   67.61Ki ±  1%        ~ (p=0.180 n=6)
Query/rate(a_100[1m])_+_rate(b_100[1m]),_range_query_with_1000_steps             74.88Ki ± 2%   70.06Ki ±  1%   -6.43% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_instant_query                         70.16Ki ± 2%   70.21Ki ±  2%        ~ (p=0.699 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_100_steps            91.64Ki ± 1%   78.77Ki ±  2%  -14.04% (p=0.002 n=6)
Query/rate(a_2000[1m])_+_rate(b_2000[1m]),_range_query_with_1000_steps           156.6Ki ± 1%   142.3Ki ±  4%   -9.14% (p=0.002 n=6)
Query/sum(a_1_+_b_1),_instant_query                                              72.66Ki ± 2%   72.62Ki ±  1%        ~ (p=0.786 n=6)
Query/sum(a_1_+_b_1),_range_query_with_100_steps                                 72.24Ki ± 2%   73.18Ki ±  1%   +1.31% (p=0.026 n=6)
Query/sum(a_1_+_b_1),_range_query_with_1000_steps                                69.58Ki ± 1%   74.69Ki ±  2%   +7.34% (p=0.002 n=6)
Query/sum(a_100_+_b_100),_instant_query                                          67.78Ki ± 2%   67.46Ki ±  1%        ~ (p=0.589 n=6)
Query/sum(a_100_+_b_100),_range_query_with_100_steps                             68.64Ki ± 2%   67.51Ki ±  1%   -1.65% (p=0.026 n=6)
Query/sum(a_100_+_b_100),_range_query_with_1000_steps                            75.36Ki ± 2%   67.62Ki ±  1%  -10.26% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_instant_query                                        70.38Ki ± 1%   70.31Ki ±  1%        ~ (p=0.556 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_100_steps                           92.66Ki ± 1%   70.03Ki ±  1%  -24.43% (p=0.002 n=6)
Query/sum(a_2000_+_b_2000),_range_query_with_1000_steps                         207.88Ki ± 1%   79.93Ki ±  1%  -61.55% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_instant_query                                  71.46Ki ± 2%   71.30Ki ±  1%        ~ (p=0.513 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_100_steps                     70.25Ki ± 2%   75.53Ki ±  0%   +7.51% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_1[1m])),_range_query_with_1000_steps                    68.05Ki ± 1%   89.29Ki ±  1%  +31.20% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_instant_query                                68.01Ki ± 1%   67.85Ki ±  1%        ~ (p=0.818 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_100_steps                   69.39Ki ± 3%   67.31Ki ±  2%   -3.00% (p=0.015 n=6)
Query/sum_by_(le)(rate(h_100[1m])),_range_query_with_1000_steps                  87.73Ki ± 1%   69.31Ki ±  2%  -21.00% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_instant_query                               73.45Ki ± 1%   73.75Ki ±  1%        ~ (p=0.394 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_100_steps                 115.38Ki ± 1%   71.73Ki ±  1%  -37.83% (p=0.002 n=6)
Query/sum_by_(le)(rate(h_2000[1m])),_range_query_with_1000_steps                266.31Ki ± 1%   73.55Ki ±  4%  -72.38% (p=0.002 n=6)
geomean                                                                          86.94Ki        80.49Ki         -7.43%

charleskorn

Changes overall look good to me 🚀

Could you please add this PR to the existing changelog entry for the new PromQL engine?

I'm not expecting any dramatic differences, but could you also please compare benchmark results for the non-native histogram selector cases for the new PromQL engine before and after this change? (eg. a_2000,_range_query_with_1000_steps) Just want to make sure there are no unexpected significant regressions.

pkg/streamingpromql/benchmarks/comparison_test.go

pkg/streamingpromql/benchmarks/ingester.go

jhesketh · 2024-05-29T04:09:02Z

Benchmarking against main shows no difference within a margin of error:


goos: linux
goarch: amd64
pkg: github.com/grafana/mimir/pkg/streamingpromql/benchmarks
cpu: 12th Gen Intel(R) Core(TM) i7-1270P
                                                                                     │ benchmark-main.txt │           benchmark-nh.txt           │
                                                                                     │       sec/op       │    sec/op     vs base                │
Query/a_2000,_instant_query/streaming-16                                                     13.61m ± ∞ ¹   18.31m ±  3%       ~ (p=0.286 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                    14.41m ± ∞ ¹   18.54m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                        26.60m ± ∞ ¹   32.01m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                       27.21m ± ∞ ¹   32.44m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                       141.6m ± ∞ ¹   155.4m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                      140.8m ± ∞ ¹   156.2m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                 5.081m ± ∞ ¹   5.729m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                                5.999m ± ∞ ¹   6.755m ±  3%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                            30.45m ± ∞ ¹   32.07m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                           31.80m ± ∞ ¹   33.00m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                               64.36m ± ∞ ¹   56.55m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                              123.1m ± ∞ ¹   106.6m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                              329.8m ± ∞ ¹   276.8m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                            1028.3m ± ∞ ¹   818.0m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                              2.964 ± ∞ ¹    2.675 ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                             9.206 ± ∞ ¹    8.693 ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                                 20.06m ± ∞ ¹   21.53m ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                                24.13m ± ∞ ¹   25.30m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                    36.30m ± ∞ ¹   36.64m ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                   75.71m ± ∞ ¹   72.59m ±  4%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                   184.2m ± ∞ ¹   173.2m ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                  543.6m ± ∞ ¹   517.3m ±  2%       ~ (p=0.286 n=1+6)

                                                                                     │ benchmark-main.txt │           benchmark-nh.txt            │
                                                                                     │        B/op        │     B/op       vs base                │
Query/a_2000,_instant_query/streaming-16                                                    2.684Mi ± ∞ ¹   2.690Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                   3.019Mi ± ∞ ¹   3.019Mi ±  0%       ~ (p=1.000 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                       3.777Mi ± ∞ ¹   3.771Mi ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                      4.010Mi ± ∞ ¹   3.965Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                      12.26Mi ± ∞ ¹   12.14Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                     12.35Mi ± ∞ ¹   12.34Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                1.921Mi ± ∞ ¹   1.903Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                               2.287Mi ± ∞ ¹   2.291Mi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                           5.942Mi ± ∞ ¹   5.940Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                          8.178Mi ± ∞ ¹   8.178Mi ±  0%       ~ (p=1.000 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                              8.071Mi ± ∞ ¹   8.094Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                             10.16Mi ± ∞ ¹   10.22Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                             24.85Mi ± ∞ ¹   24.82Mi ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                           121.77Mi ± ∞ ¹   27.94Mi ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                            701.8Mi ± ∞ ¹   700.4Mi ±  0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                           3.036Gi ± ∞ ¹   3.035Gi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                                3.937Mi ± ∞ ¹   3.930Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                               6.164Mi ± ∞ ¹   6.160Mi ±  0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                   5.301Mi ± ∞ ¹   5.339Mi ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                  7.707Mi ± ∞ ¹   7.703Mi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                  16.38Mi ± ∞ ¹   16.40Mi ±  0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                 20.78Mi ± ∞ ¹   20.74Mi ±  1%       ~ (p=0.571 n=1+6)


                                                                                     │ benchmark-main.txt │          benchmark-nh.txt           │
                                                                                     │     allocs/op      │  allocs/op   vs base                │
Query/a_2000,_instant_query/streaming-16                                                     38.80k ± ∞ ¹   38.80k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                    38.87k ± ∞ ¹   38.89k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                        46.85k ± ∞ ¹   46.84k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                       46.97k ± ∞ ¹   46.92k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                       110.9k ± ∞ ¹   110.9k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                      111.0k ± ∞ ¹   110.9k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                 30.75k ± ∞ ¹   30.76k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                                30.84k ± ∞ ¹   30.85k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                            93.63k ± ∞ ¹   93.61k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                           89.79k ± ∞ ¹   89.80k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                               109.7k ± ∞ ¹   109.7k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                              102.1k ± ∞ ¹   102.1k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                              237.8k ± ∞ ¹   237.8k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                             245.9k ± ∞ ¹   232.0k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                             1.446M ± ∞ ¹   1.446M ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                            1.485M ± ∞ ¹   1.484M ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                                 58.17k ± ∞ ¹   58.17k ± 0%       ~ (p=1.000 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                                66.59k ± ∞ ¹   66.59k ± 0%       ~ (p=1.000 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                    67.34k ± ∞ ¹   67.35k ± 0%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                   75.81k ± ∞ ¹   75.81k ± 0%       ~ (p=0.857 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                   136.5k ± ∞ ¹   136.4k ± 0%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                  173.7k ± ∞ ¹   173.7k ± 0%       ~ (p=1.000 n=1+6)

                                                                                     │ benchmark-main.txt │           benchmark-nh.txt            │
                                                                                     │         B          │       B        vs base                │
Query/a_2000,_instant_query/streaming-16                                                    68.55Ki ± ∞ ¹   68.36Ki ±  1%       ~ (p=0.857 n=1+6)
Query/a_2000,_instant_query/Prometheus-16                                                   68.18Ki ± ∞ ¹   68.68Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_100_steps/streaming-16                                       77.14Ki ± ∞ ¹   76.92Ki ±  1%       ~ (p=0.857 n=1+6)
Query/a_2000,_range_query_with_100_steps/Prometheus-16                                      75.49Ki ± ∞ ¹   75.65Ki ±  2%       ~ (p=0.857 n=1+6)
Query/a_2000,_range_query_with_1000_steps/streaming-16                                      136.2Ki ± ∞ ¹   136.0Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000,_range_query_with_1000_steps/Prometheus-16                                     134.5Ki ± ∞ ¹   135.6Ki ±  1%       ~ (p=0.571 n=1+6)
Query/a_2000[1m],_instant_query/streaming-16                                                70.96Ki ± ∞ ¹   70.34Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000[1m],_instant_query/Prometheus-16                                               70.71Ki ± ∞ ¹   70.67Ki ±  1%       ~ (p=1.000 n=1+6)
Query/a_2000_-_b_2000,_instant_query/streaming-16                                           69.43Ki ± ∞ ¹   71.12Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_instant_query/Prometheus-16                                          69.84Ki ± ∞ ¹   70.00Ki ±  1%       ~ (p=0.857 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/streaming-16                              78.62Ki ± ∞ ¹   80.00Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_100_steps/Prometheus-16                             92.00Ki ± ∞ ¹   93.87Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/streaming-16                             142.0Ki ± ∞ ¹   142.2Ki ±  2%       ~ (p=1.000 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_1000_steps/Prometheus-16                            156.4Ki ± ∞ ¹   207.9Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/streaming-16                            469.2Ki ± ∞ ¹   459.0Ki ±  3%       ~ (p=0.571 n=1+6)
Query/a_2000_-_b_2000,_range_query_with_10000_steps/Prometheus-16                           1.564Mi ± ∞ ¹   1.578Mi ±  2%       ~ (p=0.571 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/streaming-16                               102.75Ki ± ∞ ¹   97.06Ki ±  3%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_instant_query/Prometheus-16                               68.36Ki ± ∞ ¹   70.06Ki ±  2%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/streaming-16                  132.22Ki ± ∞ ¹   99.19Ki ± 32%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_100_steps/Prometheus-16                  83.38Ki ± ∞ ¹   84.11Ki ±  1%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/streaming-16                  123.1Ki ± ∞ ¹   124.4Ki ±  8%       ~ (p=0.286 n=1+6)
Query/a_2000{l=~"1..."}_-_b_2000,_range_query_with_1000_steps/Prometheus-16                 161.2Ki ± ∞ ¹   160.4Ki ±  0%       ~ (p=0.286 n=1+6)

charleskorn

🚀

* Streaming PromQL engine: native histograms Add native histogram support to instant vectors. * Use seriesPerResultBucketFactor for histogram pool * Don't defer returning slices * Refactor slice returning logic * Move returnSlices helper * rename val for consistency * Update comment from feedback * Use returnSeriesDataSlices where missed * Comment out unsupported tests for now * Enable upstream native_histogram test file * Remove duplicate test * Fix FloatHistogram selector * Add range test * Add test for mixed floats+histograms * Re-enable supported upstream tests * Load benchmark samples in batches to avoid OOM'ing * Revert "Load benchmark samples in batches to avoid OOM'ing" We can't use OOO samples with native histograms, so instead of batching by series we'll batch by samples This reverts commit 91ba5c1. * Batch by samples instead of series * Generate native histogram benchmark test data * Add native histogram benchmark test * Move compare.sh next to other benchmark scripts * Fix comparison tests for when there are no floats * Add extra samples to the histogram tests * Add note on batch size * Add quick check that histograms are loaded * Use clearer variables for indexes * Update changelog * Fix typo

Streaming PromQL engine: native histograms

a021ff7

Add native histogram support to instant vectors.

charleskorn reviewed May 14, 2024

View reviewed changes

pkg/streamingpromql/operator/pool.go Outdated Show resolved Hide resolved

charleskorn reviewed May 14, 2024

View reviewed changes

pkg/streamingpromql/query.go Outdated Show resolved Hide resolved

jhesketh added 3 commits May 14, 2024 22:13

Merge branch 'main' into jhesketh/promql

3ce17f1

Use seriesPerResultBucketFactor for histogram pool

1970ce3

Don't defer returning slices

a75e358

jhesketh commented May 14, 2024

View reviewed changes

pkg/streamingpromql/testdata/ours/native_histograms.test Show resolved Hide resolved

jhesketh marked this pull request as ready for review May 14, 2024 12:31

jhesketh requested a review from a team as a code owner May 14, 2024 12:31

charleskorn reviewed May 15, 2024

View reviewed changes

jhesketh added 3 commits May 15, 2024 23:06

Refactor slice returning logic

0b8a160

Move returnSlices helper

5c7a2cd

rename val for consistency

235a402

charleskorn reviewed May 17, 2024

View reviewed changes

pkg/streamingpromql/query.go Outdated Show resolved Hide resolved

pkg/streamingpromql/testdata/ours/native_histograms.test Show resolved Hide resolved

jhesketh added 10 commits May 21, 2024 21:41

Update comment from feedback

72493bf

Use returnSeriesDataSlices where missed

3ea8e20

Comment out unsupported tests for now

885adfa

Enable upstream native_histogram test file

84be678

Remove duplicate test

d67498b

Merge branch 'main' into jhesketh/promql

4cf927e

Fix FloatHistogram selector

229b30a

Add range test

7438138

Add test for mixed floats+histograms

999d860

Re-enable supported upstream tests

4d50e3c

charleskorn reviewed May 27, 2024

View reviewed changes

pkg/streamingpromql/testdata/ours/native_histograms.test Outdated Show resolved Hide resolved

jhesketh added 4 commits May 27, 2024 14:35

Load benchmark samples in batches to avoid OOM'ing

91ba5c1

Revert "Load benchmark samples in batches to avoid OOM'ing"

aeb1b53

We can't use OOO samples with native histograms, so instead of batching by series we'll batch by samples This reverts commit 91ba5c1.

Batch by samples instead of series

7b79add

Generate native histogram benchmark test data

1a38e40

jhesketh added 4 commits May 27, 2024 15:34

Add native histogram benchmark test

f8ae514

Move compare.sh next to other benchmark scripts

063ed83

Fix comparison tests for when there are no floats

0cff32b

Add extra samples to the histogram tests

b9adee0

charleskorn reviewed May 28, 2024

View reviewed changes

jhesketh added 5 commits May 29, 2024 13:37

Add note on batch size

7fbde86

Add quick check that histograms are loaded

14ff8bd

Use clearer variables for indexes

a08733c

Update changelog

e94b596

Merge branch 'main' into jhesketh/promql

aa095fb

Fix typo

e8e5b24

charleskorn approved these changes May 29, 2024

View reviewed changes

jhesketh merged commit 6db3385 into grafana:main May 29, 2024
29 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Streaming PromQL engine: native histograms #8121

Streaming PromQL engine: native histograms #8121

jhesketh commented May 13, 2024 •

edited

jhesketh commented May 14, 2024

charleskorn left a comment

jhesketh commented May 28, 2024 •

edited by charleskorn

charleskorn left a comment

jhesketh commented May 29, 2024 •

edited

charleskorn left a comment

Streaming PromQL engine: native histograms #8121

Streaming PromQL engine: native histograms #8121

Conversation

jhesketh commented May 13, 2024 • edited

Checklist

jhesketh commented May 14, 2024

charleskorn left a comment

Choose a reason for hiding this comment

jhesketh commented May 28, 2024 • edited by charleskorn

charleskorn left a comment

Choose a reason for hiding this comment

jhesketh commented May 29, 2024 • edited

charleskorn left a comment

Choose a reason for hiding this comment

jhesketh commented May 13, 2024 •

edited

jhesketh commented May 28, 2024 •

edited by charleskorn

jhesketh commented May 29, 2024 •

edited