Skip to content

fix(encoding): use +Inf as the implicit histogram overflow bucket bound - #343

Merged
krisztianfekete merged 3 commits into
prometheus:masterfrom
gheorghitamutu:fix/inf-overflow-bucket-bound
Sep 11, 2026
Merged

fix(encoding): use +Inf as the implicit histogram overflow bucket bound#343
krisztianfekete merged 3 commits into
prometheus:masterfrom
gheorghitamutu:fix/inf-overflow-bucket-bound

Conversation

@gheorghitamutu

@gheorghitamutu gheorghitamutu commented Sep 9, 2026

Copy link
Copy Markdown
Contributor

What

Histogram::new appends f64::MAX as the sentinel for the implicit overflow bucket, and only
encoding::text translates it back to le="+Inf". Both protobuf encoders ship it verbatim, so
consumers receive an overflow bucket whose upper bound is 1.7976931348623157e+308 — a very large
finite bound, not an infinity.

This changes the sentinel to f64::INFINITY rather than translating it in each encoder.
Histogram::new is the only construction site (new_classic_and_native and
HistogramWithExemplars::new both delegate to it), so one line covers classic, exemplar and
classic+native histograms; the protobuf encoders then need no special case, and the text encoder
only changes what it compares against.

Why it matters

1. Quantiles above the ladder return ~1e308. Prometheus recognises the +Inf bucket with
IsInf, so it never saw one: it ingested the sentinel as an ordinary finite bucket and synthesised
its own le="+Inf" series from sample_count. promql's bucketQuantile never interpolates into
the +Inf bucket and returns the second-highest bound instead — the sentinel. Every quantile above
the histogram's real ladder therefore returned ~1.1e308 instead of a value. Measured on 0.25.0
scraped by Grafana Alloy into Mimir, histogram_quantile(0.99, ...) returned
1.0968850044231772e+308, which Grafana renders as "5.58e+300 years". It is silent: the series
scrapes cleanly and the panel only goes wrong once a quantile saturates. Scraping protobuf is not
optional for anyone using native histograms, since that is the only exposition offering them.

2. +Inf observations were counted in no bucket. observe_classic routes NaN to the last
bucket explicitly, but everything else through find(|(upper_bound, _)| upper_bound >= &v).
Since f64::MAX >= f64::INFINITY is false, an infinite observation incremented sum and count
while incrementing no bucket at all, leaving the overflow bucket short of _count. Prometheus's
synthetic +Inf series masked that too. Fixing only the encoders would have made this short count
visible rather than fixing it.

3. An explicit +Inf bound would now duplicate the overflow bucket. Because the sentinel is
real infinity, Histogram::new([1.0, f64::INFINITY]) would emit two le="+Inf" series for one
metric — invalid exposition. Histogram::new therefore drops non-finite bounds from the caller's
buckets; client_golang strips an explicitly-configured +Inf bound for the same reason. NaN
and -Inf bounds are dropped too: a NaN bound can never match an observation, since every
comparison against it is false.

Why +Inf is the correct value

  • metrics.proto documents this field as holding a "+Inf bucket" (optional, inclusive).
  • client_golang writes math.Inf(1) whenever it emits that bucket, and Prometheus branches on
    math.IsInf.
  • The OpenMetrics spec requires a Histogram MetricPoint to have a bucket with an +Inf threshold,
    so the OpenMetrics encoder was in outright violation.
  • prost encodes double as a raw IEEE-754 fixed64, so infinity is bit-exact on the wire
    (00 00 00 00 00 00 F0 7F) and decodes back to +Inf on the Go side.

The native-histogram positive_upper_bound returning f64::MAX is deliberately left alone — it
mirrors client_golang's getBound and is not on any wire path.

Testing

Two regression tests, each verified to fail without the corresponding half of the change:

  • every observation, including +Inf and NaN, is counted in some bucket. Fails before this
    change with buckets=[(1.0, 1), (2.0, 0), (1.7976931348623157e308, 1)] count=3.
  • an explicit +Inf bound does not duplicate the overflow bucket, asserted both on the bucket list
    and on the rendered text exposition. Fails without the filter with
    buckets=[(1.0, 1), (inf, 0), (NaN, 0), (-inf, 0), (inf, 0)].

cargo fmt --all -- --check is clean, cargo clippy --locked --workspace --all-targets -- -D warnings
reports nothing, and cargo test --locked --all --all-features passes 92/92 with the Python
prometheus-client dependency from CONTRIBUTING.md installed.

Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
@gheorghitamutu

Copy link
Copy Markdown
Contributor Author

@mxinden @brancz I didn't create an issue for this fix as I think is quite straightforward. Do let me know if it requires one, please.

Thanks you!

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. One follow-up (can be a separate PR if you'd like as it's pre-existent): client_golang strips a trailing +Inf from user-supplied bounds since the bucket is implicit, but Histogram::new([1.0, f64::INFINITY]) now emits two le="+Inf" series.

Filtering infinite bounds in Histogram::new before appending the implicit one would fix this. Let me know if you'd like to include this one here, or if you'd open a new PR/issue for this.

@gheorghitamutu

gheorghitamutu commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

I'll include it here and tag you again when it's done/ready.

The overflow bucket is implicit and now genuinely +Inf, so an explicitly
configured f64::INFINITY bound produced two le="+Inf" series for the same
metric, which is invalid exposition. client_golang strips an
explicitly-configured +Inf bound for the same reason.

NaN and -Inf bounds are dropped as well: observe_classic matches with
upper_bound >= v, and every comparison against NaN is false, so a NaN bound
could never receive an observation.

Regression test asserts a single +Inf bucket both on the bucket list and on the
rendered text exposition. It fails without the filter with
buckets=[(1.0, 1), (inf, 0), (NaN, 0), (-inf, 0), (inf, 0)].

Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
@gheorghitamutu

Copy link
Copy Markdown
Contributor Author

@krisztianfekete modified the PR description (added the case you've said) and pushed a new commit with the changes. Please review.
Thank you!

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, small nit: the new test got inserted between the existing doc comment and infinite_observation_lands_in_the_overflow_bucket, so the comments now describe the wrong test. Could you please move it below and trim both doc comments to a line each? Also happy to see these live in the existing tests module rather than a separate one.

…omments

Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
@gheorghitamutu

Copy link
Copy Markdown
Contributor Author

Hello,
I've moved around the test and shortened its description to a one liner.

@krisztianfekete krisztianfekete left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thanks for the quick turnaround!

@krisztianfekete
krisztianfekete added this pull request to the merge queue Sep 11, 2026
Merged via the queue into prometheus:master with commit cbde720 Sep 11, 2026
10 checks passed
@gheorghitamutu
gheorghitamutu deleted the fix/inf-overflow-bucket-bound branch September 11, 2026 15:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants