fix(encoding): use +Inf as the implicit histogram overflow bucket bound - #343
Conversation
Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
krisztianfekete
left a comment
There was a problem hiding this comment.
LGTM. One follow-up (can be a separate PR if you'd like as it's pre-existent): client_golang strips a trailing +Inf from user-supplied bounds since the bucket is implicit, but Histogram::new([1.0, f64::INFINITY]) now emits two le="+Inf" series.
Filtering infinite bounds in Histogram::new before appending the implicit one would fix this. Let me know if you'd like to include this one here, or if you'd open a new PR/issue for this.
|
I'll include it here and tag you again when it's done/ready. |
The overflow bucket is implicit and now genuinely +Inf, so an explicitly configured f64::INFINITY bound produced two le="+Inf" series for the same metric, which is invalid exposition. client_golang strips an explicitly-configured +Inf bound for the same reason. NaN and -Inf bounds are dropped as well: observe_classic matches with upper_bound >= v, and every comparison against NaN is false, so a NaN bound could never receive an observation. Regression test asserts a single +Inf bucket both on the bucket list and on the rendered text exposition. It fails without the filter with buckets=[(1.0, 1), (inf, 0), (NaN, 0), (-inf, 0), (inf, 0)]. Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
|
@krisztianfekete modified the PR description (added the case you've said) and pushed a new commit with the changes. Please review. |
krisztianfekete
left a comment
There was a problem hiding this comment.
Thanks, small nit: the new test got inserted between the existing doc comment and infinite_observation_lands_in_the_overflow_bucket, so the comments now describe the wrong test. Could you please move it below and trim both doc comments to a line each? Also happy to see these live in the existing tests module rather than a separate one.
…omments Signed-off-by: Gheorghita MUTU <gheorghitamutu@gmail.com>
|
Hello, |
krisztianfekete
left a comment
There was a problem hiding this comment.
Awesome, thanks for the quick turnaround!
What
Histogram::newappendsf64::MAXas the sentinel for the implicit overflow bucket, and onlyencoding::texttranslates it back tole="+Inf". Both protobuf encoders ship it verbatim, soconsumers receive an overflow bucket whose upper bound is
1.7976931348623157e+308— a very largefinite bound, not an infinity.
This changes the sentinel to
f64::INFINITYrather than translating it in each encoder.Histogram::newis the only construction site (new_classic_and_nativeandHistogramWithExemplars::newboth delegate to it), so one line covers classic, exemplar andclassic+native histograms; the protobuf encoders then need no special case, and the text encoder
only changes what it compares against.
Why it matters
1. Quantiles above the ladder return ~1e308. Prometheus recognises the +Inf bucket with
IsInf, so it never saw one: it ingested the sentinel as an ordinary finite bucket and synthesisedits own
le="+Inf"series fromsample_count. promql'sbucketQuantilenever interpolates intothe +Inf bucket and returns the second-highest bound instead — the sentinel. Every quantile above
the histogram's real ladder therefore returned ~1.1e308 instead of a value. Measured on 0.25.0
scraped by Grafana Alloy into Mimir,
histogram_quantile(0.99, ...)returned1.0968850044231772e+308, which Grafana renders as "5.58e+300 years". It is silent: the seriesscrapes cleanly and the panel only goes wrong once a quantile saturates. Scraping protobuf is not
optional for anyone using native histograms, since that is the only exposition offering them.
2.
+Infobservations were counted in no bucket.observe_classicroutes NaN to the lastbucket explicitly, but everything else through
find(|(upper_bound, _)| upper_bound >= &v).Since
f64::MAX >= f64::INFINITYis false, an infinite observation incrementedsumandcountwhile incrementing no bucket at all, leaving the overflow bucket short of
_count. Prometheus'ssynthetic +Inf series masked that too. Fixing only the encoders would have made this short count
visible rather than fixing it.
3. An explicit
+Infbound would now duplicate the overflow bucket. Because the sentinel isreal infinity,
Histogram::new([1.0, f64::INFINITY])would emit twole="+Inf"series for onemetric — invalid exposition.
Histogram::newtherefore drops non-finite bounds from the caller'sbuckets;client_golangstrips an explicitly-configured+Infbound for the same reason.NaNand
-Infbounds are dropped too: aNaNbound can never match an observation, since everycomparison against it is false.
Why +Inf is the correct value
metrics.protodocuments this field as holding a "+Inf bucket" (optional, inclusive).client_golangwritesmath.Inf(1)whenever it emits that bucket, and Prometheus branches onmath.IsInf.+Infthreshold,so the OpenMetrics encoder was in outright violation.
doubleas a raw IEEE-754 fixed64, so infinity is bit-exact on the wire(
00 00 00 00 00 00 F0 7F) and decodes back to +Inf on the Go side.The native-histogram
positive_upper_boundreturningf64::MAXis deliberately left alone — itmirrors client_golang's
getBoundand is not on any wire path.Testing
Two regression tests, each verified to fail without the corresponding half of the change:
+InfandNaN, is counted in some bucket. Fails before thischange with
buckets=[(1.0, 1), (2.0, 0), (1.7976931348623157e308, 1)] count=3.+Infbound does not duplicate the overflow bucket, asserted both on the bucket listand on the rendered text exposition. Fails without the filter with
buckets=[(1.0, 1), (inf, 0), (NaN, 0), (-inf, 0), (inf, 0)].cargo fmt --all -- --checkis clean,cargo clippy --locked --workspace --all-targets -- -D warningsreports nothing, and
cargo test --locked --all --all-featurespasses 92/92 with the Pythonprometheus-clientdependency from CONTRIBUTING.md installed.