fix: Summary quantiles collapsing for targeted quantiles with 2*epsilon >= 1-quantile

[olegkovalenko]

Fixes #2292. Alternative to #2316, addressing the issues raised in its review.

Problem

For targeted quantile configurations with 2*epsilon >= 1 - quantile — e.g. (0.9, 0.05) or (0.99, 0.005), both taken from real-world configurations — Summary reported values from far below the requested quantile, often the minimum of all observations, regardless of the input data.

The root causes all stem from the same property: below a target quantile, the CKMS error function f(r) = 2*epsilon*(n-r)/(1-q) is of order n-r when 2*epsilon >= 1-q. Three things break:

1. compress() destroys the sketch. A single sample may span all ranks from r to n, so compress() merges away the samples that hold the information needed to answer the quantile query. With {(0.9, 0.05), (0.99, 0.005)} the sample list collapsed to 3 samples no matter how many values were inserted.

2. insertBefore() assigns misleading deltas. Freshly inserted samples get delta = f(r) - 1, so below a target their possible-rank intervals are centered near rank n regardless of the sample's actual position — indistinguishable from genuine samples near the target.

3. get() stops too early. The scan stopped at the first sample with r + g + delta > desiredRank + f(desiredRank)/2 and returned the value of the sample before it; a single wide fresh sample (always present, since get() flushes the buffer right before scanning) tripped it far before the target rank.

Fix

1. Sample widths are additionally bounded by maxWidthNotCrossingTargets(r) at both places where widths are created — merging in compress() and delta assignment in insertBefore(), via a shared effectiveMaxWidth(r) — so every target quantile keeps enough resolution around its accuracy window [q*n - eps*n, q*n + eps*n]: below a window a sample may extend at most max(windowStart - r, 2*eps*n) — it can intrude into the window but never reach the window's end — and any sample overlapping a window has width at most the window's size 2*eps*n. So no single sample can span a whole window, and the center of a sample's possible-rank interval is within eps*n of any rank the sample covers inside the window. The bound is anchored at the window's start so it does not degenerate for targets with quantile + epsilon >= 1 (e.g. (0.99, 0.01), (0.95, 0.05)), where the window's end is rank n and an end-anchored bound would be no constraint at all. For configurations with 2*epsilon < 1-quantile the bound is larger than f() near the target, so behavior there is mostly unchanged.

2. get() returns the value of the sample that minimizes the worst-case rank error: the true rank of a sample is somewhere in [r+g, r+g+delta], so picking it the rank error can be as large as max(|r+g - desiredRank|, |r+g+delta - desiredRank|) — the distance of the interval's center from the desired rank plus half the interval's width. Minimizing this cannot be derailed by a single wide sample (unlike the old stop rule), and penalizes wide samples whose interval center happens to fall near the desired rank (which an earlier revision of this PR, selecting by nearest center alone, did not — caught in review by a deterministic counterexample, now a regression test).

Relation to #2316 and its review

#2316 diagnoses the compress and get parts but bounds compress() differently (minimum of the error function over the merged interval) and leaves insert-time deltas unbounded. The review found a deterministic counterexample: single quantile (0.99, 0.005), values 1..10,000 shuffled with Random(2) — #2316 returns 9784, outside the allowed [9800, 10000].

This fix passes that case (returns 9940), and it is included as a regression test (testSingleTargetedQuantileSmallN).

Verification

The documented guarantee is q ± epsilon. Sweeping values 1..n (true rank = value) across 10 configurations × sizes {100, 257, 1k, 10k, 100k} × 100 shuffled seeds plus ascending and descending order, with pass = rank error <= eps*n + 1 rank (the +1 absorbs integer-rank quantization where eps*n < 1):

Configuration	cases	main > 1ε	this PR > 1ε
(0.5, 0.025) (review counterexample config)	510	9	2
(0.75, 0.02)	510	9	2
(0.99, 0.005)	510	299	2
(0.9, 0.05) + (0.99, 0.005)	1020	706	1
(0.5, 0.05) + (0.9, 0.01) + (0.99, 0.001)	1530	39	3
(0.9, 0.06) (strictly above boundary)	510	407	1
(0.99, 0.01) (window end = n)	510	404	0
(0.95, 0.05) (window end = n)	510	404	0
(0.5, 0.025) + (0.9, 0.05) + (0.99, 0.005)	1530	288	3
(0.5, 0.01) + (0.75, 0.01) + (0.95, 0.005) + (0.99, 0.002)	2040	22	1

Every remaining > 1ε case for this PR is descending input order (worst 1.75ε, at (0.99, 0.005), n=100k); on shuffled and ascending input there are no violations of the 1ε bound. main fails every one of those descending cases too — by up to 198ε on the collapsing configurations, and by 1.1–1.25ε even on well-behaved ones — and additionally fails dozens of shuffled cases (e.g. (0.5, 0.025) at n=10,000 with seeds 47, 52, 77), because deltas are fixed at insert time while n grows, so the paper's invariant erodes over the sketch's lifetime. In other words: this PR meets 1ε on every case main meets it, plus almost all the cases main fails. Closing the remaining descending-input gap would require maintaining the width invariant at query time, which is out of scope here; a test pins it at 2ε.

Additionally verified against exact percentiles on the evaluation grid of 2,900 cases (29 quantiles across 11 configurations × 5 distributions — uniform, heavy-tail from a production latency CDF, exponential, lognormal, gaussian — × 2 sizes × 10 seeds): worst rank error 1.60 * epsilon, no case above 2 * epsilon. Before the fix the worst rank error was ~330 * epsilon. The evaluation harness (including the instrumented query-rule variants compared before settling on this fix — the shipped rule is mode 6, minimax) is available at https://gist.github.com/olegkovalenko/83c58835a1357d3450e6538f89e2cda7.

Memory impact of the width bound is a handful of extra samples on the affected configurations (e.g. (0.99, 0.005): 3–4 → 6–11 samples after 1M inserts); well-behaved configurations such as (0.5, 0.05)(0.9, 0.01)(0.99, 0.001) are unchanged (37–40 samples).

Tests

• testTargetedQuantilesDoNotCollapse — the original reproducer from Summary quantiles collapse to the minimum observation when 2·epsilon ≥ 1−quantile #2292
• testSingleTargetedQuantileDoesNotCollapse — single targeted quantile at the boundary
• testTargetedQuantilesWithMedian — collapse still occurred with a well-behaved quantile added
• testSingleTargetedQuantileSmallN — the deterministic small-n case from the fix: prevent Summary quantiles from collapsing to the minimum observation #2316 review
• testMedianSmallN — the deterministic counterexample from this PR's review: (0.5, 0.025), values 1..257 shuffled with seed 5 (nearest-center selection returned 121, outside [122, 135]; the worst-case-error selection returns 132)
• testTargetedQuantileWindowReachingMaximum — the degenerate quantile + epsilon >= 1 family
• testTargetedQuantilesDescendingInput — descending input order, the worst case for these configurations
• testTargetedQuantilesAscendingInput — ascending input order, the counterpart of the descending test
• testTargetedQuantilesDescendingInputLargeN — pins the known remaining descending-input gap at 2ε (with a comment explaining why the 1ε bound erodes there and that main fails the same cases by far more)

validateResults now asserts the documented q ± epsilon rank bound (floor/ceil, since ranks are integers) — it previously allowed q ± 2*epsilon — and all tests above except testTargetedQuantilesDescendingInputLargeN use it at 1ε, on values 1..n inserted in the respective order (value = true rank). All existing CKMSQuantilesTest cases pass at the tightened bound (25 tests), as does the full prometheus-metrics-core suite (165 tests).

[zeitlinger]

Requesting changes based on the inline correctness issue.

[zeitlinger]

This nearest-center selection does not preserve the documented q ± epsilon error bound. On this head, inserting 1..257 shuffled with new Random(5) into a single Quantile(0.5, 0.025) returns rank 121, outside the allowed [122, 135] range; main returns rank 132. The test helper currently checks q ± 2*epsilon, masking the regression. Please preserve the public q ± epsilon guarantee and tighten the regression assertions accordingly.

[olegkovalenko]

Thanks — confirmed. Your case reproduces exactly (returns 121, allowed [122, 135]), and the
nearest-center selection was the culprit: a wide sample whose interval center happens to fall near
the desired rank could beat a narrow sample slightly further away.

Fixed by selecting the sample that minimizes the worst-case rank error instead: the true rank
of a sample is somewhere in [r+g, r+g+delta], so the worst-case error of picking it is
max(|r+g − desired|, |r+g+delta − desired|), which equals the distance of the interval's center
from the desired rank plus half the interval's width. This penalizes wide samples; your case now
returns 132. I also tightened validateResults to the documented q ± epsilon bound (floor/ceil,
since ranks are integers) and added your case as a regression test (testMedianSmallN).

Two findings from verifying this that are worth flagging:

1. The 2 * epsilon helper wasn't introduced in this PR — it's inherited from main's
   validateResults, and it appears to exist because main doesn't meet the strict q ± epsilon
   bound either. Sweeping main's unmodified code over values 1..n (true rank = value) across 10
   configurations × sizes {100, 257, 1k, 10k, 100k} × 100 shuffled seeds: main exceeds 1ε on
   dozens of shuffled cases, including your exact configuration (0.5, 0.025) at other seeds —
   e.g. n=10,000 with Random(47), Random(52), or Random(77), up to 1.11ε. (The mechanism:
   deltas are fixed at insert time while n grows, so the paper's invariant erodes over the
   sketch's lifetime.) With the selection rule above, this branch passes all of those shuffled
   and ascending cases at 1ε — every case main passes and every case it fails.

2. The remaining known gap is descending input at large n: up to 1.75ε observed (configuration
   (0.99, 0.005), n=100k). main exceeds 1ε on every descending case tested as well — by up to
   198ε on the collapsing configurations and ~1.1–1.25ε even on well-behaved ones — so this is
   not a regression, but the strict bound genuinely doesn't hold there for either implementation:
   sample widths are bounded when created, but with descending input a sample's rank grows by 1
   per insert while the accuracy windows move right by less than 1 per insert, so old samples
   drift toward the windows and their width bound erodes. A new
   testTargetedQuantilesDescendingInputLargeN documents this and pins it to 2ε; all other tests
   now assert 1ε. Making the strict bound hold under adversarial orderings would require
   maintaining the invariant at query time — a much bigger change; happy to open a follow-up
   issue for that if you think it's worth tracking.

On the evaluation grid from the PR description (2900 cases across configurations, distributions,
sizes, and seeds vs exact percentiles), the worst rank error improves from 1.78ε (nearest-center)
to 1.60ε with this rule, still with 0 cases above 2ε.