Skip to content

perf(render): coalesce concurrent response cache misses - #592

Merged
timtofan merged 1 commit into
masterfrom
atimofieiev/render-cache-coalescing
Nov 10, 2026
Merged

timtofan merged 1 commit into
masterfrom
atimofieiev/render-cache-coalescing

Conversation

@timtofan

@timtofan timtofan commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Identical concurrent response-cache misses independently fetch, decode, evaluate and serialize the same render. Large overlapping requests can multiply frontend CPU, memory and backend load precisely when the frontend is under pressure. Sharing that work reduces avoidable load without changing render semantics.

Architecture

The HTTP render handler still parses the form and checks the response cache first. On a miss, a per-App flight table uses exactly renderForm.cacheKey, the existing encoded response-cache key. The first caller executes the existing backend queues, decoder, evaluator and serializer once. Concurrent callers wait for those immutable result bytes. JSONP wrapping, response UUIDs, error rendering and access-log identity remain per caller. All render parameters already included in the cache key (including maxDataPoints, timezone and image options) remain distinct. No request header currently participates in this cache key or affects evaluation.

The shared work has an independent global timeout and retains context values. Cancelling the first caller only ends that caller's wait; remaining callers continue. When every caller leaves, the work is cancelled and removed. Completed errors are removed immediately, so later requests retry. Successful nonempty results are published to the cache before releasing the flight, closing the previous asynchronous cache-write window. Cache writes are now on the completion path and retain the cache client's configured timeout. Explicit noCache requests keep independent execution and the existing cache-population behavior.

This adds no backend protocol or cross-host coordination. Existing backend batch/routing and transport paths remain unchanged.

Before / after

Before After
N overlapping identical cache misses perform N fetch/decode/evaluation/serialization passes One execution supplies N byte-identical bodies
A disconnected request cancels its own backend work Each caller waits independently; shared work continues for remaining callers
Later requests may duplicate work while an asynchronous cache write completes Cache publication completes before the flight is released
Different response-cache keys and explicit cache bypasses execute independently Same behavior
No render-flight accounting render_leaders_total, render_coalesced_total and a bounded render_input_points histogram

Usage

Enabled directly in the existing render handler for cache-eligible misses, including null or unavailable response caches. No new configuration or dependency. The input-points histogram counts materialized input after replica merging and includes extra evaluator fetches; it is not raw wire volume across replicas.

Validation

  • make test passed with the repository's cairo/nativegzip/zlibng race suite and both backend transport build variants.
  • Real HTTP backend tests: 16 identical simultaneous misses use one backend fetch; 15 followers survive leader cancellation; different target/time/format/maxDataPoints/timezone/cacheTimeout keys and cache bypasses remain independent.
  • Shared backend errors reach current waiters only; a subsequent request retries. Tests cover abandoned-work cancellation, panic recovery, and cache publication ordering.
  • Six format checks preserve baseline output and require exact bytes between coalesced callers. Independent pickle renders already vary dictionary ordering, so their baseline values are compared after decoding. JSONP callbacks remain per caller.

Production validation passed: 61min of normal traffic, 1,589,854 requests, 4,669 coalesced renders, no process restart, and 60/60 byte-identical comparisons across two independent samples.

A natural-traffic sample found three large pairs of identical concurrent renders. Each pair read 22,553 input series (263,779,888 materialized input bytes) through one upstream render call; the follower issued zero upstream calls. No synthetic heavy load was injected.

Normal 60min window after warm-up Canary Peer control
CPU/request 12.945ms 13.851ms
Backend render responses/render 40.155 40.589
HTTP5xx 21 106
Render p50 / p99 (histogram estimates) 27.3ms / 352.5ms 27.5ms / 569.8ms

The existing transport settings and request mixes differed, so these figures are observational. Coalescing preserves independent work for different cache keys; upstream failures can still return 5xx. The instance was restored to its previous release after validation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@timtofan
timtofan marked this pull request as ready for review November 9, 2026 20:32
@timtofan
timtofan merged commit 55f2951 into master Nov 10, 2026
4 checks passed
@timtofan
timtofan deleted the atimofieiev/render-cache-coalescing branch November 10, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants