Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -63,3 +63,4 @@ src/*.html
# leftover local build artifacts (node_modules, target, dist) that remain on disk.
/crates/js/
/crates/integration-tests/
wrangler.integration.generated.toml

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick — This lands directly under the two-line comment about defunct pre-rename crate dirs, so it reads as a third entry in that block. It's unrelated — it's the Cloudflare integration harness's per-run output (crates/trusted-server-integration-tests/tests/environments/cloudflare.rs:27).

Verified in the batch scratch pass: cargo fmt --all -- --check and the docs Prettier check stay clean, no drift.

Suggested change
wrangler.integration.generated.toml
# Cloudflare integration harness output, written at test time by
# crates/trusted-server-integration-tests/tests/environments/cloudflare.rs.
wrangler.integration.generated.toml

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ignore rule is inert: the generated wrangler config is tracked on the target branch, and merging this PR re-adds it rather than untracking it.

Commit 1fa9f8c's message says 'Ignore it so local CI=1 runs cannot commit it again', but the deletion is a no-op relative to the merge base (c6235b9 never had the file) while origin/feat/request-phase-timing added it independently. A test merge of origin/feat/request-phase-timing into the PR head stages A crates/trusted-server-adapter-cloudflare/wrangler.integration.generated.toml, so after this PR lands the file is still tracked and .gitignore has no effect on an already-tracked path. Every local CI=1 run that rewrites it (crates/trusted-server-integration-tests/tests/environments/cloudflare.rs:27 GENERATED_CI_CONFIG) still shows up as a dirty tracked file. Fix needs git rm --cached on the branch, not just the ignore rule.

19 changes: 19 additions & 0 deletions crates/trusted-server-core/src/access_telemetry.rs
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,10 @@ pub fn access_event_row(
"stream_ms": timings.stream_ms,
"request_elapsed_ms": timings.request_elapsed_ms,
"resp_bytes": timings.resp_bytes,
"auction_dispatched_ms": timings.auction_dispatched_ms,
"auction_resolved_ms": timings.auction_resolved_ms,
"auction_committed_ms": timings.auction_committed_ms,
"auction_id": timings.auction_id.as_deref().unwrap_or("none"),

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 P2 / Medium: Auction API requests serialize as if no auction ran

Issue: The new fields are marked only by the split initial-page auction path. Successful /auction and /_ts/page-bids requests run auctions and emit auction_events_raw rows, but their access rows retain null offsets and the none auction ID serialized here.

Impact: Every Fastly access row for these routes loses its join to per-bidder telemetry and violates the documented meaning that null or none means no auction ran.

Evidence: POST /auction calls run_auction in auction/endpoints.rs, and GET /_ts/page-bids calls it in publisher.rs; neither path invokes any of the new mark methods. The Fastly post-send emitter still serializes the shared RequestTimings snapshot for both routes.

Suggested fix: Instrument both handlers using their AuctionObservationContext::auction_id and accurate lifecycle timestamps. If these columns intentionally cover only initial publisher navigation, document and name that narrower scope rather than using a global no-auction sentinel. Add route-level row tests.

"template_cache_state": snapshot.template_cache_state,
"country": snapshot.country,
"ts_version": snapshot.ts_version,
Expand Down Expand Up @@ -493,6 +497,9 @@ mod tests {
"stream_ms",
"request_elapsed_ms",
"resp_bytes",
"auction_dispatched_ms",
"auction_resolved_ms",
"auction_committed_ms",
] {
assert!(
parsed[field].is_null(),
Expand All @@ -518,6 +525,10 @@ mod tests {
);
}
assert_eq!(parsed["auction_wait_placement"], "none");
assert_eq!(
parsed["auction_id"], "none",
"auction_id should carry the none sentinel when no auction ran"
);
}

#[test]
Expand All @@ -536,6 +547,10 @@ mod tests {
stream_ms: Some(8),
auction_wait_placement: Some(AuctionWaitPlacement::InStream),
resp_bytes: Some(1024),
auction_dispatched_ms: Some(9),
auction_resolved_ms: Some(10),
auction_committed_ms: Some(11),
auction_id: Some("33333333-3333-3333-3333-333333333333".to_owned()),
};
let row = access_event_row(&snapshot, &timings, 1_700_000_000_000);
let parsed: serde_json::Value =
Expand All @@ -545,6 +560,10 @@ mod tests {
assert_eq!(parsed["stream_ms"], 8);
assert_eq!(parsed["resp_bytes"], 1024);
assert_eq!(parsed["auction_wait_placement"], "in_stream");
assert_eq!(parsed["auction_dispatched_ms"], 9);
assert_eq!(parsed["auction_resolved_ms"], 10);
assert_eq!(parsed["auction_committed_ms"], 11);
assert_eq!(parsed["auction_id"], "33333333-3333-3333-3333-333333333333");
}

#[test]
Expand Down
14 changes: 14 additions & 0 deletions crates/trusted-server-core/src/publisher.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3951,6 +3951,8 @@ async fn collect_non_html_auction(
params
.timings
.record_auction_wait(placement, wait_started.elapsed());
// T0-anchored timeline mark (spec section 18): final bid or timeout.
params.timings.mark_auction_resolved();

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The four new publisher call sites have no test coverage, though two existing tests already drive both collect paths and snapshot the same collector.

streaming_seam_wait_records_in_stream_placement (publisher.rs:17407) and buffered_template_miss_records_pre_header_placement (publisher.rs:17481) each build a DispatchedAuction, run the finalizer, then assert on timings.snapshot() for auction_wait_placement and auction_wait_ms. Neither was extended with assert!(snapshot.auction_resolved_ms.is_some()) / auction_committed_ms.is_some(). Deleting both params.timings.mark_auction_resolved() and timings.mark_auction_committed() leaves the entire Rust suite green, so nothing catches a future refactor that moves a mark out of the collect path or drops it in a merge conflict resolution (publisher.rs does conflict against the current base).

let delivered_winner_slots = write_bids_to_state(
&result.winning_bids,
params.price_granularity,
Expand All @@ -3960,6 +3962,9 @@ async fn collect_non_html_auction(
settings.debug.inject_adm_for_testing,
auction_id.as_deref(),
);
// T0-anchored timeline mark (spec section 18): winning bids are in page
// state, available to the response pipeline.
params.timings.mark_auction_committed();
if let (Some(observation), Some(auction_request)) =
(telemetry.observation, telemetry.auction_request.as_ref())
{
Expand Down Expand Up @@ -4010,6 +4015,8 @@ async fn collect_stream_auction(
.collect_dispatched_auction(dispatched, services, &collect_ctx)
.await;
timings.record_auction_wait(*placement, wait_started.elapsed());
// T0-anchored timeline mark (spec section 18): final bid or timeout.
timings.mark_auction_resolved();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 P1 / High: Resolved time is collection time, not bidder completion time

Issue: When bidder responses finish before the origin stream reaches </body>, nothing polls them until the seam. This line stamps auction_resolved_ms only after collect_dispatched_auction returns, potentially much later. An all-immediate provider result makes this explicit: the auction is terminal at dispatch, but this mark still waits for the seam.

Impact: R - D includes origin fetch and body-stream delay rather than auction duration. The documented overlap calculation can therefore substantially overstate auction runtime and cannot answer when the final bid landed, which is the main purpose of this change.

Evidence: Collection starts at the delayed body seam, while collect_dispatched_auction performs the first select over pending requests. The focused split_auction_accepts_an_all_immediate_no_bid_result test passes and confirms that Dispatched does not imply work remains.

Suggested fix: Capture the terminal timestamp when the final provider actually completes or times out, then pass that timestamp into RequestTimings. This likely requires polling collection concurrently or receiving completion timing from the transport. If that is unavailable, rename the field to auction_collected_ms and remove the auction-duration and overlap claims. Add a delayed-collection regression test.

log::info!(
"body_close_hold_loop: collect complete - {} winning bid(s)",
result.winning_bids.len()
Expand All @@ -4023,6 +4030,9 @@ async fn collect_stream_auction(
settings.debug.inject_adm_for_testing,
auction_id.as_deref(),
);
// T0-anchored timeline mark (spec section 18): winning bids are in page
// state, available to the response pipeline.
timings.mark_auction_committed();
if let (Some(observation), Some(auction_request)) =
(telemetry.observation, telemetry.auction_request.as_ref())
{
Expand Down Expand Up @@ -4340,6 +4350,10 @@ pub async fn handle_publisher_request(
.await
{
DispatchAuctionOutcome::Dispatched(dispatched) => {
// T0-anchored timeline mark (spec section 18): bid
// requests have left the edge. A failed dispatch never
// marks, so all three auction offsets stay null for it.
timings.mark_auction_dispatched(observation.auction_id.to_string());
Comment on lines +4353 to +4356

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrench — This comment claims an invariant the code does not hold, and spec section 18 repeats it: "The three offsets are null when no auction ran... Null means 'no auction', never 'zero'."

That is true for a failed dispatch, but not for successful dispatch followed by abandonment. Seven terminal paths mark dispatched and then never reach mark_auction_resolved / mark_auction_committed:

Reason Site
origin_proxy_error publisher.rs:4679
unexpected_origin_304 publisher.rs:4702
pass_through_response publisher.rs:4902
buffered_unmodified_response publisher.rs:4942
bodiless_response publisher.rs:1722, publisher.rs:2463
processor_init_error publisher.rs:2496
stream_process_error abandon_hold_auction, publisher.rs:1012

I confirmed this rather than inferring it. A scratch test mirroring finalizers_emit_abandoned_auction_for_bodiless_dispatched_response, driving the real publisher_response_into_streaming_response with a pre-marked RequestTimings, prints:

PROBE dispatched=Some(0) resolved=None committed=None auction_id=Some("44444444-4444-4444-4444-444444444444")

So the row carries a real auction_dispatched_ms and a real auction_id next to null resolved / committed. Consequences:

  • The derivations section 18 prescribes (R - D, C - R) silently yield nothing for these rows.
  • A dashboard filtering auction_dispatched_ms IS NOT NULL gets a population mixing completed and abandoned auctions, with no column separating them.
  • auction_id IS NOT NULL no longer implies a complete timeline.

Proposed fix (apply manually — this needs a spec edit plus a comment edit, so it cannot be a single-file suggestion). My recommendation is to document the reading rather than add a column, since the events dataset already records the abandonment reason and the join key is present on the row:

                DispatchAuctionOutcome::Dispatched(dispatched) => {
                    // T0-anchored timeline mark (spec section 18): bid
                    // requests have left the edge. A failed dispatch never
                    // marks, so all three offsets stay null for it. A
                    // *dispatched* auction that is later abandoned (bodiless
                    // response, pass-through, origin error, processor error)
                    // marks here but never resolves or commits: a non-null
                    // `auction_dispatched_ms` with null `auction_resolved_ms`
                    // reads as "dispatched, then abandoned", and `auction_id`
                    // joins to the `Abandoned` terminal row for the reason.
                    timings.mark_auction_dispatched(observation.auction_id.to_string());

Section 18's "Row changes" bullet needs the matching correction — "null when no auction ran" should become "null when no auction was dispatched; auction_resolved_ms / auction_committed_ms are additionally null when a dispatched auction was abandoned before collect."

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one of the three auction sources is instrumented, so every /auction and /_ts/page-bids access row reports auction_id 'none' and null offsets even though a full auction ran.

AuctionSource has three variants (auction/telemetry.rs:25-32): InitialNavigation, SpaNavigation (GET /_ts/page-bids), AuctionApi (POST /auction). mark_auction_dispatched is called only from handle_publisher_request's DispatchAuctionOutcome::Dispatched arm. handle_auction (auction/endpoints.rs:307) and handle_page_bids (publisher.rs:6603) both run orchestrator.run_auction(...) and both emit auction_events_raw rows with a real auction UUID, and both are RouteClass::AuctionApi in the fastly route table (adapter-fastly/src/app.rs:1302-1326), so both emit access rows. POST /auction with bidders configured -> auction_events_raw gets a row with auction_id=, while the matching access row gets auction_id='none', auction_dispatched_ms=NULL, auction_resolved_ms=NULL, auction_committed_ms=NULL. The documented join (spec section 18: 'join auction_id for the per-bidder long pole') returns nothing for the two route classes named after auctions. Both handlers already receive req: Request<EdgeBody>, which carries the RequestTimings extension, so the fix is the same one-liner used at publisher.rs:4128.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

observation.auction_id.to_string() heap-allocates on every dispatched auction even when access telemetry is disabled, which is the documented default.

timings.mark_auction_dispatched(observation.auction_id.to_string()) formats a Uuid into a 36-byte String at the call site, before the method can decide the call is a no-op (second call, or try_lock lost). Nothing gates it on settings.tinybird.access_enabled. The fastly adapter's SendContext.access_telemetry_enabled doc states the opposite rule for this exact data path: the snapshot 'costs env reads and String allocations on the pre-send path, which a disabled deployment (the default) should not pay'. On a wasm32-wasip1 guest this is one allocation per auctioned request that is thrown away in the common configuration. Store the Uuid (Copy, 16 bytes) in Inner and format it once inside snapshot(), or take &Uuid.

auction_request_for_telemetry = Some(auction_request);
auction_observation = Some(observation);
Some(dispatched)
Expand Down
134 changes: 134 additions & 0 deletions crates/trusted-server-core/src/request_timing.rs
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,19 @@ struct Inner {
/// Response body size in bytes, set via
/// [`RequestTimings::set_resp_bytes`].
resp_bytes: Option<u64>,
/// Elapsed time at the first
/// [`RequestTimings::mark_auction_dispatched`] call.
auction_dispatched: Option<Duration>,
/// Elapsed time at the first
/// [`RequestTimings::mark_auction_resolved`] call.
auction_resolved: Option<Duration>,
/// Elapsed time at the first
/// [`RequestTimings::mark_auction_committed`] call.
auction_committed: Option<Duration>,
/// Telemetry auction UUID recorded by the first
/// [`RequestTimings::mark_auction_dispatched`] call; joins the access
/// row to the per-bidder auction dataset.
auction_id: Option<String>,
}

/// Per-request phase timing collector.
Expand All @@ -128,6 +141,10 @@ impl RequestTimings {
request_elapsed: None,
auction_wait_placement: None,
resp_bytes: None,
auction_dispatched: None,
auction_resolved: None,
auction_committed: None,
auction_id: None,
})))
}

Expand Down Expand Up @@ -200,6 +217,53 @@ impl RequestTimings {
}
}

/// Stamps the elapsed time since `t0` as the auction dispatch offset and
/// records the telemetry auction id, the first time this is called.
///
/// Called when `dispatch_auction` reports the bid requests dispatched;
/// a failed dispatch never records, so all three auction offsets stay
/// `None` for it. Subsequent calls are no-ops (first call wins). Drops
/// the sample silently on lock contention or poisoning.
pub fn mark_auction_dispatched(&self, auction_id: String) {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three near-identical mark methods duplicate a body the file already has twice; a private mark-once helper collapses all five.

mark_auction_dispatched, mark_auction_resolved, mark_auction_committed are the same six lines as each other and as the pre-existing mark_headers_ready (line 197) and mark_request_elapsed (line 217), differing only in which Option field they write. That is five copies of try_lock + is_none + t0.elapsed(). It is also why the poison-recovery divergence above happened: the base branch had to edit the pattern in seven places, and the three new copies were written from the older shape. A private fn mark_once(inner: &mut Inner, slot: impl FnOnce(&mut Inner) -> &mut Option<Duration>) or a small macro removes the class of drift.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three marks are pub but have no caller outside the core crate, against the repo rule on unnecessary visibility.

CLAUDE.md (repo root), Type System: 'Consider visibility carefully - avoid unnecessary pub.' grep across crates/ finds mark_auction_dispatched / mark_auction_resolved / mark_auction_committed only in request_timing.rs and publisher.rs, both inside trusted-server-core. The comparable pre-existing mark_request_elapsed is genuinely pub because adapter-fastly/src/main.rs:628 and :659 call it. pub(crate) is the correct visibility here and keeps three more methods out of the crate's public API surface and its rustdoc.

let Ok(mut inner) = self.0.try_lock() else {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The three new marks drop the sample on a poisoned mutex while every other method on the current base recovers from poisoning, and the file auto-merges without conflict so the inconsistency lands silently.

PR head uses let Ok(mut inner) = self.0.try_lock() else { return; }; in all three new marks. origin/feat/request-phase-timing has since converted all seven lock-taking methods to match self.0.try_lock() { Ok(g) => g, Err(TryLockError::Poisoned(p)) => p.into_inner(), Err(TryLockError::WouldBlock) => return }. Verified by test merge: git merge origin/feat/request-phase-timing reports 'Auto-merging crates/trusted-server-core/src/request_timing.rs' with NO conflict, and the merged file has 7 poison-recovering sites plus exactly mark_auction_dispatched/resolved/committed that are not. After any panic that poisons the mutex (the base's module doc explicitly contemplates this), appbuild/filter/geo/kv/origin/stream/auction_wait keep recording for the rest of the request while all three auction offsets and auction_id go permanently null, and the base's module doc ('a poisoned lock recovers ... so a panic elsewhere cannot silence the rest of the request's timing') becomes false for them.

return;
};
if inner.auction_dispatched.is_none() {
inner.auction_dispatched = Some(inner.t0.elapsed());
inner.auction_id = Some(auction_id);
}
}

/// Stamps the elapsed time since `t0` as the auction resolve offset (the
/// final bid returned or the auction timed out), the first time this is
/// called.
///
/// Subsequent calls are no-ops (first call wins). Drops the sample
/// silently on lock contention or poisoning.
pub fn mark_auction_resolved(&self) {
let Ok(mut inner) = self.0.try_lock() else {
return;
};
if inner.auction_resolved.is_none() {
inner.auction_resolved = Some(inner.t0.elapsed());
}
}

/// Stamps the elapsed time since `t0` as the auction commit offset
/// (winning bids written into page state), the first time this is
/// called.
///
/// Subsequent calls are no-ops (first call wins). Drops the sample
/// silently on lock contention or poisoning.
pub fn mark_auction_committed(&self) {
let Ok(mut inner) = self.0.try_lock() else {

@jevansnyc jevansnyc Sep 14, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my own comment below: C - R is not structurally zero. Withdrawing the "drop the column" recommendation; the millisecond-truncation point still stands.

I said the span between the two marks does no real work. That is wrong. write_bids_to_state (publisher.rs:3208) calls build_bid_map_with_auction_id, which for each winning bid runs creative::process_inline_auction_creative (sanitize, URL rewrite to first-party proxies, signing) plus expand_auction_price_macro on nurl/burl. rewrite_creatives = true is the default (trusted-server.example.toml:155), so that path is on in a normal deployment. C - R therefore measures per-request creative-processing CPU, which is a real and separately actionable signal. Keep the column.

What does survive: duration_ms truncates with u32::try_from(dur.as_millis()), so on an auction with one or two small creatives C - R still floors to 0, and the spec advertises "commit latency (C - R)" as a dashboard derivation without saying that. Worth either a note in the spec that the value is whole-millisecond and often 0 on light auctions, or sub-millisecond resolution on this pair.


Original comment (the claim about "no work between the marks" is retracted):

auction_committed_ms is near-always equal to auction_resolved_ms, so the third column and the spec's 'commit latency (C - R)' derivation carry no information.

Between mark_auction_resolved and mark_auction_committed the only work is write_bids_to_state (publisher.rs:3208), which is synchronous in-memory: build_bid_map_with_auction_id plus ad_bids_state.set(bid_map), no await, no I/O. duration_ms truncates to whole milliseconds (u32::try_from(dur.as_millis())), so both marks land in the same millisecond on essentially every request and C - R evaluates to 0. The column costs a schema migration, a FORWARD_QUERY entry, and a per-row JSON key for a value that is a copy of its neighbour. Either drop it or move the commit mark to a point that can actually differ from resolve (for example after the seam rewrite completes).

return;
};
if inner.auction_committed.is_none() {
inner.auction_committed = Some(inner.t0.elapsed());
}
}

/// Records the response body size in bytes.
///
/// Drops the sample silently on lock contention or poisoning.
Expand Down Expand Up @@ -257,6 +321,10 @@ impl RequestTimings {
stream_ms: duration_ms(inner.phases[Phase::Stream.index()]),
auction_wait_placement: inner.auction_wait_placement,
resp_bytes: inner.resp_bytes,
auction_dispatched_ms: duration_ms(inner.auction_dispatched),
auction_resolved_ms: duration_ms(inner.auction_resolved),
auction_committed_ms: duration_ms(inner.auction_committed),
auction_id: inner.auction_id.clone(),
}
}
}
Expand Down Expand Up @@ -379,6 +447,18 @@ pub struct TimingSnapshot {
/// Response body size in bytes, set via
/// [`RequestTimings::set_resp_bytes`].
pub resp_bytes: Option<u64>,
/// T0 offset at which the auction dispatched (bid requests left the
/// edge), or `None` when no auction ran.
pub auction_dispatched_ms: Option<u32>,
/// T0 offset at which the auction resolved (final bid or timeout), or
/// `None` when no auction ran.
pub auction_resolved_ms: Option<u32>,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A null auction_resolved_ms/auction_committed_ms does not mean 'no auction ran' as documented; every abandoned auction sets dispatched plus a real auction_id and leaves the other two null.

There are 11 production emit_abandoned_auction call sites downstream of the dispatch mark (publisher.rs:1018, 1733, 2475, 2497, 2933, 3829, 3872, 4693 origin_proxy_error, 4715 unexpected_origin_304, 4924, 4963). On an origin fetch failure or an unexpected 304 the auction is abandoned after mark_auction_dispatched already fired, so the access row carries auction_dispatched_ms= and auction_id= with auction_resolved_ms=NULL and auction_committed_ms=NULL. The doc comments on TimingSnapshot say 'or None when no auction ran' and the plan states 'Null offsets mean "no auction ran", never zero'. A dashboard query using auction_resolved_ms IS NULL to mean 'no auction' silently folds in every abandoned auction, which is exactly the 5xx/304 population an operator investigating latency would be looking at. The same asymmetry appears if mark_auction_dispatched's try_lock loses a race while the later two win: real offsets for resolve/commit with auction_id='none'.

/// T0 offset at which winning bids were committed into page state, or
/// `None` when no auction ran.
pub auction_committed_ms: Option<u32>,
/// Telemetry auction UUID joining this row to the auction dataset, or
/// `None` when no auction ran.
pub auction_id: Option<String>,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The auction id is stored as a raw String on the collector while the rest of the codebase carries it as a Uuid, against the repo's newtype rule.

AuctionObservationContext.auction_id is Uuid (auction/telemetry.rs:98) and auction_events_raw declares the column as UUID. Inner.auction_id and TimingSnapshot.auction_id are Option<String>, and mark_auction_dispatched takes auction_id: String, so the type that guarantees the join key is well-formed is discarded at the boundary and any String reaches the row builder. CLAUDE.md (repo root), Type System: 'Create strong types with newtype patterns for domain entities.' Holding the Uuid and formatting in snapshot() keeps the invariant and removes the allocation noted above.

}

#[cfg(test)]
Expand Down Expand Up @@ -432,6 +512,60 @@ mod tests {
);
}

#[test]
fn auction_marks_are_first_call_wins_and_snapshot_maps_them() {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first-call-wins test does not actually test first-call-wins for the three offsets; it only asserts is_some().

auction_marks_are_first_call_wins_and_snapshot_maps_them calls each mark twice, then asserts snapshot.auction_dispatched_ms.is_some(), .auction_resolved_ms.is_some(), .auction_committed_ms.is_some(). Both a correct first-call-wins implementation and a broken last-call-wins one produce Some for all three, so the test passes either way. Only the auction_id assertion is a real first-call-wins check. Remove the if inner.auction_resolved.is_none() guard from mark_auction_resolved and the suite still goes green. To test what the name claims the test must capture the first value, let measurable time pass, mark again, and assert the value is unchanged.

let timings = RequestTimings::new();
timings.mark_auction_dispatched("11111111-1111-1111-1111-111111111111".to_owned());
timings.mark_auction_resolved();
timings.mark_auction_committed();
// Second calls must not overwrite the first-recorded values.
timings.mark_auction_dispatched("22222222-2222-2222-2222-222222222222".to_owned());
timings.mark_auction_resolved();
timings.mark_auction_committed();

let snapshot = timings.snapshot();
assert!(
snapshot.auction_dispatched_ms.is_some(),
"should record the dispatch offset"
);
assert!(
snapshot.auction_resolved_ms.is_some(),
"should record the resolve offset"
);
assert!(
snapshot.auction_committed_ms.is_some(),
"should record the commit offset"
);
Comment on lines +517 to +538

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ refactor — This test doesn't test what its name says for two of the three marks.

The three offset assertions are is_some(), which holds whether or not the first-call-wins guards exist. auction_id is the only witness that a guard actually fired, and it only witnesses the auction_dispatched branch — mark_auction_resolved and mark_auction_committed have no coverage of their is_none() check at all. Deleting either guard leaves this test green.

mark_headers_ready_is_first_call_wins (line 598, same module) already establishes the pattern: snapshot, sleep past the millisecond truncation in duration_ms, re-mark, compare.

Verified in a scratch worktree at this head: cargo fmt --all -- --check clean, cargo clippy-fastly clean, cargo test-fastly -p trusted-server-core --lib request_timing 12/12 pass, no post-verification drift. Also mutation-tested — removing the inner.auction_resolved.is_none() guard makes this revised test fail, while the current version still passes.

Suggested change
let timings = RequestTimings::new();
timings.mark_auction_dispatched("11111111-1111-1111-1111-111111111111".to_owned());
timings.mark_auction_resolved();
timings.mark_auction_committed();
// Second calls must not overwrite the first-recorded values.
timings.mark_auction_dispatched("22222222-2222-2222-2222-222222222222".to_owned());
timings.mark_auction_resolved();
timings.mark_auction_committed();
let snapshot = timings.snapshot();
assert!(
snapshot.auction_dispatched_ms.is_some(),
"should record the dispatch offset"
);
assert!(
snapshot.auction_resolved_ms.is_some(),
"should record the resolve offset"
);
assert!(
snapshot.auction_committed_ms.is_some(),
"should record the commit offset"
);
let timings = RequestTimings::new();
timings.mark_auction_dispatched("11111111-1111-1111-1111-111111111111".to_owned());
timings.mark_auction_resolved();
timings.mark_auction_committed();
let first = timings.snapshot();
assert!(
first.auction_dispatched_ms.is_some(),
"should record the dispatch offset"
);
assert!(
first.auction_resolved_ms.is_some(),
"should record the resolve offset"
);
assert!(
first.auction_committed_ms.is_some(),
"should record the commit offset"
);
// Sleep past `duration_ms`'s millisecond truncation so a restamp
// would change the recorded value, matching
// `mark_headers_ready_is_first_call_wins`.
std::thread::sleep(Duration::from_millis(5));
// Second calls must not overwrite the first-recorded values.
timings.mark_auction_dispatched("22222222-2222-2222-2222-222222222222".to_owned());
timings.mark_auction_resolved();
timings.mark_auction_committed();
let snapshot = timings.snapshot();
assert_eq!(
snapshot.auction_dispatched_ms, first.auction_dispatched_ms,
"should not restamp the dispatch offset"
);
assert_eq!(
snapshot.auction_resolved_ms, first.auction_resolved_ms,
"should not restamp the resolve offset"
);
assert_eq!(
snapshot.auction_committed_ms, first.auction_committed_ms,
"should not restamp the commit offset"
);

assert_eq!(
snapshot.auction_id.as_deref(),
Some("11111111-1111-1111-1111-111111111111"),
"should keep the first-recorded auction id"
);
}

#[test]
fn snapshot_without_auction_marks_yields_none_for_all_offsets() {
let timings = RequestTimings::new();
timings.mark_headers_ready();
let snapshot = timings.snapshot();
assert_eq!(
snapshot.auction_dispatched_ms, None,
"should stay None when no auction dispatched"
);
assert_eq!(
snapshot.auction_resolved_ms, None,
"should stay None when no auction resolved"
);
assert_eq!(
snapshot.auction_committed_ms, None,
"should stay None when no auction committed"
);
assert_eq!(
snapshot.auction_id, None,
"should carry no auction id when no auction ran"
);
}

#[test]
fn render_omits_unrecorded_phases_and_orders_total_first() {
let timings = RequestTimings::new();
Expand Down
64 changes: 64 additions & 0 deletions docs/superpowers/plans/2026-08-26-auction-timeline-offsets.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# Auction Timeline Offsets Implementation Plan

> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.

**Goal:** Record three T0-anchored auction milestones (dispatched, resolved, committed) plus the auction id on `RequestTimings`, and emit them as four additive columns on the `access_logs_raw` row.

**Architecture:** Follows spec section 18 exactly. All state lives in the existing `RequestTimings` inner (same `try_lock`/first-call-wins/saturating model as `mark_headers_ready`); the row builder reads the values from `TimingSnapshot`, so no new emission path and no adapter changes.

**Tech Stack:** Rust (core crate only), Tinybird datasource file.

**Spec:** `docs/superpowers/specs/2026-08-24-request-phase-timing-design.md` section 18.

## Global Constraints

- Marks are first-call-wins; `try_lock` only; a contended lock drops the sample.
- Null offsets mean "no auction ran", never zero. `auction_id` sentinel is `none`.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The plan is committed with every checkbox unticked and stale line references, and its own constraints are contradicted by what shipped in the same PR.

All 13 - [ ] steps are unchecked although the implementation landed in commits 46911a6 and later in this PR, so anyone opening the file reads it as outstanding work. The step anchors ('~line 4341', '~3952', '~4012', '~3954', '~4017') no longer match the file. The Global Constraints say 'Null offsets mean "no auction ran", never zero' (contradicted by the abandoned-auction paths) and 'no changes outside trusted-server-core and tinybird/' (the PR also changes .gitignore and two docs).

- Column names: `auction_dispatched_ms`, `auction_resolved_ms`, `auction_committed_ms`, `auction_id`; JSONPaths `json:$.<name>`; FORWARD_QUERY extended in the same order.
- Dispatch mark records only on `DispatchAuctionOutcome::Dispatched`; a failed dispatch leaves all three offsets null (the auction dataset still records the failure).
- No header emission, no config surface, no changes outside `trusted-server-core` and `tinybird/`.

---

### Task 1: RequestTimings marks and snapshot fields

**Files:**

- Modify: `crates/trusted-server-core/src/request_timing.rs`

**Interfaces:**

- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option<u32>, auction_id: Option<String>, .. }`

- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option<Duration>` and `auction_id: Option<String>` to `Inner`; initialize `None`.
- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins.
- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone.
- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`.
- [ ] `cargo test-fastly request_timing`, commit.

### Task 2: Publisher call sites

**Files:**

- Modify: `crates/trusted-server-core/src/publisher.rs`

**Interfaces:**

- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site.

- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());`
- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`.
- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`.
- [ ] `cargo test-fastly`, commit.

### Task 3: Row columns and datasource

**Files:**

- Modify: `crates/trusted-server-core/src/access_telemetry.rs`
Comment on lines +25 to +58

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrenchformat-docs CI fails on this file. Prettier 3.8.1 (the pinned docs/node_modules version) requires a blank line between a **Files:** / **Interfaces:** paragraph and the list that follows it; five are missing across the three tasks.

Reproduced locally with the pinned binary, and verified that this replacement makes prettier --check pass on both docs files in this PR with no other formatting drift.

Suggested change
**Files:**
- Modify: `crates/trusted-server-core/src/request_timing.rs`
**Interfaces:**
- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option<u32>, auction_id: Option<String>, .. }`
- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option<Duration>` and `auction_id: Option<String>` to `Inner`; initialize `None`.
- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins.
- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone.
- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`.
- [ ] `cargo test-fastly request_timing`, commit.
### Task 2: Publisher call sites
**Files:**
- Modify: `crates/trusted-server-core/src/publisher.rs`
**Interfaces:**
- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site.
- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());`
- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`.
- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`.
- [ ] `cargo test-fastly`, commit.
### Task 3: Row columns and datasource
**Files:**
- Modify: `crates/trusted-server-core/src/access_telemetry.rs`
**Files:**
- Modify: `crates/trusted-server-core/src/request_timing.rs`
**Interfaces:**
- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option<u32>, auction_id: Option<String>, .. }`
- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option<Duration>` and `auction_id: Option<String>` to `Inner`; initialize `None`.
- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins.
- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone.
- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`.
- [ ] `cargo test-fastly request_timing`, commit.
### Task 2: Publisher call sites
**Files:**
- Modify: `crates/trusted-server-core/src/publisher.rs`
**Interfaces:**
- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site.
- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());`
- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`.
- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`.
- [ ] `cargo test-fastly`, commit.
### Task 3: Row columns and datasource
**Files:**
- Modify: `crates/trusted-server-core/src/access_telemetry.rs`

- Modify: `tinybird/datasources/access_logs_raw.datasource`
Comment on lines +25 to +59

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔧 wrenchformat-docs CI is failing on this file. prettier --check flags six missing blank lines after the **Files:** / **Interfaces:** headings, which contradicts the PR body's "All CI gates pass locally."

Reproduced locally:

$ cd docs && npx prettier --check superpowers/plans/2026-08-26-auction-timeline-offsets.md
Checking formatting...
[warn] superpowers/plans/2026-08-26-auction-timeline-offsets.md
[warn] Code style issues found in the above file. Run Prettier with --write to fix.

The suggestion below is prettier --write output verbatim. Verified in a scratch worktree: with exactly these bytes, prettier --check reports "All matched files use Prettier code style!"

Suggested change
**Files:**
- Modify: `crates/trusted-server-core/src/request_timing.rs`
**Interfaces:**
- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option<u32>, auction_id: Option<String>, .. }`
- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option<Duration>` and `auction_id: Option<String>` to `Inner`; initialize `None`.
- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins.
- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone.
- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`.
- [ ] `cargo test-fastly request_timing`, commit.
### Task 2: Publisher call sites
**Files:**
- Modify: `crates/trusted-server-core/src/publisher.rs`
**Interfaces:**
- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site.
- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());`
- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`.
- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`.
- [ ] `cargo test-fastly`, commit.
### Task 3: Row columns and datasource
**Files:**
- Modify: `crates/trusted-server-core/src/access_telemetry.rs`
- Modify: `tinybird/datasources/access_logs_raw.datasource`
**Files:**
- Modify: `crates/trusted-server-core/src/request_timing.rs`
**Interfaces:**
- Produces: `mark_auction_dispatched(&self, auction_id: String)`, `mark_auction_resolved(&self)`, `mark_auction_committed(&self)`; `TimingSnapshot { auction_dispatched_ms, auction_resolved_ms, auction_committed_ms: Option<u32>, auction_id: Option<String>, .. }`
- [ ] Add `auction_dispatched`, `auction_resolved`, `auction_committed: Option<Duration>` and `auction_id: Option<String>` to `Inner`; initialize `None`.
- [ ] Add the three mark methods, first-call-wins on their own field, storing `inner.t0.elapsed()`; dispatched also stores the id first-call-wins.
- [ ] Map all four into `TimingSnapshot` via `duration_ms` / clone.
- [ ] Tests: first-call-wins per mark; snapshot maps offsets and id; unmarked snapshot yields all `None`.
- [ ] `cargo test-fastly request_timing`, commit.
### Task 2: Publisher call sites
**Files:**
- Modify: `crates/trusted-server-core/src/publisher.rs`
**Interfaces:**
- Consumes: Task 1 methods; `observation.auction_id` (`AuctionObservationContext`), in scope at the dispatch site.
- [ ] In the `DispatchAuctionOutcome::Dispatched` arm (~line 4341): `timings.mark_auction_dispatched(observation.auction_id.to_string());`
- [ ] After both `record_auction_wait` calls (collect sites ~3952 and ~4012): `mark_auction_resolved()`.
- [ ] After both `write_bids_to_state` calls (~3954 and ~4017): `mark_auction_committed()`.
- [ ] `cargo test-fastly`, commit.
### Task 3: Row columns and datasource
**Files:**
- Modify: `crates/trusted-server-core/src/access_telemetry.rs`
- Modify: `tinybird/datasources/access_logs_raw.datasource`


- [ ] `access_event_row`: add the three offset keys (nullable) and `auction_id` with `none` sentinel, after the existing phase keys.
- [ ] Extend `row_serializes_nulls_for_missing_phases` and `row_serializes_recorded_phases_as_numbers` for the new keys.
- [ ] Datasource: four schema columns with JSONPaths (`Nullable(UInt32)` ×3, `String`), appended at the end of SCHEMA and FORWARD_QUERY so existing column order stays stable.
- [ ] Full gates: fmt, clippy (all six), test-fastly/axum/cloudflare/spin, parity. Commit.
Loading
Loading