diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index fb0233b03..58e6a4ac7 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -27,6 +27,11 @@ jobs: run: echo "viceroy-version=$(grep '^viceroy ' .tool-versions | awk '{print $2}')" >> $GITHUB_OUTPUT shell: bash + - name: Retrieve Node.js version + id: node-version + run: echo "node-version=$(grep '^nodejs ' .tool-versions | awk '{print $2}')" >> $GITHUB_OUTPUT + shell: bash + - name: Set up Rust toolchain uses: actions-rust-lang/setup-rust-toolchain@v1 with: @@ -45,9 +50,20 @@ jobs: if: steps.cache-viceroy.outputs.cache-hit != 'true' run: cargo install viceroy --version "${{ steps.viceroy-version.outputs.viceroy-version }}" --locked --force + - name: Use Node.js for the served-seam contract + uses: actions/setup-node@v4 + with: + node-version: ${{ steps.node-version.outputs.node-version }} + - name: Run tests run: cargo test-fastly + - name: Run template cache ESI local harness + run: BID_DELAY=3 ./scripts/template-cache-local-test.sh esi + + - name: Run inline control harness + run: BID_DELAY=3 ./scripts/template-cache-local-test.sh inline + test-axum: name: cargo test (axum native) runs-on: ubuntu-latest @@ -214,9 +230,14 @@ jobs: run: | cargo clippy --manifest-path crates/trusted-server-cli/Cargo.toml --target "$(rustc -vV | sed -n 's/host: //p')" --all-targets -- -D warnings + - name: Set up Chrome for browser fixture tests + id: setup-chrome + uses: browser-actions/setup-chrome@v1 + - name: cargo test - run: | - cargo test --manifest-path crates/trusted-server-cli/Cargo.toml --target "$(rustc -vV | sed -n 's/host: //p')" + run: ./scripts/test-cli.sh + env: + CHROME: ${{ steps.setup-chrome.outputs.chrome-path }} test-typescript: name: vitest diff --git a/.gitignore b/.gitignore index 8ff935162..24b9e06aa 100644 --- a/.gitignore +++ b/.gitignore @@ -37,6 +37,9 @@ src/*.html .specstory .vscode +# Agent implementation worktrees +/.worktrees/ + # Claude Code — ignore all, then whitelist shared config .claude/* !.claude/settings.json diff --git a/CHANGELOG.md b/CHANGELOG.md index c00487769..6fee6e5c5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,8 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- **Breaking** — Admin Basic-auth coverage now includes `GET /_ts/admin/ec`, `GET /_ts/admin/ec/{id}`, and `GET /_ts/admin/eids`. Existing configurations whose `[[handlers]]` patterns protect only the key-management endpoints now fail startup; broaden coverage before deploying, preferably with a namespace-boundary pattern such as `^/_ts/admin(?:/|$)`. Coverage of the dynamic `/_ts/admin/ec/{id}` route is no longer inferred from ID-shaped samples: the router accepts any segment after `/_ts/admin/ec/` and Basic Auth runs on the raw path before routing, so patterns anchored to the EC ID grammar (for example `^/_ts/admin/ec/[a-f0-9]{64}[.][A-Za-z0-9]{6}$`) are rejected in favor of a prefix-level matcher. Placeholder and well-known weak handler passwords (`changeme`, `password`, `admin`, `replace-with-…`) now fail startup on every handler rather than only on handlers inferred to cover an admin endpoint, because first-match-wins handler selection lets a narrow handler shadow the admin namespace. +- Publisher HTML uses `Cache-Control: max-age=60` for successful GET document responses when server-side ad templates are structurally inactive, while preserving origin `private`/`no-store` policies and request-scoped bot, prefetch, or consent-denied responses. Cookie-bearing responses are finalized as `private, max-age=0`; CDN-specific cache headers remain unchanged for inactive templates. Set `[creative_opportunities].enabled = false` to disable publisher HTML and SPA template delivery without disabling direct `POST /auction` callers; an absent configuration, an unmatched slot, or a disabled auction also make the stack structurally inactive. An explicit `enabled = false` is not compatible with older binaries: restore the default, re-push and finalize the config before rolling back. - **Breaking** — Replaced the legacy APS contextual integration with APS OpenRTB at `/e/pb/bid`. APS configuration now uses canonical `account_id` (`pub_id` remains a compatibility alias), no longer requires APS-specific slot IDs, and defaults script creative eligibility off. Operators must update the endpoint, disable native APS demand for Trusted Server cohorts, and prepare GAM/Universal Creative targeting for `hb_bidder=aps` before rollout. `aps` entries in Prebid bidder lists are logged and stripped. APS renderer winners now preserve the upstream bid `id`, omit `crid` when APS omits it, and carry `ext.trusted_server.renderer` instead of `adm`; external `/auction` consumers must support this response shape. - **Breaking** — All auction paths now forward only a validated publisher-owned page URL as `site.page`, removing query and fragment data. APS OpenRTB omits `site.ref`; the existing Prebid Server path continues to forward the browser `Referer` as `site.ref`. Query-driven sites may lose contextual targeting and per-page reporting signals that previously came from query parameters. +- Publisher HTML now uses `Cache-Control: max-age=60` when server-side ad templates are inactive, while preserving origin `private`/`no-store` policies and CDN-specific cache headers. Set `[creative_opportunities].enabled = false` to disable publisher HTML and SPA template delivery without disabling direct `POST /auction` callers. - **Breaking** — `bid_param_zone_overrides` inner values must now be JSON objects; previously non-object or empty values (`"header" = "x"`, `"header" = {}`) were accepted and silently produced a dead rule at runtime. They now fail at startup with a configuration error. Operators upgrading should audit their `bid_param_zone_overrides` config for non-object zone entries. - **Breaking** — Integration configuration strings are no longer globally reinterpreted as JSON scalars. Operators upgrading should audit `[integrations.*]` settings and use native TOML/typed-config booleans and numbers (for example, `enabled = true`, not `enabled = "true"`); quoted numeric and boolean scalars now fail validation instead of silently converting. - **Breaking** — Sourcepoint browser module inclusion now requires explicit `[integrations.sourcepoint].enabled = true`; operators relying on the previous unconditional Sourcepoint module should enable the integration before upgrading. @@ -21,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Security +- Reserved the complete admin namespace at the publisher-fallback boundary. Percent-encoded separators (`/_ts/admin%2Fec`, `%2f`, and double-encoded forms) matched the `^/_ts/admin` Basic-auth handler but escaped the literal-slash namespace check, so an authenticated request fell through to publisher fallback and forwarded its `Authorization` header and body to the publisher origin. The reservation now spans the whole `/_ts/admin` prefix plus the retired `/admin/keys` aliases — including trailing, descendant, and encoded-separator forms — evaluated on both the raw and percent-decoded path, and applies to every adapter. - Validate synthetic ID format on inbound values from the `x-synthetic-id` header and `synthetic_id` cookie; values that do not match the expected format (`64-hex-hmac.6-alphanumeric-suffix`) are discarded and a fresh ID is generated rather than forwarded to response headers, cookies, or third-party APIs ### Fixed diff --git a/Cargo.lock b/Cargo.lock index cb8f40c68..b8edbb732 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -146,7 +146,7 @@ dependencies = [ "asn1-rs-derive", "asn1-rs-impl", "displaydoc", - "nom", + "nom 7.1.3", "num-traits", "rusticata-macros", "thiserror 1.0.69", @@ -254,6 +254,15 @@ dependencies = [ "tungstenite", ] +[[package]] +name = "atoi" +version = "2.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "f28d99ec8bfea296261ca1af174f24225171fea9664ba9003cbebee704810528" +dependencies = [ + "num-traits", +] + [[package]] name = "atomic-waker" version = "1.1.2" @@ -573,7 +582,18 @@ checksum = "c3613f74bd2eac03dad61bd53dbe620703d4371614fe0bc3b9f04dd36fe4e818" dependencies = [ "cfg-if", "cipher", - "cpufeatures", + "cpufeatures 0.2.17", +] + +[[package]] +name = "chacha20" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.1", ] [[package]] @@ -583,7 +603,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "10cd79432192d1c0f4e1a0fef9527696cc039165d729fb41b3f4f4f354c2dc35" dependencies = [ "aead", - "chacha20", + "chacha20 0.9.1", "cipher", "poly1305", "zeroize", @@ -767,7 +787,7 @@ version = "3.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "faf9468729b8cbcea668e36183cb69d317348c2e08e994829fb56ebfdfbaac34" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] @@ -916,6 +936,15 @@ dependencies = [ "libc", ] +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + [[package]] name = "crc32fast" version = "1.5.0" @@ -1041,7 +1070,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "curve25519-dalek-derive", "digest 0.10.7", "fiat-crypto", @@ -1186,7 +1215,7 @@ checksum = "5cd0a5c643689626bec213c4d8bd4d96acc8ffdb4ad4bb6bc16abf27d5f4b553" dependencies = [ "asn1-rs", "displaydoc", - "nom", + "nom 7.1.3", "num-bigint", "num-traits", "rusticata-macros", @@ -1398,7 +1427,7 @@ dependencies = [ [[package]] name = "edgezero-adapter" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "toml", ] @@ -1406,7 +1435,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-axum" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "anyhow", "async-trait", @@ -1434,7 +1463,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-cloudflare" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "anyhow", "async-trait", @@ -1449,7 +1478,7 @@ dependencies = [ "log", "serde_json", "tempfile", - "toml_edit", + "toml_edit 0.25.12+spec-1.1.0", "walkdir", "worker", ] @@ -1457,7 +1486,7 @@ dependencies = [ [[package]] name = "edgezero-adapter-fastly" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "anyhow", "async-stream", @@ -1479,14 +1508,14 @@ dependencies = [ "serde_json", "sha2 0.10.9", "thiserror 2.0.18", - "toml_edit", + "toml_edit 0.25.12+spec-1.1.0", "walkdir", ] [[package]] name = "edgezero-adapter-spin" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "anyhow", "async-trait", @@ -1506,14 +1535,14 @@ dependencies = [ "subtle", "thiserror 2.0.18", "toml", - "toml_edit", + "toml_edit 0.25.12+spec-1.1.0", "walkdir", ] [[package]] name = "edgezero-cli" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "chrono", "clap", @@ -1538,7 +1567,7 @@ dependencies = [ [[package]] name = "edgezero-core" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "anyhow", "async-compression", @@ -1569,14 +1598,14 @@ dependencies = [ [[package]] name = "edgezero-macros" version = "0.1.0" -source = "git+https://github.com/stackpop/edgezero?tag=v0.0.4#9e661ae520a8130660f18fd10f42703d7f3e050b" +source = "git+https://github.com/stackpop/edgezero?branch=feature%2Fedgezero-deploy-actions#5f3d648c3c6c38fc6e6b22b5c65c66177363aad8" dependencies = [ "log", "proc-macro2", "quote", "serde", "serde_json", - "syn 2.0.118", + "syn 3.0.3", "toml", "validator", ] @@ -1690,6 +1719,26 @@ dependencies = [ "rustc_version", ] +[[package]] +name = "esi" +version = "0.7.1" +source = "git+https://github.com/stackpop/esi.git?rev=4c53feab4d22ad9a84641b4c46f3f63bc6d197e2#4c53feab4d22ad9a84641b4c46f3f63bc6d197e2" +dependencies = [ + "atoi", + "base64", + "bytes", + "chrono", + "fastly", + "html-escape", + "log", + "md5", + "nom 8.0.0", + "percent-encoding", + "rand 0.10.2", + "regex", + "thiserror 2.0.18", +] + [[package]] name = "etcetera" version = "0.10.0" @@ -2034,6 +2083,7 @@ dependencies = [ "cfg-if", "libc", "r-efi 6.0.0", + "rand_core 0.10.1", ] [[package]] @@ -2188,6 +2238,12 @@ dependencies = [ "windows-sys 0.61.2", ] +[[package]] +name = "html-escape" +version = "0.2.15" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c9356095b4b41197bba32173600e1582792cda618f65d12f68e2e77d273413c5" + [[package]] name = "html5ever" version = "0.35.0" @@ -2914,6 +2970,12 @@ version = "0.9.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8863b587001c1b9a8a4e36008cebc6b3612cb1226fe2de94858e06092687b608" +[[package]] +name = "md5" +version = "0.8.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7ebb8d8732c6a6df3d8f032a82911cfc747e00efb95cc46e8d0acd5b5b88570c" + [[package]] name = "memchr" version = "2.8.2" @@ -2984,6 +3046,15 @@ dependencies = [ "minimal-lexical", ] +[[package]] +name = "nom" +version = "8.0.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "df9761775871bdef83bee530e60050f7e54b1105350d6884eb0fb4f46c2f9405" +dependencies = [ + "memchr", +] + [[package]] name = "num" version = "0.4.3" @@ -3477,7 +3548,7 @@ version = "0.8.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8159bd90725d2df49889a078b54f4f79e87f1f8a8444194cdca81d38f5393abf" dependencies = [ - "cpufeatures", + "cpufeatures 0.2.17", "opaque-debug", "universal-hash", ] @@ -3604,7 +3675,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf" dependencies = [ "heck", - "itertools 0.13.0", + "itertools 0.10.5", "log", "multimap", "once_cell", @@ -3624,7 +3695,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d" dependencies = [ "anyhow", - "itertools 0.13.0", + "itertools 0.10.5", "proc-macro2", "quote", "syn 2.0.118", @@ -3637,7 +3708,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf" dependencies = [ "anyhow", - "itertools 0.13.0", + "itertools 0.10.5", "proc-macro2", "quote", "syn 2.0.118", @@ -3775,6 +3846,17 @@ dependencies = [ "rand_core 0.9.5", ] +[[package]] +name = "rand" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c7f5fa3a058cd35567ef9bfa5e75732bee0f9e4c55fa90477bef2dfcdbc4be80" +dependencies = [ + "chacha20 0.10.1", + "getrandom 0.4.3", + "rand_core 0.10.1", +] + [[package]] name = "rand_chacha" version = "0.3.1" @@ -3813,6 +3895,12 @@ dependencies = [ "getrandom 0.3.4", ] +[[package]] +name = "rand_core" +version = "0.10.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "63b8176103e19a2643978565ca18b50549f6101881c443590420e4dc998a3c69" + [[package]] name = "rcgen" version = "0.13.2" @@ -4079,7 +4167,7 @@ version = "4.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632" dependencies = [ - "nom", + "nom 7.1.3", ] [[package]] @@ -4494,7 +4582,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest 0.10.7", ] @@ -4506,7 +4594,7 @@ checksum = "4d58a1e1bf39749807d89cf2d98ac2dfa0ff1cb3faa38fbb64dd88ac8013d800" dependencies = [ "block-buffer 0.9.0", "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest 0.9.0", "opaque-debug", ] @@ -4518,7 +4606,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest 0.10.7", ] @@ -4769,6 +4857,17 @@ dependencies = [ "unicode-ident", ] +[[package]] +name = "syn" +version = "3.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "53e9bae58849f64dfa4f5d5ae372c8341f7305f82a3868709269343628b659a3" +dependencies = [ + "proc-macro2", + "quote", + "unicode-ident", +] + [[package]] name = "sync_wrapper" version = "1.0.2" @@ -5074,6 +5173,19 @@ dependencies = [ "winnow 0.7.15", ] +[[package]] +name = "toml_edit" +version = "0.25.12+spec-1.1.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "d2153edc6955a6c354fad8f5efd38b6a8769bdccf9fe50f8e1329f81b0baa5d7" +dependencies = [ + "indexmap 2.14.0", + "toml_datetime 1.1.1+spec-1.1.0", + "toml_parser", + "toml_writer", + "winnow 1.0.3", +] + [[package]] name = "toml_parser" version = "1.1.2+spec-1.1.0" @@ -5273,9 +5385,11 @@ dependencies = [ "base64", "bytes", "chrono", + "derive_more", "edgezero-adapter-fastly", "edgezero-core", "error-stack", + "esi", "fastly", "fern", "futures", @@ -5322,8 +5436,11 @@ dependencies = [ "derive_more", "directories", "edgezero-cli", + "edgezero-core", "error-stack", "futures", + "glob", + "http", "http-body-util", "hyper", "hyper-util", @@ -5335,12 +5452,15 @@ dependencies = [ "scraper", "serde", "serde_json", + "similar", + "temp-env", "tempfile", "time", "tokio", "tokio-rustls", "toml", - "toml_edit", + "toml_edit 0.23.10+spec-1.0.0", + "tracing", "trusted-server-core", "url", "webpki-roots", @@ -5373,6 +5493,7 @@ dependencies = [ "hex", "hmac", "http", + "httpdate", "iab_gpp", "jose-jwk", "log", @@ -5896,7 +6017,7 @@ version = "0.1.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22" dependencies = [ - "windows-sys 0.61.2", + "windows-sys 0.48.0", ] [[package]] @@ -6325,7 +6446,7 @@ dependencies = [ "data-encoding", "der-parser", "lazy_static", - "nom", + "nom 7.1.3", "oid-registry", "ring", "rusticata-macros", diff --git a/Cargo.toml b/Cargo.toml index 7ca87e687..1d9281b47 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -54,14 +54,15 @@ criterion = { version = "0.5", default-features = false, features = ["cargo_benc derive_more = { version = "2.0", features = ["display", "error"] } directories = "5" ed25519-dalek = { version = "2.2", features = ["rand_core"] } -edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.4", default-features = false } -edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.4", default-features = false } -edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.4", default-features = false } -edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.4", default-features = false } -edgezero-cli = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.4" } -edgezero-core = { git = "https://github.com/stackpop/edgezero", tag = "v0.0.4", default-features = false } +edgezero-adapter-axum = { git = "https://github.com/stackpop/edgezero", branch = "feature/edgezero-deploy-actions", default-features = false } +edgezero-adapter-cloudflare = { git = "https://github.com/stackpop/edgezero", branch = "feature/edgezero-deploy-actions", default-features = false } +edgezero-adapter-fastly = { git = "https://github.com/stackpop/edgezero", branch = "feature/edgezero-deploy-actions", default-features = false } +edgezero-adapter-spin = { git = "https://github.com/stackpop/edgezero", branch = "feature/edgezero-deploy-actions", default-features = false } +edgezero-cli = { git = "https://github.com/stackpop/edgezero", branch = "feature/edgezero-deploy-actions" } +edgezero-core = { git = "https://github.com/stackpop/edgezero", branch = "feature/edgezero-deploy-actions", default-features = false } env_logger = "0.11" error-stack = "0.6" +esi = { git = "https://github.com/stackpop/esi.git", rev = "4c53feab4d22ad9a84641b4c46f3f63bc6d197e2" } fastly = "0.12" fern = "0.7.1" flate2 = "1.1" @@ -71,6 +72,7 @@ getrandom = "0.2" hex = "0.4.3" hmac = "0.12.1" http = "1.4.0" +httpdate = "1.0.3" http-body-util = "0.1" hyper = "1" hyper-util = "0.1" @@ -94,6 +96,7 @@ scraper = "0.24.0" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0.149" sha2 = "0.10.9" +similar = "2.7" simple_logger = "5" spin-sdk = { version = "~6.0", default-features = false, features = ["http", "key-value", "variables"] } subtle = "2.6" @@ -106,6 +109,7 @@ tokio-rustls = "0.26" toml = "1.1" toml_edit = "0.23.10" tower = "0.4" +tracing = "0.1" trusted-server-core = { path = "crates/trusted-server-core" } trusted-server-js = { path = "crates/trusted-server-js" } trusted-server-openrtb = { path = "crates/trusted-server-openrtb" } diff --git a/README.md b/README.md index b87fe61ad..81794720c 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ ts config init ts config validate # Audit a public page with Chrome/Chromium to bootstrap a draft config -ts audit https://publisher.example +ts audit generate https://publisher.example # Run tests (Fastly/WASM crates — requires Viceroy) cargo test-fastly diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index 1bed830ac..4b71d07ce 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -11,7 +11,12 @@ use edgezero_core::router::RouterService; use error_stack::Report; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::ec::EcContext; +use trusted_server_core::ec::admin::{ + admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup, +}; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; use trusted_server_core::proxy::{ @@ -178,12 +183,16 @@ async fn dispatch_fallback( services: &RuntimeServices, mut req: Request, ) -> Result> { + if let Some(response) = deny_admin_diagnostic_fallback(&req) { + return Ok(response); + } + trusted_server_core::integrations::gpt_diagnostics::prepare_request(&state.settings, &mut req)?; let path = req.uri().path().to_string(); let method = req.method().clone(); if method == Method::GET && path.starts_with("/static/tsjs=") { - return handle_tsjs_dynamic(&req, &state.registry); + return handle_tsjs_dynamic(&req, &state.registry, EdgeCacheHeader::SMaxageFallback); } if state.registry.has_route(&method, &path) { @@ -222,6 +231,7 @@ async fn dispatch_fallback( &mut ec_context, auction, req, + EdgeCacheHeader::SMaxageFallback, ) .await?; // Async finalize so the dispatched auction is collected and its bids are @@ -259,6 +269,8 @@ enum NamedRouteHandler { TrustedServerDiscovery, VerifySignature, AdminNotSupported, + AdminEcNotSupported, + AdminEidsLookup, /// Legacy `/admin/keys/*` aliases — denied locally with 404 so they never /// reach the publisher fallback (which would leak admin credentials). LegacyAdminDenied, @@ -286,7 +298,7 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ Method::DELETE, ]; -fn named_routes() -> [NamedRoute; 13] { +fn named_routes() -> [NamedRoute; 16] { [ NamedRoute { path: "/.well-known/trusted-server.json", @@ -311,6 +323,26 @@ fn named_routes() -> [NamedRoute; 13] { primary_methods: &[Method::POST], handler: NamedRouteHandler::AdminNotSupported, }, + // Admin EC lookup routes. Registered explicitly (like the key routes + // above) so they never fall through to the publisher fallback, and + // they match `Settings::ADMIN_ENDPOINTS` for auth coverage. + NamedRoute { + path: "/_ts/admin/ec", + primary_methods: &[Method::GET], + handler: NamedRouteHandler::AdminEcNotSupported, + }, + NamedRoute { + path: "/_ts/admin/ec/{id}", + primary_methods: &[Method::GET], + handler: NamedRouteHandler::AdminEcNotSupported, + }, + // Admin EIDs echo: pure request inspection (no KV), so the dev + // server serves the real handler. + NamedRoute { + path: "/_ts/admin/eids", + primary_methods: &[Method::GET], + handler: NamedRouteHandler::AdminEidsLookup, + }, // The legacy non-`/_ts` aliases (`/admin/keys/*`) are denied locally with // a 404, matching the Fastly and Cloudflare adapters: the production // basic-auth handler regex `^/_ts/admin` does not match them, and letting @@ -407,6 +439,16 @@ fn named_route_handler( ); Ok(resp) } + NamedRouteHandler::AdminEcNotSupported => { + // The EC identity graph is Fastly KV backed; the Axum + // dev server has no store to read. + Ok(admin_ec_lookup_not_supported()) + } + NamedRouteHandler::AdminEidsLookup => { + let partner_registry = + PartnerRegistry::from_config(&state.settings.ec.partners)?; + handle_admin_eids_lookup(&partner_registry, &req) + } NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()), NamedRouteHandler::Auction => { // Build the geo-aware EC context so the auction consent diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 03caa3d11..85f567a53 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -74,6 +74,9 @@ fn all_explicit_routes_are_registered() { ("POST", "/verify-signature"), ("POST", "/_ts/admin/keys/rotate"), ("POST", "/_ts/admin/keys/deactivate"), + ("GET", "/_ts/admin/ec"), + ("GET", "/_ts/admin/ec/{id}"), + ("GET", "/_ts/admin/eids"), ("POST", "/admin/keys/rotate"), ("POST", "/admin/keys/deactivate"), ("POST", "/auction"), @@ -208,6 +211,42 @@ async fn tsjs_route_prefix_is_handled_not_5xx() { ); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn tsjs_route_matching_hash_uses_s_maxage_fallback() { + let mut svc = make_service(); + let src = trusted_server_core::tsjs::tsjs_script_src(&["creative"]); + let req = Request::builder() + .method("GET") + .uri(src) + .body(AxumBody::empty()) + .expect("should build request"); + + let resp = svc + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond"); + + assert_eq!( + resp.status().as_u16(), + 200, + "matching TSJS hash should serve OK" + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|value| value.to_str().ok()), + Some("public, max-age=31536000, s-maxage=31536000, immutable"), + "Axum adapter should render the portable s-maxage fallback" + ); + assert!( + resp.headers().get("surrogate-control").is_none(), + "s-maxage fallback must not emit Fastly Surrogate-Control" + ); +} + // --------------------------------------------------------------------------- // Middleware tests // --------------------------------------------------------------------------- @@ -267,6 +306,189 @@ async fn admin_route_without_credentials_returns_401() { ); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_ec_routes_return_501() { + // The EC identity graph is Fastly KV backed, so the Axum dev server + // answers the admin EC lookup routes locally with 501 instead of letting + // them fall through to the publisher fallback. + let sample_ec_id = format!("{}.abc123", "a".repeat(64)); + for path in [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{sample_ec_id}"), + ] { + let mut svc = make_service(); + let req = Request::builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(AxumBody::empty()) + .expect("should build request"); + let resp = svc + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond"); + assert_eq!( + resp.status().as_u16(), + 501, + "{path} should report that Axum EC lookup is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json") + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn admin_ec_route_without_credentials_returns_401() { + let mut svc = make_service(); + let req = Request::builder() + .method("GET") + .uri("/_ts/admin/ec") + .body(AxumBody::empty()) + .expect("should build unauthenticated admin EC request"); + let resp = svc + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond"); + + assert_eq!(resp.status().as_u16(), 401); + assert!( + resp.headers().contains_key("www-authenticate"), + "admin EC 401 should include the Basic authentication challenge" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_eids_route_returns_200() { + // The EIDs echo is pure request inspection (no KV), so the dev server + // serves the real handler. + let mut svc = make_service(); + let req = Request::builder() + .method("GET") + .uri("/_ts/admin/eids") + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(AxumBody::empty()) + .expect("should build request"); + let resp = svc + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond"); + assert_eq!( + resp.status().as_u16(), + 200, + "/_ts/admin/eids should serve the real EIDs echo handler" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_diagnostic_fallback_is_denied_locally() { + let ec_id = format!("{}.abc123", "a".repeat(64)); + let valid_paths = [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{ec_id}"), + "/_ts/admin/eids".to_owned(), + ]; + + for path in valid_paths { + for method in ["POST", "HEAD", "OPTIONS", "PUT", "PATCH", "DELETE"] { + let request = Request::builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(AxumBody::from("sensitive-admin-body")) + .expect("should build authenticated admin request"); + let response = make_service() + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should respond"); + + assert_eq!(response.status().as_u16(), 405); + assert_eq!( + response + .headers() + .get("allow") + .and_then(|v| v.to_str().ok()), + Some("GET") + ); + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } + + for path in [ + "/_ts/admin/ec/".to_owned(), + format!("/_ts/admin/ec/{ec_id}/extra"), + "/_ts/admin/eids/".to_owned(), + "/_ts/admin/eids/extra".to_owned(), + "/_ts/admin/eids.json".to_owned(), + "/_ts/admin/ec;foo".to_owned(), + format!("/_ts/admin/ec%2F{ec_id}"), + // Percent-encoded separators match the `^/_ts/admin` basic-auth + // handler but not a literal-slash namespace check, so they must be + // reserved before publisher fallback forwards credentials upstream. + "/_ts/admin%2Fec".to_owned(), + "/_ts/admin%2fec".to_owned(), + // Retired non-`/_ts` alias namespace: only the two exact paths are + // routed to a local deny, so descendants and encoded separators must + // be reserved at the shared fallback boundary. + "/admin/keys".to_owned(), + "/admin/keys/rotate/extra".to_owned(), + "/admin/keys%2Frotate".to_owned(), + "/admin%2fkeys/rotate".to_owned(), + ] { + for method in ["GET", "POST"] { + let request = Request::builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(AxumBody::from("sensitive-admin-body")) + .expect("should build malformed admin request"); + let response = make_service() + .ready() + .await + .expect("should be ready") + .call(request) + .await + .expect("should respond"); + + assert_eq!(response.status().as_u16(), 404); + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn legacy_admin_aliases_denied_locally_not_proxied_to_publisher() { // Regression for the credential-leak finding: the production basic-auth regex diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 644676fc5..86ac86987 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -10,9 +10,15 @@ use edgezero_core::router::RouterService; use error_stack::Report; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; +use trusted_server_core::cache_policy::EdgeCacheHeader; #[cfg(target_arch = "wasm32")] use trusted_server_core::config_payload::settings_from_config_blob; use trusted_server_core::ec::EcContext; +use trusted_server_core::ec::admin::{ + admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, + deny_admin_diagnostic_fallback, handle_admin_eids_lookup, +}; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; use trusted_server_core::platform::RuntimeServices; @@ -249,6 +255,10 @@ fn admin_key_management_not_supported() -> Response { response } +fn admin_ec_lookup_not_supported() -> Response { + core_admin_ec_lookup_not_supported() +} + /// Builds the local `404 Not Found` returned for legacy `/admin/keys/*` /// aliases on the Cloudflare adapter. /// @@ -368,6 +378,9 @@ fn build_router(state: &Arc) -> RouterService { ) -> Result { let services = build_per_request_services(&ctx); let mut req = ctx.into_request(); + if let Some(response) = deny_admin_diagnostic_fallback(&req) { + return Ok(response); + } if let Err(error) = trusted_server_core::integrations::gpt_diagnostics::prepare_request( &state.settings, &mut req, @@ -380,7 +393,11 @@ fn build_router(state: &Arc) -> RouterService { let allow_tsjs = method == Method::GET; let result = if allow_tsjs && path.starts_with("/static/tsjs=") { - handle_tsjs_dynamic(&req, &state.registry) + handle_tsjs_dynamic( + &req, + &state.registry, + EdgeCacheHeader::CloudflareCdnCacheControl, + ) } else if state.registry.has_route(&method, &path) { let mut ec_context = EcContext::default(); state @@ -414,6 +431,7 @@ fn build_router(state: &Arc) -> RouterService { &mut ec_context, auction, req, + EdgeCacheHeader::CloudflareCdnCacheControl, ) .await { @@ -474,6 +492,26 @@ fn build_router(state: &Arc) -> RouterService { .post("/_ts/admin/keys/deactivate", |_ctx: RequestContext| async { Ok::(admin_key_management_not_supported()) }) + // Admin EC lookup routes. Registered explicitly (like the key + // routes above) so they never fall through to the publisher + // fallback, and they match `Settings::ADMIN_ENDPOINTS` for auth + // coverage. The EC identity graph is Fastly KV backed, so this + // adapter has no store to read. + .get("/_ts/admin/ec", |_ctx: RequestContext| async { + Ok::(admin_ec_lookup_not_supported()) + }) + .get("/_ts/admin/ec/{id}", |_ctx: RequestContext| async { + Ok::(admin_ec_lookup_not_supported()) + }) + // Admin EIDs echo: pure request inspection (no KV), so this + // adapter serves the real handler. + .get( + "/_ts/admin/eids", + make_handler(Arc::clone(&state), |s, _services, req| async move { + let partner_registry = PartnerRegistry::from_config(&s.settings.ec.partners)?; + handle_admin_eids_lookup(&partner_registry, &req) + }), + ) .post( "/auction", make_handler(Arc::clone(&state), |s, services, req| async move { diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index 09e3ed324..68fa48bf3 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -203,6 +203,43 @@ async fn tsjs_route_is_routed_not_5xx() { assert!(status < 500, "tsjs route must not 5xx: got {status}"); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn tsjs_route_emits_cloudflare_cache_header_for_matching_hash() { + let router = test_router(); + let src = trusted_server_core::tsjs::tsjs_script_src(&["creative"]); + let req = request_builder() + .method("GET") + .uri(src) + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + + let resp = route(router, req).await; + + assert_eq!( + resp.status().as_u16(), + 200, + "matching TSJS hash should serve OK" + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|value| value.to_str().ok()), + Some("public, max-age=31536000, immutable"), + "browser cache policy should be immutable for matching TSJS hash" + ); + assert_eq!( + resp.headers() + .get("cloudflare-cdn-cache-control") + .and_then(|value| value.to_str().ok()), + Some("max-age=31536000"), + "Cloudflare adapter should emit the Cloudflare-specific edge header" + ); + assert!( + resp.headers().get("surrogate-control").is_none(), + "Cloudflare adapter must not emit Fastly Surrogate-Control" + ); +} + /// Verify that every expected explicit route is registered in the route table. /// /// Uses [`RouterService::routes()`] for introspection rather than checking @@ -215,6 +252,9 @@ fn all_explicit_routes_are_registered() { ("POST", "/verify-signature"), ("POST", "/_ts/admin/keys/rotate"), ("POST", "/_ts/admin/keys/deactivate"), + ("GET", "/_ts/admin/ec"), + ("GET", "/_ts/admin/ec/{id}"), + ("GET", "/_ts/admin/eids"), ("POST", "/auction"), // SPA re-auction endpoint, plus its deprecated `/__ts/` alias. Both // paths are spelled out as literals rather than referencing @@ -275,6 +315,158 @@ async fn authenticated_admin_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_ec_routes_return_501() { + // The EC identity graph is Fastly KV backed, so Cloudflare answers the + // admin EC lookup routes locally with 501 instead of letting them fall + // through to the publisher fallback. + let sample_ec_id = format!("{}.abc123", "a".repeat(64)); + for path in [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{sample_ec_id}"), + ] { + let req = request_builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), req).await; + + assert_eq!( + resp.status().as_u16(), + 501, + "{path} should report that Cloudflare EC lookup is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json") + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn admin_ec_route_without_credentials_returns_401() { + let req = request_builder() + .method("GET") + .uri("/_ts/admin/ec") + .body(edgezero_core::body::Body::empty()) + .expect("should build unauthenticated admin EC request"); + let resp = route(test_router(), req).await; + + assert_eq!(resp.status().as_u16(), 401); + assert!( + resp.headers().contains_key("www-authenticate"), + "admin EC 401 should include the Basic authentication challenge" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_eids_route_returns_200() { + // The EIDs echo is pure request inspection (no KV), so this adapter + // serves the real handler. + let req = request_builder() + .method("GET") + .uri("/_ts/admin/eids") + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), req).await; + + assert_eq!( + resp.status().as_u16(), + 200, + "/_ts/admin/eids should serve the real EIDs echo handler" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_diagnostic_fallback_is_denied_locally() { + let ec_id = format!("{}.abc123", "a".repeat(64)); + let valid_paths = [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{ec_id}"), + "/_ts/admin/eids".to_owned(), + ]; + + for path in valid_paths { + for method in ["POST", "HEAD", "OPTIONS", "PUT", "PATCH", "DELETE"] { + let request = request_builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::from("sensitive-admin-body")) + .expect("should build authenticated admin request"); + let response = route(test_router(), request).await; + + assert_eq!(response.status().as_u16(), 405); + assert_eq!( + response + .headers() + .get("allow") + .and_then(|v| v.to_str().ok()), + Some("GET") + ); + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } + + for path in [ + "/_ts/admin/ec/".to_owned(), + format!("/_ts/admin/ec/{ec_id}/extra"), + "/_ts/admin/eids/".to_owned(), + "/_ts/admin/eids/extra".to_owned(), + "/_ts/admin/eids.json".to_owned(), + "/_ts/admin/ec;foo".to_owned(), + format!("/_ts/admin/ec%2F{ec_id}"), + // Percent-encoded separators match the `^/_ts/admin` basic-auth + // handler but not a literal-slash namespace check, so they must be + // reserved before publisher fallback forwards credentials upstream. + "/_ts/admin%2Fec".to_owned(), + "/_ts/admin%2fec".to_owned(), + // Retired non-`/_ts` alias namespace: only the two exact paths are + // routed to a local deny, so descendants and encoded separators must + // be reserved at the shared fallback boundary. + "/admin/keys".to_owned(), + "/admin/keys/rotate/extra".to_owned(), + "/admin/keys%2Frotate".to_owned(), + "/admin%2fkeys/rotate".to_owned(), + ] { + for method in ["GET", "POST"] { + let request = request_builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::from("sensitive-admin-body")) + .expect("should build malformed admin request"); + let response = route(test_router(), request).await; + + assert_eq!(response.status().as_u16(), 404); + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn admin_route_without_credentials_returns_401() { let router = test_router(); diff --git a/crates/trusted-server-adapter-fastly/Cargo.toml b/crates/trusted-server-adapter-fastly/Cargo.toml index b6bc0f1a1..47cc609b2 100644 --- a/crates/trusted-server-adapter-fastly/Cargo.toml +++ b/crates/trusted-server-adapter-fastly/Cargo.toml @@ -15,9 +15,11 @@ async-trait = { workspace = true } base64 = { workspace = true } bytes = { workspace = true } chrono = { workspace = true } +derive_more = { workspace = true } edgezero-adapter-fastly = { workspace = true, features = ["fastly"] } edgezero-core = { workspace = true } error-stack = { workspace = true } +esi = { workspace = true } fastly = { workspace = true } fern = { workspace = true } futures = { workspace = true } diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index d6090c983..41e5e65ee 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -22,6 +22,9 @@ //! | POST | `/verify-signature` | [`handle_verify_signature`] | //! | POST | `/_ts/admin/keys/rotate` | [`handle_rotate_key`] | //! | POST | `/_ts/admin/keys/deactivate` | [`handle_deactivate_key`] | +//! | GET | `/_ts/admin/ec` | [`handle_admin_ec_lookup`] | +//! | GET | `/_ts/admin/ec/{id}` | [`handle_admin_ec_lookup`] | +//! | GET | `/_ts/admin/eids` | [`handle_admin_eids_lookup`] | //! | POST | `/_ts/api/v1/batch-sync` | [`handle_batch_sync`] | //! | GET | `/_ts/api/v1/identify` | [`handle_identify`] | //! | GET | `/_ts/set-tester` | [`handle_set_tester`] | @@ -49,7 +52,8 @@ //! `route_request` (tracked in issue #495): //! //! - [`build_ec_request_state`] runs before every dispatched route (except -//! batch-sync, which uses Bearer auth) and reproduces the legacy +//! batch-sync, which uses Bearer auth, and the read-only admin diagnostics) +//! and reproduces the legacy //! pre-routing prelude: device signals, bot gate, `ts-eids`/`sharedid` //! cookie capture, geo lookup, [`EcContext`] creation, and KV-graph gating. //! - `handle_auction` and integration proxy dispatch receive the same @@ -97,8 +101,12 @@ use error_stack::Report; use trusted_server_core::auction::AuctionTelemetrySink; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; use trusted_server_core::ec::EcContext; +use trusted_server_core::ec::admin::{ + deny_admin_diagnostic_fallback, handle_admin_ec_lookup, handle_admin_eids_lookup, +}; use trusted_server_core::ec::batch_sync::handle_batch_sync; use trusted_server_core::ec::consent::ec_consent_withdrawn; use trusted_server_core::ec::device::DeviceSignals; @@ -257,6 +265,11 @@ fn build_per_request_services(state: &AppState, ctx: &RequestContext) -> Runtime .config_store(Arc::new(FastlyPlatformConfigStore)) .secret_store(Arc::new(FastlyPlatformSecretStore)) .kv_store(Arc::clone(&state.default_kv_store)) + // Spike-only (#1009). Constructed unconditionally, but only read when the + // assembly mode is a shared-template one — which defaults to Inline, so this + // is inert until an operator opts in. + .template_cache(Arc::new(crate::template_cache::FastlyTemplateCache::new())) + .template_assembler(Arc::new(crate::esi_assembly::FastlyTemplateAssembler)) .backend(Arc::new(FastlyPlatformBackend)) .http_client(Arc::new(FastlyPlatformHttpClient)) .geo(Arc::new(FastlyPlatformGeo)) @@ -525,6 +538,29 @@ async fn execute_named( return Ok(run_batch_sync(&state, &services, req)); } + // These diagnostics are read-only. Running the normal EC lifecycle would + // attach finalization state and could ingest request cookies into KV after + // the handler returns, violating that contract. + if matches!( + handler, + NamedRouteHandler::AdminEcLookup | NamedRouteHandler::AdminEidsLookup + ) { + let response = PartnerRegistry::from_config(&state.settings.ec.partners) + .and_then(|registry| match handler { + NamedRouteHandler::AdminEcLookup => { + // Deliberately do not use an EC request-state graph: that + // copy is bot-gated, while operators use curl for this + // authenticated diagnostic. + let kv = crate::maybe_identity_graph(&state.settings); + handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) + } + NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), + _ => unreachable!("admin diagnostics should use early dispatch"), + }) + .unwrap_or_else(|error| http_error(&error)); + return Ok(response); + } + if let Err(report) = trusted_server_core::integrations::gpt_diagnostics::prepare_request( &state.settings, &mut req, @@ -574,6 +610,9 @@ async fn run_named_route( } NamedRouteHandler::RotateKey => handle_rotate_key(&state.settings, services, req), NamedRouteHandler::DeactivateKey => handle_deactivate_key(&state.settings, services, req), + NamedRouteHandler::AdminEcLookup | NamedRouteHandler::AdminEidsLookup => { + unreachable!("admin diagnostics should be handled before EC setup") + } NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()), NamedRouteHandler::BatchSync => { // Dispatched by execute_named before EC state is built. @@ -708,6 +747,10 @@ async fn dispatch_fallback( services: &RuntimeServices, mut req: Request, ) -> Response { + if let Some(response) = deny_admin_diagnostic_fallback(&req) { + return response; + } + let path = req.uri().path().to_string(); let method = req.method().clone(); @@ -735,7 +778,7 @@ async fn dispatch_fallback( }; let result = if uses_dynamic_tsjs_fallback(&method, &path) { - handle_tsjs_dynamic(&req, &state.registry) + handle_tsjs_dynamic(&req, &state.registry, EdgeCacheHeader::SurrogateControl) } else if state.registry.has_route(&method, &path) { // Integration-proxy responses are not bounded by // publisher.max_buffered_body_bytes. Publisher fallback below uses the @@ -808,6 +851,7 @@ async fn dispatch_fallback( &mut ec.ec_context, auction, req, + EdgeCacheHeader::SurrogateControl, ) .await { @@ -1001,6 +1045,8 @@ enum NamedRouteHandler { VerifySignature, RotateKey, DeactivateKey, + AdminEcLookup, + AdminEidsLookup, /// Legacy `/admin/keys/*` aliases — denied locally with 404 so they never /// reach the publisher fallback (which would leak admin credentials). LegacyAdminDenied, @@ -1053,6 +1099,25 @@ const NAMED_ROUTES: &[NamedRoute] = &[ primary_methods: &[Method::POST], handler: NamedRouteHandler::DeactivateKey, }, + // Admin EC lookup: the bare route reads the EC ID from the caller's + // `ts-ec` cookie; the parameterized route takes an explicit EC ID. + NamedRoute { + path: "/_ts/admin/ec", + primary_methods: &[Method::GET], + handler: NamedRouteHandler::AdminEcLookup, + }, + NamedRoute { + path: "/_ts/admin/ec/{id}", + primary_methods: &[Method::GET], + handler: NamedRouteHandler::AdminEcLookup, + }, + // Admin EIDs echo: decodes the request's ts-eids/sharedId cookies with + // an ingestion preview. Pure request inspection — no KV access. + NamedRoute { + path: "/_ts/admin/eids", + primary_methods: &[Method::GET], + handler: NamedRouteHandler::AdminEidsLookup, + }, // The legacy non-`/_ts` aliases (`/admin/keys/*`) are denied locally with a // 404 instead of executing key operations: the production basic-auth handler // regex `^/_ts/admin` does not match them, and letting them fall through to @@ -1239,12 +1304,16 @@ mod tests { use super::{ AppState, NAMED_ROUTES, NamedRouteHandler, PAGE_BIDS_LEGACY_PATH, PAGE_BIDS_PATH, - TrustedServerApp, build_state_from_settings, startup_error_router, + TrustedServerApp, build_per_request_services, build_state_from_settings, + startup_error_router, }; + use base64::Engine as _; use bytes::Bytes; use edgezero_core::body::Body; + use edgezero_core::context::RequestContext; use edgezero_core::http::{Method, Response, StatusCode, header, request_builder}; use edgezero_core::key_value_store::NoopKvStore; + use edgezero_core::params::PathParams; use edgezero_core::router::RouterService; use std::net::{IpAddr, Ipv4Addr}; use std::sync::Mutex; @@ -1379,6 +1448,36 @@ mod tests { TrustedServerApp::routes_for_state(&state) } + #[test] + fn per_request_services_register_the_fastly_template_assembler() { + let state = build_state_from_settings(test_settings()).expect("should build test state"); + let context = RequestContext::new( + empty_request(Method::GET, "/article"), + PathParams::default(), + ); + + let services = build_per_request_services(&state, &context); + let template = format!( + "article{}", + trusted_server_core::publisher::AD_ASSEMBLY_SEAM + ); + let fragment = b""; + let assembled = services + .template_assembler() + .assemble(template.as_bytes(), fragment) + .expect("Fastly services should provide ESI assembly"); + + assert_eq!( + assembled, + template + .replace( + trusted_server_core::publisher::AD_ASSEMBLY_SEAM, + std::str::from_utf8(fragment).expect("fragment should be UTF-8") + ) + .into_bytes() + ); + } + /// Builds a router whose `AppState` uses a registry containing the given /// request filters (and no routes), so dispatch-level request-filter /// behavior can be exercised without a real integration. @@ -1651,6 +1750,44 @@ mod tests { } } + #[test] + fn admin_ec_lookup_routes_are_registered() { + // Both lookup shapes must be explicitly routed to the admin EC + // handler: the bare cookie-based route and the parameterized route. + // Leaving either unrouted would fall through to the publisher + // fallback, forwarding the caller's `Authorization` header to the + // origin. + for path in ["/_ts/admin/ec", "/_ts/admin/ec/{id}"] { + let route = NAMED_ROUTES + .iter() + .find(|route| route.path == path) + .unwrap_or_else(|| panic!("{path} must be a named route")); + assert!( + matches!(route.handler, NamedRouteHandler::AdminEcLookup), + "{path} must map to the admin EC lookup handler" + ); + assert_eq!( + route.primary_methods, + &[Method::GET], + "{path} must have GET as its only primary method" + ); + } + + let eids_route = NAMED_ROUTES + .iter() + .find(|route| route.path == "/_ts/admin/eids") + .expect("should register /_ts/admin/eids as a named route"); + assert!( + matches!(eids_route.handler, NamedRouteHandler::AdminEidsLookup), + "/_ts/admin/eids must map to the admin EIDs lookup handler" + ); + assert_eq!( + eids_route.primary_methods, + &[Method::GET], + "/_ts/admin/eids must have GET as its only primary method" + ); + } + #[test] fn page_bids_serves_canonical_path_and_deprecated_alias() { // The SPA re-auction endpoint lives at the canonical single-underscore @@ -1744,6 +1881,93 @@ mod tests { } } + #[test] + fn authenticated_admin_diagnostic_fallback_is_denied_locally() { + let router = test_router(); + let ec_id = format!("{}.abc123", "a".repeat(64)); + let valid_paths = [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{ec_id}"), + "/_ts/admin/eids".to_owned(), + ]; + + for path in valid_paths { + for method in [ + Method::POST, + Method::HEAD, + Method::OPTIONS, + Method::PUT, + Method::PATCH, + Method::DELETE, + ] { + let request = request_builder() + .method(method.clone()) + .uri(format!("https://test-publisher.com{path}")) + .header(header::AUTHORIZATION, "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(Body::from("sensitive-admin-body")) + .expect("should build authenticated admin request"); + let response = route(&router, request); + + assert_eq!(response.status(), StatusCode::METHOD_NOT_ALLOWED); + assert_eq!( + response + .headers() + .get(header::ALLOW) + .and_then(|v| v.to_str().ok()), + Some("GET") + ); + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } + + for path in [ + "/_ts/admin/ec/".to_owned(), + format!("/_ts/admin/ec/{ec_id}/extra"), + "/_ts/admin/eids/".to_owned(), + "/_ts/admin/eids/extra".to_owned(), + "/_ts/admin/eids.json".to_owned(), + "/_ts/admin/ec;foo".to_owned(), + format!("/_ts/admin/ec%2F{ec_id}"), + // Percent-encoded separators match the `^/_ts/admin` basic-auth + // handler but not a literal-slash namespace check, so they must be + // reserved before publisher fallback forwards credentials upstream. + "/_ts/admin%2Fec".to_owned(), + "/_ts/admin%2fec".to_owned(), + // Retired non-`/_ts` alias namespace: only the two exact paths are + // routed to a local deny, so descendants and encoded separators must + // be reserved at the shared fallback boundary. + "/admin/keys".to_owned(), + "/admin/keys/rotate/extra".to_owned(), + "/admin/keys%2Frotate".to_owned(), + "/admin%2fkeys/rotate".to_owned(), + ] { + for method in [Method::GET, Method::POST] { + let request = request_builder() + .method(method) + .uri(format!("https://test-publisher.com{path}")) + .header(header::AUTHORIZATION, "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(Body::from("sensitive-admin-body")) + .expect("should build malformed admin request"); + let response = route(&router, request); + + assert_eq!(response.status(), StatusCode::NOT_FOUND); + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } + } + #[test] fn dispatch_identify_options_routes_to_cors_preflight() { // Parity guard: OPTIONS /_ts/api/v1/identify must reach @@ -2029,6 +2253,103 @@ mod tests { ); } + #[test] + fn admin_eids_diagnostic_skips_ec_finalization() { + let router = test_router(); + let ec_id = format!("{}.abc123", "a".repeat(64)); + let eids = serde_json::json!([{ + "source": "example.com", + "uids": [{ "id": "example-uid", "atype": 1 }] + }]); + let eids_cookie = base64::engine::general_purpose::STANDARD.encode(eids.to_string()); + let mut request = request_builder() + .method(Method::GET) + .uri("https://test-publisher.com/_ts/admin/eids") + .header(header::AUTHORIZATION, "Basic YWRtaW46YWRtaW4tcGFzcw==") + .header( + header::COOKIE, + format!("ts-ec={ec_id}; ts-eids={eids_cookie}; sharedId=example-shared-id"), + ) + .body(Body::empty()) + .expect("should build authenticated EIDs diagnostic request"); + request.extensions_mut().insert(DeviceSignals::derive( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36", + Some("t13d1516h2_8daaf6152771_b186095e22b6"), + Some("1:65536;2:0;4:6291456;6:262144"), + )); + + let response = route(&router, request); + + assert_eq!(response.status(), StatusCode::OK); + assert!( + response + .extensions() + .get::() + .is_none(), + "admin EIDs diagnostics should not attach EC finalization state" + ); + } + + #[test] + fn admin_ec_diagnostic_skips_ec_finalization() { + let router = test_router(); + let ec_id = format!("{}.abc123", "a".repeat(64)); + let eids = serde_json::json!([{ + "source": "example.com", + "uids": [{ "id": "example-uid", "atype": 1 }] + }]); + let eids_cookie = base64::engine::general_purpose::STANDARD.encode(eids.to_string()); + let mut request = request_builder() + .method(Method::GET) + .uri(format!("https://test-publisher.com/_ts/admin/ec/{ec_id}")) + .header(header::AUTHORIZATION, "Basic YWRtaW46YWRtaW4tcGFzcw==") + .header( + header::COOKIE, + format!("ts-ec={ec_id}; ts-eids={eids_cookie}; sharedId=example-shared-id"), + ) + .body(Body::empty()) + .expect("should build authenticated EC diagnostic request"); + request.extensions_mut().insert(DeviceSignals::derive( + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 \ + (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36", + Some("t13d1516h2_8daaf6152771_b186095e22b6"), + Some("1:65536;2:0;4:6291456;6:262144"), + )); + + let response = route(&router, request); + + assert_eq!( + response.status(), + StatusCode::NOT_IMPLEMENTED, + "configured admin EC handler should run and report the unavailable test KV graph" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "admin EC diagnostics should not attach EC finalization state" + ); + assert!( + response.headers().get(header::SET_COOKIE).is_none(), + "admin EC diagnostics should not mutate the EC cookie" + ); + } + + #[test] + fn admin_ec_route_without_credentials_returns_401() { + let router = test_router(); + + let response = route(&router, empty_request(Method::GET, "/_ts/admin/ec")); + + assert_eq!(response.status(), StatusCode::UNAUTHORIZED); + assert!( + response.headers().contains_key(header::WWW_AUTHENTICATE), + "admin EC 401 should include the Basic authentication challenge" + ); + } + #[test] fn dispatch_head_on_named_get_route_falls_through_to_publisher_fallback() { // Regression guard: HEAD /first-party/proxy must reach the publisher diff --git a/crates/trusted-server-adapter-fastly/src/esi_assembly.rs b/crates/trusted-server-adapter-fastly/src/esi_assembly.rs new file mode 100644 index 000000000..f2ad29b68 --- /dev/null +++ b/crates/trusted-server-adapter-fastly/src/esi_assembly.rs @@ -0,0 +1,285 @@ +//! Fastly cold-response assembly backed by the repaired `stackpop/esi` parser. +//! +//! The shared template cache stores an inert marker. This module creates one synthetic +//! ESI include only in a request-private working copy, resolves it from an already-built +//! fragment, and never performs an HTTP request. + +use std::io::Cursor; + +use esi::{CacheConfig, Configuration, DcaMode, PendingFragmentContent, Processor}; +use fastly::http::StatusCode; +use fastly::{Request, Response}; +use trusted_server_core::platform::{ + PlatformTemplateAssembler, TemplateAssemblyError, contains_publisher_esi_directive, +}; +use trusted_server_core::publisher::AD_ASSEMBLY_SEAM; + +const INTERNAL_FRAGMENT_PATH: &str = "/_ts/internal/reader-ad-state"; +const SYNTHETIC_ESI_INCLUDE: &[u8] = b""; + +/// Why the Fastly ESI adapter refused or failed to assemble a document. +#[derive(Debug, derive_more::Display)] +enum EsiAssemblyError { + /// The inert seam marker was missing or repeated. + #[display("expected exactly one inert seam marker, found {count}")] + InvalidMarkerCount { count: usize }, + /// Publisher bytes contained ESI instructions outside TS's synthetic seam. + #[display("publisher-authored ESI directives are not allowed")] + PublisherEsiDirective, + /// The parser dispatched a URL other than TS's one synthetic fragment. + #[display("unexpected fragment request path `{path}` (query present: {has_query})")] + UnexpectedFragmentRequest { path: String, has_query: bool }, + /// The pinned parser could not process the document. + #[display("ESI processing failed: {message}")] + Processing { message: String }, + /// The parser changed bytes outside the one synthetic include. + #[display("ESI output was not an exact seam substitution")] + OutputMismatch, +} + +impl core::error::Error for EsiAssemblyError {} + +/// ESI configuration with every cache- and recursion-sensitive option explicit. +fn assembly_configuration() -> Configuration { + Configuration::default() + .with_escaped(false) + .with_default_dca(DcaMode::None) + .with_inherit_parent_dca(false) + .with_max_include_depth(1) + .with_edge_control(false) + .with_caching(CacheConfig { + is_includes_cacheable: false, + includes_default_ttl: None, + includes_force_ttl: None, + is_rendered_cacheable: false, + rendered_cache_control: false, + rendered_ttl: None, + }) +} + +fn template_with_synthetic_include(template: &[u8]) -> Result<(Vec, usize), EsiAssemblyError> { + let marker = AD_ASSEMBLY_SEAM.as_bytes(); + let positions = template + .windows(marker.len()) + .enumerate() + .filter_map(|(at, window)| (window == marker).then_some(at)) + .collect::>(); + if positions.len() != 1 { + return Err(EsiAssemblyError::InvalidMarkerCount { + count: positions.len(), + }); + } + if contains_publisher_esi_directive(template) { + return Err(EsiAssemblyError::PublisherEsiDirective); + } + + let at = positions[0]; + let mut working = + Vec::with_capacity(template.len() - marker.len() + SYNTHETIC_ESI_INCLUDE.len()); + working.extend_from_slice(&template[..at]); + working.extend_from_slice(SYNTHETIC_ESI_INCLUDE); + working.extend_from_slice(&template[at + marker.len()..]); + Ok((working, at)) +} + +fn completed_fragment_response( + request: &Request, + fragment: &[u8], +) -> Result { + let path = request.get_path().to_string(); + let has_query = request.get_url().query().is_some(); + if path != INTERNAL_FRAGMENT_PATH || has_query { + return Err(EsiAssemblyError::UnexpectedFragmentRequest { path, has_query }); + } + + Ok(PendingFragmentContent::CompletedRequest(Box::new( + Response::from_status(StatusCode::OK) + .with_header( + fastly::http::header::CONTENT_TYPE, + "text/html; charset=utf-8", + ) + .with_body(fragment.to_vec()), + ))) +} + +fn assemble_with_observer( + template: &[u8], + fragment: &[u8], + on_dispatch: F, +) -> Result, EsiAssemblyError> +where + F: Fn() + 'static, +{ + let (working, seam_at) = template_with_synthetic_include(template)?; + let fragment_len = fragment.len(); + let fragment_response = fragment.to_vec(); + let dispatcher = move |request, _index| { + on_dispatch(); + completed_fragment_response(&request, &fragment_response) + .map_err(|error| esi::ESIError::FragmentRequestError(error.to_string())) + }; + let mut processor = Processor::new(None, assembly_configuration()); + let mut output = Vec::with_capacity(template.len() + fragment_len); + processor + .process_stream(Cursor::new(working), &mut output, Some(&dispatcher), None) + .map_err(|error| EsiAssemblyError::Processing { + message: error.to_string(), + })?; + let expected_len = template.len() - AD_ASSEMBLY_SEAM.len() + fragment_len; + let output_tail_at = seam_at + fragment_len; + let template_tail_at = seam_at + AD_ASSEMBLY_SEAM.len(); + if output.len() != expected_len + || output[..seam_at] != template[..seam_at] + || &output[seam_at..output_tail_at] != fragment + || output[output_tail_at..] != template[template_tail_at..] + { + return Err(EsiAssemblyError::OutputMismatch); + } + Ok(output) +} + +fn assemble(template: &[u8], fragment: &[u8]) -> Result, EsiAssemblyError> { + assemble_with_observer(template, fragment, || {}) +} + +/// Fastly implementation of the core cold-response assembly boundary. +pub struct FastlyTemplateAssembler; + +impl PlatformTemplateAssembler for FastlyTemplateAssembler { + fn assemble(&self, template: &[u8], fragment: &[u8]) -> Result, TemplateAssemblyError> { + assemble(template, fragment).map_err(|error| TemplateAssemblyError::Failed { + message: error.to_string(), + }) + } +} + +#[cfg(test)] +mod tests { + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + + use super::*; + use trusted_server_core::publisher::AD_ASSEMBLY_SEAM; + + const FRAGMENT: &[u8] = b""; + + fn template(body: &str) -> Vec { + format!("{body}{AD_ASSEMBLY_SEAM}").into_bytes() + } + + #[test] + fn a_script_larger_than_the_parser_chunk_survives_exactly() { + let script = format!( + "", + "x".repeat(120_000) + ); + let document = template(&script); + let dispatches = Arc::new(AtomicUsize::new(0)); + let observed_dispatches = Arc::clone(&dispatches); + + let assembled = assemble_with_observer(&document, FRAGMENT, move || { + observed_dispatches.fetch_add(1, Ordering::Relaxed); + }) + .expect("should assemble a document with a large script"); + + let seam_at = document + .windows(AD_ASSEMBLY_SEAM.len()) + .position(|window| window == AD_ASSEMBLY_SEAM.as_bytes()) + .expect("should find seam"); + let mut expected = Vec::new(); + expected.extend_from_slice(&document[..seam_at]); + expected.extend_from_slice(FRAGMENT); + expected.extend_from_slice(&document[seam_at + AD_ASSEMBLY_SEAM.len()..]); + + assert_eq!( + assembled, expected, + "ESI must alter only the synthetic seam" + ); + assert_eq!(dispatches.load(Ordering::Relaxed), 1); + } + + #[test] + fn missing_and_repeated_markers_are_rejected_before_parsing() { + let missing = assemble(b"plain", FRAGMENT) + .expect_err("should reject a missing marker"); + let repeated = assemble( + format!("{AD_ASSEMBLY_SEAM}{AD_ASSEMBLY_SEAM}").as_bytes(), + FRAGMENT, + ) + .expect_err("should reject repeated markers"); + + assert!(matches!( + missing, + EsiAssemblyError::InvalidMarkerCount { count: 0 } + )); + assert!(matches!( + repeated, + EsiAssemblyError::InvalidMarkerCount { count: 2 } + )); + } + + #[test] + fn every_publisher_esi_directive_form_is_rejected_case_insensitively() { + for directive in [ + "", + "secret", + "x", + "$(HTTP_HOST)", + "text", + "", + "", + "", + ] { + let error = assemble(&template(directive), FRAGMENT) + .expect_err("should reject publisher-authored ESI"); + + assert!(matches!(error, EsiAssemblyError::PublisherEsiDirective)); + } + } + + #[test] + fn fragment_esi_is_emitted_verbatim_and_never_reparsed() { + let fragment = b""; + + let assembled = assemble(&template("article"), fragment).expect("should assemble"); + + assert!( + assembled + .windows(fragment.len()) + .any(|window| window == fragment), + "fragment bytes must remain data" + ); + } + + #[test] + fn dispatcher_rejects_every_url_except_the_synthetic_internal_one() { + let unexpected = fastly::Request::get("https://example.com/not-the-seam"); + let with_query = + fastly::Request::get("https://example.com/_ts/internal/reader-ad-state?publisher=1"); + + assert!(matches!( + completed_fragment_response(&unexpected, FRAGMENT), + Err(EsiAssemblyError::UnexpectedFragmentRequest { .. }) + )); + assert!(matches!( + completed_fragment_response(&with_query, FRAGMENT), + Err(EsiAssemblyError::UnexpectedFragmentRequest { .. }) + )); + } + + #[test] + fn configuration_cannot_cache_or_reparse_reader_state() { + let configuration = assembly_configuration(); + + assert!(!configuration.cache.is_includes_cacheable); + assert!(configuration.cache.includes_default_ttl.is_none()); + assert!(configuration.cache.includes_force_ttl.is_none()); + assert!(!configuration.cache.is_rendered_cacheable); + assert!(!configuration.cache.rendered_cache_control); + assert!(configuration.cache.rendered_ttl.is_none()); + assert_eq!(configuration.default_dca, DcaMode::None); + assert!(!configuration.inherit_parent_dca); + assert_eq!(configuration.max_include_depth, 1); + assert!(!configuration.enable_edge_control); + } +} diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index 39d35b198..1eac6df4b 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -11,6 +11,7 @@ use error_stack::Report; use fastly::http::Method as FastlyMethod; use fastly::{Request as FastlyRequest, Response as FastlyResponse}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; use trusted_server_core::ec::kv::KvIdentityGraph; @@ -23,17 +24,20 @@ use trusted_server_core::integrations::RequestFilterEffects; use trusted_server_core::platform::PlatformGeo as _; use trusted_server_core::platform::RuntimeServices; use trusted_server_core::proxy::{AssetProxyCachePolicy, stream_asset_body}; +use trusted_server_core::response_privacy::TerminalPrivateResponse; use trusted_server_core::settings::Settings; mod app; mod backend; mod compat; mod ec_kv; +mod esi_assembly; mod logging; mod management_api; mod middleware; mod platform; mod rate_limiter; +mod template_cache; mod tinybird; use crate::app::{EcFinalizeState, TrustedServerApp, load_settings_from_config_store}; @@ -202,7 +206,7 @@ fn edgezero_main(mut req: FastlyRequest) { } if let Some(policy) = asset_cache_policy { - policy.apply_after_route_finalization(&mut response); + policy.apply_after_route_finalization(&mut response, EdgeCacheHeader::SurrogateControl); } if let Some(ec_state) = ec_state { @@ -328,14 +332,8 @@ fn send_edgezero_response( mut response: HttpResponse, request_filter_effects: Option<&RequestFilterEffects>, ) { - if let Some(effects) = request_filter_effects { - effects.apply_to_response(&mut response); - } - - // Final cache guard: EC finalization and request-filter effects may have - // added a per-user Set-Cookie after `apply_finalize_headers` ran, so - // re-apply the privacy downgrade before send. - crate::middleware::enforce_set_cookie_cache_privacy(&mut response); + apply_terminal_response_effects(&mut response, request_filter_effects); + crate::middleware::enforce_uncacheable_cache_privacy(&mut response); let (parts, body) = response.into_parts(); @@ -364,6 +362,30 @@ fn send_edgezero_response( } } +/// Apply every late response mutation, then restore privacy invariants before headers commit. +fn apply_terminal_response_effects( + response: &mut HttpResponse, + request_filter_effects: Option<&RequestFilterEffects>, +) { + let must_remain_private = + trusted_server_core::response_privacy::is_private_or_no_store(response.headers()) + || response + .extensions() + .get::() + .is_some(); + if let Some(effects) = request_filter_effects { + effects.apply_to_response(response); + } + if must_remain_private { + trusted_server_core::response_privacy::enforce_private_no_store(response); + } + + // Final cache guard: EC finalization and request-filter effects may have + // added a per-user Set-Cookie after `apply_finalize_headers` ran, so + // re-apply the privacy downgrade before send. + crate::middleware::enforce_set_cookie_cache_privacy(response); +} + const FALLBACK_UNAVAILABLE: &str = "unavailable"; const FALLBACK_NOT_SENT: &str = "not sent"; const FALLBACK_NONE: &str = "none"; @@ -485,6 +507,7 @@ mod tests { use edgezero_core::http::HeaderValue; use edgezero_core::http::response_builder; use fastly::mime; + use trusted_server_core::integrations::HeaderMutation; fn test_settings() -> Settings { Settings::from_toml( @@ -557,6 +580,37 @@ mod tests { ); } + #[test] + fn late_filter_effects_cannot_make_an_assembled_response_public() { + let mut response = response_builder() + .header("cache-control", "private, no-store") + .header("etag", "\"reader-document\"") + .body(EdgeBody::empty()) + .expect("should build response"); + response.extensions_mut().insert(TerminalPrivateResponse); + let effects = RequestFilterEffects { + request_headers: Vec::new(), + response_headers: vec![ + HeaderMutation::set("cache-control", "public, s-maxage=3600"), + HeaderMutation::set("surrogate-control", "max-age=3600"), + HeaderMutation::set("cdn-cache-control", "public, max-age=3600"), + ], + }; + + apply_terminal_response_effects(&mut response, Some(&effects)); + + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|value| value.to_str().ok()), + Some("private, no-store") + ); + assert!(response.headers().get("surrogate-control").is_none()); + assert!(response.headers().get("cdn-cache-control").is_none()); + assert!(response.headers().get("etag").is_none()); + } + #[test] #[allow(clippy::panic)] fn entry_point_finalize_skips_geo_lookup_for_401() { diff --git a/crates/trusted-server-adapter-fastly/src/middleware.rs b/crates/trusted-server-adapter-fastly/src/middleware.rs index 2c00ac2ff..83ead48d3 100644 --- a/crates/trusted-server-adapter-fastly/src/middleware.rs +++ b/crates/trusted-server-adapter-fastly/src/middleware.rs @@ -235,7 +235,9 @@ pub(crate) fn apply_finalize_headers( /// entry point (`main.rs`) can re-apply it after /// [`ec_finalize_response`](trusted_server_core::ec::finalize::ec_finalize_response) /// writes the EC identity `Set-Cookie`, using the single shared implementation. -pub(crate) use trusted_server_core::response_privacy::enforce_set_cookie_cache_privacy; +pub(crate) use trusted_server_core::response_privacy::{ + enforce_set_cookie_cache_privacy, enforce_uncacheable_cache_privacy, +}; // --------------------------------------------------------------------------- // Tests @@ -429,12 +431,12 @@ mod tests { } #[test] - fn enforce_set_cookie_cache_privacy_downgrades_late_cookie() { + fn enforce_set_cookie_cache_privacy_downgrades_inactive_cache_policy() { // Mirrors the EdgeZero post-ec_finalize guard: a Set-Cookie added after - // finalize headers ran (origin-public response) must be downgraded. + // finalize headers ran must override the inactive template cache policy. let mut response = response_with_headers(&[ ("set-cookie", "ts-ec=abc; Path=/"), - ("cache-control", "public, max-age=600"), + ("cache-control", "max-age=60"), ("surrogate-control", "max-age=600"), ]); @@ -446,11 +448,11 @@ mod tests { .get("cache-control") .and_then(|v| v.to_str().ok()), Some("private, max-age=0"), - "should downgrade a late public cookie response to private" + "should downgrade an inactive cache policy on a cookie response" ); assert!( response.headers().get("surrogate-control").is_none(), - "should strip surrogate-control from the late cookie response" + "should strip surrogate-control from the inactive cookie response" ); } @@ -496,6 +498,29 @@ mod tests { ); } + #[test] + fn enforce_uncacheable_cache_privacy_handles_late_filter_headers() { + let mut response = response_with_headers(&[ + ("cache-control", "private, max-age=0"), + ("surrogate-control", "max-age=600"), + ]); + + enforce_uncacheable_cache_privacy(&mut response); + + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|value| value.to_str().ok()), + Some("private, max-age=0"), + "should preserve the late private directive" + ); + assert!( + response.headers().get("surrogate-control").is_none(), + "should strip the normalized edge header after late filter effects" + ); + } + // --------------------------------------------------------------------------- // FinalizeResponseMiddleware::handle tests // --------------------------------------------------------------------------- diff --git a/crates/trusted-server-adapter-fastly/src/template_cache.rs b/crates/trusted-server-adapter-fastly/src/template_cache.rs new file mode 100644 index 000000000..fe3148cb4 --- /dev/null +++ b/crates/trusted-server-adapter-fastly/src/template_cache.rs @@ -0,0 +1,641 @@ +//! Fastly Core Cache backing for the shared transformed-template cache. +//! +//! Only the Fastly adapter implements this; every other adapter uses +//! `UnavailableTemplateCache`, so the ESI assembly mode stays portable and only +//! the caching is Fastly-only. +//! +//! **Why Core Cache and not read-through caching.** Read-through with `after_send` + +//! `set_body_transform` looks like a better fit — it keeps HTTP semantics and derives +//! TTL and surrogate keys from origin headers for free. It is unreachable here: +//! Viceroy 0.17 stubs the entire HTTP Cache ABI and the SDK converts that into a +//! *send error*, so setting `after_send` makes every publisher origin fetch fail +//! under `fastly compute serve`, `cargo test-fastly` and the parity suite. It is also +//! silently dead whenever the origin request is in pass mode, and its closure bounds +//! (`Fn + Send + Sync`) are incompatible with a platform layer that is `!Send` by +//! construction. Recorded in the spike plan's Task 3 Step 4 so nobody re-proposes it. +//! +//! Spike-only. Remove with the spike. + +use fastly::cache::core::{CacheKey, Found, Transaction}; +use std::io::Write as _; +use std::time::Duration; +use trusted_server_core::platform::{ + PlatformTemplateCache, PlatformTemplateCacheReservation, + TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY, TemplateCacheError, TemplateCacheKey, + TemplateCacheLookup, TemplateCacheMiss, TemplateCacheReservation, TemplateEntry, + TemplateMetadata, +}; + +/// Fastly Core Cache implementation of the shared template cache. +#[derive(Default)] +pub struct FastlyTemplateCache; + +impl FastlyTemplateCache { + /// Create the Fastly Core Cache implementation. + /// + /// Entry lifetime is supplied per insert after core validates origin freshness + /// and applies the operator's configured safety ceiling. + #[must_use] + pub const fn new() -> Self { + Self + } +} + +fn backend_error(message: impl Into) -> TemplateCacheError { + TemplateCacheError::Backend { + message: message.into(), + } +} + +fn cancel_invalid_reservation( + validation_error: TemplateCacheError, + cancel: impl FnOnce() -> Result<(), E>, +) -> Result<(), TemplateCacheError> { + match cancel() { + Ok(()) => Err(validation_error), + Err(error) => Err(backend_error(format!( + "{validation_error}; cancelling invalid cache reservation also failed: {error:?}" + ))), + } +} + +enum ReadFoundError { + Invalid(TemplateCacheMiss), + Backend(TemplateCacheError), +} + +fn read_cache_body(mut reader: impl std::io::Read) -> Result, ReadFoundError> { + let mut body = Vec::new(); + reader + .read_to_end(&mut body) + .map_err(|_| ReadFoundError::Invalid(TemplateCacheMiss::Truncated))?; + Ok(body) +} + +fn read_found(found: &Found, key: &TemplateCacheKey) -> Result { + if found.is_stale() { + return Err(ReadFoundError::Invalid(TemplateCacheMiss::NotFound)); + } + + let metadata = TemplateMetadata::decode(&found.user_metadata()).ok_or( + ReadFoundError::Invalid(TemplateCacheMiss::UnreadableMetadata), + )?; + if metadata.schema_version != key.schema_version { + return Err(ReadFoundError::Invalid(TemplateCacheMiss::SchemaMismatch)); + } + if found + .known_length() + .is_some_and(|length| length != metadata.body_len) + { + return Err(ReadFoundError::Invalid(TemplateCacheMiss::Truncated)); + } + + let stream = found.to_stream().map_err(|error| { + ReadFoundError::Backend(backend_error(format!( + "opening cached template body failed: {error:?}" + ))) + })?; + let body = read_cache_body(stream)?; + if body.len() as u64 != metadata.body_len { + return Err(ReadFoundError::Invalid(TemplateCacheMiss::Truncated)); + } + Ok(TemplateEntry { metadata, body }) +} + +struct FastlyTemplateReservation { + transaction: Transaction, + surrogate_keys: Vec, +} + +impl PlatformTemplateCacheReservation for FastlyTemplateReservation { + fn insert( + self: Box, + metadata: &TemplateMetadata, + body: Vec, + max_age: Duration, + ) -> Result<(), TemplateCacheError> { + if metadata.body_len != body.len() as u64 { + let validation_error = backend_error(format!( + "metadata body_len {} does not match the {} bytes supplied", + metadata.body_len, + body.len() + )); + return cancel_invalid_reservation(validation_error, || { + self.transaction.cancel_insert_or_update() + }); + } + let encoded_metadata = match metadata.encode() { + Ok(encoded_metadata) => encoded_metadata, + Err(error) => { + let validation_error = + backend_error(format!("encoding template metadata failed: {error}")); + return cancel_invalid_reservation(validation_error, || { + self.transaction.cancel_insert_or_update() + }); + } + }; + + let mut writer = self + .transaction + .insert(max_age) + .surrogate_keys(self.surrogate_keys.iter().map(String::as_str)) + .known_length(body.len() as u64) + .user_metadata(encoded_metadata.into()) + .execute() + .map_err(|e| backend_error(format!("cache insert failed: {e:?}")))?; + writer + .write_all(&body) + .map_err(|e| backend_error(format!("writing template body failed: {e}")))?; + writer + .finish() + .map_err(|e| backend_error(format!("finishing the cached template failed: {e}")))?; + Ok(()) + } + + fn cancel(self: Box) -> Result<(), TemplateCacheError> { + self.transaction + .cancel_insert_or_update() + .map_err(|e| backend_error(format!("cancelling cache reservation failed: {e:?}"))) + } +} + +#[async_trait::async_trait(?Send)] +impl PlatformTemplateCache for FastlyTemplateCache { + async fn lookup_or_reserve( + &self, + key: &TemplateCacheKey, + ) -> Result { + let transaction = Transaction::lookup(CacheKey::from(key.to_cache_key().into_bytes())) + .execute() + .map_err(|e| backend_error(format!("transactional lookup failed: {e:?}")))?; + + if transaction.must_insert_or_update() { + return Ok(TemplateCacheLookup::Reserved( + TemplateCacheReservation::new(Box::new(FastlyTemplateReservation { + transaction, + surrogate_keys: key.surrogate_keys(), + })), + )); + } + + let found = transaction.found().ok_or_else(|| { + backend_error("transaction returned neither a hit nor an insert obligation") + })?; + Ok(match read_found(&found, key) { + Ok(entry) => TemplateCacheLookup::Hit(entry), + Err(ReadFoundError::Invalid(miss)) => TemplateCacheLookup::Invalid(miss), + Err(ReadFoundError::Backend(error)) => return Err(error), + }) + } + + async fn get(&self, key: &TemplateCacheKey) -> Result { + let cache_key = CacheKey::from(key.to_cache_key().into_bytes()); + + // A plain lookup, not a transaction: a read that does not intend to insert + // must not take an insert obligation it will never discharge, which would + // block every other client waiting on the same key until they time out. + let found = fastly::cache::core::lookup(cache_key) + .execute() + .map_err(|_| TemplateCacheMiss::NotFound)? + .ok_or(TemplateCacheMiss::NotFound)?; + + read_found(&found, key).map_err(|error| match error { + ReadFoundError::Invalid(miss) => miss, + ReadFoundError::Backend(error) => { + // This legacy method cannot expose a backend error. Production uses + // `lookup_or_reserve`, which preserves it for bounded diagnostics. + log::warn!("template_cache legacy read failed: {error}"); + TemplateCacheMiss::NotFound + } + }) + } + + async fn put( + &self, + key: &TemplateCacheKey, + metadata: &TemplateMetadata, + body: Vec, + max_age: Duration, + ) -> Result<(), TemplateCacheError> { + if metadata.body_len != body.len() as u64 { + return Err(backend_error(format!( + "metadata body_len {} does not match the {} bytes supplied; storing \ + this would make every read a truncation miss", + metadata.body_len, + body.len() + ))); + } + let encoded_metadata = metadata.encode().map_err(|error| { + backend_error(format!("encoding template metadata failed: {error}")) + })?; + + let cache_key = CacheKey::from(key.to_cache_key().into_bytes()); + + // Transactional insert so a cold key under load transforms once rather than + // once per concurrent request. + let tx = Transaction::lookup(cache_key) + .execute() + .map_err(|e| backend_error(format!("transactional lookup failed: {e:?}")))?; + + // Order matters. A STALE entry sets *both* `found()` and + // `must_insert_or_update()`. Testing `found()` first would return early on + // the stale bytes and never discharge the obligation, leaving every + // concurrent waiter blocked until timeout. + if !tx.must_insert_or_update() { + // Someone else already inserted a fresh entry. Nothing to do, and + // nothing to discharge. + return Ok(()); + } + + // `Transaction::insert` takes `self`, so from here there is no handle left to + // cancel the insert with. A write that fails part-way therefore cannot be + // retracted — which is why `TemplateMetadata::body_len` exists and `get` + // checks it. The metadata is written before the body, so a truncated entry + // still carries the length it was supposed to have. + let surrogate_keys = key.surrogate_keys(); + let mut writer = tx + .insert(max_age) + .surrogate_keys(surrogate_keys.iter().map(String::as_str)) + .known_length(body.len() as u64) + .user_metadata(encoded_metadata.into()) + .execute() + .map_err(|e| backend_error(format!("cache insert failed: {e:?}")))?; + + if let Err(e) = writer.write_all(&body) { + // Deliberately do not call `finish()`. If partial content becomes + // observable, fallible reads and the post-read check against the declared + // body length reject it. + return Err(backend_error(format!("writing template body failed: {e}"))); + } + + // Required. Without it the object never completes and its length stays + // unknown, so readers see a partial or absent entry. + writer + .finish() + .map_err(|e| backend_error(format!("finishing the cached template failed: {e}")))?; + + Ok(()) + } + + async fn purge_url(&self, key: &TemplateCacheKey) -> Result<(), TemplateCacheError> { + fastly::http::purge::purge_surrogate_key(&key.url_surrogate_key()) + .map_err(|e| backend_error(format!("purging invalid template failed: {e:?}"))) + } + + async fn purge_all(&self) -> Result<(), TemplateCacheError> { + fastly::http::purge::purge_surrogate_key(TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY) + .map_err(|e| backend_error(format!("purging templates failed: {e:?}"))) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::io; + use trusted_server_core::creative_opportunities::AssemblyMode; + use trusted_server_core::platform::TEMPLATE_SCHEMA_VERSION; + + struct FailingReader { + returned_prefix: bool, + } + + impl io::Read for FailingReader { + fn read(&mut self, buffer: &mut [u8]) -> io::Result { + if self.returned_prefix { + return Err(io::Error::other("abandoned cache stream")); + } + self.returned_prefix = true; + buffer[..3].copy_from_slice(b"abc"); + Ok(3) + } + } + + /// Distinct per test, so tests sharing the process cache cannot collide. + fn key(url: &str) -> TemplateCacheKey { + TemplateCacheKey { + url: url.to_string(), + request_host: "example.com".to_string(), + request_scheme: "https".to_string(), + origin_identity: "https://origin.example.com\0origin.example.com".to_string(), + assembly_mode: AssemblyMode::Esi, + vary_values: vec![trusted_server_core::platform::VaryHeaderValues { + name: "rsc".to_string(), + values: Some(vec![b"1".to_vec()]), + }], + template_fingerprint: "fp".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + } + } + + fn metadata_for(body: &[u8]) -> TemplateMetadata { + TemplateMetadata { + policy_headers: Vec::new(), + content_encoding: "identity".to_string(), + content_type: "text/html; charset=utf-8".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: body.len() as u64, + } + } + + /// The trait is `async_trait(?Send)` and this crate has no async test runtime, + /// so drive the futures directly. + fn run(fut: impl core::future::Future) -> T { + futures::executor::block_on(fut) + } + + fn cache() -> FastlyTemplateCache { + FastlyTemplateCache::new() + } + + #[test] + fn cache_body_read_error_is_a_truncated_miss() { + let error = read_cache_body(FailingReader { + returned_prefix: false, + }) + .expect_err("should reject an abandoned cache stream"); + + assert!( + matches!(error, ReadFoundError::Invalid(TemplateCacheMiss::Truncated)), + "should classify a cache stream read failure as truncated" + ); + } + + #[test] + fn a_stored_template_reads_back_intact() { + let cache = cache(); + let key = key("https://example.com/roundtrip"); + let body = b"template".to_vec(); + let metadata = metadata_for(&body); + + run(cache.put(&key, &metadata, body.clone(), Duration::from_secs(60))) + .expect("should store"); + + let entry = run(cache.get(&key)).expect("should read back"); + assert_eq!(entry.body, body, "bytes must survive the round trip"); + assert_eq!(entry.metadata, metadata, "metadata must survive too"); + } + + #[test] + fn transactional_lookup_reserves_before_insert_then_hits() { + let cache = cache(); + let key = key("https://example.com/pre-origin-reservation"); + let body = b"collapsed".to_vec(); + let metadata = metadata_for(&body); + + let reservation = match run(cache.lookup_or_reserve(&key)).expect("lookup should work") { + TemplateCacheLookup::Reserved(reservation) => reservation, + _ => panic!("a cold transactional lookup must assign the insert obligation"), + }; + reservation + .insert(&metadata, body.clone(), Duration::from_secs(17)) + .expect("reservation should insert"); + + match run(cache.lookup_or_reserve(&key)).expect("warm lookup should work") { + TemplateCacheLookup::Hit(entry) => assert_eq!(entry.body, body), + _ => panic!("the next transactional lookup must see the inserted template"), + } + } + + #[test] + fn length_mismatch_cancels_the_reservation_obligation() { + let cache = cache(); + let key = key("https://example.com/reservation-length-mismatch"); + let body = b"template".to_vec(); + let mut metadata = metadata_for(&body); + metadata.body_len += 1; + let reservation = match run(cache.lookup_or_reserve(&key)).expect("lookup should work") { + TemplateCacheLookup::Reserved(reservation) => reservation, + _ => panic!("a cold lookup should reserve the key"), + }; + + let error = reservation + .insert(&metadata, body, Duration::from_secs(60)) + .expect_err("length mismatch should fail insertion"); + + assert!( + error.to_string().contains("does not match"), + "should preserve the original validation reason: {error}" + ); + match run(cache.lookup_or_reserve(&key)).expect("second lookup should work") { + TemplateCacheLookup::Reserved(reservation) => reservation + .cancel() + .expect("should cancel the proof reservation"), + _ => panic!("the invalid insert should release the reservation obligation"), + } + } + + #[test] + fn metadata_encoding_failure_cancels_the_reservation_obligation() { + let cache = cache(); + let key = key("https://example.com/reservation-metadata-encoding"); + let body = b"template".to_vec(); + let mut metadata = metadata_for(&body); + metadata.content_type = "text/html\ninjected".to_string(); + let reservation = match run(cache.lookup_or_reserve(&key)).expect("lookup should work") { + TemplateCacheLookup::Reserved(reservation) => reservation, + _ => panic!("a cold lookup should reserve the key"), + }; + + let error = reservation + .insert(&metadata, body, Duration::from_secs(60)) + .expect_err("invalid metadata should fail insertion"); + + assert!( + error + .to_string() + .contains("encoding template metadata failed"), + "should preserve the original validation reason: {error}" + ); + match run(cache.lookup_or_reserve(&key)).expect("second lookup should work") { + TemplateCacheLookup::Reserved(reservation) => reservation + .cancel() + .expect("should cancel the proof reservation"), + _ => panic!("the invalid insert should release the reservation obligation"), + } + } + + #[test] + fn invalid_reservation_cancellation_preserves_both_errors() { + let error = cancel_invalid_reservation(backend_error("metadata validation failed"), || { + Err("simulated cancellation failure") + }) + .expect_err("invalid reservation should return an error"); + + let message = error.to_string(); + assert!(message.contains("metadata validation failed")); + assert!(message.contains("simulated cancellation failure")); + } + + #[test] + fn an_absent_key_is_a_miss_not_an_error() { + let miss = + run(cache().get(&key("https://example.com/never-stored"))).expect_err("should miss"); + assert_eq!(miss, TemplateCacheMiss::NotFound); + } + + #[test] + fn a_different_assembly_mode_does_not_read_the_same_entry() { + // The arms emit different bytes. If they shared an entry, one would serve + // the other's template. + let cache = cache(); + let esi = key("https://example.com/mode-split"); + let mut inline = esi.clone(); + inline.assembly_mode = AssemblyMode::Inline; + + let body = b"esi-template".to_vec(); + run(cache.put(&esi, &metadata_for(&body), body, Duration::from_secs(60))) + .expect("should store"); + + assert_eq!( + run(cache.get(&inline)).err(), + Some(TemplateCacheMiss::NotFound), + "inline must not read the ESI arm's template" + ); + } + + #[test] + fn a_schema_bump_reads_a_miss_rather_than_a_stale_shape() { + let cache = cache(); + let key_v1 = key("https://example.com/schema"); + let body = b"old-shape".to_vec(); + run(cache.put(&key_v1, &metadata_for(&body), body, Duration::from_secs(60))) + .expect("should store"); + + // A deploy that changes the transform bumps the constant. The old entry must + // not be assembled against. + let mut key_v2 = key_v1.clone(); + key_v2.schema_version = TEMPLATE_SCHEMA_VERSION + 1; + + assert_eq!( + run(cache.get(&key_v2)).err(), + Some(TemplateCacheMiss::NotFound), + "a bumped schema changes the key, so the old entry is simply not found" + ); + } + + #[test] + fn a_stale_but_present_entry_reads_as_a_miss_rather_than_being_served() { + // Stale-while-revalidate is a real option and deliberately not taken: it is a + // state machine `cache::core` does not implement for you, and serving stale here + // means serving a template built by an older transform or an older JS bundle. + // + // The entry has to be *present and stale*, not merely expired. A zero TTL with no + // `stale_while_revalidate` window is simply absent, so a test written that way + // passes without ever reaching `is_stale()` — verified: reverting the staleness + // check left that version green. The revalidate window is what keeps the object + // readable while stale, so this actually exercises the branch. + let key = key("https://example.com/stale"); + let body = b"stale-template".to_vec(); + let metadata = metadata_for(&body); + let cache_key = CacheKey::from(key.to_cache_key().into_bytes()); + + let mut writer = fastly::cache::core::insert(cache_key, Duration::from_secs(0)) + .stale_while_revalidate(Duration::from_secs(60)) + .user_metadata( + metadata + .encode() + .expect("valid metadata should encode") + .into(), + ) + .execute() + .expect("should begin insert"); + writer.write_all(&body).expect("should write body"); + writer.finish().expect("should finish insert"); + + let miss = run(cache().get(&key)).expect_err("a stale template must not be served"); + assert_eq!(miss, TemplateCacheMiss::NotFound); + } + + #[test] + fn purge_all_clears_stored_templates() { + // The rollback lever. Without this, backing out a bad template means waiting + // for the TTL. + let cache = cache(); + let key = key("https://example.com/purge"); + let body = b"template".to_vec(); + run(cache.put(&key, &metadata_for(&body), body, Duration::from_secs(60))) + .expect("should store"); + run(cache.get(&key)).expect("should be present before purge"); + + run(cache.purge_all()).expect("should purge"); + + assert!( + run(cache.get(&key)).is_err(), + "purge must clear the template, or rollback is TTL-bound" + ); + } + + #[test] + fn a_second_put_on_a_fresh_entry_is_a_no_op() { + // Exercises the `must_insert_or_update` early return: a concurrent writer + // that finds a fresh entry must neither error nor overwrite. + let cache = cache(); + let key = key("https://example.com/second-put"); + let first = b"first".to_vec(); + run(cache.put( + &key, + &metadata_for(&first), + first.clone(), + Duration::from_secs(60), + )) + .expect("first put stores"); + + let second = b"second".to_vec(); + run(cache.put( + &key, + &metadata_for(&second), + second, + Duration::from_secs(60), + )) + .expect("second put should be a no-op, not an error"); + + assert_eq!( + run(cache.get(&key)).expect("should read").body, + first, + "a fresh entry must not be overwritten by a racing writer" + ); + } + + #[test] + fn the_cache_round_trips_through_the_platform_trait_object() { + // Every other test here calls `FastlyTemplateCache` concretely. The publisher + // never does — it reaches the cache as a `dyn PlatformTemplateCache` behind + // `RuntimeServices`. That join is what `app.rs` wires, and until this test it + // was only type-checked, never executed. + let cache: std::sync::Arc = std::sync::Arc::new(cache()); + let key = key("https://example.com/via-trait-object"); + let body = b"template".to_vec(); + + run(cache.put( + &key, + &metadata_for(&body), + body.clone(), + Duration::from_secs(60), + )) + .expect("should store"); + + assert_eq!( + run(cache.get(&key)).expect("should read back").body, + body, + "the trait object must reach the same Core Cache the concrete type does" + ); + } + + #[test] + fn a_length_mismatch_is_refused_at_write_rather_than_stored() { + // Storing metadata whose length disagrees with the body would make every + // subsequent read a truncation miss — a cache that silently never hits. + // Catch it at the write instead. + let cache = cache(); + let key = key("https://example.com/length-mismatch"); + let mut metadata = metadata_for(b"12345"); + metadata.body_len = 999; + + let err = run(cache.put(&key, &metadata, b"12345".to_vec(), Duration::from_secs(60))) + .expect_err("a length mismatch must be refused"); + assert!( + matches!(err, TemplateCacheError::Backend { .. }), + "expected a backend error, got {err:?}" + ); + } +} diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index 960bafc41..06bb1a15a 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -10,7 +10,13 @@ use edgezero_core::router::RouterService; use error_stack::Report; use trusted_server_core::auction::endpoints::handle_auction; use trusted_server_core::auction::{AuctionOrchestrator, build_orchestrator}; +use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::ec::EcContext; +use trusted_server_core::ec::admin::{ + admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, + deny_admin_diagnostic_fallback, handle_admin_eids_lookup, +}; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::http_util::sanitize_forwarded_headers; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; @@ -142,12 +148,15 @@ const LEGACY_ADMIN_DENY_METHODS: &[Method] = &[ Method::DELETE, ]; -fn named_fallback_paths() -> [(&'static str, &'static [Method]); 13] { +fn named_fallback_paths() -> [(&'static str, &'static [Method]); 16] { [ ("/.well-known/trusted-server.json", &[Method::GET]), ("/verify-signature", &[Method::POST]), ("/_ts/admin/keys/rotate", &[Method::POST]), ("/_ts/admin/keys/deactivate", &[Method::POST]), + ("/_ts/admin/ec", &[Method::GET]), + ("/_ts/admin/ec/{id}", &[Method::GET]), + ("/_ts/admin/eids", &[Method::GET]), ("/admin/keys/rotate", LEGACY_ADMIN_DENY_METHODS), ("/admin/keys/deactivate", LEGACY_ADMIN_DENY_METHODS), ("/auction", &[Method::POST]), @@ -361,6 +370,10 @@ fn admin_key_management_not_supported() -> Response { response } +fn admin_ec_lookup_not_supported() -> Response { + core_admin_ec_lookup_not_supported() +} + // --------------------------------------------------------------------------- // Error helper // --------------------------------------------------------------------------- @@ -513,6 +526,23 @@ fn build_router(state: &Arc) -> RouterService { Ok::(admin_key_management_not_supported()) }; + let admin_ec_not_supported_handler = |_ctx: RequestContext| async { + Ok::(admin_ec_lookup_not_supported()) + }; + + // Admin EIDs echo: pure request inspection (no KV), so this adapter + // serves the real handler. + let s = Arc::clone(&state); + let admin_eids_handler = move |ctx: RequestContext| { + let s = Arc::clone(&s); + async move { + let req = ctx.into_request(); + let result = PartnerRegistry::from_config(&s.settings.ec.partners) + .and_then(|registry| handle_admin_eids_lookup(®istry, &req)); + Ok::(result.unwrap_or_else(|e| http_error(&e))) + } + }; + // /auction let s = Arc::clone(&state); let auction_handler = move |ctx: RequestContext| { @@ -652,6 +682,9 @@ fn build_router(state: &Arc) -> RouterService { ) -> Result { let services = build_runtime_services(&ctx); let mut req = ctx.into_request(); + if let Some(response) = deny_admin_diagnostic_fallback(&req) { + return Ok(response); + } if let Err(error) = trusted_server_core::integrations::gpt_diagnostics::prepare_request( &state.settings, &mut req, @@ -665,7 +698,7 @@ fn build_router(state: &Arc) -> RouterService { // Dynamic tsjs serving is GET-only; other methods fall through to the // integration/publisher fallback. let result = if method == Method::GET && path.starts_with("/static/tsjs=") { - handle_tsjs_dynamic(&req, &state.registry) + handle_tsjs_dynamic(&req, &state.registry, EdgeCacheHeader::SMaxageFallback) } else if state.registry.has_route(&method, &path) { let mut ec_context = EcContext::default(); state @@ -699,6 +732,7 @@ fn build_router(state: &Arc) -> RouterService { &mut ec_context, auction, req, + EdgeCacheHeader::SMaxageFallback, ) .await { @@ -758,6 +792,14 @@ fn build_router(state: &Arc) -> RouterService { // credentials and key-management payloads to the origin. .post("/_ts/admin/keys/rotate", admin_not_supported_handler) .post("/_ts/admin/keys/deactivate", admin_not_supported_handler) + // Admin EC lookup routes. Registered explicitly (like the key + // routes above) so they never fall through to the publisher + // fallback, and they match `Settings::ADMIN_ENDPOINTS` for auth + // coverage. The EC identity graph is Fastly KV backed, so this + // adapter has no store to read. + .get("/_ts/admin/ec", admin_ec_not_supported_handler) + .get("/_ts/admin/ec/{id}", admin_ec_not_supported_handler) + .get("/_ts/admin/eids", admin_eids_handler) .post("/auction", auction_handler) .get(PAGE_BIDS_PATH, page_bids_handler.clone()) .route(PAGE_BIDS_PATH, Method::OPTIONS, page_bids_options_handler) diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index 2f7b1037e..b0f5581ca 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -113,6 +113,158 @@ async fn authenticated_admin_routes_return_501() { } } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_ec_routes_return_501() { + // The EC identity graph is Fastly KV backed, so Spin answers the admin + // EC lookup routes locally with 501 instead of letting them fall through + // to the publisher fallback. + let sample_ec_id = format!("{}.abc123", "a".repeat(64)); + for path in [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{sample_ec_id}"), + ] { + let req = request_builder() + .method("GET") + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), req).await; + + assert_eq!( + resp.status().as_u16(), + 501, + "{path} should report that Spin EC lookup is unsupported" + ); + assert_eq!( + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()), + Some("application/json") + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn admin_ec_route_without_credentials_returns_401() { + let req = request_builder() + .method("GET") + .uri("/_ts/admin/ec") + .body(edgezero_core::body::Body::empty()) + .expect("should build unauthenticated admin EC request"); + let resp = route(test_router(), req).await; + + assert_eq!(resp.status().as_u16(), 401); + assert!( + resp.headers().contains_key("www-authenticate"), + "admin EC 401 should include the Basic authentication challenge" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_eids_route_returns_200() { + // The EIDs echo is pure request inspection (no KV), so this adapter + // serves the real handler. + let req = request_builder() + .method("GET") + .uri("/_ts/admin/eids") + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), req).await; + + assert_eq!( + resp.status().as_u16(), + 200, + "/_ts/admin/eids should serve the real EIDs echo handler" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_diagnostic_fallback_is_denied_locally() { + let ec_id = format!("{}.abc123", "a".repeat(64)); + let valid_paths = [ + "/_ts/admin/ec".to_owned(), + format!("/_ts/admin/ec/{ec_id}"), + "/_ts/admin/eids".to_owned(), + ]; + + for path in valid_paths { + for method in ["POST", "HEAD", "OPTIONS", "PUT", "PATCH", "DELETE"] { + let request = request_builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::from("sensitive-admin-body")) + .expect("should build authenticated admin request"); + let response = route(test_router(), request).await; + + assert_eq!(response.status().as_u16(), 405); + assert_eq!( + response + .headers() + .get("allow") + .and_then(|v| v.to_str().ok()), + Some("GET") + ); + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } + + for path in [ + "/_ts/admin/ec/".to_owned(), + format!("/_ts/admin/ec/{ec_id}/extra"), + "/_ts/admin/eids/".to_owned(), + "/_ts/admin/eids/extra".to_owned(), + "/_ts/admin/eids.json".to_owned(), + "/_ts/admin/ec;foo".to_owned(), + format!("/_ts/admin/ec%2F{ec_id}"), + // Percent-encoded separators match the `^/_ts/admin` basic-auth + // handler but not a literal-slash namespace check, so they must be + // reserved before publisher fallback forwards credentials upstream. + "/_ts/admin%2Fec".to_owned(), + "/_ts/admin%2fec".to_owned(), + // Retired non-`/_ts` alias namespace: only the two exact paths are + // routed to a local deny, so descendants and encoded separators must + // be reserved at the shared fallback boundary. + "/admin/keys".to_owned(), + "/admin/keys/rotate/extra".to_owned(), + "/admin/keys%2Frotate".to_owned(), + "/admin%2fkeys/rotate".to_owned(), + ] { + for method in ["GET", "POST"] { + let request = request_builder() + .method(method) + .uri(&path) + .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .body(edgezero_core::body::Body::from("sensitive-admin-body")) + .expect("should build malformed admin request"); + let response = route(test_router(), request).await; + + assert_eq!(response.status().as_u16(), 404); + assert_eq!( + response + .headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + } + } +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn health_route_returns_ok() { // Parity with the Fastly/Axum adapters: GET /health is a cheap liveness probe @@ -209,6 +361,36 @@ async fn tsjs_route_is_routed_not_5xx() { assert!(status < 500, "tsjs route must not 5xx: got {status}"); } +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn tsjs_route_matching_hash_uses_s_maxage_fallback() { + let router = test_router(); + let src = trusted_server_core::tsjs::tsjs_script_src(&["creative"]); + let req = request_builder() + .method("GET") + .uri(src) + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + + let resp = route(router, req).await; + + assert_eq!( + resp.status().as_u16(), + 200, + "matching TSJS hash should serve OK" + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|value| value.to_str().ok()), + Some("public, max-age=31536000, s-maxage=31536000, immutable"), + "Spin adapter should render the portable s-maxage fallback" + ); + assert!( + resp.headers().get("surrogate-control").is_none(), + "s-maxage fallback must not emit Fastly Surrogate-Control" + ); +} + #[tokio::test(flavor = "multi_thread", worker_threads = 2)] async fn verify_signature_is_routed() { let router = test_router(); diff --git a/crates/trusted-server-cli/Cargo.toml b/crates/trusted-server-cli/Cargo.toml index fe9c3664b..e08114850 100644 --- a/crates/trusted-server-cli/Cargo.toml +++ b/crates/trusted-server-cli/Cargo.toml @@ -18,16 +18,21 @@ workspace = true chromiumoxide = { workspace = true } clap = { workspace = true } edgezero-cli = { workspace = true } +edgezero-core = { workspace = true } futures = { workspace = true } +glob = { workspace = true } +http = { workspace = true } log = { workspace = true } regex = { workspace = true } scraper = { workspace = true } serde = { workspace = true } serde_json = { workspace = true } +similar = { workspace = true } tempfile = { workspace = true } tokio = { workspace = true } toml = { workspace = true } toml_edit = { workspace = true } +tracing = { workspace = true } trusted-server-core = { workspace = true } url = { workspace = true } which = { workspace = true } @@ -62,4 +67,5 @@ tokio = { workspace = true, features = ["test-util"] } x509-parser = { workspace = true } [target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies] +temp-env = { workspace = true } tempfile = { workspace = true } diff --git a/crates/trusted-server-cli/src/ad_templates/compare.rs b/crates/trusted-server-cli/src/ad_templates/compare.rs new file mode 100644 index 000000000..48ab71f3e --- /dev/null +++ b/crates/trusted-server-cli/src/ad_templates/compare.rs @@ -0,0 +1,830 @@ +//! Pure comparison of configured expected slots against browser ad evidence. +//! +//! This module is collector-independent and Chrome-free: it takes decoded +//! [`BrowserAdEvidence`] plus the [`ExpectedSlot`] set and produces a +//! [`PageVerificationResult`] with per-slot statuses, warnings, and unmatched +//! extra evidence, mirroring spec §5.3–§5.6. +//! +use serde::Deserialize; + +use trusted_server_core::auction::types::MediaType; +use trusted_server_core::creative_opportunities::RuntimeAdStackExpected; + +use crate::ad_templates::expected::ExpectedSlot; +use crate::ad_templates::output::Warning; + +/// The phase in which a piece of evidence was observed. +#[derive(Debug, Clone, Copy, Eq, PartialEq, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum EvidencePhase { + /// Observed during the initial load and settle. + InitialLoad, + /// Observed only after the deterministic scroll pass. + Scroll, +} + +/// A DOM element ID observed on the page. +#[derive(Debug, Clone, Deserialize)] +pub struct DomEvidence { + /// The element ID. + pub dom_id: String, + /// The phase it was first observed in. + pub phase: EvidencePhase, +} + +/// A GPT slot observed on the page. +#[derive(Debug, Clone, Deserialize)] +pub struct GptSlotEvidence { + /// The observed GAM ad unit path. + pub gam_unit_path: String, + /// The observed GPT slot element ID. + pub div_id: String, + /// Observed numeric sizes as `(width, height)` pairs (non-numeric dropped upstream). + pub sizes: Vec<(u32, u32)>, + /// The phase it was first observed in. + pub phase: EvidencePhase, +} + +/// An `apstag.fetchBids` call the page made, if any were recorded. +/// +/// The collector no longer hooks `apstag`: server-side APS configuration is +/// metadata rather than a client assertion, so a missing client call is not a +/// finding. The field and this shape stay for the evidence payload's schema, and +/// the list arrives empty. +#[derive(Debug, Clone, Deserialize)] +#[allow( + dead_code, + reason = "decoded for schema stability; the collector records no APS calls" +)] +pub struct ApsFetchBidsEvidence { + /// The APS slot ID requested. + pub slot_id: String, + /// Sizes requested for the slot. + pub sizes: Vec<(u32, u32)>, + /// The phase it was observed in. + pub phase: EvidencePhase, +} + +/// A `/__ts/page-bids` observation for SPA routes (spec §5.2). +/// +/// DEFERRED in Phase 1: kept as forward scaffolding so the decoded evidence shape +/// stays forward-compatible. Not populated by the collector or surfaced in JSON. +#[derive(Debug, Clone, Deserialize)] +#[allow( + dead_code, + reason = "reserved decoded shape for the optional bids phase" +)] +pub struct PageBidsEvidence { + /// The slot ID present in the page-bids response. + pub slot_id: String, + /// The phase it was observed in. + pub phase: EvidencePhase, +} + +/// All read-only ad evidence decoded from a single browser page. +#[derive(Debug, Clone, Deserialize)] +pub struct BrowserAdEvidence { + /// DOM element IDs matching configured prefixes. + pub dom_ids: Vec, + /// GPT slots observed via `defineSlot` and `getSlots()`. + pub gpt_slots: Vec, + /// `apstag.fetchBids` calls observed. + pub aps_calls: Vec, + /// `/__ts/page-bids` observations (deferred; default empty). + #[serde(default)] + #[allow(dead_code, reason = "reserved for the optional bids phase")] + pub page_bids: Vec, + /// Collector-level warnings (no page HTML/cookies/storage). + #[serde(default)] + pub warnings: Vec, +} + +/// Summary of the runtime ad-stack gate for a page. +#[derive(Debug, Clone, Copy)] +pub struct RuntimeGateSummary { + /// The three-state ad-stack expectation. + pub expected: RuntimeAdStackExpected, +} + +impl RuntimeGateSummary { + /// Builds a summary from a computed runtime expectation. + #[must_use] + pub fn from_expected(expected: RuntimeAdStackExpected) -> Self { + Self { expected } + } + + #[cfg(test)] + fn unknown_allowed() -> Self { + Self::from_expected(RuntimeAdStackExpected::Unknown) + } + + #[cfg(test)] + fn auction_disabled() -> Self { + Self::from_expected(RuntimeAdStackExpected::No) + } +} + +/// Confirmation status for a single configured slot (compare-side mirror of the +/// output `SlotStatus`). +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum SlotStatus { + /// GPT evidence matches GAM path, div, and a compatible size. + Confirmed, + /// Some evidence, but not enough to confirm. + Partial, + /// No DOM or GPT evidence confirms the slot. + Missing, + /// The checker cannot confirm this slot type; this is not page drift. + Unconfirmable, +} + +/// The verification result for one audited page. +#[derive(Debug, Clone)] +pub struct PageVerificationResult { + /// Whether the runtime ad stack was expected to run for this page. + pub runtime_ad_stack_expected: RuntimeAdStackExpected, + /// Per-slot results, in expected-slot order. + pub slots: Vec, + /// Live evidence that matched no configured slot. + pub extra_evidence: Vec, +} + +impl PageVerificationResult { + /// Whether `--strict` should fail for this page. + /// + /// False when the runtime ad stack is not expected to run (a known gate + /// suppressed it); otherwise true if any slot is missing or partial. Provider + /// warnings and extra evidence alone never fail strict. + #[must_use] + pub fn strict_failed(&self) -> bool { + if self.runtime_ad_stack_expected == RuntimeAdStackExpected::No { + return false; + } + self.slots + .iter() + .any(|slot| matches!(slot.status, SlotStatus::Missing | SlotStatus::Partial)) + } +} + +/// Per-slot verification result. +#[derive(Debug, Clone)] +pub struct SlotResult { + /// The configured slot id. + pub id: String, + /// The confirmation status. + pub status: SlotStatus, + /// The phase the confirming evidence was observed in. + pub phase: Option, + /// The live evidence observed for this slot. + pub evidence: SlotEvidence, + /// Slot-level warnings (size, provider, etc.). + pub warnings: Vec, +} + +/// Live evidence observed for a configured slot. +#[derive(Debug, Clone)] +pub struct SlotEvidence { + /// The resolved DOM element ID, if any. + pub dom_id: Option, + /// The matched GPT slot, if any. + pub gpt: Option, +} + +/// Live ad-slot evidence with no matching configured slot. +#[derive(Debug, Clone)] +pub struct ExtraEvidence { + /// Evidence kind. Only `gpt` is produced today; the field is a string so a + /// later evidence source can be added without changing the JSON schema. + pub kind: String, + /// The phase it was observed in. + pub phase: EvidencePhase, + /// The DOM element ID, if any. + pub dom_id: Option, + /// The GAM unit path, if any. + pub gam_unit_path: Option, + /// Observed numeric sizes. + pub sizes: Vec<(u32, u32)>, + /// Why this evidence is reported as extra. + pub reason: String, +} + +fn warning(code: &str, message: String) -> Warning { + Warning { + code: code.to_string(), + message, + } +} + +/// Resolves the slot root DOM element per spec §5.3. +/// +/// Exact `div_id` match first, then the first element whose ID starts with +/// `div_id`, ignoring `-container` wrappers. +fn resolve_dom<'a>(dom_ids: &'a [DomEvidence], div_id: &str) -> Option<&'a DomEvidence> { + if let Some(exact) = dom_ids.iter().find(|dom| dom.dom_id == div_id) { + return Some(exact); + } + dom_ids + .iter() + .find(|dom| dom.dom_id.starts_with(div_id) && !dom.dom_id.ends_with("-container")) +} + +/// Returns true when a GPT slot's element ID matches the resolved DOM id (or its +/// `-container`), per spec §5.4. +fn gpt_div_matches(gpt_div: &str, expected: &ExpectedSlot, resolved_dom_id: Option<&str>) -> bool { + match resolved_dom_id { + Some(dom_id) => gpt_div == dom_id || gpt_div == format!("{dom_id}-container"), + None => { + gpt_div == expected.div_id + || (gpt_div.starts_with(&expected.div_id) && !gpt_div.ends_with("-container")) + } + } +} + +fn banner_sizes(expected: &ExpectedSlot) -> Vec<(u32, u32)> { + expected + .formats + .iter() + .filter(|format| format.media_type == MediaType::Banner) + .map(|format| (format.width, format.height)) + .collect() +} + +/// Compares configured expected slots against decoded browser evidence. +#[must_use] +pub fn compare_page_evidence( + expected: &[ExpectedSlot], + evidence: &BrowserAdEvidence, + gate: RuntimeGateSummary, +) -> PageVerificationResult { + let mut consumed_gpt = vec![false; evidence.gpt_slots.len()]; + let mut slots = Vec::with_capacity(expected.len()); + + for slot in expected { + let resolved = resolve_dom(&evidence.dom_ids, &slot.div_id); + let resolved_id = resolved.map(|dom| dom.dom_id.clone()); + // An unrenderable (`None`) configured path can never match live GPT + // evidence; matching on anything else would confirm the wrong unit. + let gpt_idx = slot.gam_unit_path.as_deref().and_then(|unit_path| { + evidence.gpt_slots.iter().position(|gpt| { + gpt.gam_unit_path == unit_path + && gpt_div_matches(&gpt.div_id, slot, resolved_id.as_deref()) + }) + }); + + let banner = banner_sizes(slot); + let mut warnings = Vec::new(); + // `expected_slots_for_path` drops a slot whose template does not render, + // so on the verify path this arm is unreachable; it exists for callers + // that build expected slots directly, and as a guard if that filter ever + // changes. + if slot.gam_unit_path.is_none() { + warnings.push(warning( + "gam_unit_path_unrenderable", + format!( + "slot `{}` gam_unit_path template renders past GAM's unit-path byte limit \ + for this page's section; the runtime omits this slot on this path", + slot.id + ), + )); + } + + let (status, dom_for_evidence, gpt_for_evidence, phase) = if let Some(idx) = gpt_idx { + consumed_gpt[idx] = true; + let gpt = &evidence.gpt_slots[idx]; + let dom_id = resolved_id.clone().or_else(|| Some(gpt.div_id.clone())); + if banner.is_empty() { + warnings.push(warning( + "unsupported_format", + format!( + "slot `{}` has only non-banner formats; not confirmable in Phase 1", + slot.id + ), + )); + ( + SlotStatus::Unconfirmable, + dom_id, + Some(gpt.clone()), + Some(gpt.phase), + ) + } else if gpt.sizes.is_empty() { + warnings.push(warning( + "out_of_page_slot", + format!( + "slot `{}` matched an out-of-page GPT slot with no sizes", + slot.id + ), + )); + ( + SlotStatus::Partial, + dom_id, + Some(gpt.clone()), + Some(gpt.phase), + ) + } else if banner.iter().any(|size| gpt.sizes.contains(size)) { + let extra: Vec<(u32, u32)> = gpt + .sizes + .iter() + .copied() + .filter(|size| !banner.contains(size)) + .collect(); + if !extra.is_empty() { + warnings.push(warning( + "extra_observed_size", + format!("slot `{}` observed extra GPT sizes {extra:?}", slot.id), + )); + } + let missing: Vec<(u32, u32)> = banner + .iter() + .copied() + .filter(|size| !gpt.sizes.contains(size)) + .collect(); + if !missing.is_empty() { + warnings.push(warning( + "configured_size_not_observed", + format!( + "slot `{}` configured sizes {missing:?} were not observed", + slot.id + ), + )); + } + ( + SlotStatus::Confirmed, + dom_id, + Some(gpt.clone()), + Some(gpt.phase), + ) + } else { + warnings.push(warning( + "incompatible_sizes", + format!( + "slot `{}` GPT path and div matched but no configured size overlapped", + slot.id + ), + )); + ( + SlotStatus::Partial, + dom_id, + Some(gpt.clone()), + Some(gpt.phase), + ) + } + } else if let Some(dom) = resolved { + warnings.push(warning( + "dom_without_gpt", + "DOM element matched, but no GPT slot evidence was observed".to_string(), + )); + ( + SlotStatus::Partial, + Some(dom.dom_id.clone()), + None, + Some(dom.phase), + ) + } else { + (SlotStatus::Missing, None, None, None) + }; + + slots.push(SlotResult { + id: slot.id.clone(), + status, + phase, + evidence: SlotEvidence { + dom_id: dom_for_evidence, + gpt: gpt_for_evidence, + }, + warnings, + }); + } + + let extra_evidence = evidence + .gpt_slots + .iter() + .enumerate() + .filter(|(idx, _)| !consumed_gpt[*idx]) + .map(|(_, gpt)| ExtraEvidence { + kind: "gpt".to_string(), + phase: gpt.phase, + dom_id: Some(gpt.div_id.clone()), + gam_unit_path: Some(gpt.gam_unit_path.clone()), + sizes: gpt.sizes.clone(), + reason: "no_configured_slot_matched".to_string(), + }) + .collect(); + + PageVerificationResult { + runtime_ad_stack_expected: gate.expected, + slots, + extra_evidence, + } +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::ad_templates::expected::ExpectedFormat; + + fn dom(id: &str) -> DomEvidence { + DomEvidence { + dom_id: id.to_string(), + phase: EvidencePhase::InitialLoad, + } + } + + fn gpt_slot(gam_unit_path: &str, div_id: &str, sizes: &[(u32, u32)]) -> GptSlotEvidence { + GptSlotEvidence { + gam_unit_path: gam_unit_path.to_string(), + div_id: div_id.to_string(), + sizes: sizes.to_vec(), + phase: EvidencePhase::InitialLoad, + } + } + + fn aps(slot_id: &str, sizes: &[(u32, u32)]) -> ApsFetchBidsEvidence { + ApsFetchBidsEvidence { + slot_id: slot_id.to_string(), + sizes: sizes.to_vec(), + phase: EvidencePhase::InitialLoad, + } + } + + fn evidence( + doms: Vec, + gpts: Vec, + aps: Vec, + ) -> BrowserAdEvidence { + BrowserAdEvidence { + dom_ids: doms, + gpt_slots: gpts, + aps_calls: aps, + page_bids: Vec::new(), + warnings: Vec::new(), + } + } + + fn expected_slot( + id: &str, + div_id: &str, + gam_unit_path: &str, + sizes: &[(u32, u32)], + providers: &[&str], + ) -> ExpectedSlot { + ExpectedSlot { + id: id.to_string(), + div_id: div_id.to_string(), + gam_unit_path: Some(gam_unit_path.to_string()), + formats: sizes + .iter() + .map(|&(width, height)| ExpectedFormat { + width, + height, + media_type: MediaType::Banner, + }) + .collect(), + providers: providers.iter().copied().map(String::from).collect(), + page_patterns: Vec::new(), + } + } + + fn expected_slot_video(id: &str, div_id: &str, gam_unit_path: &str) -> ExpectedSlot { + ExpectedSlot { + id: id.to_string(), + div_id: div_id.to_string(), + gam_unit_path: Some(gam_unit_path.to_string()), + formats: vec![ExpectedFormat { + width: 0, + height: 0, + media_type: MediaType::Video, + }], + providers: Vec::new(), + page_patterns: Vec::new(), + } + } + + #[test] + fn gpt_path_div_and_size_overlap_confirms_slot() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed); + assert!( + result.slots[0].warnings.is_empty(), + "confirmed slot should carry no warnings" + ); + } + + #[test] + fn unrenderable_gam_unit_path_never_confirms() { + let mut expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + expected.gam_unit_path = None; + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!( + result.slots[0].status, + SlotStatus::Partial, + "an unrenderable configured path must not confirm against GPT evidence" + ); + assert!( + result.slots[0] + .warnings + .iter() + .any(|w| w.code == "gam_unit_path_unrenderable"), + "should explain why the slot cannot be confirmed" + ); + } + + #[test] + fn dom_only_is_partial() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence(vec![dom("ad-atf-0")], Vec::new(), Vec::new()); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Partial); + assert!( + result.slots[0] + .warnings + .iter() + .any(|w| w.code == "dom_without_gpt") + ); + } + + #[test] + fn no_dom_or_gpt_is_missing() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence(Vec::new(), Vec::new(), Vec::new()); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Missing); + } + + #[test] + fn prefix_dom_resolution_ignores_container_suffix() { + let expected = expected_slot( + "header", + "ad-header-0-", + "/123/homepage/header", + &[(728, 90)], + &[], + ); + let evidence = evidence( + vec![dom("ad-header-0--container"), dom("ad-header-0-_R_abc123")], + Vec::new(), + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!( + result.slots[0].evidence.dom_id.as_deref(), + Some("ad-header-0-_R_abc123"), + "prefix match should skip -container" + ); + assert_eq!(result.slots[0].status, SlotStatus::Partial); + } + + #[test] + fn unmatched_gpt_slot_becomes_extra_evidence() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![ + gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)]), + gpt_slot( + "/123/publisher/right-rail", + "ad-right-rail-0", + &[(300, 250)], + ), + ], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed); + assert_eq!(result.extra_evidence.len(), 1); + assert_eq!(result.extra_evidence[0].kind, "gpt"); + assert!( + !result.strict_failed(), + "extra evidence alone must not fail strict" + ); + } + + #[test] + fn auction_disabled_skips_strict_missing_failure() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence(Vec::new(), Vec::new(), Vec::new()); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::auction_disabled(), + ); + + assert_eq!(result.runtime_ad_stack_expected, RuntimeAdStackExpected::No); + assert_eq!(result.slots[0].status, SlotStatus::Missing); + assert!( + !result.strict_failed(), + "missing slot must not fail strict when ad stack is No" + ); + } + + #[test] + fn gpt_incompatible_sizes_is_partial() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(728, 90)])], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Partial); + assert!( + result.slots[0] + .warnings + .iter() + .any(|w| w.code == "incompatible_sizes") + ); + } + + #[test] + fn non_banner_only_slot_is_unconfirmable_and_does_not_fail_strict() { + let expected = expected_slot_video("video", "ad-video-", "/123/news/video"); + let evidence = evidence( + vec![dom("ad-video-0")], + vec![gpt_slot("/123/news/video", "ad-video-0", &[(640, 480)])], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Unconfirmable); + assert!( + result.slots[0] + .warnings + .iter() + .any(|w| w.code == "unsupported_format") + ); + assert!( + !result.strict_failed(), + "checker limitations should not fail strict" + ); + } + + #[test] + fn gpt_container_element_id_confirms() { + let expected = expected_slot("atf", "ad-atf-0", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0"), dom("ad-atf-0-container")], + vec![gpt_slot( + "/123/news/atf", + "ad-atf-0-container", + &[(300, 250)], + )], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!( + result.slots[0].status, + SlotStatus::Confirmed, + "container element id is a valid GPT div match" + ); + } + + #[test] + fn sizeless_live_slot_is_partial_when_config_declares_banner_sizes() { + let expected = expected_slot( + "interstitial", + "ad-oop-", + "/123/news/oop", + &[(300, 250)], + &[], + ); + let evidence = evidence( + vec![dom("ad-oop-0")], + vec![gpt_slot("/123/news/oop", "ad-oop-0", &[])], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Partial); + assert!( + result.slots[0] + .warnings + .iter() + .any(|w| w.code == "out_of_page_slot") + ); + assert!( + result.strict_failed(), + "a live sizeless slot drifting from configured banner sizes must fail strict" + ); + } + + #[test] + fn aps_match_adds_no_warning() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &["aps"]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + vec![aps("atf", &[(300, 250)])], + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed); + assert!( + !result.slots[0] + .warnings + .iter() + .any(|w| w.code.starts_with("aps_")), + "matching APS should not warn" + ); + } + + #[test] + fn server_side_aps_config_does_not_require_client_fetch_bids_evidence() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &["aps"]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + Vec::new(), + ); + + let result = compare_page_evidence( + &[expected], + &evidence, + RuntimeGateSummary::unknown_allowed(), + ); + + assert_eq!( + result.slots[0].status, + SlotStatus::Confirmed, + "missing APS does not flip status" + ); + assert!(result.slots[0].warnings.is_empty()); + assert!( + !result.strict_failed(), + "provider warning alone must not fail strict" + ); + } +} diff --git a/crates/trusted-server-cli/src/ad_templates/expected.rs b/crates/trusted-server-cli/src/ad_templates/expected.rs new file mode 100644 index 000000000..9392963ff --- /dev/null +++ b/crates/trusted-server-cli/src/ad_templates/expected.rs @@ -0,0 +1,336 @@ +//! Pure expected-slot projection from the runtime creative-opportunity matcher. +//! +//! This module owns path/URL normalization and converts the slots matched by +//! [`match_slots`] into stable, owned [`ExpectedSlot`] records for output and +//! browser-evidence comparison. It must not duplicate glob-matching semantics. + +use trusted_server_core::auction::types::MediaType; +use trusted_server_core::creative_opportunities::{CreativeOpportunitiesConfig, match_slots}; +use url::Url; + +/// The expected slots for a single page path, in configured slot order. +#[derive(Debug, Clone, PartialEq)] +pub struct ExpectedSlots { + /// The page path the slots were matched against. + pub path: String, + /// Matched slots projected into stable records, in configured order. + pub slots: Vec, +} + +/// A single configured slot expected to appear for a page path. +#[derive(Debug, Clone, PartialEq)] +pub struct ExpectedSlot { + /// The slot identifier. + pub id: String, + /// Resolved HTML `div` element ID (override or the slot id). + pub div_id: String, + /// Resolved GAM unit path: the rendered `gam_unit_path` template (or + /// `//` when the slot has none). + /// + /// `None` only for manually constructed comparison fixtures. Projection + /// omits a slot when the runtime cannot render it for this path. + pub gam_unit_path: Option, + /// Configured ad formats. + pub formats: Vec, + /// Configured provider names, in `aps`, `prebid` order. + pub providers: Vec, + /// Glob patterns configured for this slot. + pub page_patterns: Vec, +} + +/// A configured ad format as a stable width/height/media-type record. +#[derive(Debug, Clone, PartialEq)] +pub struct ExpectedFormat { + /// Creative width in pixels. + pub width: u32, + /// Creative height in pixels. + pub height: u32, + /// Configured media type. + pub media_type: MediaType, +} + +/// Projects the slots matching `path` into stable expected-slot records. +/// +/// Uses [`match_slots`] so glob semantics stay identical to the runtime, and +/// preserves configured slot order. `path` is assumed already normalized via +/// [`normalize_path_or_url`]. +/// +/// `gam_unit_path` templates are rendered against the section the runtime would +/// derive from `path` (per the config's `section_root`/`section_segment` +/// policy), so `{section}`-bearing configs project the same unit path the live +/// page requests. +// Shared projection used by the audit verifier; the static commands match slots +// directly against the runtime matcher. +#[must_use] +pub fn expected_slots_for_path(path: &str, config: &CreativeOpportunitiesConfig) -> ExpectedSlots { + let section = config.section_for_path(path); + let slots = match_slots(&config.slot, path) + .into_iter() + .filter_map(|slot| { + let gam_unit_path = slot.render_gam_unit_path(&config.gam_network_id, §ion)?; + Some(ExpectedSlot { + id: slot.id.clone(), + div_id: slot.resolved_div_id().to_string(), + gam_unit_path: Some(gam_unit_path), + formats: slot + .formats + .iter() + .map(|format| ExpectedFormat { + width: format.width, + height: format.height, + media_type: format.media_type.clone(), + }) + .collect(), + providers: provider_names(slot), + page_patterns: slot.page_patterns.clone(), + }) + }) + .collect(); + + ExpectedSlots { + path: path.to_string(), + slots, + } +} + +fn provider_names( + slot: &trusted_server_core::creative_opportunities::CreativeOpportunitySlot, +) -> Vec { + let mut providers = Vec::new(); + if slot.providers.aps.is_some() { + providers.push("aps".to_string()); + } + if slot.providers.prebid.is_some() { + providers.push("prebid".to_string()); + } + providers +} + +/// Normalizes a page path or full URL into a request path. +/// +/// Full `scheme://` inputs are parsed and reduced to their path; bare inputs have +/// query and fragment stripped and a leading `/` ensured. Empty paths become `/`. +/// +/// # Errors +/// +/// Returns a user-facing string when a `scheme://` input cannot be parsed as a URL. +pub fn normalize_path_or_url(input: &str) -> Result { + let path_input = input.split(['?', '#']).next().unwrap_or(input); + let scheme_prefix = path_input.split_once("://").map(|(scheme, _)| scheme); + let has_url_scheme = scheme_prefix.is_some_and(|scheme| { + let mut chars = scheme.chars(); + chars.next().is_some_and(|ch| ch.is_ascii_alphabetic()) + && chars.all(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '+' | '-' | '.')) + }); + if has_url_scheme { + let url = Url::parse(input).map_err(|err| format!("invalid URL `{input}`: {err}"))?; + let path = url.path(); + return Ok(if path.is_empty() { + "/".to_string() + } else { + path.to_string() + }); + } + + let base = Url::parse("https://path-normalizer.example/") + .expect("should parse static path normalization base"); + let relative = input.trim_start_matches('/'); + let normalized = base + .join(&format!("./{relative}")) + .map_err(|error| format!("invalid path `{input}`: {error}"))?; + Ok(normalized.path().to_string()) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn creative_config_with_slots(patterns: &[&str]) -> CreativeOpportunitiesConfig { + let page_patterns = patterns + .iter() + .map(|pattern| format!("\"{pattern}\"")) + .collect::>() + .join(", "); + let toml = format!( + "gam_network_id = \"123\"\n\ + \n\ + [[slot]]\n\ + id = \"atf\"\n\ + gam_unit_path = \"/123/news/atf\"\n\ + div_id = \"ad-atf-\"\n\ + page_patterns = [{page_patterns}]\n\ + formats = [{{ width = 300, height = 250 }}]\n\ + \n\ + [slot.providers.prebid]\n\ + bidders = {{}}\n" + ); + let mut config = toml::from_str::(&toml) + .expect("should deserialize creative opportunities config"); + config.compile_slots(); + config + } + + #[test] + fn expected_slots_use_runtime_matcher_and_config_order() { + let config = creative_config_with_slots(&["/news/*", "/"]); + let expected = expected_slots_for_path("/news/story", &config); + + assert_eq!(expected.path, "/news/story"); + assert_eq!( + expected + .slots + .iter() + .map(|slot| slot.id.as_str()) + .collect::>(), + ["atf"] + ); + assert_eq!(expected.slots[0].div_id, "ad-atf-"); + assert_eq!( + expected.slots[0].gam_unit_path.as_deref(), + Some("/123/news/atf") + ); + assert_eq!(expected.slots[0].providers, ["prebid"]); + assert_eq!( + expected.slots[0].formats, + vec![ExpectedFormat { + width: 300, + height: 250, + media_type: MediaType::Banner, + }] + ); + } + + #[test] + fn expected_slots_default_resolution_without_overrides() { + let toml = "gam_network_id = \"42\"\n\ + \n\ + [[slot]]\n\ + id = \"footer\"\n\ + page_patterns = [\"/\"]\n\ + formats = [{ width = 728, height = 90 }]\n"; + let mut config = + toml::from_str::(toml).expect("should deserialize"); + config.compile_slots(); + + let expected = expected_slots_for_path("/", &config); + assert_eq!(expected.slots[0].div_id, "footer"); + assert_eq!( + expected.slots[0].gam_unit_path.as_deref(), + Some("/42/footer") + ); + assert!(expected.slots[0].providers.is_empty()); + } + + #[test] + fn expected_slots_render_section_templates_per_path() { + let toml = "gam_network_id = \"99999\"\n\ + section_root = \"homepage\"\n\ + \n\ + [[slot]]\n\ + id = \"ad-header-0\"\n\ + gam_unit_path = \"/{network_id}/example/{section}\"\n\ + page_patterns = [\"/\", \"/news\", \"/news/*\"]\n\ + formats = [{ width = 728, height = 90 }]\n"; + let mut config = + toml::from_str::(toml).expect("should deserialize"); + config.compile_slots(); + + // A path with a section segment renders that segment. + assert_eq!( + expected_slots_for_path("/news/story", &config).slots[0] + .gam_unit_path + .as_deref(), + Some("/99999/example/news"), + "a section template should render the path's section" + ); + // The site root falls back to the configured section_root. + assert_eq!( + expected_slots_for_path("/", &config).slots[0] + .gam_unit_path + .as_deref(), + Some("/99999/example/homepage"), + "the root path should render section_root" + ); + } + + #[test] + fn expected_slots_omit_dynamic_template_the_runtime_cannot_render() { + // A `{section}` template that renders past GAM's 100-byte unit-path + // limit. The runtime omits this slot for the request path, so diagnostics + // must not match it against a truncated or otherwise different path. + let toml = "gam_network_id = \"99999\"\n\ + section_root = \"homepage\"\n\ + \n\ + [[slot]]\n\ + id = \"ad-header-0\"\n\ + gam_unit_path = \"/{section}/{section}\"\n\ + page_patterns = [\"/*\"]\n\ + formats = [{ width = 728, height = 90 }]\n"; + let mut config = + toml::from_str::(toml).expect("should deserialize"); + config.compile_slots(); + + let long_path = format!("/{}", "a".repeat(60)); + let expected = expected_slots_for_path(&long_path, &config); + + assert!( + expected.slots.is_empty(), + "the runtime omits an over-limit dynamic slot on this path" + ); + } + + #[test] + fn normalize_path_or_url_strips_query_and_fragment() { + assert_eq!( + normalize_path_or_url("https://www.example.com/news/story?x=1#top") + .expect("should normalize"), + "/news/story" + ); + assert_eq!( + normalize_path_or_url("news/story?x=1").expect("should normalize"), + "/news/story" + ); + } + + #[test] + fn normalize_path_or_url_roots_empty_input() { + assert_eq!( + normalize_path_or_url("https://www.example.com").expect("should normalize"), + "/" + ); + assert_eq!(normalize_path_or_url("").expect("should normalize"), "/"); + } + + #[test] + fn normalize_path_or_url_uses_identical_url_rules_for_bare_paths() { + assert_eq!( + normalize_path_or_url("/a/../b").expect("should normalize bare dot segment"), + "/b" + ); + assert_eq!( + normalize_path_or_url("https://example.com/a/../b") + .expect("should normalize URL dot segment"), + "/b" + ); + assert_eq!( + normalize_path_or_url("/a b").expect("should encode bare path"), + "/a%20b" + ); + assert_eq!( + normalize_path_or_url("/r?to=https://example.com") + .expect("query URL should not change input classification"), + "/r" + ); + assert_eq!( + normalize_path_or_url("/news:latest").expect("colon should stay in bare path"), + "/news:latest", + "a colon in the first segment must not be parsed as a URL scheme" + ); + assert_eq!( + normalize_path_or_url("https://example.com/news:latest") + .expect("colon should stay in URL path"), + "/news:latest", + "bare and absolute forms should normalize identically" + ); + } +} diff --git a/crates/trusted-server-cli/src/ad_templates/mod.rs b/crates/trusted-server-cli/src/ad_templates/mod.rs new file mode 100644 index 000000000..3c26bf121 --- /dev/null +++ b/crates/trusted-server-cli/src/ad_templates/mod.rs @@ -0,0 +1,7 @@ +//! Pure, host-only ad-template CLI logic shared by the static `ts config +//! ad-templates ...` commands and the browser-backed `ts audit ad-templates +//! verify` command. + +pub mod compare; +pub mod expected; +pub mod output; diff --git a/crates/trusted-server-cli/src/ad_templates/output.rs b/crates/trusted-server-cli/src/ad_templates/output.rs new file mode 100644 index 000000000..e12c9eebc --- /dev/null +++ b/crates/trusted-server-cli/src/ad_templates/output.rs @@ -0,0 +1,480 @@ +//! Stable, serializable output model for ad-template diagnostics. +//! +//! These types mirror the `--json` contract in +//! `docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md` §8. +//! Field names and declaration order are load-bearing: `serde` serializes struct +//! fields in declaration order, so the order here must match the spec examples. +//! +//! The model is consumed by the `ts audit ad-templates verify` orchestrator, +//! which assembles these wire types from the URL/gate context and comparison result. + +use std::borrow::Cow; + +use serde::{Deserialize, Serialize}; + +use trusted_server_core::creative_opportunities::RuntimeAdStackExpected; + +/// Escapes control characters in page-controlled text bound for a terminal. +/// +/// Page titles and collector warning messages are attacker-controlled: an +/// audited page can put ANSI/OSC escape sequences in `document.title` and drive +/// the operator's terminal (cursor movement, clipboard writes, forged output) +/// when the value is printed verbatim. Every C0 control (including ESC), DEL, +/// and the C1 range are rendered as `\u{XXXX}` so the text stays inert. JSON +/// output is unaffected — `serde_json` escapes these already. +/// +/// Returns a borrowed `Cow` when the input needs no escaping. +#[must_use] +pub fn escape_terminal_text(value: &str) -> Cow<'_, str> { + if !value.chars().any(is_terminal_control) { + return Cow::Borrowed(value); + } + let mut escaped = String::with_capacity(value.len()); + for ch in value.chars() { + if is_terminal_control(ch) { + escaped.push_str(&format!("\\u{{{:04X}}}", ch as u32)); + } else { + escaped.push(ch); + } + } + Cow::Owned(escaped) +} + +/// Whether `ch` can act as a terminal control code (C0, DEL, or C1). +fn is_terminal_control(ch: char) -> bool { + let code = ch as u32; + code < 0x20 + || (0x7f..=0x9f).contains(&code) + || (0x202a..=0x202e).contains(&code) + || (0x2066..=0x2069).contains(&code) +} + +/// Confirmation status for a single configured slot. +#[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum SlotStatus { + /// GPT evidence matches GAM path, div, and a compatible size. + Confirmed, + /// Some evidence, but not enough to confirm. + Partial, + /// No DOM or GPT evidence confirms the slot. + Missing, + /// The checker does not support confirming this slot type. + Unconfirmable, +} + +/// JSON rendering of the runtime ad-stack expectation. +#[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum RuntimeAdStackExpectedJson { + /// The server-side ad stack is expected to run. + Yes, + /// A known gate blocks the server-side ad stack. + No, + /// Consent or another gate is unprovable. + Unknown, +} + +impl From for RuntimeAdStackExpectedJson { + fn from(value: RuntimeAdStackExpected) -> Self { + match value { + RuntimeAdStackExpected::Yes => Self::Yes, + RuntimeAdStackExpected::No => Self::No, + RuntimeAdStackExpected::Unknown => Self::Unknown, + } + } +} + +/// State of a single runtime gate. +#[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum GateState { + /// The gate passed. + Pass, + /// The gate blocked the ad stack. + Fail, + /// The gate state could not be proven. + Unknown, +} + +/// Evidence-collection phase, rendered for JSON output. +#[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum EvidencePhaseJson { + /// Observed during the initial page load and settle. + InitialLoad, + /// Observed only after the deterministic scroll pass. + Scroll, +} + +/// A structured warning with a stable machine code and human message. +/// +/// `Serialize` for output; `Deserialize` because the browser collector payload +/// carries warning objects decoded into the comparison input. +#[derive(Debug, Clone, Eq, PartialEq, Serialize, Deserialize)] +pub struct Warning { + /// Stable machine-readable code (e.g. `dom_without_gpt`). + pub code: String, + /// Human-readable message; JSON consumers must not parse this. + pub message: String, +} + +/// Top-level `--json` document for `ts audit ad-templates verify`. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct VerificationReport { + /// True when no strict failure and no page-level error occurred. + pub ok: bool, + /// Whether `--strict` was set. + pub strict: bool, + /// One entry per requested URL, in input order. + pub pages: Vec, + /// Run-level warnings not attributable to a single page. + /// + /// Always empty today — every warning the verifier raises belongs to a page + /// or a slot. Kept because the JSON schema declares it, so a consumer can + /// read it unconditionally. + pub warnings: Vec, +} + +/// A single audited page result. +/// +/// `error` is declared immediately after `path` so the serialized key order +/// matches the spec §8 `navigation_failed` shape; on normal pages it is `None` +/// and skipped, leaving the runtime/gates fields in §8 order. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct PageJson { + /// The requested URL. + pub url: String, + /// The final URL after redirects, or `null` on navigation failure. + pub final_url: Option, + /// The requested URL's path. + pub requested_path: String, + /// The final path used for matching, or `null` on navigation failure. + pub path: Option, + /// Present only on a page-level collection failure. + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, + /// Three-state runtime ad-stack expectation; absent on error pages. + #[serde(skip_serializing_if = "Option::is_none")] + pub runtime_ad_stack_expected: Option, + /// Per-gate evidence; absent on error pages. + #[serde(skip_serializing_if = "Option::is_none")] + pub gates: Option, + /// Number of configured slots matched for the final path; absent on error pages. + #[serde(skip_serializing_if = "Option::is_none")] + pub matched_slot_count: Option, + /// Per-slot verification results. + pub slots: Vec, + /// Live ad-slot evidence with no matching configured slot. + pub extra_evidence: Vec, + /// Page-level warnings. + pub warnings: Vec, +} + +/// Runtime gate states for a page, one field per spec §5.2 gate. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct Gates { + /// Request method is `GET`. + pub method_get: GateState, + /// Request is a top-level navigation. + pub navigation: GateState, + /// Request is not a prefetch. + pub not_prefetch: GateState, + /// Request is not from a known bot. + pub not_bot: GateState, + /// At least one configured slot matched the final path. + pub matched_slots: GateState, + /// The `[auction].enabled` kill switch is on. + pub auction_enabled: GateState, + /// Consent allows the auction (often `unknown` for live requests). + pub consent_allows_auction: GateState, +} + +/// A single configured slot's verification result. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct SlotJson { + /// The configured slot id. + pub id: String, + /// The slot's confirmation status. + pub status: SlotStatus, + /// The phase the confirming evidence was observed in. + #[serde(skip_serializing_if = "Option::is_none")] + pub phase: Option, + /// The configured shape of the slot (no `id`/`page_patterns` per §8). + pub configured: ConfiguredJson, + /// The live evidence observed for this slot. + pub evidence: SlotEvidenceJson, + /// Slot-level warnings (e.g. provider or size warnings). + pub warnings: Vec, +} + +/// The configured shape of a slot, as rendered in §8 `configured`. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct ConfiguredJson { + /// Resolved div element ID. + pub div_id: String, + /// Resolved GAM unit path, or `null` when a dynamic template renders past + /// GAM's unit-path byte limit for this page's section. + pub gam_unit_path: Option, + /// Configured formats. + pub formats: Vec, + /// Configured provider names. + pub providers: Vec, +} + +/// A configured format, as rendered in §8. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct FormatJson { + /// Creative width in pixels. + pub width: u32, + /// Creative height in pixels. + pub height: u32, + /// Media type string (`banner`, `video`, `native`). + pub media_type: String, +} + +/// Live evidence observed for a configured slot. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct SlotEvidenceJson { + /// The resolved DOM element ID observed, if any. + pub dom_id: Option, + /// GPT slot evidence, if any (no `phase` key per §8). + pub gpt: Option, +} + +/// GPT slot evidence, as rendered in §8 `evidence.gpt`. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct GptEvidenceJson { + /// The observed GAM ad unit path. + pub gam_unit_path: String, + /// The observed GPT slot element ID. + pub div_id: String, + /// Observed numeric sizes as `[width, height]` pairs. + pub sizes: Vec<[u32; 2]>, +} + +/// Live ad-slot evidence with no matching configured slot. +#[derive(Debug, Clone, PartialEq, Serialize)] +pub struct ExtraEvidenceJson { + /// Evidence kind: `dom`, `gpt`, or `aps`. + pub kind: String, + /// The phase the evidence was observed in. + pub phase: EvidencePhaseJson, + /// The DOM element ID, if any. + pub dom_id: Option, + /// The GAM unit path, if any. + pub gam_unit_path: Option, + /// Observed numeric sizes as `[width, height]` pairs. + pub sizes: Vec<[u32; 2]>, + /// Why this evidence is reported as extra. + pub reason: String, +} + +#[cfg(test)] +impl VerificationReport { + fn example_confirmed_with_extra_evidence() -> Self { + VerificationReport { + ok: true, + strict: false, + pages: vec![PageJson { + url: "https://www.example.com/news/story".to_string(), + final_url: Some("https://www.example.com/news/story".to_string()), + requested_path: "/news/story".to_string(), + path: Some("/news/story".to_string()), + error: None, + runtime_ad_stack_expected: Some(RuntimeAdStackExpectedJson::Unknown), + gates: Some(Gates { + method_get: GateState::Pass, + navigation: GateState::Pass, + not_prefetch: GateState::Pass, + not_bot: GateState::Pass, + matched_slots: GateState::Pass, + auction_enabled: GateState::Pass, + consent_allows_auction: GateState::Unknown, + }), + matched_slot_count: Some(1), + slots: vec![SlotJson { + id: "atf".to_string(), + status: SlotStatus::Confirmed, + phase: Some(EvidencePhaseJson::InitialLoad), + configured: ConfiguredJson { + div_id: "ad-atf-".to_string(), + gam_unit_path: Some("/123/news/atf".to_string()), + formats: vec![FormatJson { + width: 300, + height: 250, + media_type: "banner".to_string(), + }], + providers: vec!["aps".to_string()], + }, + evidence: SlotEvidenceJson { + dom_id: Some("ad-atf-0".to_string()), + gpt: Some(GptEvidenceJson { + gam_unit_path: "/123/news/atf".to_string(), + div_id: "ad-atf-0".to_string(), + sizes: vec![[300, 250]], + }), + }, + warnings: Vec::new(), + }], + extra_evidence: vec![ExtraEvidenceJson { + kind: "gpt".to_string(), + phase: EvidencePhaseJson::InitialLoad, + dom_id: Some("ad-right-rail-0".to_string()), + gam_unit_path: Some("/123/publisher/right-rail".to_string()), + sizes: vec![[300, 250]], + reason: "no_configured_slot_matched".to_string(), + }], + warnings: vec![Warning { + code: "redirected".to_string(), + message: "navigation redirected to the final path".to_string(), + }], + }], + warnings: Vec::new(), + } + } + + fn example_navigation_failed() -> Self { + VerificationReport { + ok: false, + strict: false, + pages: vec![PageJson { + url: "https://www.example.com/broken".to_string(), + final_url: None, + requested_path: "/broken".to_string(), + path: None, + error: Some(Warning { + code: "navigation_failed".to_string(), + message: "failed to read main document navigation response".to_string(), + }), + runtime_ad_stack_expected: None, + gates: None, + matched_slot_count: None, + slots: Vec::new(), + extra_evidence: Vec::new(), + warnings: Vec::new(), + }], + warnings: Vec::new(), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn escape_terminal_text_passes_through_ordinary_titles() { + assert!( + matches!( + escape_terminal_text("Example News — Story"), + Cow::Borrowed(_) + ), + "text with no control characters should not allocate" + ); + assert_eq!( + escape_terminal_text("Example News — Story"), + "Example News — Story" + ); + } + + #[test] + fn escape_terminal_text_neutralizes_control_sequences() { + // ESC-based CSI/OSC sequences and a raw newline are the terminal-driving + // primitives a hostile page would put in `document.title`. + assert_eq!( + escape_terminal_text("a\u{1b}]0;pwned\u{7}b"), + "a\\u{001B}]0;pwned\\u{0007}b", + "ESC and BEL should be rendered inert" + ); + assert_eq!( + escape_terminal_text("line\nforged: ok"), + "line\\u{000A}forged: ok", + "a newline should not let a title forge an output line" + ); + assert_eq!( + escape_terminal_text("del\u{7f}c1\u{9b}"), + "del\\u{007F}c1\\u{009B}", + "DEL and the C1 range should be escaped too" + ); + assert_eq!( + escape_terminal_text("safe\u{202E}forged\u{2066}tail"), + "safe\\u{202E}forged\\u{2066}tail", + "Unicode bidi controls should be rendered inert" + ); + } + + #[test] + fn verification_json_contains_gate_state_and_extra_evidence() { + let result = VerificationReport::example_confirmed_with_extra_evidence(); + let value = serde_json::to_value(&result).expect("should serialize"); + + assert_eq!(value["ok"], true); + assert_eq!(value["pages"][0]["requested_path"], "/news/story"); + assert_eq!(value["pages"][0]["runtime_ad_stack_expected"], "unknown"); + assert_eq!( + value["pages"][0]["gates"]["consent_allows_auction"], + "unknown" + ); + assert_eq!(value["pages"][0]["slots"][0]["status"], "confirmed"); + assert_eq!( + value["pages"][0]["slots"][0]["evidence"]["gpt"]["sizes"][0][0], + 300 + ); + assert_eq!(value["pages"][0]["extra_evidence"][0]["kind"], "gpt"); + assert_eq!(value["pages"][0]["warnings"][0]["code"], "redirected"); + // `configured` excludes id/page_patterns per §8. + assert!(value["pages"][0]["slots"][0]["configured"]["id"].is_null()); + assert!(value["pages"][0]["slots"][0]["configured"]["page_patterns"].is_null()); + // `evidence.gpt` has no `phase` key per §8. + assert!(value["pages"][0]["slots"][0]["evidence"]["gpt"]["phase"].is_null()); + } + + #[test] + fn page_error_json_matches_navigation_failed_shape() { + let result = VerificationReport::example_navigation_failed(); + let value = serde_json::to_value(&result).expect("should serialize"); + let page = &value["pages"][0]; + + assert_eq!(page["error"]["code"], "navigation_failed"); + assert!(page["final_url"].is_null(), "final_url should be null"); + assert!(page["path"].is_null(), "path should be null"); + assert!( + page.get("runtime_ad_stack_expected").is_none(), + "runtime field absent on error page" + ); + assert!(page.get("gates").is_none(), "gates absent on error page"); + assert!( + page.get("matched_slot_count").is_none(), + "matched_slot_count absent on error page" + ); + assert_eq!(value["ok"], false); + } + + #[test] + fn missing_slot_json_omits_evidence_phase() { + let slot = SlotJson { + id: "missing".to_string(), + status: SlotStatus::Missing, + phase: None, + configured: ConfiguredJson { + div_id: "ad-missing-".to_string(), + gam_unit_path: Some("/123/publisher/missing".to_string()), + formats: Vec::new(), + providers: Vec::new(), + }, + evidence: SlotEvidenceJson { + dom_id: None, + gpt: None, + }, + warnings: Vec::new(), + }; + + let value = serde_json::to_value(slot).expect("should serialize missing slot"); + + assert!( + value.get("phase").is_none(), + "missing evidence should not claim an initial-load phase" + ); + } +} diff --git a/crates/trusted-server-cli/src/app_config.rs b/crates/trusted-server-cli/src/app_config.rs new file mode 100644 index 000000000..bee536146 --- /dev/null +++ b/crates/trusted-server-cli/src/app_config.rs @@ -0,0 +1,171 @@ +//! Shared effective Trusted Server app-config loading for the `ts` CLI. +//! +//! Both the static `ts config ad-templates ...` commands and the browser-backed +//! `ts audit ad-templates verify` command load the same effective app config +//! through [`load_settings`], so config-path resolution and the `EdgeZero` +//! environment overlay stay consistent across command families. + +use std::path::{Path, PathBuf}; + +use clap::Args; +use edgezero_core::app_config::{self, AppConfigLoadOptions}; +use edgezero_core::manifest::ManifestLoader; +use trusted_server_core::config::TrustedServerAppConfig; +use trusted_server_core::settings::Settings; + +/// Shared local app-config flags accepted by every config/audit ad-template command. +#[derive(Clone, Debug, Args)] +pub struct AppConfigArgs { + /// Path to `trusted-server.toml`. Defaults to `.toml` beside `edgezero.toml`. + #[arg(long)] + pub app_config: Option, + /// Path to `edgezero.toml`. + #[arg(long, default_value = "edgezero.toml")] + pub manifest: PathBuf, + /// Skip app-config environment overlay. + #[arg(long)] + pub no_env: bool, +} + +/// Effective settings plus the resolved app-config path they were loaded from. +#[derive(Debug)] +pub struct LoadedSettings { + /// The `trusted-server.toml` path the settings were loaded from. + pub app_config_path: PathBuf, + /// The deserialized effective settings. + pub settings: Settings, +} + +/// Loads the effective Trusted Server settings described by `args`. +/// +/// Resolves the app-config path from `args` (or the manifest's `.toml` +/// default), applies the `EdgeZero` environment overlay unless `no_env` is set, and +/// returns the deserialized [`Settings`]. +/// +/// # Errors +/// +/// Returns a user-facing string when the manifest cannot be loaded, has no +/// `[app].name`, or the resolved app-config file cannot be read or parsed. When an +/// explicit `--app-config` path is given and is missing, the error names that +/// exact path rather than silently falling back. +pub fn load_settings(args: &AppConfigArgs) -> Result { + load_settings_with_env_overlay(args, !args.no_env) +} + +/// Loads Trusted Server settings from the resolved app-config file without +/// applying environment overlays. +/// +/// Mutating commands use this path so environment-only values are never +/// persisted into the operator-owned TOML file. +/// +/// # Errors +/// +/// Returns the same path-resolution, read, and parse errors as +/// [`load_settings`]. +#[cfg(test)] +pub(crate) fn load_file_settings(args: &AppConfigArgs) -> Result { + load_settings_with_env_overlay(args, false) +} + +/// Resolves the operator-owned app-config path without deserializing settings. +/// +/// Mutating recovery commands use this when the existing config may already be +/// invalid but still needs a narrowly scoped structural repair. +/// +/// # Errors +/// +/// Returns a user-facing string when the manifest cannot be loaded or has no +/// `[app].name` and no explicit config path was supplied. +pub fn resolve_app_config_file(args: &AppConfigArgs) -> Result { + if let Some(path) = &args.app_config { + return Ok(path.clone()); + } + let manifest_loader = ManifestLoader::from_path(&args.manifest) + .map_err(|err| format!("failed to load {}: {err}", args.manifest.display()))?; + let app_name = manifest_loader.manifest().app.name.clone().ok_or_else(|| { + format!( + "{} has no [app].name; cannot resolve trusted-server.toml", + args.manifest.display() + ) + })?; + Ok(resolve_app_config_path(None, &args.manifest, &app_name)) +} + +fn load_settings_with_env_overlay( + args: &AppConfigArgs, + env_overlay: bool, +) -> Result { + let manifest_loader = ManifestLoader::from_path(&args.manifest) + .map_err(|err| format!("failed to load {}: {err}", args.manifest.display()))?; + let app_name = manifest_loader.manifest().app.name.clone().ok_or_else(|| { + format!( + "{} has no [app].name; cannot resolve trusted-server.toml", + args.manifest.display() + ) + })?; + let app_config_path = + resolve_app_config_path(args.app_config.as_deref(), &args.manifest, &app_name); + + let mut opts = AppConfigLoadOptions::default(); + opts.env_overlay = env_overlay; + let app_config = app_config::deserialize_app_config_with_options::( + &app_config_path, + &app_name, + &opts, + ) + .map_err(|err| format!("failed to load {}: {err}", app_config_path.display()))?; + + Ok(LoadedSettings { + app_config_path, + settings: app_config.into_settings(), + }) +} + +fn resolve_app_config_path( + explicit: Option<&Path>, + manifest_path: &Path, + app_name: &str, +) -> PathBuf { + if let Some(path) = explicit { + return path.to_path_buf(); + } + let file_name = format!("{app_name}.toml"); + if let Some(parent) = manifest_path + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + { + parent.join(file_name) + } else { + PathBuf::from(file_name) + } +} + +#[cfg(test)] +mod tests { + use std::fs; + + use tempfile::TempDir; + + use super::*; + + #[test] + fn explicit_missing_app_config_does_not_fall_back() { + let temp = TempDir::new().expect("should create temp dir"); + let manifest_path = temp.path().join("edgezero.toml"); + fs::write(&manifest_path, "[app]\nname = \"trusted-server\"\n") + .expect("should write manifest"); + let missing_path = temp.path().join("missing.toml"); + + let args = AppConfigArgs { + app_config: Some(missing_path.clone()), + manifest: manifest_path, + no_env: true, + }; + + let err = load_settings(&args).expect_err("should reject missing explicit config"); + assert!( + err.contains(missing_path.to_string_lossy().as_ref()), + "error should mention the explicit missing path" + ); + } +} diff --git a/crates/trusted-server-cli/src/commands/audit/ad_template_collector.js b/crates/trusted-server-cli/src/commands/audit/ad_template_collector.js new file mode 100644 index 000000000..6938808f5 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/audit/ad_template_collector.js @@ -0,0 +1,241 @@ +// Bounded ad-template evidence collector, injected before publisher scripts run. +// +// This body runs inside an IIFE that defines `__TS_CONFIG` (the configured div +// prefixes). It records evidence into `window.__tsAdTemplateEvidence` +// and never captures page HTML, cookies, storage, request bodies, or arbitrary DOM. +// It always calls original page functions with unchanged arguments and never +// spoofs the browser automation flag. + +const __ts_config = typeof __TS_CONFIG === "object" && __TS_CONFIG ? __TS_CONFIG : {} +const __ts_prefixes = Array.isArray(__ts_config.div_prefixes) ? __ts_config.div_prefixes : [] + +const __ts_ev = (window.__tsAdTemplateEvidence = window.__tsAdTemplateEvidence || { + dom_ids: [], + gpt_slots: [], + aps_calls: [], + warnings: [] +}) + +const __ts_phase = () => (window.__tsScrollPhase ? "scroll" : "initial_load") + +// Hard cap per evidence list so a hostile page cannot grow the store without +// bound; the page controls how many slots/elements/warnings it produces. +const __ts_max_entries = 128 +const __ts_max_string_length = 512 +const __ts_wrapped_googletags = new WeakSet() + +function __ts_text(value) { + return String(value).slice(0, __ts_max_string_length) +} + +// Truncation has to be visible: surplus configured slots classify Missing, and +// `--strict` counts that, so a silent drop is indistinguishable from real drift. +let __ts_truncated = false +function __ts_push(list, entry) { + if (list.length < __ts_max_entries) { + list.push(entry) + return + } + if (__ts_truncated) return + __ts_truncated = true + if (__ts_ev.warnings.length < __ts_max_entries) { + __ts_ev.warnings.push({ + code: "evidence_truncated", + message: "an evidence list hit the " + __ts_max_entries + "-entry cap; results are incomplete" + }) + } +} + +function __ts_warn(code, error) { + __ts_push(__ts_ev.warnings, { code, message: __ts_text(error) }) +} + +// GPT sizes reach Rust as u32 pairs, so anything non-integral (fluid slots, +// NaN, negative or fractional dimensions) must be dropped here — a single bad +// pair would fail deserialization of the whole evidence payload and discard +// every other slot's otherwise valid evidence. +function __ts_size_pair(width, height) { + if (!Number.isInteger(width) || !Number.isInteger(height)) return null + if (width < 0 || height < 0 || width > 4294967295 || height > 4294967295) return null + return [width, height] +} + +function __ts_warn_ignored_size(width, height) { + const numeric = Number.isInteger(width) && Number.isInteger(height) + const outOfRange = + numeric && (width < 0 || height < 0 || width > 4294967295 || height > 4294967295) + __ts_push(__ts_ev.warnings, { + code: outOfRange ? "size_out_of_range" : "fluid_size_ignored", + message: outOfRange ? "GPT size outside u32 range ignored" : "non-integer GPT size ignored" + }) +} + +function __ts_normalize_sizes(sizes) { + const out = [] + if (!Array.isArray(sizes)) return out + // Accept [w, h] or [[w, h], ...]; treat numeric-leading arrays as a single pair. + const pairs = typeof sizes[0] === "number" ? [sizes] : sizes + for (const size of pairs) { + if (out.length >= __ts_max_entries) break + const pair = Array.isArray(size) ? __ts_size_pair(size[0], size[1]) : null + if (pair) { + out.push(pair) + } else { + __ts_warn_ignored_size( + Array.isArray(size) ? size[0] : undefined, + Array.isArray(size) ? size[1] : undefined + ) + } + } + return out +} + +function __ts_record_define_slot(adUnitPath, sizes, divId) { + __ts_push(__ts_ev.gpt_slots, { + gam_unit_path: __ts_text(adUnitPath), + div_id: __ts_text(divId), + sizes: __ts_normalize_sizes(sizes), + phase: __ts_phase() + }) +} + +function __ts_wrap_googletag(googletag) { + if (!googletag || (typeof googletag !== "object" && typeof googletag !== "function")) { + return googletag + } + if (__ts_wrapped_googletags.has(googletag)) return googletag + __ts_wrapped_googletags.add(googletag) + // Wrap defineSlot so both direct calls and calls dispatched from the cmd queue + // are recorded (queued callbacks call this same wrapped function). + const originalDefineSlot = googletag.defineSlot + if (typeof originalDefineSlot === "function") { + try { + const descriptor = Object.getOwnPropertyDescriptor(googletag, "defineSlot") + Object.defineProperty(googletag, "defineSlot", { + configurable: true, + enumerable: descriptor ? descriptor.enumerable : true, + writable: true, + value: function (adUnitPath, sizes, divId) { + const slot = originalDefineSlot.apply(this, arguments) + try { + __ts_record_define_slot(adUnitPath, sizes, divId) + } catch (error) { + __ts_warn("define_slot_capture_failed", error) + } + return slot + } + }) + } catch (error) { + __ts_warn("define_slot_wrap_failed", error) + } + } + return googletag +} + +// Wrap an existing global or intercept a later assignment of it. +function __ts_install(name, wrap) { + if (window[name]) { + try { + wrap(window[name]) + } catch (error) { + __ts_warn(name + "_wrap_failed", error) + } + return + } + let internal + Object.defineProperty(window, name, { + configurable: true, + // A real `window.googletag` is an ordinary enumerable global; matching that + // keeps `Object.keys(window)` identical with and without the collector. + enumerable: true, + get() { + return internal + }, + set(value) { + internal = value + try { + internal = wrap(value) + } catch (error) { + __ts_warn(name + "_wrap_failed", error) + } + } + }) +} + +__ts_install("googletag", __ts_wrap_googletag) + +// On-demand DOM + getSlots scrape, invoked by the collector after settle/scroll. +window.__tsCollectAdTemplateEvidence = function () { + try { + const seen = new Set(__ts_ev.dom_ids.map((entry) => entry.dom_id)) + for (const element of document.querySelectorAll("[id]")) { + const id = __ts_text(element.id) + if (id.endsWith("-container")) continue + if (__ts_prefixes.some((prefix) => id.startsWith(prefix)) && !seen.has(id)) { + __ts_push(__ts_ev.dom_ids, { dom_id: id, phase: __ts_phase() }) + seen.add(id) + } + } + const googletag = window.googletag + if (googletag && typeof googletag.pubads === "function") { + const pubads = googletag.pubads() + const slots = typeof pubads.getSlots === "function" ? pubads.getSlots() : [] + for (const slot of slots) { + try { + const path = typeof slot.getAdUnitPath === "function" ? slot.getAdUnitPath() : "" + const divId = typeof slot.getSlotElementId === "function" ? slot.getSlotElementId() : "" + const rawSizes = typeof slot.getSizes === "function" ? slot.getSizes() : [] + const sizes = [] + for (const size of rawSizes) { + if (sizes.length >= __ts_max_entries) break + let pair = null + if ( + size && + typeof size.getWidth === "function" && + typeof size.getHeight === "function" + ) { + // A fluid GPT size answers getWidth()/getHeight() with a + // non-numeric value rather than throwing. + pair = __ts_size_pair(size.getWidth(), size.getHeight()) + } else if (Array.isArray(size)) { + pair = __ts_size_pair(size[0], size[1]) + } + if (pair) { + sizes.push(pair) + } else { + const width = + size && typeof size.getWidth === "function" + ? size.getWidth() + : Array.isArray(size) + ? size[0] + : undefined + const height = + size && typeof size.getHeight === "function" + ? size.getHeight() + : Array.isArray(size) + ? size[1] + : undefined + __ts_warn_ignored_size(width, height) + } + } + const exists = __ts_ev.gpt_slots.some( + (entry) => entry.gam_unit_path === __ts_text(path) && entry.div_id === __ts_text(divId) + ) + if (!exists) { + __ts_push(__ts_ev.gpt_slots, { + gam_unit_path: __ts_text(path), + div_id: __ts_text(divId), + sizes, + phase: __ts_phase() + }) + } + } catch (error) { + __ts_warn("gpt_scrape_failed", error) + } + } + } + } catch (error) { + __ts_warn("collect_failed", error) + } + return __ts_ev +} diff --git a/crates/trusted-server-cli/src/commands/audit/ad_templates.rs b/crates/trusted-server-cli/src/commands/audit/ad_templates.rs new file mode 100644 index 000000000..cb11a0a0c --- /dev/null +++ b/crates/trusted-server-cli/src/commands/audit/ad_templates.rs @@ -0,0 +1,1007 @@ +//! Browser-backed `ts audit ad-templates verify` orchestration. +//! +//! For each URL: collect live evidence through an [`AuditCollector`], match +//! configured slots against the **final** (post-redirect) path, evaluate the +//! runtime gate, compare evidence, and assemble the stable §8 wire result. The +//! orchestration is collector-agnostic so it is fully tested with an in-memory +//! fake collector, with no Chrome dependency. + +use std::io::{self, Write}; + +use trusted_server_core::auction::types::MediaType; +use trusted_server_core::creative_opportunities::{ + AdStackGateInput, CreativeOpportunitiesConfig, evaluate_ad_stack_gate, +}; + +use crate::ad_templates::compare::{ + BrowserAdEvidence, EvidencePhase, ExtraEvidence, RuntimeGateSummary, SlotEvidence, SlotResult, + SlotStatus as CompareStatus, compare_page_evidence, +}; +use crate::ad_templates::expected::{ExpectedSlot, expected_slots_for_path, normalize_path_or_url}; +use crate::ad_templates::output::{ + ConfiguredJson, EvidencePhaseJson, ExtraEvidenceJson, FormatJson, GateState, Gates, + GptEvidenceJson, PageJson, RuntimeAdStackExpectedJson, SlotEvidenceJson, SlotJson, SlotStatus, + VerificationReport, Warning, escape_terminal_text, +}; +use crate::commands::audit::AuditAdTemplatesVerifyArgs; +use crate::commands::audit::collector::{ + AdTemplateCollectorConfig, AuditCollector, BrowserCollectRequest, build_ad_template_init_script, +}; +use crate::run::RunOutcome; + +/// Verifies configured ad-template slots against live page evidence. +/// +/// # Errors +/// +/// Returns a user-facing string when config loading fails, or when verification +/// surfaces a page-level error or a `--strict` failure (after writing output). +pub(crate) fn run_verify(args: &AuditAdTemplatesVerifyArgs) -> Result { + args.browser.validate()?; + validate_cookie_scope(&args.urls, &args.cookies)?; + let loaded = crate::app_config::load_settings(&args.config)?; + let collector = crate::commands::audit::browser::BrowserCollector::from_opts(&args.browser); + let report = build_report( + &collector, + loaded.settings.creative_opportunities.as_ref(), + loaded.settings.auction.enabled, + &args.urls, + VerifyOptions { + strict: args.strict, + scroll: args.scroll, + allow_cross_origin_redirect: args.allow_cross_origin_redirect, + }, + &args.cookies, + )?; + + let stdout = io::stdout(); + let mut out = stdout.lock(); + if args.json { + write_json(&mut out, &report)?; + } else { + write_human(&mut out, &report)?; + } + + if report.pages.iter().any(|page| page.error.is_some()) { + Err("ad-template verification reported problems".to_string()) + } else if report.ok { + Ok(RunOutcome::Success) + } else { + Ok(RunOutcome::AssertionFailed) + } +} + +fn validate_cookie_scope(urls: &[url::Url], cookies: &[(String, String)]) -> Result<(), String> { + if cookies.is_empty() { + return Ok(()); + } + let origins: std::collections::BTreeSet = urls + .iter() + .map(|url| url.origin().ascii_serialization()) + .collect(); + if origins.len() > 1 { + return Err( + "--cookie may be used only when every verification URL has one origin; split this run so credentials are never copied to another origin" + .to_string(), + ); + } + Ok(()) +} + +/// Run-level verification switches. +#[derive(Debug, Clone, Copy)] +struct VerifyOptions { + /// Exit non-zero when a matched slot is missing or only partially confirmed. + strict: bool, + /// Perform a deterministic scroll pass after the initial settle. + scroll: bool, + /// Accept evidence from a page that redirected to a different origin. + allow_cross_origin_redirect: bool, +} + +/// Builds the verification report for `urls` using `collector`. +/// +/// `creative` is the effective `[creative_opportunities]` config (if any) and +/// `auction_enabled` is the `[auction].enabled` kill switch. +fn build_report( + collector: &dyn AuditCollector, + creative: Option<&CreativeOpportunitiesConfig>, + auction_enabled: bool, + urls: &[url::Url], + options: VerifyOptions, + cookies: &[(String, String)], +) -> Result { + let init_script = build_init_script(creative)?; + + let requests: Vec<_> = urls + .iter() + .map(|url| BrowserCollectRequest { + url: url.clone(), + init_scripts: vec![init_script.clone()], + scroll: options.scroll, + collect_ad_evidence: true, + cookies: cookies.to_vec(), + }) + .collect(); + let collected_pages = collector.collect_pages(&requests); + + let mut pages = Vec::with_capacity(urls.len()); + let mut any_error = false; + let mut any_strict_fail = false; + + for (url, collected) in urls.iter().zip(collected_pages) { + match collected { + Err(message) => { + any_error = true; + pages.push(error_page(url, &message)); + } + // Slots are matched on the *final* path, so a redirect to a + // different origin would let an unrelated site's evidence satisfy + // `--strict` — and the path-equality redirect warning would not even + // fire when the paths happen to agree. Reject unless opted in. + Ok(collected) + if !options.allow_cross_origin_redirect + && origin_changed(url, &collected.final_url) => + { + any_error = true; + pages.push(cross_origin_page(url, &collected.final_url)); + } + Ok(collected) => { + let (page, strict_failed) = build_page(url, &collected, creative, auction_enabled); + if options.strict && strict_failed { + any_strict_fail = true; + } + pages.push(page); + } + } + } + + let ok = !(any_error || (options.strict && any_strict_fail)); + Ok(VerificationReport { + ok, + strict: options.strict, + pages, + warnings: Vec::new(), + }) +} + +/// The URL without its fragment, for comparisons the server can observe. +pub(super) fn without_fragment(url: &url::Url) -> url::Url { + let mut url = url.clone(); + url.set_fragment(None); + url +} + +/// Whether navigation left the requested URL's origin (scheme, host, or port). +/// +/// A same-host default-port `http:80` to `https:443` redirect is *not* a change: +/// the host is the cookie boundary, and that upgrade is the ordinary canonical +/// redirect. Host changes, port changes, and HTTPS downgrades all are. +pub(super) fn origin_changed(requested: &url::Url, final_url: &url::Url) -> bool { + if requested.host_str() != final_url.host_str() { + return true; + } + + match (requested.scheme(), final_url.scheme()) { + ("http", "https") => { + requested.port_or_known_default() != Some(80) + || final_url.port_or_known_default() != Some(443) + } + (requested_scheme @ ("http" | "https"), final_scheme) + if requested_scheme == final_scheme => + { + requested.port_or_known_default() != final_url.port_or_known_default() + } + // Refuse HTTPS downgrades and any unexpected scheme transition. + _ => true, + } +} + +/// Builds the read-only collector init script from the configured slots. +fn build_init_script(creative: Option<&CreativeOpportunitiesConfig>) -> Result { + let config = AdTemplateCollectorConfig { + div_prefixes: creative + .map(|creative| { + creative + .slot + .iter() + .map(|slot| slot.resolved_div_id().to_string()) + .collect() + }) + .unwrap_or_default(), + }; + build_ad_template_init_script(&config) +} + +/// Assembles a successful page result, returning the wire `PageJson` and whether +/// the page would fail `--strict`. +fn build_page( + requested: &url::Url, + collected: &crate::commands::audit::collector::CollectedPage, + creative: Option<&CreativeOpportunitiesConfig>, + auction_enabled: bool, +) -> (PageJson, bool) { + let requested_path = normalize_path_or_url(requested.as_str()).unwrap_or_else(|_| "/".into()); + let final_url = &collected.final_url; + let final_path = normalize_path_or_url(final_url.as_str()).unwrap_or_else(|_| "/".into()); + + let expected = creative + .map(|creative| expected_slots_for_path(&final_path, creative).slots) + .unwrap_or_default(); + let matched = !expected.is_empty(); + + let gate = evaluate_ad_stack_gate(AdStackGateInput { + method_get: true, + navigation: true, + prefetch: false, + bot: false, + matched_slots: matched, + consent_allows_auction: None, + auction_enabled, + }); + + let evidence = collected.ad_evidence.clone().unwrap_or_else(empty_evidence); + let result = compare_page_evidence( + &expected, + &evidence, + RuntimeGateSummary::from_expected(gate.expected), + ); + let strict_failed = result.strict_failed(); + + let mut warnings: Vec = collected.warnings.to_vec(); + warnings.extend(evidence.warnings.iter().map(|warning| Warning { + code: format!("page_{}", warning.code), + message: warning.message.clone(), + })); + // Fragments never reach the server, so a fragment-only difference is not a + // redirect and slots match on the path either way. + if without_fragment(requested) != without_fragment(final_url) { + warnings.push(Warning { + code: "redirected".to_string(), + message: format!("navigation redirected from {requested} to {final_url}"), + }); + } + + let slots = expected + .iter() + .zip(result.slots.iter()) + .map(|(expected_slot, slot_result)| to_slot_json(expected_slot, slot_result)) + .collect(); + let extra_evidence = result.extra_evidence.iter().map(to_extra_json).collect(); + + let page = PageJson { + url: requested.to_string(), + final_url: Some(final_url.to_string()), + requested_path, + path: Some(final_path), + error: None, + runtime_ad_stack_expected: Some(RuntimeAdStackExpectedJson::from( + result.runtime_ad_stack_expected, + )), + gates: Some(to_gates(matched, auction_enabled)), + matched_slot_count: Some(expected.len()), + slots, + extra_evidence, + warnings, + }; + (page, strict_failed) +} + +/// Builds a page-level navigation-failure result (spec §8 `navigation_failed`). +fn error_page(requested: &url::Url, message: &str) -> PageJson { + let requested_path = normalize_path_or_url(requested.as_str()).unwrap_or_else(|_| "/".into()); + PageJson { + url: requested.to_string(), + final_url: None, + requested_path, + path: None, + error: Some(Warning { + code: "navigation_failed".to_string(), + message: message.to_string(), + }), + runtime_ad_stack_expected: None, + gates: None, + matched_slot_count: None, + slots: Vec::new(), + extra_evidence: Vec::new(), + warnings: Vec::new(), + } +} + +/// Builds a page-level cross-origin-redirect refusal. +/// +/// The final URL is reported so the operator can re-run against it explicitly +/// (or pass `--allow-cross-origin-redirect`) once they have confirmed it is +/// their own property. +fn cross_origin_page(requested: &url::Url, final_url: &url::Url) -> PageJson { + let requested_path = normalize_path_or_url(requested.as_str()).unwrap_or_else(|_| "/".into()); + PageJson { + url: requested.to_string(), + final_url: Some(final_url.to_string()), + requested_path, + path: None, + error: Some(Warning { + code: "cross_origin_redirect".to_string(), + message: format!( + "navigation left the requested origin ({} -> {}); \ + evidence from another origin is not accepted as verification. \ + Re-run against the final URL, or pass --allow-cross-origin-redirect", + requested.origin().ascii_serialization(), + final_url.origin().ascii_serialization(), + ), + }), + runtime_ad_stack_expected: None, + gates: None, + matched_slot_count: None, + slots: Vec::new(), + extra_evidence: Vec::new(), + warnings: Vec::new(), + } +} + +fn empty_evidence() -> BrowserAdEvidence { + BrowserAdEvidence { + dom_ids: Vec::new(), + gpt_slots: Vec::new(), + aps_calls: Vec::new(), + page_bids: Vec::new(), + warnings: Vec::new(), + } +} + +fn to_gates(matched: bool, auction_enabled: bool) -> Gates { + let pass_if = |cond: bool| { + if cond { + GateState::Pass + } else { + GateState::Fail + } + }; + Gates { + method_get: GateState::Pass, + navigation: GateState::Pass, + not_prefetch: GateState::Pass, + not_bot: GateState::Pass, + matched_slots: pass_if(matched), + auction_enabled: pass_if(auction_enabled), + // Live consent is not provable from a browser navigation in Phase 1. + consent_allows_auction: GateState::Unknown, + } +} + +fn to_slot_json(expected: &ExpectedSlot, result: &SlotResult) -> SlotJson { + SlotJson { + id: result.id.clone(), + status: to_status(result.status), + phase: result.phase.map(to_phase), + configured: ConfiguredJson { + div_id: expected.div_id.clone(), + gam_unit_path: expected.gam_unit_path.clone(), + formats: expected + .formats + .iter() + .map(|format| FormatJson { + width: format.width, + height: format.height, + media_type: media_type_label(&format.media_type).to_string(), + }) + .collect(), + providers: expected.providers.clone(), + }, + evidence: to_slot_evidence(&result.evidence), + warnings: result.warnings.clone(), + } +} + +fn to_slot_evidence(evidence: &SlotEvidence) -> SlotEvidenceJson { + SlotEvidenceJson { + dom_id: evidence.dom_id.clone(), + gpt: evidence.gpt.as_ref().map(|gpt| GptEvidenceJson { + gam_unit_path: gpt.gam_unit_path.clone(), + div_id: gpt.div_id.clone(), + sizes: gpt.sizes.iter().map(|&(w, h)| [w, h]).collect(), + }), + } +} + +fn to_extra_json(extra: &ExtraEvidence) -> ExtraEvidenceJson { + ExtraEvidenceJson { + kind: extra.kind.clone(), + phase: to_phase(extra.phase), + dom_id: extra.dom_id.clone(), + gam_unit_path: extra.gam_unit_path.clone(), + sizes: extra.sizes.iter().map(|&(w, h)| [w, h]).collect(), + reason: extra.reason.clone(), + } +} + +fn to_status(status: CompareStatus) -> SlotStatus { + match status { + CompareStatus::Confirmed => SlotStatus::Confirmed, + CompareStatus::Partial => SlotStatus::Partial, + CompareStatus::Missing => SlotStatus::Missing, + CompareStatus::Unconfirmable => SlotStatus::Unconfirmable, + } +} + +fn to_phase(phase: EvidencePhase) -> EvidencePhaseJson { + match phase { + EvidencePhase::InitialLoad => EvidencePhaseJson::InitialLoad, + EvidencePhase::Scroll => EvidencePhaseJson::Scroll, + } +} + +fn media_type_label(media_type: &MediaType) -> &'static str { + match media_type { + MediaType::Banner => "banner", + MediaType::Video => "video", + MediaType::Native => "native", + } +} + +fn write_json(out: &mut dyn Write, report: &VerificationReport) -> Result<(), String> { + let json = serde_json::to_string_pretty(report) + .map_err(|error| format!("failed to serialize verification report: {error}"))?; + writeln!(out, "{json}").map_err(write_err) +} + +fn write_human(out: &mut dyn Write, report: &VerificationReport) -> Result<(), String> { + // Warning codes and messages can originate in the audited page (the + // collector forwards `String(error)` from page scripts), so escape control + // characters before writing them to the operator's terminal. + let write_warning = |out: &mut dyn Write, indent: &str, warning: &Warning| { + writeln!( + out, + "{indent}warning [{}]: {}", + escape_terminal_text(&warning.code), + escape_terminal_text(&warning.message) + ) + .map_err(write_err) + }; + + for warning in &report.warnings { + write_warning(out, "", warning)?; + } + for page in &report.pages { + writeln!(out, "url: {}", escape_terminal_text(&page.url)).map_err(write_err)?; + if let Some(error) = &page.error { + writeln!( + out, + " error [{}]: {}", + escape_terminal_text(&error.code), + escape_terminal_text(&error.message) + ) + .map_err(write_err)?; + continue; + } + if let Some(path) = &page.path { + writeln!(out, " path: {}", escape_terminal_text(path)).map_err(write_err)?; + } + if let Some(expected) = page.runtime_ad_stack_expected { + writeln!(out, " runtime ad stack: {}", runtime_label(expected)).map_err(write_err)?; + } + if let Some(count) = page.matched_slot_count { + writeln!(out, " matched slots: {count}").map_err(write_err)?; + } + if let Some(gates) = &page.gates { + writeln!(out, " gates: {}", gates_label(gates)).map_err(write_err)?; + } + for slot in &page.slots { + writeln!( + out, + " slot {}: {}", + escape_terminal_text(&slot.id), + status_label(slot.status) + ) + .map_err(write_err)?; + for warning in &slot.warnings { + write_warning(out, " ", warning)?; + } + } + for extra in &page.extra_evidence { + writeln!( + out, + " extra {} evidence: div={} gam={} sizes={:?} ({})", + escape_terminal_text(&extra.kind), + escape_terminal_text(extra.dom_id.as_deref().unwrap_or("-")), + escape_terminal_text(extra.gam_unit_path.as_deref().unwrap_or("-")), + extra.sizes, + escape_terminal_text(&extra.reason), + ) + .map_err(write_err)?; + } + for warning in &page.warnings { + write_warning(out, " ", warning)?; + } + } + writeln!(out, "ok: {}", report.ok).map_err(write_err) +} + +fn status_label(status: SlotStatus) -> &'static str { + match status { + SlotStatus::Confirmed => "confirmed", + SlotStatus::Partial => "partial", + SlotStatus::Missing => "missing", + SlotStatus::Unconfirmable => "unconfirmable", + } +} + +fn runtime_label(expected: RuntimeAdStackExpectedJson) -> &'static str { + match expected { + RuntimeAdStackExpectedJson::Yes => "yes", + RuntimeAdStackExpectedJson::No => "no", + RuntimeAdStackExpectedJson::Unknown => "unknown", + } +} + +fn gate_label(gate: GateState) -> &'static str { + match gate { + GateState::Pass => "pass", + GateState::Fail => "fail", + GateState::Unknown => "unknown", + } +} + +fn gates_label(gates: &Gates) -> String { + format!( + "method_get={} navigation={} not_prefetch={} not_bot={} matched_slots={} auction_enabled={} consent={}", + gate_label(gates.method_get), + gate_label(gates.navigation), + gate_label(gates.not_prefetch), + gate_label(gates.not_bot), + gate_label(gates.matched_slots), + gate_label(gates.auction_enabled), + gate_label(gates.consent_allows_auction), + ) +} + +#[allow( + clippy::needless_pass_by_value, + reason = "used as a map_err fn that receives io::Error by value" +)] +fn write_err(error: io::Error) -> String { + format!("failed to write command output: {error}") +} + +#[cfg(test)] +mod tests { + use std::cell::Cell; + use std::collections::HashMap; + + use super::*; + use crate::ad_templates::compare::{DomEvidence, GptSlotEvidence}; + use crate::commands::audit::collector::CollectedPage; + + struct FakeCollector { + pages: HashMap>, + batch_calls: Cell, + } + + impl FakeCollector { + fn page(requested: &str, final_url: &str, evidence: BrowserAdEvidence) -> Self { + let mut pages = HashMap::new(); + pages.insert( + requested.to_string(), + Ok(CollectedPage { + final_url: url::Url::parse(final_url).expect("should parse final URL"), + title: String::new(), + script_count: 0, + resource_count: 0, + warnings: Vec::new(), + ad_evidence: Some(evidence), + }), + ); + Self { + pages, + batch_calls: Cell::new(0), + } + } + + fn with_error(mut self, requested: &str, message: &str) -> Self { + self.pages + .insert(requested.to_string(), Err(message.to_string())); + self + } + } + + impl AuditCollector for FakeCollector { + fn collect_page(&self, request: BrowserCollectRequest) -> Result { + self.pages + .get(request.url.as_str()) + .cloned() + .unwrap_or_else(|| Err(format!("no fake page for {}", request.url))) + } + + fn collect_pages( + &self, + requests: &[BrowserCollectRequest], + ) -> Vec> { + self.batch_calls.set(self.batch_calls.get() + 1); + requests + .iter() + .cloned() + .map(|request| self.collect_page(request)) + .collect() + } + } + + fn news_config() -> CreativeOpportunitiesConfig { + let toml = "gam_network_id = \"123\"\n\ + \n\ + [[slot]]\n\ + id = \"atf\"\n\ + gam_unit_path = \"/123/news/atf\"\n\ + div_id = \"ad-atf-\"\n\ + page_patterns = [\"/news/*\"]\n\ + formats = [{ width = 300, height = 250 }]\n"; + let mut config = + toml::from_str::(toml).expect("should deserialize"); + config.compile_slots(); + config + } + + fn confirmed_news_evidence() -> BrowserAdEvidence { + BrowserAdEvidence { + dom_ids: vec![DomEvidence { + dom_id: "ad-atf-0".to_string(), + phase: EvidencePhase::InitialLoad, + }], + gpt_slots: vec![GptSlotEvidence { + gam_unit_path: "/123/news/atf".to_string(), + div_id: "ad-atf-0".to_string(), + sizes: vec![(300, 250)], + phase: EvidencePhase::InitialLoad, + }], + aps_calls: Vec::new(), + page_bids: Vec::new(), + warnings: Vec::new(), + } + } + + fn report_for( + collector: &dyn AuditCollector, + auction_enabled: bool, + strict: bool, + urls: &[&str], + ) -> VerificationReport { + report_for_with_options( + collector, + auction_enabled, + urls, + VerifyOptions { + strict, + scroll: false, + allow_cross_origin_redirect: false, + }, + ) + } + + fn report_for_with_options( + collector: &dyn AuditCollector, + auction_enabled: bool, + urls: &[&str], + options: VerifyOptions, + ) -> VerificationReport { + let config = news_config(); + let parsed: Vec = urls + .iter() + .map(|url| url::Url::parse(url).expect("should parse URL")) + .collect(); + build_report( + collector, + Some(&config), + auction_enabled, + &parsed, + options, + &[], + ) + .expect("typed collector configuration should serialize") + } + + #[test] + fn verify_uses_final_url_for_matching_after_redirect() { + let collector = FakeCollector::page( + "https://www.example.com/", + "https://www.example.com/news/story", + confirmed_news_evidence(), + ); + let report = report_for(&collector, true, false, &["https://www.example.com/"]); + let json = serde_json::to_value(&report).expect("should serialize"); + + assert_eq!(json["pages"][0]["path"], "/news/story"); + assert_eq!(json["pages"][0]["slots"][0]["status"], "confirmed"); + let warnings = json["pages"][0]["warnings"] + .as_array() + .expect("should have warnings array"); + assert!( + warnings.iter().any(|w| w["code"] == "redirected"), + "redirect should emit a `redirected` warning" + ); + } + + #[test] + fn cross_origin_redirect_is_rejected_even_when_paths_match() { + // Same path on a different origin: the redirect warning would not fire, + // so without the origin check this unrelated page's evidence would + // satisfy --strict. + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://impostor.example.net/news/story", + confirmed_news_evidence(), + ); + let report = report_for( + &collector, + true, + true, + &["https://www.example.com/news/story"], + ); + + assert!(!report.ok, "a cross-origin redirect must not report ok"); + let json = serde_json::to_value(&report).expect("should serialize"); + assert_eq!(json["pages"][0]["error"]["code"], "cross_origin_redirect"); + assert!( + json["pages"][0]["slots"] + .as_array() + .expect("should have slots array") + .is_empty(), + "off-origin evidence must not be reported as slot verification" + ); + } + + #[test] + fn cross_origin_redirect_is_accepted_with_explicit_opt_in() { + let collector = FakeCollector::page( + "https://example.com/news/story", + "https://www.example.com/news/story", + confirmed_news_evidence(), + ); + let report = report_for_with_options( + &collector, + true, + &["https://example.com/news/story"], + VerifyOptions { + strict: true, + scroll: false, + allow_cross_origin_redirect: true, + }, + ); + + assert!( + report.ok, + "an opted-in apex -> www redirect should verify normally" + ); + assert_eq!(report.pages[0].matched_slot_count, Some(1)); + } + + #[test] + fn same_origin_path_redirect_still_verifies() { + let collector = FakeCollector::page( + "https://www.example.com/", + "https://www.example.com/news/story", + confirmed_news_evidence(), + ); + let report = report_for(&collector, true, true, &["https://www.example.com/"]); + + assert!( + report.ok, + "a same-origin redirect should still be verified, not refused" + ); + } + + #[test] + fn same_host_http_to_https_upgrade_is_accepted() { + let collector = FakeCollector::page( + "http://www.example.com/news/story", + "https://www.example.com/news/story", + confirmed_news_evidence(), + ); + let report = report_for( + &collector, + true, + true, + &["http://www.example.com/news/story"], + ); + + assert!(report.ok, "a default-port HTTPS upgrade should be accepted"); + } + + #[test] + fn downgrade_and_port_changes_are_rejected() { + for (requested, final_url) in [ + ( + "https://www.example.com/news/story", + "http://www.example.com/news/story", + ), + ( + "https://www.example.com:8443/news/story", + "https://www.example.com:9443/news/story", + ), + ( + "http://www.example.com:8080/news/story", + "https://www.example.com:8443/news/story", + ), + ] { + let collector = FakeCollector::page(requested, final_url, confirmed_news_evidence()); + let report = report_for(&collector, true, true, &[requested]); + assert!(!report.ok, "redirect {requested} -> {final_url} must fail"); + } + } + + #[test] + fn confirmed_page_is_ok_in_default_mode() { + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://www.example.com/news/story", + confirmed_news_evidence(), + ); + let report = report_for( + &collector, + true, + false, + &["https://www.example.com/news/story"], + ); + + assert!(report.ok, "confirmed page should be ok"); + assert_eq!(report.pages[0].matched_slot_count, Some(1)); + } + + #[test] + fn verifier_surfaces_injected_collector_warnings() { + let mut evidence = confirmed_news_evidence(); + evidence.warnings.push(Warning { + code: "fluid_size_ignored".to_string(), + message: "a fluid size could not be compared".to_string(), + }); + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://www.example.com/news/story", + evidence, + ); + + let report = report_for( + &collector, + true, + false, + &["https://www.example.com/news/story"], + ); + + assert!( + report.pages[0] + .warnings + .iter() + .any(|warning| warning.code == "page_fluid_size_ignored"), + "collector warning should be visible in the page report" + ); + } + + #[test] + fn human_output_includes_runtime_and_extra_evidence_diagnostics() { + let mut evidence = confirmed_news_evidence(); + evidence.gpt_slots.push(GptSlotEvidence { + gam_unit_path: "/123/publisher/extra".to_string(), + div_id: "ad-extra-0".to_string(), + sizes: vec![(728, 90)], + phase: EvidencePhase::InitialLoad, + }); + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://www.example.com/news/story", + evidence, + ); + let report = report_for( + &collector, + true, + false, + &["https://www.example.com/news/story"], + ); + let mut output = Vec::new(); + + write_human(&mut output, &report).expect("should write human report"); + let output = String::from_utf8(output).expect("should be UTF-8 output"); + + assert!(output.contains("runtime ad stack: unknown")); + assert!(output.contains("matched slots: 1")); + assert!(output.contains("gates: method_get=pass")); + assert!(output.contains("extra gpt evidence")); + } + + #[test] + fn strict_missing_slot_fails() { + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://www.example.com/news/story", + empty_evidence(), + ); + let report = report_for( + &collector, + true, + true, + &["https://www.example.com/news/story"], + ); + + assert!( + !report.ok, + "strict mode with a missing slot should not be ok" + ); + } + + #[test] + fn auction_disabled_skips_strict_missing_failure() { + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://www.example.com/news/story", + empty_evidence(), + ); + // auction disabled -> runtime expected No -> strict does not fail on missing. + let report = report_for( + &collector, + false, + true, + &["https://www.example.com/news/story"], + ); + + assert!( + report.ok, + "missing slot must not fail strict when auction is disabled" + ); + assert_eq!( + report.pages[0].runtime_ad_stack_expected, + Some(RuntimeAdStackExpectedJson::No) + ); + } + + #[test] + fn multi_url_page_error_sets_ok_false() { + let collector = FakeCollector::page( + "https://www.example.com/news/story", + "https://www.example.com/news/story", + confirmed_news_evidence(), + ) + .with_error("https://www.example.com/broken", "navigation failed"); + let report = report_for( + &collector, + true, + false, + &[ + "https://www.example.com/news/story", + "https://www.example.com/broken", + ], + ); + + assert!(!report.ok, "a page-level error sets ok=false"); + assert_eq!( + collector.batch_calls.get(), + 1, + "all verifier URLs should use one collector batch" + ); + let json = serde_json::to_value(&report).expect("should serialize"); + assert_eq!(json["pages"][1]["error"]["code"], "navigation_failed"); + assert!(json["pages"][1]["final_url"].is_null()); + } + + #[test] + fn supplied_cookies_are_rejected_for_multiple_origins() { + let urls = [ + url::Url::parse("https://a.example/x").expect("should parse first URL"), + url::Url::parse("https://b.example/y").expect("should parse second URL"), + ]; + + let error = validate_cookie_scope(&urls, &[("session".to_string(), "secret".to_string())]) + .expect_err("should not replicate one cookie across origins"); + + assert!( + error.contains("one origin"), + "the refusal should explain cookie scope, got {error}" + ); + } + + #[test] + fn supplied_cookies_are_allowed_for_same_origin_urls() { + let urls = [ + url::Url::parse("https://a.example/x").expect("should parse first URL"), + url::Url::parse("https://a.example/y").expect("should parse second URL"), + ]; + + validate_cookie_scope(&urls, &[("session".to_string(), "secret".to_string())]) + .expect("same-origin URLs share the intended cookie scope"); + } +} diff --git a/crates/trusted-server-cli/src/commands/audit/browser.rs b/crates/trusted-server-cli/src/commands/audit/browser.rs new file mode 100644 index 000000000..b6bbdacd5 --- /dev/null +++ b/crates/trusted-server-cli/src/commands/audit/browser.rs @@ -0,0 +1,1189 @@ +//! Chrome/Chromium-backed implementation of [`AuditCollector`] using +//! `chromiumoxide` (CDP). +//! +//! The collector installs optional pre-navigation init scripts, sets any +//! operator-supplied cookies, navigates, waits for the page to settle, optionally +//! scrolls, and reads back a bounded set of evidence. It never *captures* page +//! HTML, cookies, or storage; supplied cookies are only *sent* to carry an +//! existing session past origin gates. + +use std::time::Duration; + +use chromiumoxide::browser::{Browser, BrowserConfig}; +use chromiumoxide::cdp::browser_protocol::network::CookieParam; +use chromiumoxide::handler::viewport::Viewport; +use chromiumoxide::page::Page; +use futures::StreamExt as _; + +use crate::ad_templates::compare::BrowserAdEvidence; +use crate::ad_templates::output::Warning; +use crate::commands::audit::collector::{ + AuditCollector, BrowserCollectRequest, BrowserOpts, BrowserProfile, CollectedPage, +}; + +/// Candidate Chrome/Chromium executable names searched on `PATH`. +pub(crate) const CHROME_NAMES: &[&str] = &[ + "google-chrome", + "google-chrome-stable", + "chromium", + "chromium-browser", + "chrome", + "Google Chrome", + "Google Chrome for Testing", +]; + +/// Poll interval while waiting for the page network to settle, in milliseconds. +const SETTLE_POLL_MS: u64 = 250; +/// Hard cap on page navigation so a stalled load cannot hang the audit. +const NAVIGATION_TIMEOUT: Duration = Duration::from_secs(30); +/// Bound for each CDP operation after navigation. +const CDP_OPERATION_TIMEOUT: Duration = Duration::from_secs(5); +/// Hard cap per decoded evidence list, so a hostile page cannot inflate CLI +/// memory. +/// +/// Must equal `__ts_max_entries` in `ad_template_collector.js`. The collector +/// already caps each list, but the evidence object lives on `window`, so a page +/// that appends to it directly is bounded here instead. Anything the collector +/// itself dropped is reported as an `evidence_truncated` warning. +const MAX_EVIDENCE_ENTRIES: usize = 128; +/// Hard cap on the UTF-8 JSON payload before CDP transfers it back to Rust. +const MAX_EVIDENCE_PAYLOAD_BYTES: usize = 1024 * 1024; +/// Hard cap on browser teardown so a wedged Chrome cannot hang the audit. +const BROWSER_CLOSE_TIMEOUT: Duration = Duration::from_secs(5); +/// Default quiet window (no new resources) marking the page settled. +const DEFAULT_SETTLE_QUIET_MS: u64 = 750; +/// Default hard cap on settling so slow/ad-heavy pages still terminate. +const DEFAULT_SETTLE_MAX_MS: u64 = 10_000; + +/// Page-settle timing thresholds. +#[derive(Debug, Clone, Copy)] +struct SettleConfig { + /// Quiet window with no new resources marking the page settled. + quiet: Duration, + /// Hard cap on total settle time. + max: Duration, +} + +/// Immutable browser/session settings shared by every URL in one audit batch. +struct BrowserSessionOptions<'a> { + chrome: &'a std::path::Path, + profile_dir: &'a std::path::Path, + settle: SettleConfig, + accept_invalid_certs: bool, + headful: bool, + assume_consent: bool, + proxy: Option<&'a str>, + profile: BrowserProfile, +} + +/// A `chromiumoxide`-backed page collector launching a local Chrome/Chromium. +#[derive(Debug, Clone)] +pub struct BrowserCollector { + /// Explicit Chrome/Chromium executable override (else `$CHROME`, else auto-detect). + chrome: Option, + /// Quiet window marking the page settled. + settle_quiet: Duration, + /// Hard cap on settling. + settle_max: Duration, + /// Navigate to origins with invalid TLS certificates (dangerous opt-in). + accept_invalid_certs: bool, + /// Run visible Chrome rather than new headless Chrome. + headful: bool, + /// Install the standard consent API stub before publisher scripts. + assume_consent: bool, + /// Optional browser proxy endpoint. + proxy: Option, + /// Device viewport/user-agent profile. + profile: BrowserProfile, +} + +impl Default for BrowserCollector { + fn default() -> Self { + Self::new() + } +} + +impl BrowserCollector { + /// Creates a collector with default tuning and auto-detected Chrome. + #[must_use] + pub fn new() -> Self { + Self { + chrome: None, + settle_quiet: Duration::from_millis(DEFAULT_SETTLE_QUIET_MS), + settle_max: Duration::from_millis(DEFAULT_SETTLE_MAX_MS), + accept_invalid_certs: false, + headful: false, + assume_consent: true, + proxy: None, + profile: BrowserProfile::Desktop, + } + } + + /// Creates a collector from operator-supplied browser options. + #[must_use] + pub fn from_opts(opts: &BrowserOpts) -> Self { + Self { + chrome: opts.chrome.clone(), + settle_quiet: Duration::from_millis(opts.settle_quiet_ms), + settle_max: Duration::from_millis(opts.settle_max_ms), + accept_invalid_certs: opts.danger_accept_invalid_certs, + headful: opts.headful, + assume_consent: !opts.no_assume_consent, + proxy: opts.browser_proxy.clone(), + profile: opts.profile, + } + } +} + +/// Pre-document consent behavior shared with the generation crawler. +pub(crate) const CONSENT_STUB_SCRIPT: &str = include_str!("consent_stub.js"); + +/// Shared browser launch inputs used by both audit collectors. +pub(crate) struct BrowserLaunchOptions<'a> { + pub(crate) chrome: &'a std::path::Path, + pub(crate) profile_dir: &'a std::path::Path, + pub(crate) headful: bool, + pub(crate) proxy: Option<&'a str>, + pub(crate) accept_invalid_certs: bool, + pub(crate) viewport: Viewport, + pub(crate) user_agent: Option<&'a str>, +} + +/// Builds the common Chrome configuration for all browser-backed audits. +pub(crate) fn build_browser_config( + options: BrowserLaunchOptions<'_>, +) -> Result { + let mut builder = BrowserConfig::builder() + .chrome_executable(options.chrome) + .user_data_dir(options.profile_dir); + if !options.accept_invalid_certs { + builder = builder.respect_https_errors(); + } + if let Some(proxy) = options.proxy { + let endpoint = if proxy.contains("://") { + proxy.to_string() + } else { + format!("http://{proxy}") + }; + builder = builder + .arg(("proxy-server", endpoint.as_str())) + .arg(("proxy-bypass-list", "<-loopback>")); + } + builder = if options.headful { + builder.with_head() + } else { + builder.new_headless_mode() + }; + builder = builder + .window_size(options.viewport.width, options.viewport.height) + .viewport(options.viewport); + if let Some(user_agent) = options.user_agent { + builder = builder.arg(("user-agent", user_agent)); + } + builder + .build() + .map_err(|error| format!("failed to build browser config: {error}")) +} + +fn browser_profile(profile: BrowserProfile) -> (Viewport, Option<&'static str>) { + match profile { + BrowserProfile::Desktop => ( + Viewport { + width: 1280, + height: 800, + device_scale_factor: Some(1.0), + emulating_mobile: false, + is_landscape: true, + has_touch: false, + }, + None, + ), + BrowserProfile::Mobile => ( + Viewport { + width: 390, + height: 844, + device_scale_factor: Some(3.0), + emulating_mobile: true, + is_landscape: false, + has_touch: true, + }, + Some( + "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) \ + AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1", + ), + ), + } +} + +/// Resolves the Chrome/Chromium executable to launch. +/// +/// Precedence: explicit `--chrome` override, then the `CHROME` environment +/// variable, then auto-detection on `PATH` and standard install locations. +pub(crate) fn resolve_chrome( + override_path: Option<&std::path::Path>, +) -> Result { + if let Some(path) = override_path { + return if path.is_file() { + Ok(path.to_path_buf()) + } else { + Err(format!( + "--chrome path does not point to a file: {}", + path.display() + )) + }; + } + if let Ok(env_path) = std::env::var("CHROME") { + let path = std::path::PathBuf::from(&env_path); + return if path.is_file() { + Ok(path) + } else { + Err(format!("CHROME={env_path} does not point to a file")) + }; + } + find_chrome() +} + +/// Builds a host-only cookie that applies to every path on `url`'s host. +/// +/// Scoped by origin rather than by the full URL: only the origin is load-bearing +/// for a host-only cookie, and a full URL would carry the path, query, and any +/// `user:password@` into CDP and into this function's error message. +pub(crate) fn host_cookie(name: &str, value: &str, url: &url::Url) -> Result { + let origin = url.origin(); + if !origin.is_tuple() { + return Err(format!( + "cannot scope cookie `{name}` because the audited URL has no host" + )); + } + let mut cookie = CookieParam::new(name.to_string(), value.to_string()); + cookie.url = Some(origin.ascii_serialization()); + cookie.path = Some("/".to_string()); + cookie.secure = Some(url.scheme() == "https"); + Ok(cookie) +} + +fn format_cookie_install_error(name: &str, _error: impl std::fmt::Display) -> String { + // Do not forward the CDP error: a browser implementation may include the + // rejected cookie value in its diagnostic. + format!("failed to set cookie `{name}`") +} + +/// Installs host-only, root-scoped cookies before a page has an origin. +pub(crate) async fn set_browser_cookies( + browser: &Browser, + cookies: &[(String, String)], + url: &url::Url, +) -> Result<(), String> { + for (name, value) in cookies { + let cookie = host_cookie(name, value, url)?; + browser + .set_cookies(vec![cookie]) + .await + .map_err(|error| format_cookie_install_error(name, error))?; + } + Ok(()) +} + +/// Auto-detects a Chrome/Chromium executable. +/// +/// Searches `PATH` by common names first, then well-known per-OS install +/// locations (e.g. the macOS `.app` bundle, which is not on `PATH`). +fn find_chrome() -> Result { + if let Some(path) = CHROME_NAMES.iter().find_map(|name| which::which(name).ok()) { + return Ok(path); + } + if let Some(path) = well_known_chrome_paths() + .into_iter() + .find(|path| path.is_file()) + { + return Ok(path); + } + Err(format!( + "could not find Chrome/Chromium on PATH or in standard install locations (looked for: {})", + CHROME_NAMES.join(", ") + )) +} + +/// Well-known absolute Chrome/Chromium install locations for the host OS. +fn well_known_chrome_paths() -> Vec { + let mut paths = Vec::new(); + + #[cfg(target_os = "macos")] + { + const APPS: &[&str] = &[ + "Google Chrome.app/Contents/MacOS/Google Chrome", + "Google Chrome Canary.app/Contents/MacOS/Google Chrome Canary", + "Chromium.app/Contents/MacOS/Chromium", + ]; + for app in APPS { + paths.push(std::path::PathBuf::from(format!("/Applications/{app}"))); + if let Ok(home) = std::env::var("HOME") { + paths.push(std::path::PathBuf::from(format!( + "{home}/Applications/{app}" + ))); + } + } + } + + #[cfg(target_os = "linux")] + { + for path in [ + "/usr/bin/google-chrome", + "/usr/bin/google-chrome-stable", + "/usr/bin/chromium", + "/usr/bin/chromium-browser", + "/snap/bin/chromium", + ] { + paths.push(std::path::PathBuf::from(path)); + } + } + + #[cfg(target_os = "windows")] + { + for path in [ + r"C:\Program Files\Google\Chrome\Application\chrome.exe", + r"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe", + ] { + paths.push(std::path::PathBuf::from(path)); + } + } + + paths +} + +impl AuditCollector for BrowserCollector { + fn collect_page(&self, request: BrowserCollectRequest) -> Result { + self.collect_pages(std::slice::from_ref(&request)) + .into_iter() + .next() + .expect("should return one result for one browser request") + } + + fn collect_pages( + &self, + requests: &[BrowserCollectRequest], + ) -> Vec> { + if requests.is_empty() { + return Vec::new(); + } + // HTTP(S) scheme is enforced by the CLI value parser before we get here. + let chrome = match resolve_chrome(self.chrome.as_deref()) { + Ok(chrome) => chrome, + Err(error) => return vec![Err(error); requests.len()], + }; + let profile = match tempfile::tempdir() { + Ok(profile) => profile, + Err(error) => { + let error = format!("failed to create browser profile dir: {error}"); + return vec![Err(error); requests.len()]; + } + }; + + let runtime = match tokio::runtime::Builder::new_current_thread() + .enable_all() + .build() + { + Ok(runtime) => runtime, + Err(error) => { + let error = format!("failed to build browser runtime: {error}"); + return vec![Err(error); requests.len()]; + } + }; + + let settle = SettleConfig { + quiet: self.settle_quiet, + max: self.settle_max, + }; + + let accept_invalid_certs = self.accept_invalid_certs; + let headful = self.headful; + let assume_consent = self.assume_consent; + let proxy = self.proxy.clone(); + let browser_profile = self.profile; + let request_count = requests.len(); + let requests = requests.to_vec(); + let result = runtime.block_on(async move { + let options = BrowserSessionOptions { + chrome: &chrome, + profile_dir: profile.path(), + settle, + accept_invalid_certs, + headful, + assume_consent, + proxy: proxy.as_deref(), + profile: browser_profile, + }; + collect(requests, &options).await + }); + match result { + Ok(results) => results, + Err(error) => vec![Err(error); request_count], + } + } +} + +/// Drives a single page collection on the current-thread runtime. +async fn collect( + requests: Vec, + options: &BrowserSessionOptions<'_>, +) -> Result>, String> { + // chromiumoxide defaults to ignoring TLS errors. The audit sends + // operator-supplied session cookies and treats what it reads back as + // verification evidence, so a certificate-invalid impersonator could both + // harvest the session and fabricate the evidence. Validate certificates + // unless the operator explicitly opts out. + let (viewport, user_agent) = browser_profile(options.profile); + let config = build_browser_config(BrowserLaunchOptions { + chrome: options.chrome, + profile_dir: options.profile_dir, + headful: options.headful, + proxy: options.proxy, + accept_invalid_certs: options.accept_invalid_certs, + viewport, + user_agent, + })?; + + let (mut browser, mut handler) = Browser::launch(config) + .await + .map_err(|error| format!("failed to launch browser: {error}"))?; + + // Drive the CDP event loop for the duration of the session. + let handler_task = tokio::spawn(async move { while handler.next().await.is_some() {} }); + + let mut results = Vec::with_capacity(requests.len()); + for request in requests { + results.push( + collect_with_browser(&browser, request, options.settle, options.assume_consent).await, + ); + } + + // Best-effort teardown; ignore errors since we already have a result, but + // bound it so a Chrome that ignores `close` cannot hang the command. + let _ = tokio::time::timeout(BROWSER_CLOSE_TIMEOUT, browser.close()).await; + let _ = tokio::time::timeout(BROWSER_CLOSE_TIMEOUT, browser.wait()).await; + handler_task.abort(); + + Ok(results) +} + +async fn collect_with_browser( + browser: &Browser, + request: BrowserCollectRequest, + settle_config: SettleConfig, + assume_consent: bool, +) -> Result { + set_browser_cookies(browser, &request.cookies, &request.url).await?; + + // Open a blank page first so init scripts are installed before the real + // document loads (evaluate-on-new-document applies to subsequent navigations). + let page = browser + .new_page("about:blank") + .await + .map_err(|error| format!("failed to open browser page: {error}"))?; + + let result = collect_open_page(&page, &request, settle_config, assume_consent).await; + let close_result = tokio::time::timeout(BROWSER_CLOSE_TIMEOUT, page.close()).await; + + match (result, close_result) { + (Err(error), _) => Err(error), + (Ok(mut collected), Err(_)) => { + collected.warnings.push(Warning { + code: "page_close_timeout".to_string(), + message: "timed out closing the browser tab after collection".to_string(), + }); + Ok(collected) + } + (Ok(mut collected), Ok(Err(error))) => { + collected.warnings.push(Warning { + code: "page_close_failed".to_string(), + message: format!("failed to close the browser tab after collection: {error}"), + }); + Ok(collected) + } + (Ok(collected), Ok(Ok(_))) => Ok(collected), + } +} + +/// Collects from an open tab. The caller owns tab teardown so every return path, +/// including an error from this function, closes the page before continuing. +async fn collect_open_page( + page: &Page, + request: &BrowserCollectRequest, + settle_config: SettleConfig, + assume_consent: bool, +) -> Result { + let mut warnings = Vec::new(); + + if assume_consent { + page.evaluate_on_new_document(CONSENT_STUB_SCRIPT) + .await + .map_err(|error| format!("failed to install consent init script: {error}"))?; + warnings.push(Warning { + code: "consent_stub_active".to_string(), + message: "audit consent APIs were stubbed; re-run with --no-assume-consent to observe the publisher CMP without substitution".to_string(), + }); + } + page.evaluate_on_new_document("performance.setResourceTimingBufferSize(100000)") + .await + .map_err(|error| format!("failed to increase resource timing buffer: {error}"))?; + + for script in &request.init_scripts { + page.evaluate_on_new_document(script.clone()) + .await + .map_err(|error| format!("failed to install init script: {error}"))?; + } + + tokio::time::timeout(NAVIGATION_TIMEOUT, page.goto(request.url.as_str())) + .await + .map_err(|_| format!("navigation to {} timed out", request.url))? + .map_err(|error| format!("failed to navigate to {}: {error}", request.url))?; + match tokio::time::timeout(NAVIGATION_TIMEOUT, page.wait_for_navigation()).await { + Ok(Ok(_)) => {} + Ok(Err(error)) => warnings.push(Warning { + code: "navigation_wait_failed".to_string(), + message: format!( + "navigation load event could not be read ({error}); continuing with settled page evidence" + ), + }), + Err(_) => warnings.push(Warning { + code: "navigation_wait_timeout".to_string(), + message: format!( + "navigation did not fire its load event within {} seconds; continuing with settled page evidence", + NAVIGATION_TIMEOUT.as_secs() + ), + }), + } + + settle(page, settle_config, &mut warnings).await; + + if request.scroll { + if request.collect_ad_evidence { + // Snapshot evidence before scrolling so entries already present at + // initial load keep phase "load"; the store dedups first-seen, so + // the post-scroll scrape only adds genuinely scroll-phase entries. + if tokio::time::timeout( + CDP_OPERATION_TIMEOUT, + page.evaluate( + "(typeof window.__tsCollectAdTemplateEvidence === 'function' \ + && window.__tsCollectAdTemplateEvidence(), null)", + ), + ) + .await + .is_err() + { + warnings.push(Warning { + code: "ad_evidence_snapshot_timeout".to_string(), + message: "timed out snapshotting ad evidence before scroll".to_string(), + }); + } + } + scroll_page(page, &mut warnings).await; + settle(page, settle_config, &mut warnings).await; + } + + let final_url_text = tokio::time::timeout(CDP_OPERATION_TIMEOUT, page.url()) + .await + .map_err(|_| "timed out reading final page URL".to_string())? + .map_err(|error| format!("failed to read final page URL: {error}"))? + .ok_or_else(|| "browser page URL was empty after navigation".to_string())?; + let final_url = url::Url::parse(&final_url_text).map_err(|error| { + format!("browser returned invalid final URL `{final_url_text}`: {error}") + })?; + let title = match tokio::time::timeout(CDP_OPERATION_TIMEOUT, page.get_title()).await { + Ok(Ok(title)) => title.unwrap_or_default(), + Ok(Err(error)) => { + warnings.push(Warning { + code: "page_title_failed".to_string(), + message: format!("failed to read page title: {error}"), + }); + String::new() + } + Err(_) => { + warnings.push(Warning { + code: "page_title_timeout".to_string(), + message: "timed out reading page title".to_string(), + }); + String::new() + } + }; + let script_count = eval_usize(page, "document.querySelectorAll('script').length") + .await + .unwrap_or_else(|message| { + warnings.push(Warning { + code: "script_count_failed".to_string(), + message, + }); + 0 + }); + let resource_count = resource_count(page).await.unwrap_or_else(|message| { + warnings.push(Warning { + code: "resource_count_failed".to_string(), + message, + }); + 0 + }); + + if resource_count >= 250 { + warnings.push(Warning { + code: "resource_timing_heavy".to_string(), + message: format!("page recorded {resource_count} network resources"), + }); + } + + if let Ok(Ok(frames)) = tokio::time::timeout(CDP_OPERATION_TIMEOUT, page.frames()).await + && frames.len() > 1 + { + warnings.push(Warning { + code: "child_frames_not_inspected".to_string(), + message: format!( + "ad-template evidence inspected only the main frame; {} child frame(s) were present", + frames.len() - 1 + ), + }); + } + + let ad_evidence = if request.collect_ad_evidence { + extract_ad_evidence(page, &mut warnings).await + } else { + None + }; + + Ok(CollectedPage { + final_url, + title, + script_count, + resource_count, + warnings, + ad_evidence, + }) +} + +/// Waits for the page network to go quiet after navigation or scroll. +/// +/// Polls the resource-entry count and returns once it stays unchanged for a +/// quiet window, or when the hard cap elapses — so ad-heavy pages finish loading +/// before evidence is read, without hanging on pages that never go idle. +async fn settle(page: &Page, config: SettleConfig, warnings: &mut Vec) { + let start = std::time::Instant::now(); + let mut last = None; + let mut quiet_since = None; + + loop { + if start.elapsed() >= config.max { + warnings.push(Warning { + code: "settle_timeout".to_string(), + message: "page did not settle before the configured maximum wait".to_string(), + }); + return; + } + + let ready_state = match eval_string(page, "document.readyState").await { + Ok(state) => state, + Err(message) => { + warnings.push(Warning { + code: "settle_read_failed".to_string(), + message, + }); + return; + } + }; + let current = match resource_count(page).await { + Ok(count) => count, + Err(message) => { + warnings.push(Warning { + code: "settle_read_failed".to_string(), + message, + }); + return; + } + }; + let ready = matches!(ready_state.as_str(), "interactive" | "complete"); + if ready && last == Some(current) { + let quiet_start = quiet_since.get_or_insert_with(std::time::Instant::now); + if quiet_start.elapsed() >= config.quiet { + return; + } + } else { + quiet_since = None; + } + last = Some(current); + + let remaining_max = config.max.saturating_sub(start.elapsed()); + let remaining_quiet = quiet_since + .map(|quiet_start| config.quiet.saturating_sub(quiet_start.elapsed())) + .unwrap_or(config.quiet); + let sleep_for = Duration::from_millis(SETTLE_POLL_MS) + .min(remaining_max) + .min(remaining_quiet.max(Duration::from_millis(1))); + tokio::time::sleep(sleep_for).await; + } +} + +/// Reads the number of resource timing entries observed so far. +async fn resource_count(page: &Page) -> Result { + eval_usize(page, "performance.getEntriesByType('resource').length").await +} + +/// Performs a deterministic stepped scroll to trigger lazy ad loading. +async fn scroll_page(page: &Page, warnings: &mut Vec) { + // Mark subsequent observations as scroll-phase for the collector. + eval_discard(page, "window.__tsScrollPhase = true", warnings).await; + for fraction in ["0.33", "0.66", "1"] { + let script = format!( + "window.scrollTo(0, Math.floor(Math.max(document.body.scrollHeight, \ + document.documentElement.scrollHeight) * {fraction}))" + ); + eval_discard(page, script, warnings).await; + tokio::time::sleep(Duration::from_millis(250)).await; + } + eval_discard(page, "window.scrollTo(0, 0)", warnings).await; +} + +async fn eval_discard(page: &Page, expression: impl Into, warnings: &mut Vec) { + match tokio::time::timeout(CDP_OPERATION_TIMEOUT, page.evaluate(expression.into())).await { + Ok(Ok(_)) => {} + Ok(Err(error)) => warnings.push(Warning { + code: "page_evaluation_failed".to_string(), + message: format!("browser page evaluation failed: {error}"), + }), + Err(_) => warnings.push(Warning { + code: "page_evaluation_timeout".to_string(), + message: "browser page evaluation timed out".to_string(), + }), + } +} + +async fn eval_usize(page: &Page, expression: &str) -> Result { + tokio::time::timeout(CDP_OPERATION_TIMEOUT, page.evaluate(expression)) + .await + .map_err(|_| format!("timed out evaluating `{expression}`"))? + .map_err(|error| format!("failed to evaluate `{expression}`: {error}"))? + .into_value::() + .map_err(|error| format!("failed to decode `{expression}`: {error}")) +} + +async fn eval_string(page: &Page, expression: &str) -> Result { + tokio::time::timeout(CDP_OPERATION_TIMEOUT, page.evaluate(expression)) + .await + .map_err(|_| format!("timed out evaluating `{expression}`"))? + .map_err(|error| format!("failed to evaluate `{expression}`: {error}"))? + .into_value::() + .map_err(|error| format!("failed to decode `{expression}`: {error}")) +} + +/// Reads and decodes `window.__tsAdTemplateEvidence`, warning (not failing) on a +/// decode error. +async fn extract_ad_evidence( + page: &Page, + warnings: &mut Vec, +) -> Option { + // Serialize and size-check in the page so a hostile publisher-controlled + // evidence object cannot force an unbounded CDP response and Rust decode. + let evaluation = tokio::time::timeout( + CDP_OPERATION_TIMEOUT, + page.evaluate(format!( + r#"(() => {{ + const evidence = typeof window.__tsCollectAdTemplateEvidence === 'function' + ? window.__tsCollectAdTemplateEvidence() + : (window.__tsAdTemplateEvidence || null) + if (evidence === null) return {{ kind: 'absent' }} + try {{ + const json = JSON.stringify(evidence) + const bytes = new TextEncoder().encode(json).byteLength + if (bytes > {MAX_EVIDENCE_PAYLOAD_BYTES}) return {{ kind: 'too_large' }} + return {{ kind: 'evidence', json }} + }} catch (error) {{ + return {{ + kind: 'serialization_failed', + message: String(error).slice(0, 512), + }} + }} + }})()"# + )), + ) + .await; + + let envelope = match evaluation { + Ok(Ok(result)) => match result.into_value::() { + Ok(envelope) => Some(envelope), + Err(error) => { + warnings.push(Warning { + code: "ad_evidence_decode_failed".to_string(), + message: format!("failed to decode ad-template evidence envelope: {error}"), + }); + return None; + } + }, + Ok(Err(error)) => { + warnings.push(Warning { + code: "ad_evidence_read_failed".to_string(), + message: format!("failed to read ad-template evidence: {error}"), + }); + return None; + } + Err(_) => { + warnings.push(Warning { + code: "ad_evidence_read_timeout".to_string(), + message: "timed out reading ad-template evidence".to_string(), + }); + return None; + } + }; + + match envelope { + Some(envelope) => decode_ad_evidence_envelope(envelope, warnings), + None => { + warnings.push(Warning { + code: "ad_evidence_absent".to_string(), + message: "no ad-template evidence was collected from the page".to_string(), + }); + None + } + } +} + +#[derive(Debug, serde::Deserialize)] +#[serde(tag = "kind", rename_all = "snake_case")] +enum EvidenceEnvelope { + Absent, + TooLarge, + Evidence { json: String }, + SerializationFailed { message: String }, +} + +fn decode_ad_evidence_envelope( + envelope: EvidenceEnvelope, + warnings: &mut Vec, +) -> Option { + match envelope { + EvidenceEnvelope::Absent => { + warnings.push(Warning { + code: "ad_evidence_absent".to_string(), + message: "no ad-template evidence was collected from the page".to_string(), + }); + None + } + EvidenceEnvelope::TooLarge => { + warnings.push(Warning { + code: "ad_evidence_too_large".to_string(), + message: format!( + "ad-template evidence exceeded the {MAX_EVIDENCE_PAYLOAD_BYTES}-byte limit" + ), + }); + None + } + EvidenceEnvelope::SerializationFailed { message } => { + warnings.push(Warning { + code: "ad_evidence_encode_failed".to_string(), + message: format!("failed to serialize ad-template evidence in the page: {message}"), + }); + None + } + EvidenceEnvelope::Evidence { json } => { + match serde_json::from_str::(&json) { + Ok(mut evidence) => { + // Defense in depth: the injected script caps these lists, but the + // page owns that store, so re-cap after decode. + evidence.dom_ids.truncate(MAX_EVIDENCE_ENTRIES); + evidence.gpt_slots.truncate(MAX_EVIDENCE_ENTRIES); + evidence.aps_calls.truncate(MAX_EVIDENCE_ENTRIES); + evidence.warnings.truncate(MAX_EVIDENCE_ENTRIES); + Some(evidence) + } + Err(error) => { + warnings.push(Warning { + code: "ad_evidence_decode_failed".to_string(), + message: format!("failed to decode ad-template evidence: {error}"), + }); + None + } + } + } + } +} + +/// Whether a Chrome/Chromium fixture is available for browser-backed tests. +/// +/// Skips optional local runs, but makes the scripted/CI contract fail loudly. +/// Shared with the generation collector's tests so the contract has one +/// definition. +#[cfg(test)] +pub(crate) fn browser_fixture_available() -> bool { + if resolve_chrome(None).is_ok() { + return true; + } + assert!( + std::env::var_os("TS_AUDIT_BROWSER_TESTS").is_none(), + "TS_AUDIT_BROWSER_TESTS requires Chrome/Chromium; set CHROME to its executable" + ); + false +} + +#[cfg(test)] +mod tests { + use std::io::{Read as _, Write as _}; + use std::net::TcpListener; + use std::sync::mpsc; + + use super::*; + use crate::commands::audit::collector::{ + AdTemplateCollectorConfig, build_ad_template_init_script, + }; + + #[test] + fn well_known_chrome_paths_are_known_for_this_os() { + // macOS/Linux/Windows each have candidate paths; guards the cfg branches. + assert!( + !well_known_chrome_paths().is_empty(), + "supported OSes should list candidate Chrome install paths" + ); + } + + #[test] + fn oversized_ad_evidence_is_an_explicit_warning() { + let mut warnings = Vec::new(); + let evidence = decode_ad_evidence_envelope(EvidenceEnvelope::TooLarge, &mut warnings); + + assert!(evidence.is_none()); + assert_eq!(warnings.len(), 1); + assert_eq!(warnings[0].code, "ad_evidence_too_large"); + } + + #[test] + fn supplied_cookie_is_host_only_and_root_scoped() { + let url = + url::Url::parse("https://publisher.example/news/story").expect("should parse test URL"); + let cookie = host_cookie("clearance", "token", &url).expect("should build cookie"); + + assert!(cookie.domain.is_none(), "host-only cookies omit Domain"); + assert_eq!(cookie.path.as_deref(), Some("/")); + assert_eq!( + cookie.url.as_deref(), + Some("https://publisher.example"), + "the origin scopes a host-only cookie before first navigation" + ); + assert_eq!(cookie.secure, Some(true), "HTTPS cookies must be Secure"); + } + + #[test] + fn cookie_install_error_identifies_name_without_a_value() { + let error = format_cookie_install_error( + "datadome", + "invalid cookie value operator-secret-cookie-value", + ); + + assert_eq!(error, "failed to set cookie `datadome`"); + assert!(!error.contains("operator-secret-cookie-value")); + } + + #[test] + #[ignore = "requires local Chrome/Chromium; run through scripts/test-cli.sh"] + fn supplied_cookie_reaches_first_navigation() { + if !browser_fixture_available() { + return; + } + + let listener = TcpListener::bind("127.0.0.1:0").expect("should bind fixture server"); + let address = listener.local_addr().expect("should read fixture address"); + let (request_tx, request_rx) = mpsc::channel(); + std::thread::spawn(move || { + let (mut stream, _) = listener.accept().expect("should accept browser request"); + stream + .set_read_timeout(Some(Duration::from_secs(10))) + .expect("should set fixture read timeout"); + let mut request = Vec::new(); + while !request.ends_with(b"\r\n\r\n") { + let mut chunk = [0_u8; 1024]; + let chunk_len = stream.read(&mut chunk).expect("should read HTTP request"); + assert!(chunk_len > 0, "request should contain complete headers"); + request.extend_from_slice(&chunk[..chunk_len]); + assert!( + request.len() <= 16 * 1024, + "request headers should be bounded" + ); + } + request_tx + .send(String::from_utf8_lossy(&request).into_owned()) + .expect("should send captured request"); + + let body = b"cookie fixture"; + write!( + stream, + "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", + body.len() + ) + .expect("should write fixture headers"); + stream.write_all(body).expect("should write fixture body"); + }); + + let collector = BrowserCollector { + settle_quiet: Duration::from_millis(100), + settle_max: Duration::from_secs(1), + ..BrowserCollector::new() + }; + collector + .collect_page(BrowserCollectRequest { + url: url::Url::parse(&format!("http://{address}/")) + .expect("should parse fixture URL"), + init_scripts: Vec::new(), + scroll: false, + collect_ad_evidence: false, + cookies: vec![("clearance".to_string(), "token".to_string())], + }) + .expect("cookie should be installed before first navigation"); + + let request = request_rx + .recv_timeout(Duration::from_secs(5)) + .expect("fixture should receive the first navigation"); + assert!( + request.lines().any(|line| { + line.split_once(':').is_some_and(|(name, value)| { + name.eq_ignore_ascii_case("cookie") + && value + .trim() + .split(';') + .any(|cookie| cookie.trim() == "clearance=token") + }) + }), + "first navigation should carry the supplied cookie; request was {request:?}" + ); + } + + /// A self-contained page that stubs just enough of GPT (no network) for the + /// collector to observe a defined slot via the wrapped `defineSlot` and the + /// `getSlots()` scrape. + const GPT_FIXTURE: &str = r#" + + + +
+ + + +"#; + + #[test] + #[ignore = "requires local Chrome/Chromium; run through scripts/test-cli.sh"] + fn collects_gpt_slot_from_local_fixture() { + if !browser_fixture_available() { + // Browser fixture test requires a local Chrome/Chromium; skipping. + return; + } + let mut fixture = tempfile::Builder::new() + .suffix(".html") + .tempfile() + .expect("should create fixture file"); + fixture + .write_all(GPT_FIXTURE.as_bytes()) + .expect("should write fixture"); + let url = url::Url::from_file_path(fixture.path()).expect("should build file url"); + + let script = build_ad_template_init_script(&AdTemplateCollectorConfig { + div_prefixes: vec!["ad-atf-".to_string()], + }) + .expect("should build init script"); + + let collector = BrowserCollector::new(); + let page = collector + .collect_page(BrowserCollectRequest { + url, + init_scripts: vec![script], + scroll: false, + collect_ad_evidence: true, + cookies: Vec::new(), + }) + .expect("should collect fixture page"); + + let evidence = page.ad_evidence.expect("fixture should yield ad evidence"); + assert!( + evidence + .gpt_slots + .iter() + .any(|slot| slot.gam_unit_path == "/123/news/atf"), + "should capture the defined GPT slot" + ); + assert!( + evidence.dom_ids.iter().any(|dom| dom.dom_id == "ad-atf-0"), + "should capture the configured-prefix DOM id" + ); + } + + #[test] + #[ignore = "requires local Chrome/Chromium; run through scripts/test-cli.sh"] + fn scroll_pass_keeps_initial_load_phase_for_load_time_evidence() { + if !browser_fixture_available() { + // Browser fixture test requires a local Chrome/Chromium; skipping. + return; + } + let mut fixture = tempfile::Builder::new() + .suffix(".html") + .tempfile() + .expect("should create fixture file"); + fixture + .write_all(GPT_FIXTURE.as_bytes()) + .expect("should write fixture"); + let url = url::Url::from_file_path(fixture.path()).expect("should build file url"); + + let script = build_ad_template_init_script(&AdTemplateCollectorConfig { + div_prefixes: vec!["ad-atf-".to_string()], + }) + .expect("should build init script"); + + let collector = BrowserCollector::new(); + let page = collector + .collect_page(BrowserCollectRequest { + url, + init_scripts: vec![script], + scroll: true, + collect_ad_evidence: true, + cookies: Vec::new(), + }) + .expect("should collect fixture page"); + + // The slot and DOM id exist at load time, so the pre-scroll snapshot + // must record them as initial-load even though a scroll pass ran. + let evidence = page.ad_evidence.expect("fixture should yield ad evidence"); + assert!( + evidence.dom_ids.iter().any(|dom| dom.dom_id == "ad-atf-0" + && dom.phase == crate::ad_templates::compare::EvidencePhase::InitialLoad), + "load-time DOM id should keep phase initial_load under --scroll" + ); + assert!( + evidence.gpt_slots.iter().any(|slot| { + slot.gam_unit_path == "/123/news/atf" + && slot.phase == crate::ad_templates::compare::EvidencePhase::InitialLoad + }), + "load-time GPT slot should keep phase initial_load under --scroll" + ); + } +} diff --git a/crates/trusted-server-cli/src/commands/audit/browser_collector.rs b/crates/trusted-server-cli/src/commands/audit/browser_collector.rs deleted file mode 100644 index 87a2ccc2c..000000000 --- a/crates/trusted-server-cli/src/commands/audit/browser_collector.rs +++ /dev/null @@ -1,435 +0,0 @@ -use std::path::{Path, PathBuf}; -use std::time::Duration; - -use chromiumoxide::ArcHttpRequest; -use chromiumoxide::browser::{Browser, BrowserConfig}; -use futures::StreamExt as _; -use serde::Deserialize; -use tempfile::TempDir; -use tokio::runtime::Builder; -use tokio::time::{sleep, timeout}; -use url::Url; -use which::which; - -use crate::commands::audit::collector::{ - AuditCollector, CollectedPage, CollectedRequest, CollectedScriptTag, -}; -use crate::error::{CliResult, report_error}; - -const SETTLE_QUIET_PERIOD: Duration = Duration::from_millis(750); -const SETTLE_POLL_INTERVAL: Duration = Duration::from_millis(250); -const SETTLE_MAX_WAIT: Duration = Duration::from_secs(6); -const NAVIGATION_TIMEOUT: Duration = Duration::from_secs(30); -const BROWSER_CLOSE_TIMEOUT: Duration = Duration::from_secs(5); -const RESOURCE_TIMING_BUFFER_WARNING_THRESHOLD: usize = 250; -const RESOURCE_TIMING_BUFFER_WARNING: &str = - "browser resource timing buffer reached its default size; some network assets may be missing"; - -#[derive(Default)] -pub(crate) struct BrowserAuditCollector; - -impl AuditCollector for BrowserAuditCollector { - fn collect_page(&self, target_url: &Url) -> CliResult { - let runtime = Builder::new_current_thread() - .enable_all() - .build() - .map_err(|error| { - report_error(format!( - "failed to build Tokio runtime for browser audit: {error}" - )) - })?; - - runtime.block_on(collect_page_via_browser_async(target_url)) - } -} - -async fn collect_page_via_browser_async(target_url: &Url) -> CliResult { - let chrome_executable = find_browser_executable()?; - let user_data_dir = TempDir::new().map_err(|error| { - report_error(format!( - "failed to create temporary browser profile for audit: {error}" - )) - })?; - let config = BrowserConfig::builder() - .chrome_executable(chrome_executable) - .user_data_dir(user_data_dir.path()) - .new_headless_mode() - .build() - .map_err(|error| { - report_error(format!( - "failed to build Chromium configuration for audit: {error}" - )) - })?; - - let (mut browser, mut handler) = Browser::launch(config).await.map_err(|error| { - report_error(format!( - "failed to launch Chrome/Chromium for audit: {error}" - )) - })?; - - let handler_task = tokio::spawn(async move { - while let Some(event) = handler.next().await { - if event.is_err() { - break; - } - } - }); - - let result = collect_page_from_browser(&mut browser, target_url).await; - - let close_result = timeout(BROWSER_CLOSE_TIMEOUT, browser.close()) - .await - .map_err(|_| report_error("timed out closing browser after audit")) - .and_then(|result| { - result.map_err(|error| { - report_error(format!("failed to close browser after audit: {error}")) - }) - }); - if close_result.is_err() { - handler_task.abort(); - } - let _ = handler_task.await; - - match (result, close_result) { - (Ok(collected), Ok(_)) => Ok(collected), - (Ok(_), Err(error)) | (Err(error), _) => Err(error), - } -} - -async fn collect_page_from_browser( - browser: &mut Browser, - target_url: &Url, -) -> CliResult { - let page = browser.new_page("about:blank").await.map_err(|error| { - report_error(format!("failed to create browser page for audit: {error}")) - })?; - - timeout(NAVIGATION_TIMEOUT, page.goto(target_url.as_str())) - .await - .map_err(|_| report_error(format!("timed out navigating to `{target_url}`")))? - .map_err(|error| report_error(format!("failed to navigate to `{target_url}`: {error}")))?; - - let navigation_response = timeout(NAVIGATION_TIMEOUT, page.wait_for_navigation_response()) - .await - .map_err(|_| { - report_error(format!( - "timed out waiting for main document navigation response from `{target_url}`" - )) - })? - .map_err(|error| { - report_error(format!( - "failed to read main document navigation response: {error}" - )) - })?; - - let mut warnings = Vec::new(); - if let Some(warning) = validate_navigation_response(navigation_response)? { - warnings.push(warning); - } - if !wait_for_page_settle(&page).await? { - warnings.push( - "browser audit timed out while waiting for the page to settle; results may be partial" - .to_string(), - ); - } - - let final_url = page - .url() - .await - .map_err(|error| report_error(format!("failed to read final page URL: {error}")))? - .ok_or_else(|| report_error("browser page URL was empty after navigation"))?; - let page_title = page - .get_title() - .await - .map_err(|error| report_error(format!("failed to read page title: {error}")))?; - let html = page - .content() - .await - .map_err(|error| report_error(format!("failed to read rendered page HTML: {error}")))?; - - let script_tags: Vec = page - .evaluate( - r#"() => Array.from(document.scripts).map((script) => ({ - src: script.src || null, - inline_text: script.src ? null : (script.textContent || null), - }))"#, - ) - .await - .map_err(|error| report_error(format!("failed to read rendered script tags: {error}")))? - .into_value() - .map_err(|error| { - report_error(format!( - "failed to decode rendered script tag data: {error}" - )) - })?; - - let network_requests: Vec = page - .evaluate( - r#"() => performance.getEntriesByType('resource').map((entry) => ({ - url: entry.name, - initiator_type: entry.initiatorType || null, - }))"#, - ) - .await - .map_err(|error| { - report_error(format!( - "failed to read browser performance resource entries: {error}" - )) - })? - .into_value() - .map_err(|error| { - report_error(format!( - "failed to decode browser performance resource data: {error}" - )) - })?; - - if let Some(warning) = resource_timing_buffer_warning(network_requests.len()) { - warnings.push(warning.to_string()); - } - - Ok(CollectedPage { - requested_url: target_url.to_string(), - final_url, - page_title: page_title.filter(|title| !title.trim().is_empty()), - html, - script_tags: script_tags - .into_iter() - .map(|script| CollectedScriptTag { - src: script.src, - inline_text: script.inline_text.filter(|text| !text.trim().is_empty()), - }) - .collect(), - network_requests: network_requests - .into_iter() - .map(|entry| CollectedRequest { - url: entry.url, - resource_type: entry.initiator_type, - }) - .collect(), - warnings, - }) -} - -async fn wait_for_page_settle(page: &chromiumoxide::Page) -> CliResult { - let mut elapsed = Duration::ZERO; - let mut previous_count = None; - let mut stable_for = Duration::ZERO; - - while elapsed < SETTLE_MAX_WAIT { - let ready_state: String = page - .evaluate("document.readyState") - .await - .map_err(|error| report_error(format!("failed to read document ready state: {error}")))? - .into_value() - .map_err(|error| { - report_error(format!("failed to decode document ready state: {error}")) - })?; - let resource_count: usize = page - .evaluate("performance.getEntriesByType('resource').length") - .await - .map_err(|error| report_error(format!("failed to read resource count: {error}")))? - .into_value() - .map_err(|error| report_error(format!("failed to decode resource count: {error}")))?; - - if ready_state == "complete" { - if previous_count == Some(resource_count) { - stable_for += SETTLE_POLL_INTERVAL; - } else { - stable_for = Duration::ZERO; - } - - if stable_for >= SETTLE_QUIET_PERIOD { - return Ok(true); - } - } - - previous_count = Some(resource_count); - sleep(SETTLE_POLL_INTERVAL).await; - elapsed += SETTLE_POLL_INTERVAL; - } - - Ok(false) -} - -fn validate_navigation_response(navigation_response: ArcHttpRequest) -> CliResult> { - let request = navigation_response - .ok_or_else(|| report_error("browser audit did not capture the main document response"))?; - - if let Some(failure_text) = &request.failure_text { - return Err(report_error(format!( - "main document request failed: {failure_text}" - ))); - } - - let response = request.response.as_ref().ok_or_else(|| { - report_error("browser audit did not capture the main document HTTP response") - })?; - - if is_successful_navigation_status(response.status) { - return Ok(None); - } - - Ok(Some(format!( - "audit request returned HTTP {} {} for `{}`; results may be partial", - response.status, response.status_text, response.url - ))) -} - -fn is_successful_navigation_status(status: i64) -> bool { - (200..400).contains(&status) -} - -fn resource_timing_buffer_warning(resource_count: usize) -> Option<&'static str> { - (resource_count >= RESOURCE_TIMING_BUFFER_WARNING_THRESHOLD) - .then_some(RESOURCE_TIMING_BUFFER_WARNING) -} - -fn find_browser_executable() -> CliResult { - for candidate in browser_executable_path_candidates() { - if let Ok(path) = which(candidate) { - return Ok(path); - } - } - - for candidate in browser_executable_fallbacks() { - let candidate_path = Path::new(candidate); - if candidate_path.is_file() { - return Ok(candidate_path.to_path_buf()); - } - } - - Err(report_error( - "Chrome/Chromium was not found on PATH or in the standard local install locations checked by `ts audit`. Install a local Chrome or Chromium binary before running `ts audit`.", - )) -} - -fn browser_executable_path_candidates() -> &'static [&'static str] { - &[ - "google-chrome", - "google-chrome-stable", - "chromium", - "chromium-browser", - "chrome", - "Google Chrome", - "Google Chrome for Testing", - ] -} - -fn browser_executable_fallbacks() -> &'static [&'static str] { - #[cfg(target_os = "macos")] - { - &[ - "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome", - "/Applications/Chromium.app/Contents/MacOS/Chromium", - "/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing", - ] - } - - #[cfg(target_os = "linux")] - { - &[ - "/usr/bin/google-chrome", - "/usr/bin/google-chrome-stable", - "/usr/bin/chromium", - "/usr/bin/chromium-browser", - "/snap/bin/chromium", - ] - } - - #[cfg(not(any(target_os = "macos", target_os = "linux")))] - { - &[] - } -} - -#[derive(Debug, Deserialize)] -struct BrowserScriptTag { - src: Option, - inline_text: Option, -} - -#[derive(Debug, Deserialize)] -struct BrowserPerformanceEntry { - url: String, - initiator_type: Option, -} - -#[cfg(test)] -mod tests { - use std::sync::Arc; - - use chromiumoxide::cdp::browser_protocol::network::{Headers, RequestId, Response}; - use chromiumoxide::cdp::browser_protocol::security::SecurityState; - use chromiumoxide::handler::http::HttpRequest; - - use super::*; - - #[test] - fn successful_navigation_status_allows_redirects_but_rejects_errors() { - assert!(is_successful_navigation_status(200)); - assert!(is_successful_navigation_status(302)); - assert!(is_successful_navigation_status(399)); - assert!(!is_successful_navigation_status(199)); - assert!(!is_successful_navigation_status(400)); - assert!(!is_successful_navigation_status(500)); - } - - #[test] - fn navigation_response_returns_warning_for_http_error_status() { - let warning = - validate_navigation_response(navigation_response_with_status(403, "Forbidden")) - .expect("should validate navigation response") - .expect("should return warning for HTTP error status"); - - assert_eq!( - warning, - "audit request returned HTTP 403 Forbidden for `https://example.com/`; results may be partial", - "should warn and continue when the main document returns an HTTP error" - ); - } - - #[test] - fn resource_timing_buffer_warning_starts_at_threshold() { - assert_eq!( - resource_timing_buffer_warning(RESOURCE_TIMING_BUFFER_WARNING_THRESHOLD - 1), - None, - "should not warn before the resource timing buffer threshold" - ); - assert_eq!( - resource_timing_buffer_warning(RESOURCE_TIMING_BUFFER_WARNING_THRESHOLD), - Some(RESOURCE_TIMING_BUFFER_WARNING), - "should warn when the resource timing buffer reaches the threshold" - ); - } - - #[test] - fn browser_path_candidates_include_common_names() { - let candidates = browser_executable_path_candidates(); - - assert!(candidates.contains(&"google-chrome")); - assert!(candidates.contains(&"chromium")); - assert!(candidates.contains(&"Google Chrome for Testing")); - } - - fn navigation_response_with_status(status: i64, status_text: &str) -> ArcHttpRequest { - let mut request = - HttpRequest::new(RequestId::new("request-1"), None, None, false, Vec::new()); - request.response = Some( - Response::builder() - .url("https://example.com/") - .status(status) - .status_text(status_text) - .headers(Headers::default()) - .mime_type("text/html") - .charset("utf-8") - .connection_reused(false) - .connection_id(1.0) - .encoded_data_length(0.0) - .security_state(SecurityState::Secure) - .build() - .expect("should build navigation response"), - ); - - Some(Arc::new(request)) - } -} diff --git a/crates/trusted-server-cli/src/commands/audit/collector.rs b/crates/trusted-server-cli/src/commands/audit/collector.rs index 314ae54fc..6ab427b2c 100644 --- a/crates/trusted-server-cli/src/commands/audit/collector.rs +++ b/crates/trusted-server-cli/src/commands/audit/collector.rs @@ -1,41 +1,327 @@ -use serde::{Deserialize, Serialize}; -use url::Url; +//! Collector abstraction shared by the generic page audit and the ad-template +//! verifier. +//! +//! Decoupling collection behind [`AuditCollector`] lets the verifier orchestration +//! (Task 9) be tested with an in-memory fake collector, with no Chrome dependency. -use crate::error::CliResult; +use std::path::PathBuf; -pub(crate) trait AuditCollector { - fn collect_page(&self, target_url: &Url) -> CliResult; +use clap::{Args, ValueEnum}; + +use crate::ad_templates::compare::BrowserAdEvidence; + +/// Operator-tunable browser options shared by `ts audit page` and +/// `ts audit ad-templates verify`. +/// +/// These are audit-tool knobs, not publisher runtime config, so they live on the +/// CLI (flags / `CHROME` env) rather than in `trusted-server.toml`. +#[derive(Debug, Clone, Args)] +pub struct BrowserOpts { + /// Path to the Chrome/Chromium executable. Falls back to `$CHROME`, then + /// auto-detection on `PATH` and standard install locations. + #[arg(long)] + pub chrome: Option, + /// Browser device profile used for viewport and user-agent emulation. + #[arg(long = "browser-profile", value_enum, default_value_t = BrowserProfile::Desktop)] + pub profile: BrowserProfile, + /// Run a visible browser instead of Chrome's new headless mode. + #[arg(long)] + pub headful: bool, + /// Do not answer the standard IAB consent APIs for the fresh audit profile. + #[arg(long)] + pub no_assume_consent: bool, + /// Route the browser through this proxy, as `host:port` or a full URL. + #[arg(long, value_name = "HOST:PORT")] + pub browser_proxy: Option, + /// Quiet window in milliseconds (no new network resources) that marks the + /// page settled. + #[arg(long, default_value_t = 750)] + pub settle_quiet_ms: u64, + /// Hard cap in milliseconds on waiting for the page to settle. + #[arg(long, default_value_t = 10_000)] + pub settle_max_ms: u64, + /// Navigate to origins whose TLS certificate does not validate. + /// + /// DANGEROUS: the audit sends any `--cookie` session to the origin and + /// treats what it reads back as verification evidence, so an invalid + /// certificate could mean an impersonator is harvesting the session and + /// fabricating the evidence. Use only against a host you control with a + /// known self-signed certificate. + #[arg(long)] + pub danger_accept_invalid_certs: bool, +} + +/// Browser options for generation, whose device selection is controlled by +/// `--profiles` rather than the verifier's singular `--browser-profile`. +#[derive(Debug, Clone, Args)] +pub struct GenerateBrowserOpts { + /// Path to the Chrome/Chromium executable. Falls back to `$CHROME`, then auto-detection. + #[arg(long)] + pub chrome: Option, + /// Run a visible browser instead of Chrome's new headless mode. + #[arg(long)] + pub headful: bool, + /// Do not answer the standard IAB consent APIs for the fresh audit profile. + #[arg(long)] + pub no_assume_consent: bool, + /// Route the browser through this proxy, as `host:port` or a full URL. + #[arg(long, value_name = "HOST:PORT")] + pub browser_proxy: Option, + /// Quiet window in milliseconds that marks the page settled. + #[arg(long, default_value_t = 750)] + pub settle_quiet_ms: u64, + /// Hard cap in milliseconds on waiting for the page to settle. + #[arg(long, default_value_t = 10_000)] + pub settle_max_ms: u64, + /// Navigate to origins whose TLS certificate does not validate. + /// + /// DANGEROUS: the audit sends any `--cookie` session to the origin and + /// treats what it reads back as the evidence it writes config from, so an + /// invalid certificate could mean an impersonator is harvesting the session + /// and fabricating the evidence. Use only against a host you control with a + /// known self-signed certificate. + #[arg(long)] + pub danger_accept_invalid_certs: bool, +} + +/// Defaults mirroring the `#[arg(default_value_t)]` values above, so a path that +/// builds these options in code (the legacy `ts audit ` form) behaves like +/// the parsed command. +impl Default for GenerateBrowserOpts { + fn default() -> Self { + Self { + chrome: None, + headful: false, + no_assume_consent: false, + browser_proxy: None, + settle_quiet_ms: 750, + settle_max_ms: 10_000, + danger_accept_invalid_certs: false, + } + } +} + +impl GenerateBrowserOpts { + /// Validates relationships between independently parsed browser flags. + pub fn validate(&self) -> Result<(), String> { + validate_settle_window(self.settle_quiet_ms, self.settle_max_ms) + } +} + +/// Browser device profile shared by page audits and ad-template verification. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, ValueEnum)] +pub enum BrowserProfile { + /// Desktop Chrome at 1280×800. + #[default] + Desktop, + /// Mobile-sized viewport with a mobile user agent. + Mobile, +} + +impl BrowserOpts { + /// Validates relationships between independently parsed browser flags. + pub fn validate(&self) -> Result<(), String> { + validate_settle_window(self.settle_quiet_ms, self.settle_max_ms) + } +} + +fn validate_settle_window(quiet_ms: u64, max_ms: u64) -> Result<(), String> { + if quiet_ms > max_ms { + return Err(format!( + "--settle-quiet-ms ({quiet_ms}) cannot exceed --settle-max-ms ({max_ms})" + )); + } + Ok(()) +} + +/// A request to collect a single page. +#[derive(Debug, Clone)] +pub struct BrowserCollectRequest { + /// The URL to navigate to. + pub url: url::Url, + /// Pre-navigation init scripts (evaluate-on-new-document). Empty for a plain + /// page audit; the ad-template verifier supplies the read-only collector here. + pub init_scripts: Vec, + /// Whether to perform the deterministic scroll pass after settle. + pub scroll: bool, + /// Whether to extract `window.__tsAdTemplateEvidence` after settle/scroll. + pub collect_ad_evidence: bool, + /// Operator-supplied `(name, value)` cookies set on the browser context + /// before navigation, scoped to the request URL. Used to carry an existing + /// authenticated session (e.g. a valid bot-protection clearance cookie) so + /// the origin serves the real page instead of a challenge. The collector + /// only sends these; it never reads cookies back. + pub cookies: Vec<(String, String)>, +} + +/// The result of collecting a single page. +#[derive(Debug, Clone)] +pub struct CollectedPage { + /// The final URL after redirects. + pub final_url: url::Url, + /// The page title. + pub title: String, + /// Number of `"#; let state = std::sync::Arc::new(std::sync::Mutex::new(Some(bids_script.to_string()))); let config = HtmlProcessorConfig { + csp_nonce_observed: None, + body_close: BodyCloseInjection::InlineBids, origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), @@ -1786,6 +1912,8 @@ mod tests { let bids_script = r#""#; let state = std::sync::Arc::new(std::sync::Mutex::new(Some(bids_script.to_string()))); let config = HtmlProcessorConfig { + csp_nonce_observed: None, + body_close: BodyCloseInjection::InlineBids, origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), @@ -1824,6 +1952,8 @@ mod tests { let request_host = "proxy.test-publisher.example.com"; let config = HtmlProcessorConfig { + csp_nonce_observed: None, + body_close: BodyCloseInjection::None, origin_host: "origin.test-publisher.example.com".to_string(), request_host: request_host.to_string(), request_scheme: "https".to_string(), @@ -1876,6 +2006,8 @@ mod tests { // (state is None) — e.g. auction timed out with zero bids. Fallback to {}. let state = std::sync::Arc::new(std::sync::Mutex::new(None)); let config = HtmlProcessorConfig { + csp_nonce_observed: None, + body_close: BodyCloseInjection::InlineBids, origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), @@ -1906,6 +2038,8 @@ mod tests { // unmodified (spec §8: "Existing client-side Prebid/GPT flow runs unmodified"). let state = std::sync::Arc::new(std::sync::Mutex::new(None)); let config = HtmlProcessorConfig { + csp_nonce_observed: None, + body_close: BodyCloseInjection::None, origin_host: "origin.example.com".to_string(), request_host: "example.com".to_string(), request_scheme: "https".to_string(), @@ -1927,6 +2061,176 @@ mod tests { ); } + fn marker_mode_config(marker: &str, observer: Option>) -> HtmlProcessorConfig { + HtmlProcessorConfig { + csp_nonce_observed: observer, + body_close: BodyCloseInjection::Marker(marker.to_string()), + origin_host: "origin.example.com".to_string(), + request_host: "example.com".to_string(), + request_scheme: "https".to_string(), + integrations: IntegrationRegistry::empty_for_tests(), + ad_slots_script: None, + ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), + max_buffered_body_bytes: 16 * 1024 * 1024, + gpt_diagnostics: None, + suppress_datadome_client_side_tag: false, + } + } + + fn render_marker_mode(marker: &str, source: &str) -> String { + let mut processor = create_html_processor(marker_mode_config(marker, None)); + let output = processor + .process_chunk(source.as_bytes(), true) + .expect("should process the document"); + String::from_utf8(output).expect("output should be utf8") + } + + #[test] + fn marker_mode_ignores_a_body_close_written_in_script_data() { + // A reverse byte search for `` picks this string literal, because the + // document has no structural close at all. Splicing a `` inside the publisher's script and corrupts the document — + // and, once stored, every warm reader of it. Only the parser can tell the + // difference, so the parser places the marker. + const MARKER: &str = ""; + let source = + r#"

a

"#; + + let html = render_marker_mode(MARKER, source); + + assert!( + html.contains(r#"const marker = "";"#), + "should leave the publisher's script data byte for byte: {html}" + ); + assert_eq!( + html.matches(MARKER).count(), + 1, + "should emit exactly one transform-owned marker: {html}" + ); + assert!( + html.ends_with(MARKER), + "a document with no structural body close takes the terminal marker: {html}" + ); + } + + #[test] + fn marker_mode_prefers_the_structural_body_close_over_trailing_comment_data() { + // A reverse byte search takes the *last* `` sequence, which here lives in + // trailing comment data, so the marker landed after the document's real end. + const MARKER: &str = ""; + let source = "

a

"; + + let html = render_marker_mode(MARKER, source); + + assert!( + html.contains(&format!("

a

{MARKER}")), + "should place the marker at the structural body close: {html}" + ); + assert!( + html.contains(""), + "should leave the publisher's trailing comment untouched: {html}" + ); + assert_eq!( + html.matches(MARKER).count(), + 1, + "should emit exactly one transform-owned marker: {html}" + ); + } + + #[test] + fn a_nonce_bearing_meta_policy_is_observed() { + let observed = Arc::new(AtomicBool::new(false)); + let mut processor = + create_html_processor(marker_mode_config("", Some(Arc::clone(&observed)))); + + processor + .process_chunk( + br#"a"#, + true, + ) + .expect("should process the document"); + + assert!( + observed.load(Ordering::SeqCst), + "a policy delivered in markup is invisible to the response-header gate" + ); + } + + #[test] + fn a_nonce_attribute_is_observed() { + let observed = Arc::new(AtomicBool::new(false)); + let mut processor = + create_html_processor(marker_mode_config("", Some(Arc::clone(&observed)))); + + processor + .process_chunk( + b"a", + true, + ) + .expect("should process the document"); + + assert!( + observed.load(Ordering::SeqCst), + "a document written for a per-response nonce must not be shared" + ); + } + + #[test] + fn the_word_nonce_in_script_text_is_not_observed() { + // The reason this is structural rather than a byte scan over the output. + let observed = Arc::new(AtomicBool::new(false)); + let mut processor = + create_html_processor(marker_mode_config("", Some(Arc::clone(&observed)))); + + processor + .process_chunk( + br#"a"#, + true, + ) + .expect("should process the document"); + + assert!( + !observed.load(Ordering::SeqCst), + "ordinary script text must not cost a cacheable page its shared template" + ); + } + + #[test] + fn bodyless_marker_mode_emits_an_owned_terminal_seam_even_after_source_bytes() { + const MARKER: &str = ""; + let config = HtmlProcessorConfig { + csp_nonce_observed: None, + body_close: BodyCloseInjection::Marker(MARKER.to_string()), + origin_host: "origin.example.com".to_string(), + request_host: "example.com".to_string(), + request_scheme: "https".to_string(), + integrations: IntegrationRegistry::empty_for_tests(), + ad_slots_script: None, + ad_bids_state: std::sync::Arc::new(std::sync::Mutex::new(None)), + max_buffered_body_bytes: 16 * 1024 * 1024, + gpt_diagnostics: None, + suppress_datadome_client_side_tag: false, + }; + let source = + format!(r#""#); + + let mut processor = create_html_processor(config); + let output = processor + .process_chunk(source.as_bytes(), true) + .expect("should process bodyless HTML"); + let html = std::str::from_utf8(&output).expect("should be utf8"); + + assert_eq!( + html.matches(MARKER).count(), + 2, + "one source occurrence plus the transform-owned terminal seam must survive processing; repeated markers are rejected before template caching" + ); + assert!( + html.ends_with(MARKER), + "the transform-owned template-cache fallback must be unambiguously terminal" + ); + } + #[test] fn response_size_does_not_grow_disproportionately() { // Processing must not expand HTML by more than 1.1× (accounts for the diff --git a/crates/trusted-server-core/src/http_util.rs b/crates/trusted-server-core/src/http_util.rs index 5ad7011fe..1b0693e56 100644 --- a/crates/trusted-server-core/src/http_util.rs +++ b/crates/trusted-server-core/src/http_util.rs @@ -4,8 +4,10 @@ use edgezero_core::body::Body as EdgeBody; use error_stack::Report; use http::{Request, Response, StatusCode, header}; use sha2::{Digest as _, Sha256}; +use std::time::Duration; use subtle::ConstantTimeEq as _; +use crate::cache_policy::{CachePolicy, EdgeCacheHeader}; use crate::constants::INTERNAL_HEADERS; use crate::error::TrustedServerError; use crate::platform::ClientInfo; @@ -274,43 +276,41 @@ pub fn serve_static_with_etag( body: &str, req: &Request, content_type: &str, + edge_header: EdgeCacheHeader, ) -> Response { - // Compute ETag for conditional caching let hash = Sha256::digest(body.as_bytes()); let etag = format!("\"sha256-{}\"", hex::encode(hash)); + let short_policy = CachePolicy::public_short_with_stale( + Duration::from_secs(300), + Duration::from_secs(60), + Duration::from_secs(86_400), + ); - // If-None-Match handling for 304 responses if let Some(if_none_match) = req .headers() .get(header::IF_NONE_MATCH) .and_then(|h| h.to_str().ok()) && if_none_match == etag { - return Response::builder() - .status(StatusCode::NOT_MODIFIED) - .header(header::ETAG, &etag) - .header( - header::CACHE_CONTROL, - "public, max-age=300, s-maxage=300, stale-while-revalidate=60, stale-if-error=86400", - ) - .header("surrogate-control", "max-age=300") - .header(header::VARY, "Accept-Encoding") - .body(EdgeBody::empty()) - .expect("should build 304 static response"); - } - - Response::builder() + let mut response = Response::builder() + .status(StatusCode::NOT_MODIFIED) + .header(header::ETAG, &etag) + .header(header::VARY, "Accept-Encoding") + .body(EdgeBody::empty()) + .expect("should build 304 static response"); + short_policy.apply_to_headers(response.headers_mut(), edge_header); + return response; + } + + let mut response = Response::builder() .status(StatusCode::OK) .header(header::CONTENT_TYPE, content_type) - .header( - header::CACHE_CONTROL, - "public, max-age=300, s-maxage=300, stale-while-revalidate=60, stale-if-error=86400", - ) - .header("surrogate-control", "max-age=300") .header(header::ETAG, &etag) .header(header::VARY, "Accept-Encoding") .body(EdgeBody::from(body.as_bytes())) - .expect("should build static response") + .expect("should build static response"); + short_policy.apply_to_headers(response.headers_mut(), edge_header); + response } /// Encrypts a URL using XChaCha20-Poly1305 with a key derived from the publisher `proxy_secret`. diff --git a/crates/trusted-server-core/src/integrations/aps.rs b/crates/trusted-server-core/src/integrations/aps.rs index 4fed9278d..aba550097 100644 --- a/crates/trusted-server-core/src/integrations/aps.rs +++ b/crates/trusted-server-core/src/integrations/aps.rs @@ -23,8 +23,8 @@ use crate::auction::types::{ }; use crate::error::TrustedServerError; use crate::integrations::{ - IntegrationEndpoint, IntegrationProxy, IntegrationRegistration, - UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_response_bounded, + IntegrationEndpoint, IntegrationHeadInjector, IntegrationHtmlContext, IntegrationProxy, + IntegrationRegistration, UPSTREAM_RTB_MAX_RESPONSE_BYTES, collect_response_bounded, ensure_integration_backend_with_timeout, predict_integration_backend_name, }; use crate::openrtb::{ @@ -50,6 +50,7 @@ const APS_RENDERER_CSP: &str = "default-src 'none'; sandbox allow-forms allow-po const APS_RENDERER_DOCUMENT: &str = r#" + "#; +/// Rendering owner for selected APS bids. +#[derive(Debug, Clone, Copy, Default, Deserialize, Serialize, PartialEq, Eq)] +#[serde(rename_all = "snake_case")] +pub enum ApsRenderingMode { + /// Render through Trusted Server's opaque static renderer route. + #[default] + TrustedServer, + /// Render through the injected APS runner in a publisher-origin friendly frame. + PublisherNative, +} + /// Configuration for the APS `OpenRTB` integration. #[derive(Debug, Clone, Deserialize, Serialize, Validate)] #[validate(schema(function = "validate_inventory_identity_override"))] @@ -141,6 +153,9 @@ pub struct ApsConfig { /// Whether APS script creatives are eligible before winner selection. #[serde(default)] pub allow_script_creatives: bool, + /// Rendering owner for selected APS bids. + #[serde(default)] + pub rendering_mode: ApsRenderingMode, /// APS-authorized inventory domain used instead of the deployment hostname. #[serde(default, skip_serializing_if = "Option::is_none")] #[validate(custom(function = "validate_inventory_domain"))] @@ -314,6 +329,7 @@ impl Default for ApsConfig { timeout_ms: default_timeout_ms(), debug: false, allow_script_creatives: false, + rendering_mode: ApsRenderingMode::TrustedServer, inventory_domain: None, inventory_page_origin: None, } @@ -1184,7 +1200,9 @@ impl AuctionProvider for ApsAuctionProvider { } #[derive(Debug)] -struct ApsRendererIntegration; +struct ApsRendererIntegration { + rendering_mode: ApsRenderingMode, +} #[async_trait(?Send)] impl IntegrationProxy for ApsRendererIntegration { @@ -1193,7 +1211,10 @@ impl IntegrationProxy for ApsRendererIntegration { } fn routes(&self) -> Vec { - vec![IntegrationEndpoint::get(APS_RENDERER_ROUTE)] + (self.rendering_mode == ApsRenderingMode::TrustedServer) + .then(|| IntegrationEndpoint::get(APS_RENDERER_ROUTE)) + .into_iter() + .collect() } async fn handle( @@ -1225,6 +1246,22 @@ impl IntegrationProxy for ApsRendererIntegration { } } +impl IntegrationHeadInjector for ApsRendererIntegration { + fn integration_id(&self) -> &'static str { + APS_INTEGRATION_ID + } + + fn head_inserts(&self, _ctx: &IntegrationHtmlContext<'_>) -> Vec { + (self.rendering_mode == ApsRenderingMode::PublisherNative) + .then(|| { + "" + .to_string() + }) + .into_iter() + .collect() + } +} + /// Register the APS static renderer endpoint when APS is enabled. /// /// # Errors @@ -1233,16 +1270,21 @@ impl IntegrationProxy for ApsRendererIntegration { pub fn register( settings: &Settings, ) -> Result, Report> { - let Some(_config) = settings.integration_config::(APS_INTEGRATION_ID)? else { + let Some(config) = settings.integration_config::(APS_INTEGRATION_ID)? else { return Ok(None); }; - let integration = Arc::new(ApsRendererIntegration); - Ok(Some( - IntegrationRegistration::builder(APS_INTEGRATION_ID) - .with_proxy(integration) - .without_js() - .build(), - )) + let integration = Arc::new(ApsRendererIntegration { + rendering_mode: config.rendering_mode, + }); + let registration = IntegrationRegistration::builder(APS_INTEGRATION_ID) + .without_js() + .with_head_injector(integration.clone()); + let registration = if config.rendering_mode == ApsRenderingMode::TrustedServer { + registration.with_proxy(integration) + } else { + registration + }; + Ok(Some(registration.build())) } /// Register the APS auction provider when enabled. @@ -1273,6 +1315,7 @@ mod tests { UserInfo, }; use crate::consent::ConsentContext; + use crate::integrations::IntegrationDocumentState; use crate::openrtb::{Eid, Uid}; use crate::platform::GeoInfo; use crate::platform::test_support::{ @@ -1289,6 +1332,7 @@ mod tests { timeout_ms: 800, debug: false, allow_script_creatives: false, + rendering_mode: ApsRenderingMode::TrustedServer, inventory_domain: None, inventory_page_origin: None, } @@ -1405,6 +1449,7 @@ mod tests { assert!(!canonical.debug); assert!(debug.debug); assert!(!canonical.allow_script_creatives); + assert_eq!(canonical.rendering_mode, ApsRenderingMode::TrustedServer); assert!(canonical.endpoint.ends_with("/e/pb/bid")); } @@ -1466,6 +1511,14 @@ mod tests { })) .is_err() ); + assert!( + serde_json::from_value::(json!({ + "account_id": "example-account", + "rendering_mode": "unsupported" + })) + .is_err(), + "should reject an unknown APS rendering mode" + ); for endpoint in [ "http://aps.example/e/pb/bid", "https://", @@ -2319,7 +2372,9 @@ mod tests { #[test] fn registers_and_serves_only_static_renderer_route() { - let integration = ApsRendererIntegration; + let integration = ApsRendererIntegration { + rendering_mode: ApsRenderingMode::TrustedServer, + }; let routes = integration.routes(); assert_eq!(routes.len(), 1, "should register one route"); assert_eq!(routes[0].method, Method::GET); @@ -2374,9 +2429,55 @@ mod tests { assert_eq!(registration.integration_id, APS_INTEGRATION_ID); assert_eq!(registration.proxies.len(), 1); + assert_eq!(registration.head_injectors.len(), 1); assert!(registration.js_disabled); } + #[test] + fn publisher_native_config_registers_runner_mode_without_renderer_route() { + let mut settings = create_test_settings(); + settings + .integrations + .insert_config( + APS_INTEGRATION_ID, + &json!({ + "enabled": true, + "account_id": "example-account", + "rendering_mode": "publisher_native" + }), + ) + .expect("should insert native APS config"); + + let registration = register(&settings) + .expect("should register APS") + .expect("should return enabled registration"); + assert!( + registration.proxies.is_empty(), + "should not register the static renderer" + ); + assert_eq!(registration.head_injectors.len(), 1); + + let integration = ApsRendererIntegration { + rendering_mode: ApsRenderingMode::PublisherNative, + }; + assert!( + integration.routes().is_empty(), + "should expose no renderer route" + ); + let document_state = IntegrationDocumentState::default(); + let context = IntegrationHtmlContext { + request_host: "publisher.example", + request_scheme: "https", + origin_host: "origin.example", + document_state: &document_state, + }; + assert_eq!( + integration.head_inserts(&context), + vec![""], + "should inject only the native-mode marker" + ); + } + #[test] fn config_without_enabled_does_not_register_provider_or_renderer() { let mut settings = create_test_settings(); @@ -2429,6 +2530,10 @@ mod tests { assert!(APS_RENDERER_DOCUMENT.contains("message.nonce!==expected")); assert!(APS_RENDERER_DOCUMENT.contains("prebid/creative/render")); assert!(APS_RENDERER_DOCUMENT.contains("window._aps instanceof Map")); + assert!( + APS_RENDERER_DOCUMENT + .contains("html,body{margin:0;padding:0}body>iframe{display:block}") + ); assert!(APS_RENDERER_DOCUMENT.contains("store:new Map([['listeners',new Map()]])")); assert!(APS_RENDERER_DOCUMENT.contains("account.queue.push(new CustomEvent")); assert!( diff --git a/crates/trusted-server-core/src/integrations/datadome.rs b/crates/trusted-server-core/src/integrations/datadome.rs index d95ee35ee..d47999a68 100644 --- a/crates/trusted-server-core/src/integrations/datadome.rs +++ b/crates/trusted-server-core/src/integrations/datadome.rs @@ -68,7 +68,6 @@ use serde_json::Value as JsonValue; use url::Url; use validator::Validate; -use crate::constants::ENV_FASTLY_IS_STAGING; use crate::error::TrustedServerError; use crate::integrations::{ AttributeRewriteAction, INTEGRATION_MAX_BODY_BYTES, IntegrationAttributeContext, @@ -90,7 +89,7 @@ pub use protection_scope::{ use protection_scope::ProtectionScope; pub(crate) const DATADOME_INTEGRATION_ID: &str = "datadome"; -/// Fixed request header used by the staging-only protection test bypass. +/// Fixed request header used by the configuration-gated protection test bypass. pub(crate) const HEADER_DATADOME_TEST_BYPASS: &str = "x-ts-datadome-bypass"; /// Request marker indicating that Trusted Server should omit its automatic @@ -122,7 +121,7 @@ static DATADOME_URL_PATTERN: LazyLock = LazyLock::new(|| { /// Temporary static-header bypass for server-side `DataDome` protection. /// -/// This is intended only for an access-controlled staging environment. A +/// This is intended only for an access-controlled test environment. A /// matching `x-ts-datadome-bypass` header bypasses the server-side Protection /// API and is removed before the publisher origin receives the request. The /// credential itself is loaded from the Secret Store at runtime. @@ -224,7 +223,7 @@ pub struct DataDomeConfig { )] pub protection_exclusion_rules: Vec, - /// Temporary static-header bypass for access-controlled staging tests. + /// Temporary static-header bypass for access-controlled tests. #[serde(default)] pub protection_test_bypass: Option, @@ -478,10 +477,6 @@ impl DataDomeIntegration { } fn active_protection_test_bypass(&self) -> Option<&ProtectionTestBypassConfig> { - if std::env::var(ENV_FASTLY_IS_STAGING).as_deref() != Ok("1") { - return None; - } - self.config .protection_test_bypass .as_ref() @@ -945,17 +940,7 @@ fn build( }; let integration = DataDomeIntegration::try_new(config)?; - let protection_test_bypass_configured = integration - .config - .protection_test_bypass - .as_ref() - .is_some_and(|bypass| bypass.enabled); let protection_test_bypass_active = integration.active_protection_test_bypass().is_some(); - if protection_test_bypass_configured && !protection_test_bypass_active { - log::warn!( - "[datadome] DataDome test bypass is configured but inactive because FASTLY_IS_STAGING is not 1" - ); - } log::info!( "[datadome] Registering integration (sdk_origin: {}, rewrite_sdk: {}, enable_protection: {}, protection_test_bypass: {})", integration.config.sdk_origin, @@ -963,8 +948,6 @@ fn build( integration.config.enable_protection, if protection_test_bypass_active { "active" - } else if protection_test_bypass_configured { - "configured-inactive" } else { "disabled" }, diff --git a/crates/trusted-server-core/src/integrations/datadome/protection.rs b/crates/trusted-server-core/src/integrations/datadome/protection.rs index 75de88afb..4d1988b6a 100644 --- a/crates/trusted-server-core/src/integrations/datadome/protection.rs +++ b/crates/trusted-server-core/src/integrations/datadome/protection.rs @@ -836,7 +836,7 @@ fn truncate_utf8(value: &str, limit: i32) -> String { mod tests { use std::collections::HashMap; use std::net::{IpAddr, Ipv4Addr}; - use std::sync::{Arc, Mutex}; + use std::sync::Arc; use crate::integrations::datadome::{ DataDomeConfig, ProtectionExclusionRuleConfig, ProtectionMatcherConfig, @@ -852,8 +852,6 @@ mod tests { use super::*; - static FASTLY_IS_STAGING_ENV_LOCK: Mutex<()> = Mutex::new(()); - fn protection_integration() -> Arc { let config = DataDomeConfig { enabled: true, @@ -871,24 +869,19 @@ mod tests { .expect("should build filter request") } - fn filter_with_staging( + fn filter_request( integration: &DataDomeIntegration, settings: &Settings, services: &RuntimeServices, request: &mut Request, ) -> RequestFilterDecision { - let _guard = FASTLY_IS_STAGING_ENV_LOCK - .lock() - .expect("should lock staging environment test guard"); - temp_env::with_var(crate::constants::ENV_FASTLY_IS_STAGING, Some("1"), || { - futures::executor::block_on(integration.filter_protection_request(RequestFilterInput { - settings, - services, - request, - geo_info: None, - is_integration_route: false, - })) - }) + futures::executor::block_on(integration.filter_protection_request(RequestFilterInput { + settings, + services, + request, + geo_info: None, + is_integration_route: false, + })) } fn filter_marks_request( @@ -973,7 +966,7 @@ mod tests { edgezero_core::http::HeaderValue::from_static("temporary-test-credential-32-bytes!"), ); - let decision = filter_with_staging(&integration, &settings, &services, &mut request); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!( matches!(decision, RequestFilterDecision::Continue(_)), @@ -1036,7 +1029,7 @@ mod tests { edgezero_core::http::HeaderValue::from_static("stale-test-credential"), ); - let decision = filter_with_staging(&integration, &settings, &services, &mut request); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!( matches!(decision, RequestFilterDecision::Continue(_)), @@ -1062,7 +1055,7 @@ mod tests { } #[test] - fn protection_test_bypass_is_inactive_outside_staging() { + fn protection_test_bypass_works_without_staging_environment() { let config = DataDomeConfig { enabled: true, enable_protection: true, @@ -1075,20 +1068,11 @@ mod tests { }; let integration = DataDomeIntegration::try_new(config).expect("should create integration"); let mut secrets = HashMap::new(); - secrets.insert( - "datadome_server_side_key".to_string(), - b"server-side-key".to_vec(), - ); secrets.insert( "datadome_test_bypass".to_string(), b"temporary-test-credential-32-bytes!".to_vec(), ); let http_client = Arc::new(StubHttpClient::new()); - http_client.push_response_with_headers( - 200, - Vec::new(), - vec![(HEADER_DATADOME_RESPONSE, "200")], - ); let services = build_services_with_secret_and_http_client( HashMapSecretStore::new(secrets), http_client.clone(), @@ -1100,44 +1084,26 @@ mod tests { edgezero_core::http::HeaderValue::from_static("temporary-test-credential-32-bytes!"), ); - let _guard = FASTLY_IS_STAGING_ENV_LOCK - .lock() - .expect("should lock staging environment test guard"); - let decision = temp_env::with_var( - crate::constants::ENV_FASTLY_IS_STAGING, - None::<&str>, - || { - futures::executor::block_on(integration.filter_protection_request( - RequestFilterInput { - settings: &settings, - services: &services, - request: &mut request, - geo_info: None, - is_integration_route: false, - }, - )) - }, - ); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!( matches!(decision, RequestFilterDecision::Continue(_)), - "an allowed Protection API response should continue" + "a matching test credential should continue without a staging environment" + ); + assert!( + has_client_tag_suppression_marker(&request), + "the bypass should suppress the automatic DataDome client tag" ); assert!( request .headers() .get(super::super::HEADER_DATADOME_TEST_BYPASS) .is_none(), - "the bypass credential must be stripped outside staging" + "the bypass credential must not reach the publisher origin" ); assert!( - !has_client_tag_suppression_marker(&request), - "the bypass must not suppress the DataDome client tag outside staging" - ); - assert_eq!( - http_client.recorded_backend_names().len(), - 1, - "the bypass must still call the Protection API outside staging" + http_client.recorded_backend_names().is_empty(), + "a matching test credential must not call the Protection API" ); } @@ -1179,7 +1145,7 @@ mod tests { edgezero_core::http::HeaderValue::from_static("temporary-test-credential-32-bytes!"), ); - let decision = filter_with_staging(&integration, &settings, &services, &mut request); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!( matches!(decision, RequestFilterDecision::Continue(_)), @@ -1234,7 +1200,7 @@ mod tests { edgezero_core::http::HeaderValue::from_static("wrong-credential"), ); - let decision = filter_with_staging(&integration, &settings, &services, &mut request); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!( matches!(decision, RequestFilterDecision::Continue(_)), @@ -1302,7 +1268,7 @@ mod tests { ); } - let decision = filter_with_staging(&integration, &settings, &services, &mut request); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!(matches!(decision, RequestFilterDecision::Continue(_))); assert!( @@ -1368,7 +1334,7 @@ mod tests { .expect("should build bypass header"), ); - let decision = filter_with_staging(&integration, &settings, &services, &mut request); + let decision = filter_request(&integration, &settings, &services, &mut request); assert!(matches!(decision, RequestFilterDecision::Continue(_))); assert_eq!(has_client_tag_suppression_marker(&request), should_match); diff --git a/crates/trusted-server-core/src/integrations/gpt_bootstrap.js b/crates/trusted-server-core/src/integrations/gpt_bootstrap.js index eae59d79c..2475c5082 100644 --- a/crates/trusted-server-core/src/integrations/gpt_bootstrap.js +++ b/crates/trusted-server-core/src/integrations/gpt_bootstrap.js @@ -117,9 +117,16 @@ // and deliberately identical to the bundle scheduler — the impression is // spent on a viewed tab, and the post-hydration guarantee holds whenever // the request is actually issued. - ts.scheduleInitialAdInit = function (initialBids) { - if ((ts.navGeneration || 0) !== 0) return; - if (initialBids) ts.bids = initialBids; + ts.scheduleInitialAdInit = function (initialBids, initialSlots) { + // The bundle may replace this scheduler after the fallback claims the initial + // pass. Keep the latch on the shared document API so replacement cannot reset it. + if ((ts.navGeneration || 0) !== 0 || ts.initialAdInitScheduled) return; + ts.initialAdInitScheduled = true; + // Slots are generation-guarded for the same reason the bids are: the + // shared-template seam sends both, and an assignment made before this call + // would overwrite a committed SPA navigation's slots. + if (initialSlots !== undefined) ts.adSlots = initialSlots; + if (initialBids !== undefined) ts.bids = initialBids; var fire = function () { if ((ts.navGeneration || 0) !== 0) return; if (typeof ts.adInit === "function") ts.adInit(); diff --git a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs index 5bd86d19e..94e4607cd 100644 --- a/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs +++ b/crates/trusted-server-core/src/integrations/gpt_diagnostics.rs @@ -14,7 +14,7 @@ use edgezero_core::body::Body as EdgeBody; use crate::error::TrustedServerError; use crate::http_util::is_navigation_request; -use crate::response_privacy::CDN_CACHE_HEADERS; +use crate::response_privacy::enforce_synthesized_html_cache_privacy; use crate::settings::{IntegrationConfig, Settings}; use crate::tsjs; @@ -112,6 +112,87 @@ impl GptDiagnosticsRequestDecision { ) }) } + /// An active decision, for tests in other modules that need one. + /// + /// The fields are private and built by `prepare_request` from a cookie or query + /// parameter; there is no other way to obtain an active decision across a module + /// boundary. + #[cfg(test)] + pub(crate) fn active_for_tests() -> Self { + Self { + active: true, + clean_browser_path_and_query: None, + cookie_action: GptDiagnosticsCookieAction::None, + } + } +} + +#[cfg(test)] +mod head_seam_invariant_tests { + use super::*; + + /// Every combination of the three fields the decision carries. + fn all_decisions() -> Vec { + let mut out = Vec::new(); + for active in [false, true] { + for clean in [None, Some("/clean".to_string())] { + for cookie_action in [ + GptDiagnosticsCookieAction::None, + GptDiagnosticsCookieAction::SetSession, + GptDiagnosticsCookieAction::ClearSession, + ] { + out.push(GptDiagnosticsRequestDecision { + active, + clean_browser_path_and_query: clean.clone(), + cookie_action, + }); + } + } + } + out + } + + #[test] + fn requires_private_no_store_is_a_superset_of_injection() { + // Load-bearing relationship, not an incidental one. Whenever this decision + // injects anything into ``, the response must also be stamped + // `private, no-store` — which is what keeps request-scoped diagnostics out + // of a shared cache if the explicit assembly-mode gate in + // `create_html_stream_processor` is ever removed or bypassed. + // + // If a future change makes a script emit without also requiring the stamp, + // this fails here rather than silently in a cached template. + for decision in all_decisions() { + let injects = + decision.bootstrap_script().is_some() || decision.module_script_tag().is_some(); + if injects { + assert!( + decision.requires_private_no_store(), + "decision injects into but does not require private/no-store: \ + {decision:?}" + ); + } + } + } + + #[test] + fn a_default_decision_injects_nothing() { + let decision = GptDiagnosticsRequestDecision::default(); + assert_eq!( + decision.bootstrap_script(), + None, + "should not inject a bootstrap for an inert decision" + ); + assert_eq!( + decision.module_script_tag(), + None, + "should not inject a module for an inert decision" + ); + assert!( + !decision.requires_private_no_store(), + "an inert decision should not force the response private" + ); + } } #[derive(Clone, Copy, Debug, PartialEq, Eq)] @@ -253,13 +334,12 @@ pub fn finalize_response( } if decision.requires_private_no_store() { - response.headers_mut().insert( - header::CACHE_CONTROL, - HeaderValue::from_static("private, no-store"), - ); - for name in CDN_CACHE_HEADERS { - response.headers_mut().remove(*name); - } + // Marks the response terminal-private as well as stamping it. Stamping alone + // left the policy at the mercy of whatever ran later: a late + // `RequestFilterEffects` mutation such as `Cache-Control: public` replaced it, + // and the adapter's terminal guard had no marker to re-enforce from, so + // request-scoped diagnostics HTML became shared-cacheable. + enforce_synthesized_html_cache_privacy(response); } } @@ -497,6 +577,8 @@ mod tests { let decision = prepare_request(&settings(true), &mut request).expect("should prepare"); let mut response = Response::builder() .header(header::CACHE_CONTROL, "public, max-age=60") + .header(header::ETAG, "\"origin\"") + .header(header::LAST_MODIFIED, "Wed, 12 Aug 2026 00:00:00 GMT") .header("surrogate-control", "max-age=60") .header("fastly-surrogate-control", "max-age=60") .header("cloudflare-cdn-cache-control", "public, max-age=60") @@ -507,7 +589,8 @@ mod tests { assert_eq!( response.headers()[header::CACHE_CONTROL], - "private, no-store" + "private, no-store", + "should stamp diagnostics responses non-storable" ); assert_eq!(response.headers()[header::SET_COOKIE], SET_CONSOLE_COOKIE); assert!(!response.headers().contains_key("surrogate-control")); @@ -517,6 +600,74 @@ mod tests { .headers() .contains_key("cloudflare-cdn-cache-control") ); + assert!( + !response.headers().contains_key(header::ETAG), + "should drop the origin validator with the shared-cache policy" + ); + assert!( + !response.headers().contains_key(header::LAST_MODIFIED), + "should drop the origin validator with the shared-cache policy" + ); + } + + #[test] + fn an_active_no_cookie_action_response_is_marked_terminal_private() { + // The session-cookie activation path: active, but nothing new to set. Stamping + // `Cache-Control` alone left this response defenceless against a later mutation, + // because the adapter's terminal guard keys on the marker, not on the stamp, and + // the `Set-Cookie` privacy net never sees a response that sets no cookie. + let mut request = navigation("https://publisher.example/", Some("__Host-ts-console=1")); + let decision = prepare_request(&settings(true), &mut request).expect("should prepare"); + assert!(decision.active(), "the session cookie should activate"); + assert_eq!( + decision.cookie_action, + GptDiagnosticsCookieAction::None, + "an already-established session sets no new cookie" + ); + let mut response = Response::builder() + .body(EdgeBody::empty()) + .expect("should build response"); + + finalize_response(&decision, &mut response); + + assert!( + !response.headers().contains_key(header::SET_COOKIE), + "should not set a cookie for an established session" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should mark request-scoped diagnostics HTML for terminal re-enforcement" + ); + } + + #[test] + fn an_inactive_decision_leaves_the_origin_cache_policy_alone() { + let mut request = navigation("https://publisher.example/", None); + let decision = prepare_request(&settings(true), &mut request).expect("should prepare"); + assert!(!decision.requires_private_no_store()); + let mut response = Response::builder() + .header(header::CACHE_CONTROL, "public, max-age=60") + .header(header::ETAG, "\"origin\"") + .body(EdgeBody::empty()) + .expect("should build response"); + + finalize_response(&decision, &mut response); + + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "public, max-age=60", + "should not downgrade a response the integration did not touch" + ); + assert!( + response + .extensions() + .get::() + .is_none(), + "should not mark an untouched response terminal-private" + ); } #[test] diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index 4cc10f8da..c9b3f5ded 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -23,6 +23,7 @@ use crate::auction::provider::{AuctionProvider, ProviderRequestOutcome}; use crate::auction::types::{ AuctionContext, AuctionRequest, AuctionResponse, Bid as AuctionBid, MediaType, }; +use crate::cache_policy::{CacheControlPolicy, EdgeCacheHeader}; use crate::consent_config::ConsentForwardingMode; use crate::cookies::{CONSENT_COOKIE_NAMES, strip_cookies}; use crate::error::TrustedServerError; @@ -754,14 +755,16 @@ impl PrebidIntegration { ) -> Result, Report> { let body = "// Script overridden by Trusted Server\n"; - http::Response::builder() + let mut response = http::Response::builder() .status(StatusCode::OK) .header(header::CONTENT_TYPE, PREBID_BUNDLE_CONTENT_TYPE) - .header(header::CACHE_CONTROL, "public, max-age=31536000") .body(EdgeBody::from(body)) .change_context(TrustedServerError::Prebid { message: "Failed to build Prebid script handler response".to_string(), - }) + })?; + CacheControlPolicy::NoStorePrivate + .apply_to_headers(response.headers_mut(), EdgeCacheHeader::None); + Ok(response) } fn external_bundle_script_src(&self) -> String { @@ -3557,7 +3560,14 @@ external_bundle_sri = "sha384-AAAA" .get(header::CACHE_CONTROL) .and_then(|value| value.to_str().ok()) .expect("should have cache-control"); - assert!(cache_control.contains("max-age=31536000")); + assert_eq!( + cache_control, "no-store, private", + "neutralized stable shim must not be cached for a year" + ); + assert!( + response.headers().get("surrogate-control").is_none(), + "neutralized shim must not emit edge-cache headers" + ); let body = String::from_utf8( response diff --git a/crates/trusted-server-core/src/integrations/testlight.rs b/crates/trusted-server-core/src/integrations/testlight.rs index 888427e52..80b2c4dfa 100644 --- a/crates/trusted-server-core/src/integrations/testlight.rs +++ b/crates/trusted-server-core/src/integrations/testlight.rs @@ -264,8 +264,9 @@ fn default_timeout_ms() -> u32 { } fn default_shim_src() -> String { - // Testlight is included in the unified bundle, so we return the unified script source. - // Uses conservative all-module hash since the registry is unavailable at config time. + // Testlight is included in the unified bundle, so return the registry-free + // unified script source. It intentionally omits `?v=` because the exact + // enabled module set is unavailable at config-default time. tsjs::tsjs_unified_script_src() } diff --git a/crates/trusted-server-core/src/lib.rs b/crates/trusted-server-core/src/lib.rs index 70a4d6cfd..48e92faed 100644 --- a/crates/trusted-server-core/src/lib.rs +++ b/crates/trusted-server-core/src/lib.rs @@ -35,6 +35,7 @@ pub(crate) mod asset_image_optimizer; pub mod auction; pub mod auction_config_types; pub mod auth; +pub mod cache_policy; pub mod config; pub mod config_payload; pub mod consent; diff --git a/crates/trusted-server-core/src/platform/mod.rs b/crates/trusted-server-core/src/platform/mod.rs index 287f1accf..1c5bf4c2a 100644 --- a/crates/trusted-server-core/src/platform/mod.rs +++ b/crates/trusted-server-core/src/platform/mod.rs @@ -12,6 +12,8 @@ //! - [`PlatformBackend`] — dynamic backend registration //! - [`PlatformHttpClient`] — outbound HTTP client //! - [`PlatformGeo`] — geographic information lookup +//! - [`PlatformTemplateAssembler`] — cold-response shared-template assembly +//! - [`PlatformTemplateCache`] — shared transformed-template caching //! //! ## Platform-Agnostic Components //! @@ -36,6 +38,8 @@ mod error; mod http; mod image_optimizer; mod kv; +mod template_assembly; +mod template_cache; #[cfg(test)] pub(crate) mod test_support; mod traits; @@ -52,6 +56,17 @@ pub use image_optimizer::{ PlatformImageOptimizerParams, PlatformImageOptimizerRegion, }; pub use kv::UnavailableKvStore; +pub use template_assembly::{ + PlatformTemplateAssembler, TemplateAssemblyError, UnavailableTemplateAssembler, + contains_publisher_esi_directive, +}; +pub use template_cache::{ + PlatformTemplateCache, PlatformTemplateCacheReservation, REPLAYABLE_POLICY_HEADERS, + TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY, TEMPLATE_SCHEMA_VERSION, TemplateCacheError, + TemplateCacheKey, TemplateCacheLookup, TemplateCacheMiss, TemplateCacheReservation, + TemplateEntry, TemplateMetadata, TemplateMetadataEncodeError, UnavailableTemplateCache, + VaryHeaderValues, VarySpec, +}; pub use traits::{PlatformBackend, PlatformConfigStore, PlatformGeo, PlatformSecretStore}; pub use types::{ ClientInfo, GeoInfo, PlatformBackendSpec, RuntimeServices, RuntimeServicesBuilder, StoreId, diff --git a/crates/trusted-server-core/src/platform/template_assembly.rs b/crates/trusted-server-core/src/platform/template_assembly.rs new file mode 100644 index 000000000..f56179f1f --- /dev/null +++ b/crates/trusted-server-core/src/platform/template_assembly.rs @@ -0,0 +1,112 @@ +//! Platform boundary for assembling a shared template with reader-specific state. +//! +//! Core owns the cache-safety ordering and the portable byte-seam fallback. An adapter +//! may provide a richer assembler for the cold response after the reader-neutral +//! template has been stored. + +use core::fmt; + +/// Whether publisher bytes contain an ESI directive form understood by the parser. +/// +/// Both ordinary `` elements and `` comment blocks are active +/// parser input. The conservative byte scan also rejects these sequences inside scripts: +/// bypassing shared processing is safer than treating publisher data as edge instructions. +#[must_use] +pub fn contains_publisher_esi_directive(bytes: &[u8]) -> bool { + [b" Result, TemplateAssemblyError>; +} + +impl fmt::Debug for dyn PlatformTemplateAssembler { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("PlatformTemplateAssembler") + } +} + +/// Default assembler used by adapters that do not provide platform assembly. +#[derive(Debug, Default, Clone, Copy)] +pub struct UnavailableTemplateAssembler; + +impl PlatformTemplateAssembler for UnavailableTemplateAssembler { + fn assemble( + &self, + _template: &[u8], + _fragment: &[u8], + ) -> Result, TemplateAssemblyError> { + Err(TemplateAssemblyError::Unsupported) + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn unavailable_assembler_refuses_the_document() { + let error = UnavailableTemplateAssembler + .assemble(b"", b"") + .expect_err("should refuse when platform assembly is unavailable"); + + assert!(matches!(error, TemplateAssemblyError::Unsupported)); + } + + #[test] + fn assembler_contract_is_object_safe() { + let assembler: Box = Box::new(UnavailableTemplateAssembler); + + assert!(matches!( + assembler.assemble(b"template", b"fragment"), + Err(TemplateAssemblyError::Unsupported) + )); + } + + #[test] + fn publisher_esi_detection_covers_elements_and_comment_blocks() { + for directive in [ + b"".as_slice(), + b"secret".as_slice(), + b"".as_slice(), + b"".as_slice(), + ] { + assert!( + contains_publisher_esi_directive(directive), + "should detect publisher ESI bytes: {directive:?}" + ); + } + assert!( + !contains_publisher_esi_directive(b""), + "should not classify the inert TS seam as publisher ESI" + ); + } +} diff --git a/crates/trusted-server-core/src/platform/template_cache.rs b/crates/trusted-server-core/src/platform/template_cache.rs new file mode 100644 index 000000000..e3bbc42da --- /dev/null +++ b/crates/trusted-server-core/src/platform/template_cache.rs @@ -0,0 +1,1214 @@ +//! The shared transformed-template cache for the #1009 ESI validation spike. +//! +//! Three caches are in play and conflating them is what produced the original wrong +//! conclusion in the design doc, so this module names which one it is: +//! +//! | Cache | Contents | Owner | +//! | ----- | --------------------------------- | ------------------------------ | +//! | C1 | raw origin bytes | Fastly read-through. Not this. | +//! | Template cache | post-`lol_html`, pre-assembly | **This module.** | +//! | Final response | final per-user assembled response | **Must never exist.** | +//! +//! The template cache holds a *shared template*: no per-user bytes, and no decisions that depend on +//! the request. What may and may not live in it is +//! [§6.7 of the design doc](../../../../docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md), +//! and the invariant is enforced by the rendered-document byte-identity tests in +//! `publisher`. +//! +//! Spike-only. Remove with the spike. + +use core::fmt; +use std::collections::HashSet; + +use crate::creative_opportunities::AssemblyMode; + +/// Version of the transform that produced a cached template. +/// +/// Bump on **any** change to what the transform emits. Without it a deploy reads +/// yesterday's template shape and assembles against markers that moved, which fails +/// as a rendering bug far from its cause rather than as a cache miss. +/// +/// | Version | Transform | +/// | ------- | --------- | +/// | 1 | `` seam used an executable ESI include tag targeting the old fragment endpoint | +/// | 2 | Marker became the inert comment ``; the seam hands slots to `scheduleInitialAdInit` instead of assigning them | +/// | 3 | Marker became ``; canonical collision-safe key, explicit origin freshness, and complete repeated document-policy metadata | +/// | 4 | Marker is the shorter, accurate [`AD_ASSEMBLY_SEAM`](crate::publisher::AD_ASSEMBLY_SEAM) | +pub const TEMPLATE_SCHEMA_VERSION: u32 = 4; + +/// Surrogate key attached to every template so an incident can purge the template cache globally. +pub const TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY: &str = "ts-template"; + +/// Inputs that select one cached template. +/// +/// Every field changes the emitted bytes for the same URL. A signal that changes the +/// bytes and is **not** here produces cross-served templates; a signal that is +/// per-user does not belong here at all — it belongs out of the template entirely. +/// That distinction is the whole design: the key holds per-*variant* signals, and +/// per-*user* signals are excluded from the template rather than keyed on. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TemplateCacheKey { + /// Full request URL, stated explicitly rather than inherited from an ambient + /// request, so the key cannot silently depend on what the caller happened to + /// mutate first. + pub url: String, + /// Host and scheme. The post-processed output is host-dependent by construction: + /// both reach `IntegrationHtmlContext` and drive URL rewriting. + pub request_host: String, + /// See [`Self::request_host`]. + pub request_scheme: String, + /// Publisher origin identity, including the outbound Host override. Two virtual + /// hosts can share a connection target while producing unrelated documents. + pub origin_identity: String, + /// Inline and ESI modes emit different template bytes. Without this they poison + /// each other's entries. + pub assembly_mode: AssemblyMode, + /// Values of the request headers the **origin** declares it varies on, in the + /// order the origin listed them. Not a fixed list: the origin is authoritative, + /// and hard-coding one here would silently drift when the origin's changes. + pub vary_values: Vec, + /// Digest of every setting that can shape the transformed template plus the tsjs + /// bundle. Over-invalidating is safe; omitting a shaping input cross-serves bytes. + pub template_fingerprint: String, + /// See [`TEMPLATE_SCHEMA_VERSION`]. + pub schema_version: u32, +} + +impl TemplateCacheKey { + /// Render a fixed-size opaque key for the platform cache. + /// + /// The canonical input is length-prefixed before hashing, so neither delimiters nor + /// raw request values can collide or leak into cache diagnostics. + #[must_use] + pub fn to_cache_key(&self) -> String { + use sha2::Digest as _; + + fn push(out: &mut Vec, part: &[u8]) { + out.extend_from_slice(&(part.len() as u64).to_be_bytes()); + out.extend_from_slice(part); + } + + let mut canonical = Vec::new(); + push(&mut canonical, b"ts-template-cache"); + push(&mut canonical, &self.schema_version.to_be_bytes()); + push( + &mut canonical, + match self.assembly_mode { + AssemblyMode::Inline => b"inline", + AssemblyMode::Esi => b"esi", + }, + ); + push(&mut canonical, self.request_scheme.as_bytes()); + push(&mut canonical, self.request_host.as_bytes()); + push(&mut canonical, self.origin_identity.as_bytes()); + push(&mut canonical, self.url.as_bytes()); + push(&mut canonical, self.template_fingerprint.as_bytes()); + push( + &mut canonical, + &(self.vary_values.len() as u64).to_be_bytes(), + ); + for varied in &self.vary_values { + push(&mut canonical, varied.name.to_ascii_lowercase().as_bytes()); + match &varied.values { + None => push(&mut canonical, b"absent"), + Some(values) => { + push(&mut canonical, b"present"); + push(&mut canonical, &(values.len() as u64).to_be_bytes()); + for value in values { + push(&mut canonical, value); + } + } + } + } + + let digest = sha2::Sha256::digest(canonical); + format!( + "ts-template-cache-v{}-{}", + self.schema_version, + hex::encode(digest) + ) + } + + /// Surrogate keys to attach at insert, for purge-based rollback. + /// + /// `ts-template` purges every template at once, which is the rollback lever. + /// The per-URL key allows targeted invalidation. Both are needed: the broad one + /// for an incident, the narrow one for ordinary invalidation. + #[must_use] + pub fn surrogate_keys(&self) -> Vec { + vec![ + TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY.to_string(), + self.url_surrogate_key(), + ] + } + + /// Surrogate key for every variant of this publisher URL. + /// + /// Used to evict a malformed object without flushing unrelated article templates. + #[must_use] + pub fn url_surrogate_key(&self) -> String { + format!("ts-template-url-{}", digest_hex(self.url.as_bytes())) + } +} + +fn digest_hex(bytes: &[u8]) -> String { + use sha2::Digest as _; + hex::encode(sha2::Sha256::digest(bytes)) +} + +/// One configured `Vary` input exactly as it appeared on the request. +/// +/// `None` means absent. `Some(vec![vec![]])` means present with one empty field +/// value. Repeated fields stay separate and ordered; no UTF-8 conversion is involved. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VaryHeaderValues { + /// Validated, lowercase header name. + pub name: String, + /// Every raw field value in wire order, or `None` when absent. + pub values: Option>>, +} + +/// Origin response headers safe to store with a shared template and replay on a hit. +/// +/// Every one is a per-URL policy statement, identical for every reader. Nothing +/// per-reader (`Set-Cookie`) and nothing cache-controlling (`Cache-Control`, `ETag`, +/// `Surrogate-Control`) appears here, and it is an allowlist so a new origin header is +/// excluded until someone decides otherwise. +pub const REPLAYABLE_POLICY_HEADERS: &[&str] = &[ + "content-security-policy", + "content-security-policy-report-only", + "permissions-policy", + "referrer-policy", + "strict-transport-security", + "cross-origin-opener-policy", + "cross-origin-embedder-policy", + "cross-origin-resource-policy", + "origin-agent-cluster", + "reporting-endpoints", + "report-to", + "link", + "x-frame-options", + "x-content-type-options", + "content-language", + "x-robots-tag", +]; + +/// Headers the key covers by construction, whatever the operator configured. +/// +/// The shared path stores decoded identity bytes and negotiates the reader representation +/// only after assembly, so an origin declaring `Vary: Accept-Encoding` is covered without +/// reader input. This assumes those origin variants differ only by HTTP content coding; +/// operators must leave ESI disabled if an origin changes document semantics instead. +/// Without this carve-out, the ordinary declaration sent by any compressing origin reads +/// as an uncovered gap and disqualifies the response, so **the template cache would never store anything +/// against a real origin** unless the operator redundantly listed a header the transform +/// already normalizes. Found by review before it could make the spike measure a hit rate +/// of approximately zero and read that as a result. +const STRUCTURALLY_COVERED: &[&str] = &["accept-encoding"]; + +/// Request headers to include in the cache key, and where the list comes from. +/// +/// # The chicken-and-egg this resolves +/// +/// The key must cover everything the origin varies on, or two requests needing +/// different templates share one entry. But a **lookup happens before the fetch**, +/// so on a cold key the origin's `Vary` is not yet known. +/// +/// Three ways out, and the trade-off is real: +/// +/// 1. **Configure the list** — what this does. One lookup, no extra round trip, and +/// the operator states what the origin varies on. Cost: it drifts silently if the +/// origin's `Vary` changes and nobody updates config. +/// 2. **Two-phase lookup** — fetch a URL-keyed record holding the last-seen `Vary`, +/// then key properly. Correct, but doubles the lookups on every request. +/// 3. **Store the list alongside** and re-key on mismatch. Same cost as (2) plus +/// complexity. +/// +/// (1) is chosen for the spike because Step A already measured the origin's actual +/// `Vary`, the origin response is checked for drift before storage, and the configured +/// template-cache ceiling bounds how long a newly introduced mismatch can survive. +/// **This is a spike-grade choice, not a production one** — see the drift guard below. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct VarySpec { + /// Header names, lowercased, in a fixed order. + names: Vec, +} + +impl VarySpec { + /// Build from configured header names. + /// + /// # Panics + /// + /// Panics when a name is not a valid HTTP field name. Runtime configuration is + /// validated with [`Self::try_new`] before this constructor is used. + #[must_use] + pub fn new(names: impl IntoIterator) -> Self { + Self::try_new(names).expect("VarySpec names should be validated at configuration load") + } + + /// Build from configured names, validating and deduplicating them. + /// + /// # Errors + /// + /// Returns the offending name when it is not a valid HTTP field name. + pub fn try_new(names: impl IntoIterator) -> Result { + let mut seen = HashSet::new(); + let mut normalized = Vec::new(); + for raw in names { + let name = http::header::HeaderName::from_bytes(raw.as_bytes()) + .map_err(|_| raw.clone())? + .as_str() + .to_string(); + if STRUCTURALLY_COVERED.contains(&name.as_str()) { + continue; + } + if seen.insert(name.clone()) { + normalized.push(name); + } + } + Ok(Self { names: normalized }) + } + + /// Configured names, lowercased. + #[must_use] + pub fn names(&self) -> &[String] { + &self.names + } + + /// Extract the key inputs from a request's headers. + /// + /// A header the origin varies on but the request omits still contributes an + /// entry, with an empty value — otherwise "absent" and "present but empty" + /// would collide, and those are different requests to the origin. + #[must_use] + pub fn values_from(&self, headers: &http::HeaderMap) -> Vec { + self.names + .iter() + .map(|name| { + let values = headers.contains_key(name.as_str()).then(|| { + headers + .get_all(name.as_str()) + .iter() + .map(|value| value.as_bytes().to_vec()) + .collect() + }); + VaryHeaderValues { + name: name.clone(), + values, + } + }) + .collect() + } + + /// Whether the origin's declared `Vary` contains anything this spec omits. + /// + /// The drift guard for choice (1) above. Called **after** the origin responds, + /// when its `Vary` is finally known: if the origin varies on something the key + /// did not cover, the template just built is unsafe to store, because a request + /// differing only in that header would read it. + /// + /// Returns the uncovered names, so the caller can log precisely which config is + /// stale rather than reporting a generic refusal. + #[must_use] + pub fn uncovered_by<'a>(&self, origin_vary: impl IntoIterator) -> Vec { + origin_vary + .into_iter() + .flat_map(|value| value.split(',')) + .map(|name| name.trim().to_ascii_lowercase()) + .filter(|name| !name.is_empty() && name != "*") + .filter(|name| !STRUCTURALLY_COVERED.contains(&name.as_str())) + .filter(|name| !self.names.contains(name)) + .collect() + } +} + +/// Metadata stored alongside the template bytes. +/// +/// `cache::core` carries **no HTTP semantics** — status, headers, encoding and +/// revalidation are all the caller's. Rather than storing origin headers and +/// replaying them, store only what is needed to rebuild a response from scratch. +/// +/// That choice is deliberate and load-bearing: the publisher path forces +/// `private, no-store` and strips validators *after* the origin send, so replaying a +/// stored origin header would fight it. Rebuilding every header on a hit means no +/// origin header is ever replayed and the `Set-Cookie` privacy net stays trivially +/// safe. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TemplateMetadata { + /// Encoding of the stored bytes. The template cache writes only `identity`; retaining the field in + /// metadata makes corrupt or stale representations fail validation on read. + pub content_encoding: String, + /// Content type to rebuild the response with. + pub content_type: String, + /// Schema version the bytes were produced under. Checked on read: a mismatch is + /// a miss, not an error, so a rollback to an older binary degrades to + /// re-transforming rather than misassembling. + pub schema_version: u32, + /// Length of the template bytes as written. + /// + /// Guards against a partially written entry. `Transaction::insert` consumes the + /// transaction, so a write that fails part-way cannot cancel the insert — there + /// is no handle left to cancel it with. Recording the intended length and + /// checking it on read makes a truncated entry a miss instead of a silently + /// short template that would assemble into a broken page. + pub body_len: u64, + /// Origin response headers that are policy, not per-reader state. + /// + /// Reconstructing headers from scratch on a hit keeps origin `Set-Cookie` and caching + /// directives out of a shared cache — but it also dropped `Content-Security-Policy`, + /// framing protection and `Content-Language`, weakening the page. These are + /// per-URL and identical for every reader, so they belong with the template. + /// + /// Deliberately an allowlist: anything per-reader or cache-controlling is excluded by + /// construction rather than by remembering to strip it. + pub policy_headers: Vec<(String, String)>, +} + +/// Why public template metadata could not be represented safely. +#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] +#[display("template metadata field `{field}` contains a line break")] +pub struct TemplateMetadataEncodeError { + field: &'static str, +} + +impl core::error::Error for TemplateMetadataEncodeError {} + +impl TemplateMetadata { + /// Serialize for `user_metadata`. Deliberately a tiny hand-rolled format rather + /// than JSON — one allocation, no dependency, and a parse failure is + /// unambiguous. + /// + /// # Errors + /// + /// Returns an error when any public string field contains CR or LF, which would + /// otherwise inject another record into the newline-delimited representation. + pub fn encode(&self) -> Result, TemplateMetadataEncodeError> { + fn reject_line_breaks( + field: &'static str, + value: &str, + ) -> Result<(), TemplateMetadataEncodeError> { + if value.contains(['\r', '\n']) { + return Err(TemplateMetadataEncodeError { field }); + } + Ok(()) + } + + reject_line_breaks("content_encoding", &self.content_encoding)?; + reject_line_breaks("content_type", &self.content_type)?; + for (name, value) in &self.policy_headers { + reject_line_breaks("policy_header_name", name)?; + reject_line_breaks("policy_header_value", value)?; + } + + let mut out = format!( + "v={}\nce={}\nct={}\nlen={}", + self.schema_version, self.content_encoding, self.content_type, self.body_len + ); + for (name, value) in &self.policy_headers { + // Line breaks were rejected above before constructing the delimited form. + out.push_str(&format!("\nh={name}:{value}")); + } + Ok(out.into_bytes()) + } + + /// Parse `user_metadata`. Returns `None` on anything unexpected, which callers + /// must treat as a cache miss. + #[must_use] + pub fn decode(raw: &[u8]) -> Option { + let text = core::str::from_utf8(raw).ok()?; + let mut schema_version = None; + let mut policy_headers = Vec::new(); + let mut content_encoding = None; + let mut content_type = None; + let mut body_len = None; + for line in text.lines() { + let (key, value) = line.split_once('=')?; + match key { + "v" => { + if schema_version.replace(value.parse().ok()?).is_some() { + return None; + } + } + "ce" => { + if content_encoding.replace(value.to_string()).is_some() { + return None; + } + } + "h" => { + let (name, header_value) = value.split_once(':')?; + let name = http::header::HeaderName::from_bytes(name.as_bytes()).ok()?; + if !REPLAYABLE_POLICY_HEADERS.contains(&name.as_str()) { + return None; + } + http::HeaderValue::from_bytes(header_value.as_bytes()).ok()?; + policy_headers.push((name.as_str().to_string(), header_value.to_string())); + } + "ct" => { + if content_type.replace(value.to_string()).is_some() { + return None; + } + } + "len" => { + if body_len.replace(value.parse().ok()?).is_some() { + return None; + } + } + _ => return None, + } + } + let content_encoding = content_encoding?; + // Every template is decoded before insert. Accepting another value here would + // let corrupt metadata label plaintext bytes as gzip on a warm hit. + if content_encoding != "identity" { + return None; + } + let content_type = content_type?; + http::HeaderValue::from_bytes(content_type.as_bytes()).ok()?; + if !content_type + .split(';') + .next() + .is_some_and(|media_type| media_type.trim().eq_ignore_ascii_case("text/html")) + { + return None; + } + Some(Self { + schema_version: schema_version?, + policy_headers, + content_encoding, + content_type, + body_len: body_len?, + }) + } +} + +/// Why a template read did not produce usable bytes. +#[derive(Debug, Clone, Copy, PartialEq, Eq, derive_more::Display)] +pub enum TemplateCacheMiss { + /// No entry for this key. + #[display("no cached template for this key")] + NotFound, + /// Found, but produced by a different transform version. + #[display("cached template has a different schema version")] + SchemaMismatch, + /// Found, but its metadata could not be parsed. + #[display("cached template metadata is unreadable")] + UnreadableMetadata, + /// Found, but shorter than the metadata says it should be — a write that failed + /// part-way. See [`TemplateMetadata::body_len`]. + #[display("cached template is truncated")] + Truncated, + /// This platform has no template cache. + #[display("no template cache on this platform")] + Unsupported, +} + +impl core::error::Error for TemplateCacheMiss {} + +/// Errors a template cache write can produce. +#[derive(Debug, derive_more::Display)] +pub enum TemplateCacheError { + /// This platform has no template cache. + #[display("no template cache on this platform")] + Unsupported, + /// The platform rejected the operation. + #[display("template cache backend error: {message}")] + Backend { + /// What the backend reported. + message: String, + }, +} + +impl core::error::Error for TemplateCacheError {} + +/// Result of the pre-origin cache transaction. +pub enum TemplateCacheLookup { + /// A fresh usable template. + Hit(TemplateEntry), + /// This request owns the obligation to provide or cancel the cold object. + Reserved(TemplateCacheReservation), + /// This adapter deliberately has no shared-template cache. + Unsupported, + /// A cache object existed but failed schema, metadata, or length validation. + Invalid(TemplateCacheMiss), +} + +/// Platform-owned insert obligation. Dropping it cancels, making every early-return +/// path safe without an async cleanup ladder in the publisher pipeline. +pub struct TemplateCacheReservation { + inner: Option>, +} + +impl core::fmt::Debug for TemplateCacheReservation { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("TemplateCacheReservation") + .finish_non_exhaustive() + } +} + +impl TemplateCacheReservation { + /// Wrap a platform reservation. + #[must_use] + pub fn new(inner: Box) -> Self { + Self { inner: Some(inner) } + } + + /// Fulfil the reservation with a validated template. + /// + /// # Errors + /// + /// Returns the platform cache error when the reservation cannot be fulfilled. + pub fn insert( + mut self, + metadata: &TemplateMetadata, + body: Vec, + max_age: std::time::Duration, + ) -> Result<(), TemplateCacheError> { + self.inner + .take() + .ok_or_else(|| TemplateCacheError::Backend { + message: "template reservation was already consumed".to_string(), + })? + .insert(metadata, body, max_age) + } + + /// Explicitly give up the reservation. Drop performs the same operation as a net. + /// + /// # Errors + /// + /// Returns the platform cache error when the reservation cannot be cancelled. + pub fn cancel(mut self) -> Result<(), TemplateCacheError> { + self.inner + .take() + .ok_or_else(|| TemplateCacheError::Backend { + message: "template reservation was already consumed".to_string(), + })? + .cancel() + } +} + +impl Drop for TemplateCacheReservation { + fn drop(&mut self) { + if let Some(inner) = self.inner.take() + && let Err(err) = inner.cancel() + { + log::warn!("template_cache reservation cancellation failed: {err}"); + } + } +} + +/// Adapter-specific ownership token returned by a transactional lookup. +pub trait PlatformTemplateCacheReservation: Send { + /// Insert and discharge the obligation. + /// + /// # Errors + /// + /// Returns an adapter-specific cache error when the insert fails. + fn insert( + self: Box, + metadata: &TemplateMetadata, + body: Vec, + max_age: std::time::Duration, + ) -> Result<(), TemplateCacheError>; + + /// Cancel and allow a waiting request to take ownership. + /// + /// # Errors + /// + /// Returns an adapter-specific cache error when cancellation fails. + fn cancel(self: Box) -> Result<(), TemplateCacheError>; +} + +impl fmt::Debug for dyn PlatformTemplateCache { + fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { + f.write_str("PlatformTemplateCache") + } +} + +/// A platform's shared-template cache. +/// +/// Only the Fastly adapter implements this; every other adapter uses +/// [`UnavailableTemplateCache`], which reports [`TemplateCacheMiss::Unsupported`] so +/// the caller transforms every time rather than failing. +/// +/// `Send + Sync` on the trait, `?Send` on the futures: `RuntimeServices` is held in a +/// `LazyLock` static, so the trait object must cross threads even though the futures +/// themselves never do — the platform layer is `!Send` by construction. +#[async_trait::async_trait(?Send)] +pub trait PlatformTemplateCache: Send + Sync { + /// Transactionally look up a template before origin work begins. + /// + /// This compatibility default exists for implementations with no transactional + /// reservation support. It reports ordinary cold misses as `Unsupported`; an + /// adapter that supports template-cache reservations must override it so cold requests can + /// return [`TemplateCacheLookup::Reserved`]. + async fn lookup_or_reserve( + &self, + key: &TemplateCacheKey, + ) -> Result { + Ok(match self.get(key).await { + Ok(entry) => TemplateCacheLookup::Hit(entry), + Err(TemplateCacheMiss::Unsupported | TemplateCacheMiss::NotFound) => { + TemplateCacheLookup::Unsupported + } + Err(miss) => TemplateCacheLookup::Invalid(miss), + }) + } + + /// Read a template. `Err` is a miss, not a failure — every variant means + /// "transform it yourself". + async fn get(&self, key: &TemplateCacheKey) -> Result; + + /// Store a template. + /// + /// Callers must not call this without having consulted the template-cache eligibility gate + /// first: this method stores what it is given and cannot tell a shared template + /// from a per-user one. + async fn put( + &self, + key: &TemplateCacheKey, + metadata: &TemplateMetadata, + body: Vec, + max_age: std::time::Duration, + ) -> Result<(), TemplateCacheError>; + + /// Purge every cached variant for one publisher URL. + async fn purge_url(&self, key: &TemplateCacheKey) -> Result<(), TemplateCacheError>; + + /// Purge every stored template. The rollback lever. + async fn purge_all(&self) -> Result<(), TemplateCacheError>; +} + +/// A template read from the cache. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct TemplateEntry { + /// Metadata stored at insert. + pub metadata: TemplateMetadata, + /// The transformed template bytes. + pub body: Vec, +} + +/// The null object, used by every adapter without a template cache. +/// +/// Reporting [`TemplateCacheMiss::Unsupported`] rather than erroring means the +/// ESI assembly mode degrades to transforming per request on Cloudflare, Axum and Spin +/// instead of failing — the mode stays portable, only the caching is not. +pub struct UnavailableTemplateCache; + +#[async_trait::async_trait(?Send)] +impl PlatformTemplateCache for UnavailableTemplateCache { + async fn lookup_or_reserve( + &self, + _key: &TemplateCacheKey, + ) -> Result { + Ok(TemplateCacheLookup::Unsupported) + } + + async fn get(&self, _key: &TemplateCacheKey) -> Result { + Err(TemplateCacheMiss::Unsupported) + } + + async fn put( + &self, + _key: &TemplateCacheKey, + _metadata: &TemplateMetadata, + _body: Vec, + _max_age: std::time::Duration, + ) -> Result<(), TemplateCacheError> { + Err(TemplateCacheError::Unsupported) + } + + async fn purge_url(&self, _key: &TemplateCacheKey) -> Result<(), TemplateCacheError> { + Err(TemplateCacheError::Unsupported) + } + + async fn purge_all(&self) -> Result<(), TemplateCacheError> { + Err(TemplateCacheError::Unsupported) + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::Arc; + use std::sync::atomic::{AtomicUsize, Ordering}; + + fn key() -> TemplateCacheKey { + TemplateCacheKey { + url: "https://example.com/news/article".to_string(), + request_host: "example.com".to_string(), + request_scheme: "https".to_string(), + origin_identity: "https://origin.example.com\0origin.example.com".to_string(), + assembly_mode: AssemblyMode::Esi, + vary_values: vec![VaryHeaderValues { + name: "rsc".to_string(), + values: Some(vec![b"1".to_vec()]), + }], + template_fingerprint: "abc123".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + } + } + + struct CountingReservation(Arc); + + impl PlatformTemplateCacheReservation for CountingReservation { + fn insert( + self: Box, + _metadata: &TemplateMetadata, + _body: Vec, + _max_age: std::time::Duration, + ) -> Result<(), TemplateCacheError> { + Ok(()) + } + + fn cancel(self: Box) -> Result<(), TemplateCacheError> { + self.0.fetch_add(1, Ordering::SeqCst); + Ok(()) + } + } + + #[test] + fn dropping_an_unfulfilled_reservation_cancels_exactly_once() { + let cancellations = Arc::new(AtomicUsize::new(0)); + drop(TemplateCacheReservation::new(Box::new( + CountingReservation(Arc::clone(&cancellations)), + ))); + assert_eq!(cancellations.load(Ordering::SeqCst), 1); + } + + #[test] + fn fulfilling_a_reservation_does_not_also_cancel_on_drop() { + let cancellations = Arc::new(AtomicUsize::new(0)); + TemplateCacheReservation::new(Box::new(CountingReservation(Arc::clone(&cancellations)))) + .insert( + &TemplateMetadata { + content_encoding: "identity".to_string(), + policy_headers: Vec::new(), + content_type: "text/html".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 0, + }, + Vec::new(), + std::time::Duration::from_secs(1), + ) + .expect("should fulfil the reservation"); + + assert_eq!( + cancellations.load(Ordering::SeqCst), + 0, + "should discharge a fulfilled reservation without cancelling it" + ); + } + + /// Every field must change the key. A field that does not is a cross-serving + /// bug: two requests needing different templates would share one entry. + #[test] + fn every_field_changes_the_key() { + let base = key().to_cache_key(); + + let mut mode = key(); + mode.assembly_mode = AssemblyMode::Inline; + assert_ne!( + mode.to_cache_key(), + base, + "assembly mode must change the key" + ); + + let mut url = key(); + url.url = "https://example.com/other".to_string(); + assert_ne!(url.to_cache_key(), base, "url must change the key"); + + let mut host = key(); + host.request_host = "other.example.com".to_string(); + assert_ne!(host.to_cache_key(), base, "host must change the key"); + + let mut scheme = key(); + scheme.request_scheme = "http".to_string(); + assert_ne!(scheme.to_cache_key(), base, "scheme must change the key"); + + let mut origin = key(); + origin.origin_identity = "https://origin.example.com\0other.example.com".to_string(); + assert_ne!( + origin.to_cache_key(), + base, + "origin Host identity must change the key" + ); + + let mut fingerprint = key(); + fingerprint.template_fingerprint = "def456".to_string(); + assert_ne!( + fingerprint.to_cache_key(), + base, + "template fingerprint must change the key" + ); + + let mut schema = key(); + schema.schema_version = TEMPLATE_SCHEMA_VERSION + 1; + assert_ne!( + schema.to_cache_key(), + base, + "schema version must change the key" + ); + + let mut vary = key(); + vary.vary_values = vec![VaryHeaderValues { + name: "rsc".to_string(), + values: Some(vec![b"0".to_vec()]), + }]; + assert_ne!(vary.to_cache_key(), base, "vary values must change the key"); + } + + /// The reason for length prefixes rather than a delimiter. + #[test] + fn values_containing_delimiters_cannot_collide() { + let mut a = key(); + a.request_host = "a".to_string(); + a.url = "b:c".to_string(); + + let mut b = key(); + b.request_host = "a:b".to_string(); + b.url = "c".to_string(); + + assert_ne!( + a.to_cache_key(), + b.to_cache_key(), + "field values containing the delimiter must not produce the same key; a \ + collision here serves one visitor's template to another" + ); + } + + #[test] + fn rendered_key_is_fixed_size_and_contains_no_request_material() { + let rendered = key().to_cache_key(); + assert_eq!( + rendered, + "ts-template-cache-v4-54431eb4ea82644d6378717a8c3f18302fafbf739e684598da79e392b16900a6" + ); + assert!(rendered.starts_with("ts-template-cache-v4-")); + assert_eq!(rendered.len(), 85); + for sensitive in ["example.com", "/news/article", "rsc", "abc123"] { + assert!( + !rendered.contains(sensitive), + "key leaked `{sensitive}`: {rendered}" + ); + } + } + + #[test] + fn vary_header_names_are_matched_case_insensitively() { + let mut upper = key(); + upper.vary_values = vec![VaryHeaderValues { + name: "RSC".to_string(), + values: Some(vec![b"1".to_vec()]), + }]; + assert_eq!( + upper.to_cache_key(), + key().to_cache_key(), + "header names are case-insensitive, so casing must not split the cache" + ); + } + + #[test] + fn vary_values_are_order_sensitive() { + // The origin lists them in a fixed order and the caller preserves it, so a + // differing order means differing inputs rather than the same request. + let mut a = key(); + a.vary_values = vec![ + VaryHeaderValues { + name: "rsc".to_string(), + values: Some(vec![b"1".to_vec()]), + }, + VaryHeaderValues { + name: "x-route".to_string(), + values: Some(vec![b"article".to_vec()]), + }, + ]; + let mut b = key(); + b.vary_values = vec![ + VaryHeaderValues { + name: "x-route".to_string(), + values: Some(vec![b"article".to_vec()]), + }, + VaryHeaderValues { + name: "rsc".to_string(), + values: Some(vec![b"1".to_vec()]), + }, + ]; + assert_ne!(a.to_cache_key(), b.to_cache_key()); + } + + #[test] + fn surrogate_keys_carry_a_global_and_a_per_url_lever() { + let keys = key().surrogate_keys(); + assert!( + keys.contains(&TEMPLATE_CACHE_PURGE_ALL_SURROGATE_KEY.to_string()), + "a global purge lever is what makes rollback possible" + ); + assert_eq!(keys.len(), 2, "global plus per-URL"); + assert!( + !keys[1].contains(char::is_whitespace), + "surrogate keys are space-delimited; whitespace would purge more than \ + intended, got {:?}", + keys[1] + ); + assert!( + !keys[1].contains('/') && !keys[1].contains(':'), + "URL punctuation must be reduced, got {:?}", + keys[1] + ); + } + + #[test] + fn punctuation_distinct_urls_have_distinct_surrogate_keys() { + let mut slash = key(); + slash.url = "https://example.com/a/b".to_string(); + let mut colon = key(); + colon.url = "https://example.com/a:b".to_string(); + assert_ne!(slash.surrogate_keys()[1], colon.surrogate_keys()[1]); + } + + #[test] + fn an_absent_vary_header_is_distinct_from_an_empty_one() { + // "absent" and "present but empty" are different requests to the origin, so + // they must not share a template. + let spec = VarySpec::new(["RSC".to_string()]); + let absent_headers = http::HeaderMap::new(); + let absent = spec.values_from(&absent_headers); + let mut empty_headers = http::HeaderMap::new(); + empty_headers.insert("rsc", http::HeaderValue::from_static("")); + let empty = spec.values_from(&empty_headers); + assert_ne!(absent, empty); + + // The distinction that does matter: a present value differs from both. + let mut present_headers = http::HeaderMap::new(); + present_headers.insert("rsc", http::HeaderValue::from_static("1")); + let present = spec.values_from(&present_headers); + assert_ne!(present, absent); + } + + #[test] + fn repeated_and_non_utf8_vary_values_are_preserved() { + let spec = VarySpec::new(["x-route".to_string()]); + let mut headers = http::HeaderMap::new(); + headers.append("x-route", http::HeaderValue::from_static("first")); + headers.append( + "x-route", + http::HeaderValue::from_bytes(b"\xffsecond").expect("obs-text is valid field data"), + ); + assert_eq!( + spec.values_from(&headers), + vec![VaryHeaderValues { + name: "x-route".to_string(), + values: Some(vec![b"first".to_vec(), b"\xffsecond".to_vec()]), + }] + ); + } + + #[test] + fn vary_spec_lowercases_configured_names() { + assert_eq!( + VarySpec::new(["RSC".to_string(), "Accept-Encoding".to_string()]).names(), + ["rsc"] + ); + } + + #[test] + fn vary_spec_rejects_invalid_names_and_deduplicates_case_insensitively() { + assert_eq!( + VarySpec::try_new(["not a header".to_string()]), + Err("not a header".to_string()) + ); + assert_eq!( + VarySpec::try_new(["RSC".to_string(), "rsc".to_string()]) + .expect("valid names") + .names(), + ["rsc"] + ); + } + + #[test] + fn drift_is_detected_when_the_origin_varies_on_something_unconfigured() { + // The failure mode configured-Vary has: the origin adds a header to its Vary, + // nobody updates config, and requests differing only in that header start + // sharing a template. + let spec = VarySpec::new(["rsc".to_string()]); + + assert!( + spec.uncovered_by(["rsc"]).is_empty(), + "a fully covered Vary is not drift" + ); + assert_eq!( + spec.uncovered_by(["rsc, next-router-prefetch, Accept-Encoding"]), + vec!["next-router-prefetch"], + "uncovered names must be reported so the stale config is identifiable; \ + accept-encoding is excluded because the key covers it structurally" + ); + } + + #[test] + fn a_key_field_counts_as_coverage_without_being_configured() { + // The failure this prevents is silent and total: every compressing origin sends + // `Vary: Accept-Encoding`, so treating it as a gap means the cache never stores + // anything, and a spike measuring hit rate would report ~0 and look like a + // finding rather than a bug. + let spec = VarySpec::new([]); + + assert!( + spec.uncovered_by(["Accept-Encoding"]).is_empty(), + "the shared path uses one upstream encoding offer and stores identity bytes" + ); + assert_eq!( + spec.uncovered_by(["accept-encoding, rsc"]), + vec!["rsc"], + "only the genuinely uncovered name should be reported" + ); + } + + #[test] + fn a_wildcard_vary_is_not_reported_as_a_named_gap() { + // `Vary: *` means uncacheable, which the eligibility gate handles. Reporting + // it here would produce a nonsense "configure a header called *". + let spec = VarySpec::new(["rsc".to_string()]); + assert!(spec.uncovered_by(["*"]).is_empty()); + } + + #[test] + fn metadata_round_trips() { + let metadata = TemplateMetadata { + content_encoding: "identity".to_string(), + policy_headers: vec![ + ( + "content-security-policy".to_string(), + "default-src 'self'".to_string(), + ), + ( + "content-security-policy".to_string(), + "script-src 'self'".to_string(), + ), + ( + "link".to_string(), + "; rel=preload; as=script".to_string(), + ), + ], + content_type: "text/html; charset=utf-8".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 42, + }; + let encoded = metadata.encode().expect("valid metadata should encode"); + let decoded = TemplateMetadata::decode(&encoded).expect("should decode what it encoded"); + assert_eq!(decoded, metadata); + } + + #[test] + fn metadata_encoding_rejects_line_break_injection() { + for metadata in [ + TemplateMetadata { + content_encoding: "identity\nh=link:".to_string(), + policy_headers: Vec::new(), + content_type: "text/html".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 0, + }, + TemplateMetadata { + content_encoding: "identity".to_string(), + policy_headers: vec![( + "content-security-policy".to_string(), + "default-src 'self'\r\nh=link:".to_string(), + )], + content_type: "text/html".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 0, + }, + TemplateMetadata { + content_encoding: "identity".to_string(), + policy_headers: vec![( + "content-security-policy\rh=link".to_string(), + "default-src 'self'".to_string(), + )], + content_type: "text/html".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 0, + }, + TemplateMetadata { + content_encoding: "identity".to_string(), + policy_headers: Vec::new(), + content_type: "text/html\nh=link:".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 0, + }, + ] { + assert!( + metadata.encode().is_err(), + "should reject metadata fields that can inject another line" + ); + } + } + + #[test] + fn unparseable_metadata_is_a_miss_not_a_panic() { + for raw in [ + &b"not-key-value"[..], + &b"v=notanumber\nce=gzip\nct=text/html\nlen=1"[..], + &b"v=1\nce=gzip\nct=text/html"[..], + &b"v=1\nce=gzip\nct=text/html\nlen=1\nunexpected=1"[..], + &b"v=1\nv=1\nce=identity\nct=text/html\nlen=1"[..], + &b"v=1\nce=identity\nct=text/html\nlen=1\nh=cache-control:public"[..], + &b"v=1\nce=identity\nct=text/html\nlen=1\nh=not-a-policy:value"[..], + &b"v=1\nce=identity\nct=text/html\nlen=1\nh=malformed"[..], + &b"v=1\nce=identity\nct=application/json\nlen=1"[..], + &[0xff, 0xfe][..], + ] { + assert_eq!( + TemplateMetadata::decode(raw), + None, + "malformed metadata must be a miss, not a partial read: {raw:?}" + ); + } + } + + #[test] + fn the_policy_allowlist_covers_document_security_and_delivery_headers() { + for required in [ + "strict-transport-security", + "cross-origin-opener-policy", + "cross-origin-embedder-policy", + "cross-origin-resource-policy", + "origin-agent-cluster", + "reporting-endpoints", + "report-to", + "link", + ] { + assert!( + REPLAYABLE_POLICY_HEADERS.contains(&required), + "warm ESI hits must preserve {required}" + ); + } + } + + #[tokio::test] + async fn the_null_object_reports_unsupported_rather_than_failing() { + // Degrading to per-request transformation keeps the shared modes portable on + // adapters with no cache; erroring would make them Fastly-only outright. + let cache = UnavailableTemplateCache; + assert_eq!( + cache.get(&key()).await.err(), + Some(TemplateCacheMiss::Unsupported) + ); + assert!(matches!( + cache + .put( + &key(), + &TemplateMetadata { + content_encoding: "identity".to_string(), + policy_headers: Vec::new(), + content_type: "text/html".to_string(), + schema_version: TEMPLATE_SCHEMA_VERSION, + body_len: 0, + }, + Vec::new(), + std::time::Duration::from_secs(1) + ) + .await, + Err(TemplateCacheError::Unsupported) + )); + } +} diff --git a/crates/trusted-server-core/src/platform/types.rs b/crates/trusted-server-core/src/platform/types.rs index a39a26430..7a3d09334 100644 --- a/crates/trusted-server-core/src/platform/types.rs +++ b/crates/trusted-server-core/src/platform/types.rs @@ -168,6 +168,16 @@ pub struct RuntimeServices { /// per-request basis by cloning [`RuntimeServices`] with /// [`RuntimeServices::with_kv_store`]. pub(crate) kv_store: Arc, + /// Shared transformed-template cache. Defaults to + /// [`UnavailableTemplateCache`], so adapters without one degrade to transforming + /// per request rather than failing. Spike-only; see + /// [`crate::platform::template_cache`]. + pub(crate) template_cache: Arc, + /// Platform-specific cold-response template assembler. + /// + /// Defaults to [`super::UnavailableTemplateAssembler`]. Core retains a portable + /// byte-seam fallback when this service is unavailable or rejects a document. + pub(crate) template_assembler: Arc, /// Dynamic backend registration and name prediction. pub(crate) backend: Arc, /// Outbound HTTP client abstraction. @@ -223,6 +233,18 @@ impl RuntimeServices { &*self.kv_store } + /// The shared transformed-template cache. Spike-only. + #[must_use] + pub fn template_cache(&self) -> &dyn super::PlatformTemplateCache { + &*self.template_cache + } + + /// Returns the platform-specific cold-response template assembler. + #[must_use] + pub fn template_assembler(&self) -> &dyn super::PlatformTemplateAssembler { + &*self.template_assembler + } + /// Returns the dynamic backend service. #[must_use] pub fn backend(&self) -> &dyn PlatformBackend { @@ -272,6 +294,29 @@ impl RuntimeServices { ..self } } + + /// Returns a clone of this instance with the template cache replaced. + /// + /// Spike-only (#1009). + #[must_use] + pub fn with_template_cache(self, cache: Arc) -> Self { + Self { + template_cache: cache, + ..self + } + } + + /// Returns a clone of this instance with the template assembler replaced. + #[must_use] + pub fn with_template_assembler( + self, + assembler: Arc, + ) -> Self { + Self { + template_assembler: assembler, + ..self + } + } } impl fmt::Debug for RuntimeServices { @@ -290,6 +335,8 @@ pub struct RuntimeServicesBuilder { config_store: Option>, secret_store: Option>, kv_store: Option>, + template_cache: Option>, + template_assembler: Option>, backend: Option>, http_client: Option>, geo: Option>, @@ -303,6 +350,8 @@ impl RuntimeServicesBuilder { config_store: None, secret_store: None, kv_store: None, + template_cache: None, + template_assembler: None, backend: None, http_client: None, geo: None, @@ -325,6 +374,23 @@ impl RuntimeServicesBuilder { self } + /// Set the shared transformed-template cache. Spike-only. + #[must_use] + pub fn template_cache(mut self, cache: Arc) -> Self { + self.template_cache = Some(cache); + self + } + + /// Set the platform-specific cold-response template assembler. + #[must_use] + pub fn template_assembler( + mut self, + assembler: Arc, + ) -> Self { + self.template_assembler = Some(assembler); + self + } + /// Set the KV store implementation. #[must_use] pub fn kv_store(mut self, kv_store: Arc) -> Self { @@ -387,6 +453,14 @@ impl RuntimeServicesBuilder { kv_store: self .kv_store .expect("should set kv_store before building RuntimeServices"), + // Defaulted rather than required: an adapter with no template cache + // should degrade to transforming per request, not fail to build. + template_cache: self + .template_cache + .unwrap_or_else(|| Arc::new(super::UnavailableTemplateCache)), + template_assembler: self + .template_assembler + .unwrap_or_else(|| Arc::new(super::UnavailableTemplateAssembler)), backend: self .backend .expect("should set backend before building RuntimeServices"), diff --git a/crates/trusted-server-core/src/proxy.rs b/crates/trusted-server-core/src/proxy.rs index ea0a0cf8d..14485328a 100644 --- a/crates/trusted-server-core/src/proxy.rs +++ b/crates/trusted-server-core/src/proxy.rs @@ -13,6 +13,11 @@ use std::sync::{Arc, LazyLock, Mutex}; use std::time::Duration; use web_time::{SystemTime, UNIX_EPOCH}; +use crate::cache_policy::{ + CachePolicy, EdgeCacheHeader, NO_STORE_PRIVATE_CACHE_CONTROL, + apply_no_store_private_to_headers, cache_control_headers_are_private_or_no_store, + remove_edge_cache_headers, +}; use crate::constants::{ HEADER_ACCEPT, HEADER_ACCEPT_ENCODING, HEADER_ACCEPT_LANGUAGE, HEADER_REFERER, HEADER_USER_AGENT, HEADER_X_FORWARDED_FOR, @@ -96,7 +101,7 @@ const ASSET_PROXY_STRIP_RESPONSE_HEADERS: [&str; 3] = ["set-cookie", "strict-transport-security", "clear-site-data"]; /// Cache-control value used when asset proxy responses must not be stored. -pub const ASSET_NO_STORE_PRIVATE_CACHE_CONTROL: &str = "no-store, private"; +pub const ASSET_NO_STORE_PRIVATE_CACHE_CONTROL: &str = NO_STORE_PRIVATE_CACHE_CONTROL; /// Cache policy metadata emitted by the asset proxy handler. /// @@ -109,13 +114,32 @@ pub enum AssetProxyCachePolicy { OriginControlled, /// Reapply `Cache-Control: no-store, private` after standard finalization. NoStorePrivate, + /// Reapply an operator-selected normalized cache policy after finalization. + /// + /// The adapter must call [`Self::apply_after_route_finalization`] after + /// standard response and privacy finalization, passing its runtime + /// [`EdgeCacheHeader`]. Asset rehosting is Fastly-only today; a future + /// adapter must preserve this finalization step to emit its edge directive. + Normalized(CachePolicy), } impl AssetProxyCachePolicy { /// Apply protected cache headers after route-level response finalization. - pub fn apply_after_route_finalization(self, response: &mut Response) { - if self == Self::NoStorePrivate { - apply_no_store_cache_control(response); + pub fn apply_after_route_finalization( + self, + response: &mut Response, + edge_header: EdgeCacheHeader, + ) { + match self { + Self::OriginControlled => {} + Self::NoStorePrivate => apply_no_store_cache_control(response), + Self::Normalized(policy) => { + if cache_control_headers_are_private_or_no_store(response.headers()) { + remove_edge_cache_headers(response.headers_mut()); + } else { + policy.apply_to_headers(response.headers_mut(), edge_header); + } + } } } } @@ -169,6 +193,11 @@ impl AssetProxyResponse { apply_no_store_cache_control(&mut self.response); } + fn apply_normalized_cache_policy(&mut self, policy: CachePolicy) { + self.cache_policy = AssetProxyCachePolicy::Normalized(policy); + policy.apply_to_headers(self.response.headers_mut(), EdgeCacheHeader::None); + } + /// Return cache policy metadata for router finalization. #[must_use] pub fn cache_policy(&self) -> AssetProxyCachePolicy { @@ -1020,10 +1049,7 @@ fn strip_asset_proxy_response_headers(response: &mut Response) { } fn apply_no_store_cache_control(response: &mut Response) { - response.headers_mut().insert( - header::CACHE_CONTROL, - HeaderValue::from_static(ASSET_NO_STORE_PRIVATE_CACHE_CONTROL), - ); + apply_no_store_private_to_headers(response.headers_mut()); } fn should_preflight_s3( @@ -1206,6 +1232,13 @@ pub async fn handle_asset_proxy_request( let mut response = platform_response_to_fastly_asset(platform_resp); strip_asset_proxy_response_headers(response.response_mut()); + let status = response.response().status(); + if (status.is_success() || status == StatusCode::NOT_MODIFIED) + && let Some(policy) = settings.asset_cache_policy_for_path(incoming_path)? + { + response.apply_normalized_cache_policy(policy); + } + Ok(response) } @@ -2167,6 +2200,7 @@ mod tests { use std::io; use std::rc::Rc; use std::sync::{Arc, Mutex}; + use std::time::Duration; use super::{ AssetProxyCachePolicy, IMAGE_FALLBACK_CONTENT_TYPE, ProxyRequestConfig, @@ -2177,6 +2211,7 @@ mod tests { proxy_request, rebuild_response_with_body, reconstruct_and_validate_signed_target, redirect_is_permitted, stream_asset_body, }; + use crate::cache_policy::{CachePolicy, EdgeCacheHeader}; use crate::constants::{HEADER_ACCEPT, HEADER_X_FORWARDED_FOR}; use crate::creative; use crate::error::{IntoHttpResponse, TrustedServerError}; @@ -2191,9 +2226,9 @@ mod tests { use crate::settings::{ AssetImageOptimizerConfig, AssetOriginAuth, ImageOptimizerAspectRatioConfig, ImageOptimizerCropOffsetsConfig, ImageOptimizerProfileSet, ImageOptimizerSettings, - OriginQueryPolicy, ProxyAssetRoute, S3SigV4AuthConfig, UnknownProfilePolicy, + OriginQueryPolicy, ProxyAssetRoute, S3SigV4AuthConfig, Settings, UnknownProfilePolicy, }; - use crate::test_support::tests::create_test_settings; + use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; use bytes::Bytes; use edgezero_core::body::Body as EdgeBody; use edgezero_core::http::response_builder as edge_response_builder; @@ -4304,6 +4339,167 @@ mod tests { }); } + #[test] + fn handle_asset_proxy_request_replaces_third_party_cache_policy_for_rehosted_asset() { + futures::executor::block_on(async { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"asset".to_vec(), + vec![(header::CACHE_CONTROL.as_str(), "no-store")], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let settings = Settings::from_toml(&format!( + r#"{} + + [[cache.asset_rules]] + id = "fingerprinted-assets" + enabled = true + path_globs = ["/assets/**/*.js"] + fingerprint_style = "hex" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + )) + .expect("should parse settings with cache asset rule"); + let req = build_http_request( + Method::GET, + "https://www.example.com/assets/app.0123abcd.js", + ); + let route = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); + + let asset_response = handle_asset_proxy_request(&settings, &services, req, &route) + .await + .expect("should proxy asset request"); + assert_eq!( + asset_response.cache_policy(), + AssetProxyCachePolicy::Normalized(CachePolicy::public_immutable( + Duration::from_secs(31_536_000) + )), + "should carry normalized cache policy metadata" + ); + + let mut response = asset_response + .into_response() + .expect("should return buffered asset response"); + assert_eq!( + response_header(&response, header::CACHE_CONTROL), + Some("public, max-age=31536000, immutable"), + "configured rehost policy should replace the third-party no-store directive" + ); + assert!( + response.headers().get("surrogate-control").is_none(), + "runtime-specific edge header should wait for adapter finalization" + ); + + AssetProxyCachePolicy::Normalized(CachePolicy::public_immutable(Duration::from_secs( + 31_536_000, + ))) + .apply_after_route_finalization(&mut response, EdgeCacheHeader::SurrogateControl); + assert_eq!( + response + .headers() + .get("surrogate-control") + .and_then(|value| value.to_str().ok()), + Some("max-age=31536000"), + "Fastly finalization should render Surrogate-Control" + ); + }); + } + + #[test] + fn normalized_asset_policy_preserves_final_private_or_no_store_directives() { + for cache_control in ["private, max-age=0", "no-store"] { + let mut response = edge_response_builder() + .header(header::CACHE_CONTROL, cache_control) + .header("surrogate-control", "max-age=31536000") + .header("cdn-cache-control", "max-age=31536000") + .header("cloudflare-cdn-cache-control", "max-age=31536000") + .body(EdgeBody::empty()) + .expect("should build asset response"); + + AssetProxyCachePolicy::Normalized(CachePolicy::public_immutable(Duration::from_secs( + 31_536_000, + ))) + .apply_after_route_finalization(&mut response, EdgeCacheHeader::SurrogateControl); + + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some(cache_control), + "final privacy directive should veto normalized cache policy" + ); + assert!( + [ + "surrogate-control", + "cdn-cache-control", + "cloudflare-cdn-cache-control", + ] + .iter() + .all(|name| !response.headers().contains_key(*name)), + "final privacy directive should remove every edge-cache header" + ); + } + } + + #[test] + fn handle_asset_proxy_request_leaves_non_matching_assets_origin_controlled() { + futures::executor::block_on(async { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"asset".to_vec(), + vec![(header::CACHE_CONTROL.as_str(), "public, max-age=60")], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let settings = Settings::from_toml(&format!( + r#"{} + + [[cache.asset_rules]] + id = "fingerprinted-assets" + enabled = true + path_globs = ["/assets/**/*.js"] + fingerprint_style = "hex" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + )) + .expect("should parse settings with cache asset rule"); + let req = build_http_request(Method::GET, "https://www.example.com/assets/app.js"); + let route = ProxyAssetRoute::new("/assets/", "https://assets.example.com"); + + let asset_response = handle_asset_proxy_request(&settings, &services, req, &route) + .await + .expect("should proxy asset request"); + + assert_eq!( + asset_response.cache_policy(), + AssetProxyCachePolicy::OriginControlled, + "non-fingerprinted file should not receive normalized immutable policy" + ); + let response = asset_response + .into_response() + .expect("should return buffered asset response"); + assert_eq!( + response_header(&response, header::CACHE_CONTROL), + Some("public, max-age=60"), + "origin-controlled response should preserve origin cache header" + ); + }); + } + fn test_profile_set() -> ImageOptimizerProfileSet { let mut profiles = HashMap::new(); profiles.insert("default".to_string(), "width=1920".to_string()); diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index d95f82567..abdd962b9 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -20,8 +20,9 @@ use std::borrow::Cow; use std::io::Write; +use std::sync::atomic::{AtomicBool, Ordering}; use std::sync::{Arc, Mutex}; -use std::time::Duration; +use std::time::{Duration, Instant, SystemTime}; use brotli::Decompressor; use brotli::enc::BrotliEncoderParams; @@ -39,7 +40,9 @@ use crate::auction::endpoints::{ }; use crate::auction::formats::sanitize_publisher_page_url; use crate::auction::orchestrator::{ - AuctionOrchestrator, DispatchAuctionOutcome, DispatchedAuction, + AuctionOrchestrator, DispatchAuctionOutcome, DispatchedAuction, ERROR_TYPE_ALL, + ERROR_TYPE_HTTP_STATUS, ERROR_TYPE_LAUNCH_FAILED, ERROR_TYPE_PARSE_RESPONSE, + ERROR_TYPE_TIMEOUT, ERROR_TYPE_TRANSPORT, }; use crate::auction::telemetry::{ AuctionObservationContext, AuctionSource, AuctionTerminalOutcome, build_auction_events, @@ -48,20 +51,33 @@ use crate::auction::telemetry::{ use crate::auction::types::{ AuctionContext, AuctionRequest, Bid, DeviceInfo, PublisherInfo, SiteInfo, UserInfo, }; +use crate::cache_policy::{ + CachePolicy, EdgeCacheHeader, cache_control_headers_are_private_or_no_store, +}; use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; use crate::constants::{COOKIE_TS_EIDS, HEADER_X_COMPRESS_HINT}; use crate::cookies::handle_request_cookies; +use crate::creative_opportunities::{AssemblyMode, CreativeOpportunitiesConfig}; use crate::ec::EcContext; use crate::ec::kv::KvIdentityGraph; use crate::ec::registry::PartnerRegistry; use crate::error::TrustedServerError; +use crate::html_processor::BodyCloseInjection; use crate::http_util::{RequestInfo, is_navigation_request, serve_static_with_etag}; use crate::integrations::IntegrationRegistry; -use crate::platform::{GeoInfo, PlatformBackendSpec, PlatformHttpRequest, RuntimeServices}; +use crate::platform::{ + GeoInfo, PlatformBackendSpec, PlatformHttpRequest, RuntimeServices, VarySpec, + contains_publisher_esi_directive, +}; use crate::price_bucket::{PriceGranularity, price_bucket}; -use crate::response_privacy::enforce_synthesized_html_cache_privacy; +use crate::response_privacy::{ + enforce_synthesized_html_cache_privacy, enforce_terminal_private_cache_privacy, +}; use crate::rsc_flight::RscFlightUrlRewriter; -use crate::settings::Settings; +use crate::settings::{ + AUCTION_DEBUG_METADATA_ALLOWLIST, AUCTION_DEBUG_UPSTREAM_METADATA_KEYS, + AuctionDebugCommentFormat, AuctionDebugCommentOptions, AuctionDebugCommentVerbosity, Settings, +}; use crate::streaming_processor::{ BodyStreamDecoder, BodyStreamEncoder, Compression, GzipDecodeReader, PipelineConfig, STREAM_CHUNK_SIZE, StreamProcessor, StreamingPipeline, @@ -70,6 +86,71 @@ use crate::streaming_replacer::create_url_replacer; const SUPPORTED_ENCODING_VALUES: [&str; 3] = ["gzip", "deflate", "br"]; const DEFAULT_PUBLISHER_FIRST_BYTE_TIMEOUT: Duration = Duration::from_secs(15); +const HEADER_X_TS_TEMPLATE_CACHE: &str = "x-ts-template-cache"; +const HEADER_X_TS_ASSEMBLY: &str = "x-ts-assembly"; + +#[derive(Clone, Copy, PartialEq, Eq)] +enum TemplateCacheResponseState { + Hit, + MissReserved, + MissStored, + MissStoreError, + BypassRequest, + BypassResponse, + Unsupported, + Invalid, + BackendError, +} + +impl TemplateCacheResponseState { + const fn as_str(self) -> &'static str { + match self { + Self::Hit => "hit", + Self::MissReserved => "miss-reserved", + Self::MissStored => "miss-stored", + Self::MissStoreError => "miss-store-error", + Self::BypassRequest => "bypass-request", + Self::BypassResponse => "bypass-response", + Self::Unsupported => "unsupported", + Self::Invalid => "invalid", + Self::BackendError => "backend-error", + } + } +} + +fn set_template_cache_response_state( + response: &mut Response, + state: TemplateCacheResponseState, +) { + response.headers_mut().insert( + HEADER_X_TS_TEMPLATE_CACHE, + HeaderValue::from_static(state.as_str()), + ); +} + +#[derive(Clone, Copy, PartialEq, Eq)] +enum AssemblyResponseState { + EsiParser, + ByteSeamFallback, + ByteSeam, +} + +impl AssemblyResponseState { + const fn as_str(self) -> &'static str { + match self { + Self::EsiParser => "esi-parser", + Self::ByteSeamFallback => "byte-seam-fallback", + Self::ByteSeam => "byte-seam", + } + } +} + +fn set_assembly_response_state(response: &mut Response, state: AssemblyResponseState) { + response.headers_mut().insert( + HEADER_X_TS_ASSEMBLY, + HeaderValue::from_static(state.as_str()), + ); +} fn body_as_reader( body: EdgeBody, @@ -201,11 +282,16 @@ fn restrict_accept_encoding(req: &mut Request) { // origin responds without compression. Adding encodings here would cause the // origin to compress its response even though the client never asked for it, // and the client would then receive content it cannot decode. + if !req.headers().contains_key(header::ACCEPT_ENCODING) { + return; + } let Some(current) = req .headers() - .get(header::ACCEPT_ENCODING) - .and_then(|value| value.to_str().ok()) - .map(str::to_owned) + .get_all(header::ACCEPT_ENCODING) + .iter() + .map(|value| value.to_str().ok()) + .collect::>>() + .map(|values| values.join(", ")) else { return; }; @@ -273,6 +359,158 @@ fn accept_encoding_qvalue(header_value: &str, target: &str) -> Option { matched_qvalue } +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +enum ReaderEncodingError { + Malformed, + NoAcceptableEncoding, +} + +fn parse_quality_value(value: &str) -> Option { + let value = value.trim(); + let (whole, fraction) = value + .split_once('.') + .map_or((value, None), |(whole, fraction)| (whole, Some(fraction))); + let fraction_is_valid = fraction.is_none_or(|fraction| { + fraction.len() <= 3 && fraction.bytes().all(|byte| byte.is_ascii_digit()) + }); + if !fraction_is_valid { + return None; + } + match whole { + "0" => value.parse().ok(), + "1" if fraction.is_none_or(|fraction| fraction.bytes().all(|byte| byte == b'0')) => { + Some(1.0) + } + _ => None, + } +} + +fn negotiate_reader_compression( + headers: &edgezero_core::http::HeaderMap, +) -> Result { + if !headers.contains_key(header::ACCEPT_ENCODING) { + return Ok(Compression::None); + } + + let mut qualities = Vec::<(String, f32)>::new(); + for field in headers.get_all(header::ACCEPT_ENCODING) { + let field = field.to_str().map_err(|_| ReaderEncodingError::Malformed)?; + for item in field + .split(',') + .map(str::trim) + .filter(|item| !item.is_empty()) + { + let mut parts = item.split(';'); + let token = parts + .next() + .map(str::trim) + .filter(|token| !token.is_empty()) + .ok_or(ReaderEncodingError::Malformed)? + .to_ascii_lowercase(); + if token != "*" && http::HeaderName::from_bytes(token.as_bytes()).is_err() { + return Err(ReaderEncodingError::Malformed); + } + let mut quality = 1.0; + let mut saw_quality = false; + for parameter in parts { + let (name, value) = parameter + .trim() + .split_once('=') + .ok_or(ReaderEncodingError::Malformed)?; + if !name.trim().eq_ignore_ascii_case("q") || saw_quality { + return Err(ReaderEncodingError::Malformed); + } + quality = parse_quality_value(value).ok_or(ReaderEncodingError::Malformed)?; + saw_quality = true; + } + if qualities.iter().any(|(seen, _)| seen == &token) { + return Err(ReaderEncodingError::Malformed); + } + qualities.push((token, quality)); + } + } + + let explicit = |name: &str| { + qualities + .iter() + .find_map(|(candidate, quality)| (candidate == name).then_some(*quality)) + }; + let wildcard = explicit("*"); + let quality_for = |name: &str| explicit(name).or(wildcard).unwrap_or(0.0); + // Identity is implicitly acceptable at q=1 unless explicitly excluded, or a + // wildcard q=0 excludes every unlisted coding. + let identity_quality = + explicit("identity").unwrap_or_else(|| if wildcard == Some(0.0) { 0.0 } else { 1.0 }); + + let candidates = [ + (Compression::Brotli, quality_for("br")), + (Compression::Gzip, quality_for("gzip")), + (Compression::Deflate, quality_for("deflate")), + (Compression::None, identity_quality), + ]; + let mut selected = None; + for (compression, quality) in candidates { + if quality > 0.0 && selected.is_none_or(|(_, best)| quality > best) { + selected = Some((compression, quality)); + } + } + selected + .map(|(compression, _)| compression) + .ok_or(ReaderEncodingError::NoAcceptableEncoding) +} + +fn set_response_compression(response: &mut Response, compression: Compression) { + let encoding = match compression { + Compression::None => None, + Compression::Gzip => Some("gzip"), + Compression::Deflate => Some("deflate"), + Compression::Brotli => Some("br"), + }; + if let Some(encoding) = encoding { + response + .headers_mut() + .insert(header::CONTENT_ENCODING, HeaderValue::from_static(encoding)); + } else { + response.headers_mut().remove(header::CONTENT_ENCODING); + } + let varies_on_encoding = response + .headers() + .get_all(header::VARY) + .iter() + .any(|value| { + value.to_str().is_ok_and(|value| { + value + .split(',') + .any(|name| name.trim().eq_ignore_ascii_case("accept-encoding")) + }) + }); + if !varies_on_encoding { + response + .headers_mut() + .append(header::VARY, HeaderValue::from_static("Accept-Encoding")); + } + response.headers_mut().remove(header::CONTENT_LENGTH); +} + +fn response_compression(response: &Response) -> Compression { + response + .headers() + .get(header::CONTENT_ENCODING) + .and_then(|value| value.to_str().ok()) + .map(Compression::from_content_encoding) + .unwrap_or(Compression::None) +} + +fn encode_complete_body( + body: Vec, + compression: Compression, +) -> Result, Report> { + let mut encoder = BodyStreamEncoder::new(compression); + let mut encoded = encoder.encode_chunk(body)?; + encoded.extend_from_slice(&encoder.finish()?); + Ok(encoded) +} + /// Unified tsjs static serving: `/static/tsjs=` /// /// Serves two types of bundles: @@ -287,6 +525,7 @@ fn accept_encoding_qvalue(header_value: &str, target: &str) -> Option { pub fn handle_tsjs_dynamic( req: &Request, integration_registry: &IntegrationRegistry, + edge_header: EdgeCacheHeader, ) -> Result, Report> { const PREFIX: &str = "/static/tsjs="; const UNIFIED_FILENAMES: &[&str] = &["tsjs-unified.js", "tsjs-unified.min.js"]; @@ -298,13 +537,11 @@ pub fn handle_tsjs_dynamic( let filename = &path[PREFIX.len()..]; if UNIFIED_FILENAMES.contains(&filename) { - // Serve core + immediate modules (excludes deferred like prebid) + // Serve core + immediate modules (excludes deferred like prebid). let module_ids = integration_registry.js_module_ids_immediate(); let body = trusted_server_js::concatenate_modules(&module_ids); - let mut resp = serve_static_with_etag(&body, req, "application/javascript; charset=utf-8"); - resp.headers_mut() - .insert(HEADER_X_COMPRESS_HINT, HeaderValue::from_static("on")); - return Ok(resp); + let hash = trusted_server_js::concatenated_hash(&module_ids); + return Ok(serve_tsjs_static(req, &body, &hash, edge_header)); } if let Some(module_id) = parse_single_module_filename(filename) { @@ -318,18 +555,46 @@ pub fn handle_tsjs_dynamic( if !deferred_ids.contains(&module_id) && !diagnostics_standalone { return Ok(not_found_response()); } - if let Some(content) = trusted_server_js::module_bundle(module_id) { - let mut resp = - serve_static_with_etag(content, req, "application/javascript; charset=utf-8"); - resp.headers_mut() - .insert(HEADER_X_COMPRESS_HINT, HeaderValue::from_static("on")); - return Ok(resp); + if let (Some(content), Some(hash)) = ( + trusted_server_js::module_bundle(module_id), + trusted_server_js::single_module_hash(module_id), + ) { + return Ok(serve_tsjs_static(req, content, hash, edge_header)); } } Ok(not_found_response()) } +fn serve_tsjs_static( + req: &Request, + body: &str, + expected_hash: &str, + edge_header: EdgeCacheHeader, +) -> Response { + let mut response = serve_static_with_etag( + body, + req, + "application/javascript; charset=utf-8", + edge_header, + ); + if request_version_hash(req).is_some_and(|hash| hash == expected_hash) { + CachePolicy::public_immutable(Duration::from_secs(31_536_000)) + .apply_to_headers(response.headers_mut(), edge_header); + } + response + .headers_mut() + .insert(HEADER_X_COMPRESS_HINT, HeaderValue::from_static("on")); + response +} + +fn request_version_hash(req: &Request) -> Option<&str> { + req.uri().query()?.split('&').find_map(|pair| { + let (name, value) = pair.split_once('=')?; + (name == "v").then_some(value) + }) +} + /// Extract a module ID from a deferred-module filename like `tsjs-sourcepoint.min.js`. /// /// Returns `Some(&'static str)` if the filename matches a known JS module ID, @@ -361,6 +626,10 @@ struct ProcessResponseParams<'a> { suppress_datadome_client_side_tag: bool, gpt_diagnostics: Option<&'a crate::integrations::gpt_diagnostics::GptDiagnosticsRequestDecision>, + /// See [`HtmlStreamProcessorParams::shared_template_authorized`]. + shared_template_authorized: bool, + /// See [`HtmlStreamProcessorParams::csp_nonce_observed`]. + csp_nonce_observed: Option<&'a Arc>, } struct PublisherBodyProcessor { @@ -384,9 +653,11 @@ impl PublisherBodyProcessor { settings, integration_registry, ad_slots_script: params.ad_slots_script.as_deref().map(str::to_string), - ad_bids_state: Arc::clone(¶ms.ad_bids_state), + ad_bids_state: Arc::clone(params.ad_bids_state.script_cell()), suppress_datadome_client_side_tag: params.suppress_datadome_client_side_tag, gpt_diagnostics: params.gpt_diagnostics.clone(), + shared_template_authorized: params.template_cache_key.is_some(), + csp_nonce_observed: params.csp_nonce_observed.clone(), })?) } else if is_rsc_flight { Box::new(RscFlightUrlRewriter::new( @@ -428,6 +699,7 @@ fn process_response_streaming( body: EdgeBody, output: &mut W, params: &ProcessResponseParams, + output_compression: Compression, ) -> Result<(), Report> { let is_html = is_html_content_type(params.content_type); let is_rsc_flight = @@ -443,7 +715,7 @@ fn process_response_streaming( let compression = Compression::from_content_encoding(params.content_encoding); let config = PipelineConfig { input_compression: compression, - output_compression: compression, + output_compression, chunk_size: 8192, }; // Bound how much decoded gzip output may sit in the heap at once, using the @@ -465,6 +737,8 @@ fn process_response_streaming( ad_bids_state: params.ad_bids_state.clone(), suppress_datadome_client_side_tag: params.suppress_datadome_client_side_tag, gpt_diagnostics: params.gpt_diagnostics.cloned(), + shared_template_authorized: params.shared_template_authorized, + csp_nonce_observed: params.csp_nonce_observed.cloned(), })?; StreamingPipeline::new(config, processor) .with_max_pending_decoded_bytes(max_pending_decoded_bytes) @@ -509,13 +783,21 @@ async fn process_response_streaming_async( params.content_encoding ); - let compression = Compression::from_content_encoding(¶ms.content_encoding); + let input_compression = Compression::from_content_encoding(¶ms.content_encoding); + // A template-cache response is always identity bytes. Decode during the transform instead of + // recompressing and immediately decoding the entire buffered result afterwards. + let output_compression = if params.template_cache_key.is_some() { + Compression::None + } else { + input_compression + }; let mut processor = PublisherBodyProcessor::new(params, settings, integration_registry)?; process_body_chunks_async( body, output, &mut processor, - compression, + input_compression, + output_compression, settings.publisher.max_buffered_body_bytes, ) .await @@ -557,11 +839,12 @@ async fn process_body_chunks_async( body: EdgeBody, writer: &mut W, processor: &mut P, - compression: Compression, + input_compression: Compression, + output_compression: Compression, max_body_bytes: usize, ) -> Result<(), Report> { - let mut decoder = BodyStreamDecoder::new(compression, max_body_bytes); - let mut encoder = BodyStreamEncoder::new(compression); + let mut decoder = BodyStreamDecoder::new(input_compression, max_body_bytes); + let mut encoder = BodyStreamEncoder::new(output_compression); let mut source = BodyChunkSource::new(body, STREAM_CHUNK_SIZE).with_max_bytes(max_body_bytes); while let Some(segments) = @@ -950,6 +1233,154 @@ struct HtmlStreamProcessorParams<'a> { ad_bids_state: Arc>>, suppress_datadome_client_side_tag: bool, gpt_diagnostics: Option, + /// Whether a shared template was authorized for this response. + /// + /// Carried rather than re-derived so both seams see the same answer. See + /// [`effective_assembly_mode`]. + shared_template_authorized: bool, + /// Where the transform records a response-bound CSP nonce, when one matters. + csp_nonce_observed: Option>, +} + +/// The diagnostics decision the template may carry. +/// +/// Diagnostics is request-scoped — activated by a cookie or query parameter, and +/// documented as an immutable per-request decision — so it must not reach a shared +/// template. +/// +/// It does not leak today even without this gate, but only by coincidence: +/// `requires_private_no_store()` is a strict superset of the conditions under which +/// a script is emitted, and that stamp lands before the template cache gate reads response +/// headers, so the gate refuses. Two independent conditions that happen to align, +/// with nothing enforcing the relationship. This makes the guarantee explicit; +/// `requires_private_no_store_is_a_superset_of_injection` keeps the coincidence as a +/// backstop if this gate is ever removed. +pub(crate) fn template_gpt_diagnostics( + mode: AssemblyMode, + decision: Option, +) -> Option { + match mode { + AssemblyMode::Inline => decision, + AssemblyMode::Esi => None, + } +} + +/// The marker emitted at the `` seam under [`AssemblyMode::Esi`], reserving the +/// place this reader's slots and bids are spliced into. +/// +/// An inert HTML comment, deliberately. Template schema v1 used an executable ESI include +/// tag here, when the `esi` crate resolved it at the edge. That crate was removed from the +/// render path because it truncates any element larger than its 16 KB chunk size, and +/// nothing has parsed ESI since. What remained was a tag that *looked* executable, would +/// have been executed by any ESI-enabled layer in front of us, and renders as text in a +/// browser if assembly is ever skipped. A comment cannot do any of those things: an +/// unassembled template degrades to a page with no ads rather than a page with a visible +/// tag. +/// +/// Carries no URL. Every byte here is a byte every reader of the shared template +/// receives, so nothing request-scoped may appear, and keeping a URL out also removes +/// any escaping question at the seam. +pub const AD_ASSEMBLY_SEAM: &str = ""; + +/// Transform-owned stand-in for the seam, emitted at the document's structural body end. +/// +/// Deliberately *not* [`AD_ASSEMBLY_SEAM`]. The payload that ends up in the seam is not +/// known until the completed transform has been checked for publisher collisions, and the +/// position is not knowable from the output bytes: a reverse search for `` selects +/// a string literal in `` when the document has +/// no real close, and prefers a `` sequence in trailing comment data over the real +/// closing tag. Only the parser knows which one is structural, so the parser marks the +/// spot and the substitution below fills it in. +/// +/// Keeping it distinct from [`AD_ASSEMBLY_SEAM`] is what lets a publisher document that +/// contains the seam bytes still receive correctly positioned bids: that collision +/// revokes the shared reservation without disturbing this placeholder. +pub(crate) const TEMPLATE_SEAM_PLACEHOLDER: &str = ""; + +/// The mode the operator asked for, before availability is taken into account. +/// +/// Spelled once, because the mode has to mean the same thing at the cache key, at the +/// seam, and at both hit finalizers. Every one of those re-derived it from the same +/// `Option` chain, and the finalizers had no way to ask at all — which is why they +/// demanded a seam marker of a mode that emits none. +fn configured_assembly_mode(settings: &Settings) -> AssemblyMode { + settings + .creative_opportunities + .as_ref() + .map(CreativeOpportunitiesConfig::assembly_mode) + .unwrap_or_default() +} + +/// Whether this mode's completed template receives [`AD_ASSEMBLY_SEAM`]. +/// +/// The property that decides whether a template is *expected* to have a hole in it, and +/// therefore whether the absence of one is a defect or the design. Only `Esi` splices per +/// reader. +/// +/// Matched exhaustively rather than compared against `Esi`, so a new mode has to state +/// its answer here instead of silently inheriting one. +fn mode_emits_seam_marker(mode: AssemblyMode) -> bool { + match mode { + AssemblyMode::Inline => false, + AssemblyMode::Esi => true, + } +} + +/// The assembly mode this response will actually be delivered under. +/// +/// The configured mode says what the operator wants; the cache key says whether it is +/// available. A shared mode with no key means the gate refused this response — the +/// origin set a cookie, declared a `Vary` the key does not cover, returned a non-200, +/// and so on — so there is no shared template to build and nothing downstream will +/// assemble one. +/// +/// When that happens the request falls back to [`AssemblyMode::Inline`] **entirely**, +/// at every seam. Falling back at one seam and not another is what produced the failure +/// this function exists to prevent: the `` seam emitted a legacy ESI tag because +/// the mode was `Esi`, while assembly was skipped because there was no key, so the reader +/// received a document with unresolved executable ESI markup in it and no bids at all. +/// +/// Bypassing is the *normal* case against a real origin, not an edge case, so this path +/// runs far more often than the shared one. +fn effective_assembly_mode(settings: &Settings, shared_template_authorized: bool) -> AssemblyMode { + let configured = configured_assembly_mode(settings); + if matches!(configured, AssemblyMode::Inline) || shared_template_authorized { + return configured; + } + log::debug!( + "assembly mode {configured:?} is unavailable for this response (no shared template \ + was authorized); falling back to inline" + ); + AssemblyMode::Inline +} + +/// What the streaming HTML processor should inject at ``. +/// +/// Explicit rather than inferred. The previous shape read +/// `ad_slots_script.is_some()` inside the element handler, which silently coupled +/// two independent decisions: once [`template_ad_slots_script`] stopped emitting a +/// head script under a shared mode, body-close injection stopped with it. +/// +/// `Esi` emits [`TEMPLATE_SEAM_PLACEHOLDER`], not the seam itself. What goes into the +/// seam is still decided after the completed transform has been checked for publisher +/// collisions and ESI directives — but *where* it goes has to be decided here, by the +/// parser, because the output bytes cannot distinguish a structural `` from one +/// written inside a script string or a trailing comment. +pub(crate) fn body_close_injection( + mode: AssemblyMode, + head_script_present: bool, +) -> BodyCloseInjection { + match mode { + // Per-navigation and never shared, so gating on slot presence is correct. + AssemblyMode::Inline => { + if head_script_present { + BodyCloseInjection::InlineBids + } else { + BodyCloseInjection::None + } + } + AssemblyMode::Esi => BodyCloseInjection::Marker(TEMPLATE_SEAM_PLACEHOLDER.to_string()), + } } fn create_html_stream_processor( @@ -963,10 +1394,26 @@ fn create_html_stream_processor( params.origin_host, params.request_host, params.request_scheme, - ) - .with_ad_state(params.ad_slots_script, params.ad_bids_state) - .with_gpt_diagnostics(params.gpt_diagnostics) - .with_datadome_client_tag_suppression(params.suppress_datadome_client_side_tag); + ); + + let assembly_mode = effective_assembly_mode(params.settings, params.shared_template_authorized); + let body_close = body_close_injection(assembly_mode, params.ad_slots_script.is_some()); + + let gpt_diagnostics = template_gpt_diagnostics(assembly_mode, params.gpt_diagnostics); + + // Only a response that can be stored has a consumer for the observation, so the + // handlers are not registered for ordinary inline traffic. + let csp_nonce_observed = params + .shared_template_authorized + .then_some(params.csp_nonce_observed) + .flatten(); + + let config = config + .with_ad_state(params.ad_slots_script, params.ad_bids_state) + .with_gpt_diagnostics(gpt_diagnostics) + .with_body_close(body_close) + .with_csp_nonce_observer(csp_nonce_observed) + .with_datadome_client_tag_suppression(params.suppress_datadome_client_side_tag); Ok(create_html_processor(config)) } @@ -1000,6 +1447,27 @@ pub enum PublisherResponse { /// Parameters for [`process_response_streaming`]. params: Box, }, + /// A shared template read from template cache, to be assembled on the way out. + /// + /// Distinct from [`Self::Stream`] because the bytes are **already transformed** — + /// running them through `lol_html` again would inject a second tsjs `"#.to_string() + }))); + + let config = HtmlProcessorConfig { + csp_nonce_observed: None, + origin_host: "origin.example.com".to_string(), + request_host: "example.com".to_string(), + request_scheme: "https".to_string(), + integrations: IntegrationRegistry::empty_for_tests(), + ad_slots_script, + ad_bids_state, + max_buffered_body_bytes: 16 * 1024 * 1024, + gpt_diagnostics, + body_close, + suppress_datadome_client_side_tag: false, + }; - struct RangeAwareHttpClient { - stub: StubHttpClient, + let mut processor = create_html_processor(config); + let out = processor + .process_chunk(DOCUMENT, true) + .expect("should process the document"); + String::from_utf8(out).expect("output should be utf8") } - impl RangeAwareHttpClient { - fn new() -> Self { - Self { - stub: StubHttpClient::new(), + fn every_shape() -> Vec { + let mut shapes = Vec::new(); + for ad_stack_ran in [false, true] { + for diagnostics_active in [false, true] { + for bids_available in [false, true] { + shapes.push(RequestShape { + ad_stack_ran, + diagnostics_active, + bids_available, + }); + } } } + shapes } - #[async_trait::async_trait(?Send)] - impl PlatformHttpClient for RangeAwareHttpClient { - async fn send( - &self, - request: PlatformHttpRequest, - ) -> Result> { - if request.request.headers().contains_key(header::RANGE) { - self.stub.push_response_with_headers( - 206, - b"partial".to_vec(), - vec![ - ("content-type", "text/html; charset=utf-8"), - ("content-range", "bytes 0-18/39"), - ], - ); - } else { - self.stub.push_response_with_headers( - 200, - b"origin".to_vec(), - vec![("content-type", "text/html; charset=utf-8")], - ); - } - self.stub.send(request).await - } + #[test] + fn shared_template_ad_seam_is_readable_and_versioned() { + assert_eq!( + (crate::platform::TEMPLATE_SCHEMA_VERSION, AD_ASSEMBLY_SEAM,), + (4, ""), + "the readable seam and its cache schema must move together" + ); + assert_eq!( + body_close_injection(AssemblyMode::Esi, false), + BodyCloseInjection::Marker(TEMPLATE_SEAM_PLACEHOLDER.to_string()), + "the seam's position must come from the parser, never from a byte search" + ); + assert_ne!( + TEMPLATE_SEAM_PLACEHOLDER, AD_ASSEMBLY_SEAM, + "a publisher document carrying the seam bytes must still receive \ + correctly positioned bids, which a shared marker would make impossible" + ); + } - async fn send_async( - &self, - request: PlatformHttpRequest, - ) -> Result> { + #[test] + fn shared_modes_render_byte_identical_documents_for_every_request_shape() { + let mode = AssemblyMode::Esi; + let shapes = every_shape(); + let baseline = render(mode, shapes[0]); + + for shape in &shapes[1..] { + let rendered = render(mode, *shape); + assert_eq!( + rendered, baseline, + "{mode:?}: rendered template differs for {shape:?}. A shared \ + template that varies by request freezes the first-filling \ + request's decision for every later reader." + ); + } + } + + #[test] + fn shared_mode_templates_contain_no_request_scoped_markers() { + // Byte-identity alone would be satisfied by rendering the same wrong + // thing every time, so also assert the specific things that must be + // absent. + let mode = AssemblyMode::Esi; + let rendered = render( + mode, + RequestShape { + ad_stack_ran: true, + diagnostics_active: true, + bids_available: true, + }, + ); + for forbidden in [ + ".adSlots", + ".bids=", + "__tsjs_gpt_diagnostics_active", + "history.replaceState", + ] { + assert!( + !rendered.contains(forbidden), + "{mode:?}: template contains request-scoped `{forbidden}`:\n{rendered}" + ); + } + } + + #[test] + fn inline_still_varies_by_request_as_it_must() { + // The shared-mode assertions would also pass if rendering were broken + // everywhere. Inline responses are per-navigation and never shared, so + // they *should* differ — this proves the test can tell the difference. + let with_ads = render( + AssemblyMode::Inline, + RequestShape { + ad_stack_ran: true, + diagnostics_active: false, + bids_available: true, + }, + ); + let without = render( + AssemblyMode::Inline, + RequestShape { + ad_stack_ran: false, + diagnostics_active: false, + bids_available: false, + }, + ); + assert_ne!( + with_ads, without, + "inline must still vary by request; if it does not, this harness is \ + not rendering what it claims to" + ); + assert!( + with_ads.contains(".adSlots"), + "inline with a matched slot should carry adSlots" + ); + } + } + + mod template_fingerprint_tests { + use super::*; + + /// Base settings with one integration's config replaced. + /// + /// Edits the parsed `[integrations]` map rather than appending TOML, so the two + /// fixtures differ in exactly the field under test — the base settings already + /// declare `[integrations.prebid]`, and a second table would not parse. + fn settings_with_prebid(enabled: bool, timeout_ms: u32) -> Settings { + let mut settings = create_test_settings(); + settings.integrations.insert( + "prebid".to_string(), + serde_json::json!({ + "enabled": enabled, + "server_url": "https://prebid.example.com/openrtb2/auction", + "external_bundle_url": "https://assets.example.com/prebid/bundle.js", + "timeout": timeout_ms, + }), + ); + settings + } + + #[test] + fn disabling_an_integration_changes_the_fingerprint() { + // The fingerprint was `concatenated_hash(all_module_ids())` — every module + // compiled into the binary, so a constant for that binary. Turning an + // integration off changed the injected `") + .expect("should substitute the placeholder the transform emitted"); + + assert_eq!( + replaced, br#"

article

"#, + "should splice the payload exactly where the parser marked the body end" + ); + } + + #[test] + fn a_body_close_written_in_script_data_does_not_attract_the_seam() { + // The reverse byte search this replaced picked the string literal, spliced a + // `` terminated the + // publisher's script — in the served page and in the stored template alike. + let document = format!( + r#"

article

{TEMPLATE_SEAM_PLACEHOLDER}"# + ) + .into_bytes(); + + let replaced = replace_seam_placeholder(document, b"") + .expect("should substitute the placeholder the transform emitted"); + + assert_eq!( + replaced, + br#"

article

"#, + "should leave a body-close sequence inside script data untouched" + ); + } + + #[test] + fn a_publisher_copy_of_the_placeholder_refuses_substitution() { + let document = format!( + "

{TEMPLATE_SEAM_PLACEHOLDER}

article{TEMPLATE_SEAM_PLACEHOLDER}" + ) + .into_bytes(); + + let (returned, error) = + replace_seam_placeholder(document.clone(), b"") + .expect_err("should refuse a document that collides with the placeholder"); + + assert!( + matches!(error, SeamError::Repeated), + "should name the collision rather than guess an occurrence" + ); + assert_eq!( + returned, document, + "should hand back the publisher document byte for byte" + ); + } + + #[test] + fn a_document_without_the_placeholder_refuses_substitution() { + let document = b"

article

".to_vec(); + + let (returned, error) = + replace_seam_placeholder(document.clone(), b"") + .expect_err("should refuse a document the transform did not mark"); + + assert!( + matches!(error, SeamError::Missing), + "should name the absent placeholder rather than append blindly" + ); + assert_eq!( + returned, document, + "should hand back the document unchanged" + ); + } + + #[test] + fn parser_validation_does_not_change_the_cached_schema() { + assert_eq!(crate::platform::TEMPLATE_SCHEMA_VERSION, 4); + assert_eq!(AD_ASSEMBLY_SEAM, ""); + assert!(!contains_publisher_esi_directive( + AD_ASSEMBLY_SEAM.as_bytes() + )); + } + + /// Shareable HTML that already contains the seam marker. + /// + /// The marker is reserved, but publisher content can still contain it. The + /// transform adds its own terminal placeholder; repeated markers then make the + /// response bypass the template cache rather than requiring normalization. + fn queue_html_that_collides_with_the_marker(stub: &StubHttpClient) { + stub.push_response_with_headers( + 200, + format!( + "

origin

" + ) + .into_bytes(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ], + ); + } + + #[tokio::test] + async fn origin_marker_collision_bypasses_template_cache_without_mutating_publisher_bytes() + { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_html_that_collides_with_the_marker(&stub); + queue_html_that_collides_with_the_marker(&stub); + + let cold = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("cold document should be UTF-8"); + let warm = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("warm document should be UTF-8"); + + assert!( + cold.contains("origin") && cold.contains("window.tsjs"), + "a reserved-comment collision must not turn a valid origin 200 into a 500" + ); + for document in [&cold, &warm] { + assert!( + document.contains(&format!("window.publisherMarker=\"{AD_ASSEMBLY_SEAM}\"")), + "should preserve a publisher marker inside script data: {document}" + ); + } + assert_eq!(stub.recorded_request_uris().len(), 2); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "should not store a template with a publisher marker collision" + ); + } + + #[tokio::test] + async fn html_without_an_explicit_body_gets_a_terminal_seam_instead_of_a_500() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + stub.push_response_with_headers( + 200, + b"
origin fragment
".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ], + ); + + let cold = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("cold fragment should be UTF-8"); + let warm = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("warm fragment should be UTF-8"); + + for document in [&cold, &warm] { + assert!(document.contains("origin fragment")); + assert!(!document.contains(AD_ASSEMBLY_SEAM)); + } + assert_eq!(stub.recorded_request_uris().len(), 1); + assert!( + !cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "the recovered template should remain cacheable" + ); + } + + #[tokio::test] + async fn an_unsplittable_cached_template_is_a_miss_before_any_header_commits() { + // The hit path used to check only that a marker existed *somewhere* and + // leave the exactly-one check to the finalizer. By then the 200 and its + // headers were committed and, on the streaming adapter, the document head + // was already on the wire — so the only available failure was a truncated + // response. Falling back to the origin is a slower correct page. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + // Fill the cache legitimately, then corrupt the stored body in place so the + // entry is found under exactly the key the next request derives. + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + { + let stored_key = cache + .stored_keys + .lock() + .expect("should lock stored keys") + .first() + .cloned() + .expect("the cold request should have stored a template"); + let mut entries = cache.entries.lock().expect("should lock entries"); + let entry = entries + .get_mut(&stored_key.to_cache_key()) + .expect("the stored template should be readable"); + entry.body = format!( + "origin{AD_ASSEMBLY_SEAM}\ + {AD_ASSEMBLY_SEAM}" + ) + .into_bytes(); + } + + // The fall-back must be able to reach the origin. + queue_shareable_html(&stub); + let response = run(&settings, &services, navigation_request()).await; + let status = response.status(); + assert_eq!( + response + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()), + Some("invalid"), + "an invalid-object recovery must be observable without exposing its key" + ); + let served = String::from_utf8(body_of(response).await) + .expect("served document should be UTF-8"); + + assert_eq!( + status, + StatusCode::OK, + "the reader should get a complete page, not a committed-then-broken one" + ); + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "an unusable entry must be treated as a miss and refetched" + ); + assert!( + !served.contains(AD_ASSEMBLY_SEAM), + "no marker may survive into the served page: {served}" + ); + assert!( + served.contains("window.tsjs"), + "and the fallback must still deliver the seam: {served}" + ); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "an unusable fresh object must be purged instead of forcing inline fallback \ + until its TTL expires" + ); + } + + /// The schema version whose templates carried an executable ESI tag at the seam. + /// + /// Pinned as a literal rather than derived from [`TEMPLATE_SCHEMA_VERSION`]: the + /// point is that the current version is *not* this one, and a derived value + /// would move with it and assert nothing. + const ESI_INCLUDE_SCHEMA_VERSION: u32 = 1; + + /// The exact schema-v1 marker retained only as a cache-compatibility fixture. + const LEGACY_ESI_INCLUDE: &str = ""; + + #[tokio::test] + async fn a_template_written_under_the_previous_schema_version_is_never_read() { + // v1 put an executable ESI include at the seam. v2 puts an inert comment + // there and hands slots to the scheduler, so a v1 entry has no marker this + // binary can find. `schema_version` is the only thing keeping the two apart + // — nothing purges template cache on deploy. + assert_ne!( + crate::platform::TEMPLATE_SCHEMA_VERSION, + ESI_INCLUDE_SCHEMA_VERSION, + "the seam marker changed shape, so the schema version must have moved \ + off the value under which the old marker was stored" + ); + + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + // Fill the cache the ordinary way, then plant a predecessor's entry + // alongside it: same request, previous schema version, previous marker. + // Re-keyed from the key the store actually used, so the fixture cannot + // drift from what the request derives. + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + { + let stored_key = cache + .stored_keys + .lock() + .expect("should lock stored keys") + .first() + .cloned() + .expect("the cold request should have stored a template"); + let mut entries = cache.entries.lock().expect("should lock entries"); + let mut predecessor = entries + .get(&stored_key.to_cache_key()) + .cloned() + .expect("the stored template should be readable"); + predecessor.body = + format!("origin{LEGACY_ESI_INCLUDE}") + .into_bytes(); + predecessor.metadata.schema_version = ESI_INCLUDE_SCHEMA_VERSION; + let mut old_key = stored_key; + old_key.schema_version = ESI_INCLUDE_SCHEMA_VERSION; + entries.insert(old_key.to_cache_key(), predecessor); + } + + let served = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("served document should be UTF-8"); + + assert!( + cache + .lookups + .lock() + .expect("should lock lookups") + .iter() + .all(|key| key.schema_version != ESI_INCLUDE_SCHEMA_VERSION), + "no lookup may name a schema version whose templates this binary cannot \ + assemble" + ); + assert!( + !served.contains(", + ) -> Result, Report> { + Ok(Some(GeoInfo { + city: self.0.to_string(), + country: self.0.to_string(), + continent: self.0.to_string(), + latitude: 1.0, + longitude: 2.0, + metro_code: 3, + region: Some(self.0.to_string()), + asn: Some(4), + })) + } + } + + /// One synthetic user: an identity, a consent posture, and a location. + struct SyntheticUser { + ec_id: &'static str, + jurisdiction: crate::consent::jurisdiction::Jurisdiction, + geo_marker: &'static str, + } + + /// Runs one synthetic user against a fresh cache and returns the stored template. + /// + /// Fresh cache per user deliberately: the point is to compare what each *would* + /// store, so sharing a cache would let the first user's entry answer for the + /// second and the comparison would prove nothing. + async fn stored_template_for(user: &SyntheticUser) -> Vec { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = RuntimeServices::builder() + .config_store(Arc::new(NoopConfigStore)) + .secret_store(Arc::new(NoopSecretStore)) + .kv_store(Arc::new(edgezero_core::key_value_store::NoopKvStore)) + .backend(Arc::new(StubBackend)) + .http_client(Arc::clone(&stub) as Arc) + .geo(Arc::new(StubGeo(user.geo_marker))) + .client_info(ClientInfo::default()) + .template_cache( + Arc::clone(&cache) as Arc + ) + .build(); + queue_shareable_html(&stub); + + let orchestrator = Arc::new(AuctionOrchestrator::new(settings.auction.clone())); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let consent = crate::consent::ConsentContext { + jurisdiction: user.jurisdiction.clone(), + ..Default::default() + }; + let mut ec_context = EcContext::new_for_test(Some(user.ec_id.to_string()), consent); + let publisher_response = handle_publisher_request( + &settings, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[article_slot()], + registry: None, + }, + navigation_request(), + EdgeCacheHeader::SMaxageFallback, + ) + .await + .expect("should proxy publisher request"); + let _ = publisher_response_into_streaming_response( + publisher_response, + &Method::GET, + Arc::clone(&settings), + ®istry, + orchestrator, + services.clone(), + ) + .await + .expect("should finalize publisher response"); + + let entries = cache.entries.lock().expect("should lock entries"); + entries + .values() + .next() + .expect("a template should have been stored") + .body + .clone() + } + + #[tokio::test] + async fn two_users_differing_in_identity_consent_and_geo_store_the_same_template() { + // The gate the whole design rests on. The template is shared between + // visitors, so anything request-scoped that reaches it is one visitor's data + // served to the next. Byte-identity is the assertion because it does not + // depend on guessing which field might leak. + let alice = SyntheticUser { + ec_id: "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.alice1", + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + geo_marker: "AliceCity", + }; + let bob = SyntheticUser { + ec_id: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb.bobbb1", + jurisdiction: crate::consent::jurisdiction::Jurisdiction::Gdpr, + geo_marker: "BobCity", + }; + + let alice_template = stored_template_for(&alice).await; + let bob_template = stored_template_for(&bob).await; + + assert_eq!( + alice_template, bob_template, + "two users differing in identity, consent and geo must produce the same \ + shared template" + ); + + // Belt and braces: byte-identity would also hold if *both* templates leaked + // the same wrong thing, so name the values that must be absent. + let template = String::from_utf8(alice_template).expect("template should be utf-8"); + for forbidden in [ + alice.ec_id, + bob.ec_id, + alice.geo_marker, + bob.geo_marker, + "adSlots", + "window.tsjs", + ] { + assert!( + !template.contains(forbidden), + "`{forbidden}` must not appear in a shared template: {template}" + ); + } + } + + #[tokio::test] + async fn inline_mode_never_reads_or_writes_the_cache() { + // The shipped path. If this ever cached, per-user ad state would be shared + // between visitors — the exact failure the whole design exists to avoid. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("inline")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, navigation_request()).await; + let _ = run(&settings, &services, navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "inline must fetch the origin every time" + ); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "inline must never write a shared template" + ); + } + + /// [`settings_with_mode`], with one integration configured a stated way. + /// + /// Edits the parsed `[integrations]` map rather than appending TOML, so two + /// fixtures differ in exactly the field under test. + fn settings_with_prebid_timeout(mode: &str, timeout_ms: u32) -> Settings { + let mut settings = settings_with_mode(mode); + settings.integrations.insert( + "prebid".to_string(), + serde_json::json!({ + "enabled": true, + "server_url": "https://prebid.example.com/openrtb2/auction", + "external_bundle_url": "https://assets.example.com/prebid/bundle.js", + "timeout": timeout_ms, + }), + ); + settings + } + + /// Every key the cache was asked to store, rendered. + fn stored_cache_keys(cache: &MemoryTemplateCache) -> Vec { + cache + .stored_keys + .lock() + .expect("should lock stored keys") + .iter() + .map(crate::platform::TemplateCacheKey::to_cache_key) + .collect() + } + + /// Every key the cache was asked to read, rendered. + fn looked_up_cache_keys(cache: &MemoryTemplateCache) -> Vec { + cache + .lookups + .lock() + .expect("should lock lookups") + .iter() + .map(crate::platform::TemplateCacheKey::to_cache_key) + .collect() + } + + #[tokio::test] + async fn two_integration_configurations_never_share_a_template() { + // `template_fingerprint` folds the complete typed config, and its own + // tests call it directly — so reverting the *call site* back to the + // bundle-only hash, a constant for a given binary, left the entire suite + // green while every configuration silently shared one template. + // + // This drives the real request path and asserts on the keys the cache was + // actually handed, which is the only place that mutation is visible. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + let first = Arc::new(settings_with_prebid_timeout("esi", 1000)); + let second = Arc::new(settings_with_prebid_timeout("esi", 2500)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = run(&first, &services, navigation_request()).await; + let _ = run(&second, &services, navigation_request()).await; + + let stored = stored_cache_keys(&cache); + assert_eq!( + stored.len(), + 2, + "each configuration must store its own template, got {stored:?}" + ); + assert_ne!( + stored[0], stored[1], + "two `[integrations]` configurations must key different templates; one key \ + serves the first configuration's injected markup to the second" + ); + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "the second configuration must have missed and fetched the origin rather \ + than reading the first's template" + ); + } + + #[tokio::test] + async fn one_integration_configuration_keys_one_template() { + // The converse, and the failure mode a fingerprint fix can introduce: + // over-invalidating is as total as under-invalidating. A fingerprint that + // moves between two equal configurations is a cache that never hits, which + // the spike would report as "no measurable benefit" rather than as a bug. + // + // The two `Settings` are parsed independently, so their `[integrations]` + // maps iterate in different orders — which is what exercises the sort. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + let first = Arc::new(settings_with_prebid_timeout("esi", 1000)); + let second = Arc::new(settings_with_prebid_timeout("esi", 1000)); + queue_shareable_html(&stub); + + let _ = run(&first, &services, navigation_request()).await; + let _ = run(&second, &services, navigation_request()).await; + + let looked_up = looked_up_cache_keys(&cache); + assert_eq!( + looked_up.len(), + 2, + "both requests must consult the cache, got {looked_up:?}" + ); + assert_eq!( + looked_up[0], looked_up[1], + "equal configurations must name one key, or the cache never hits" + ); + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "the second request must be served from the first's template" + ); + } + + fn cookie_navigation_request() -> Request { + HttpRequest::builder() + .method(Method::GET) + .uri("https://ts.example.com/article") + .header(header::HOST, "ts.example.com") + .header("sec-fetch-dest", "document") + .header("sec-fetch-mode", "navigate") + .header(header::COOKIE, "ts-ec=abc123") + .body(EdgeBody::empty()) + .expect("should build cookie-bearing request") + } + + #[tokio::test] + async fn by_default_a_cookie_bearing_request_uses_no_shared_cache() { + // The shipped default, and the reason the cache is nearly inert on real + // traffic: TS sets its own identity cookie, so essentially every repeat + // visitor arrives carrying one and is excluded in both directions. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, cookie_navigation_request()).await; + let _ = run(&settings, &services, cookie_navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "both requests must reach the origin" + ); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "and neither may store a template" + ); + } + + #[tokio::test] + async fn a_declared_cookie_independent_origin_lets_repeat_visitors_share() { + // The opt-in. Without it the spike can only ever measure first-ever page + // views, which is not the population the issue cares about. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut raw = settings_with_mode("esi"); + raw.creative_opportunities + .as_mut() + .expect("fixture configures creative opportunities") + .origin_is_cookie_independent = Some(true); + let settings = Arc::new(raw); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, cookie_navigation_request()).await; + let _ = run(&settings, &services, cookie_navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "the second cookie-bearing request should be served from the cache" + ); + } + + #[tokio::test] + async fn an_active_diagnostics_request_never_stores_a_template() { + // An independent review reintroduced a diagnostics leak scoped to + // A request-private diagnostics mutation could otherwise leak through a + // shared template. `requires_private_no_store()` is a + // strict superset of the condition under which diagnostics markup is + // emitted, and that stamp lands *before* the template cache gate reads response headers, + // so such a request never stores a template at all. + // + // That is a coincidence between two independent conditions, and the whole + // protection rests on it. This pins the consequence directly, so the + // relationship is checked rather than merely reasoned about. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, diagnostics_navigation_request()).await; + + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "a reader running diagnostics must not contribute to a shared cache" + ); + } + + #[tokio::test] + async fn active_diagnostics_bypass_an_already_warm_template() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + let response = run(&settings, &services, diagnostics_navigation_request()).await; + let document = String::from_utf8(body_of(response).await) + .expect("diagnostics document should be UTF-8"); + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "request-private diagnostics must reach the origin even when an ordinary \ + shared template is warm" + ); + assert_eq!( + cache.lookups.lock().expect("should lock lookups").len(), + 1, + "the diagnostics request must not consult template cache at all" + ); + assert!( + document.contains("__tsjs_gpt_diagnostics_active"), + "origin fallback must retain the request-private diagnostics bootstrap" + ); + } + + #[tokio::test] + async fn datadome_suppressed_request_bypasses_a_warm_shared_template() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut raw = settings_with_mode("esi"); + raw.integrations + .insert_config( + "datadome", + &serde_json::json!({ + "enabled": true, + "client_side_key": "test-client-key", + }), + ) + .expect("should configure DataDome integration"); + let settings = Arc::new(raw); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let ordinary = run(&settings, &services, navigation_request()).await; + let ordinary_document = String::from_utf8(body_of(ordinary).await) + .expect("ordinary document should be UTF-8"); + assert!( + ordinary_document.contains("/integrations/datadome/tags.js"), + "the warm template fixture must contain the ordinary DataDome tag" + ); + + let mut suppressed_request = navigation_request(); + suppressed_request + .headers_mut() + .insert("sec-fetch-dest", HeaderValue::from_static("script")); + suppressed_request + .extensions_mut() + .insert(crate::integrations::datadome::DataDomeClientTagSuppressed); + let suppressed = run(&settings, &services, suppressed_request).await; + assert_eq!( + suppressed + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()), + Some("bypass-request"), + "request-scoped tag suppression must not read a shared template" + ); + let suppressed_document = String::from_utf8(body_of(suppressed).await) + .expect("suppressed document should be UTF-8"); + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "the suppressed navigation must reach the origin even when template cache is warm" + ); + assert_eq!( + cache.lookups.lock().expect("should lock lookups").len(), + 1, + "the suppressed request must bypass template cache before lookup" + ); + assert!( + !suppressed_document.contains("/integrations/datadome/tags.js"), + "the request-scoped suppression decision must survive origin processing" + ); + } + + #[tokio::test] + async fn real_diagnostics_query_bypasses_template_cache_and_keeps_its_private_bootstrap() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut raw = settings_with_mode("esi"); + raw.integrations + .insert_config("gpt_diagnostics", &serde_json::json!({ "enabled": true })) + .expect("should configure diagnostics"); + let settings = Arc::new(raw); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let mut request = navigation_request(); + *request.uri_mut() = "https://ts.example.com/article?ts_console=1" + .parse() + .expect("should parse diagnostics URI"); + let response = run(&settings, &services, request).await; + assert_eq!( + response + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()), + Some("bypass-request") + ); + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("private, no-store") + ); + assert!(response.headers().contains_key(header::SET_COOKIE)); + let document = String::from_utf8(body_of(response).await) + .expect("diagnostics document should be UTF-8"); + assert!(document.contains("__tsjs_gpt_diagnostics_active")); + assert!(document.contains("tsjs-gpt_diagnostics.min.js")); + assert!( + cache + .lookups + .lock() + .expect("should lock lookups") + .is_empty(), + "the real query activation must bypass lookup before origin work" + ); + } + + #[tokio::test] + async fn real_diagnostics_cookie_bypasses_a_warm_cookie_independent_template() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let mut raw = settings_with_mode("esi"); + raw.creative_opportunities + .as_mut() + .expect("fixture configures creative opportunities") + .origin_is_cookie_independent = Some(true); + raw.integrations + .insert_config("gpt_diagnostics", &serde_json::json!({ "enabled": true })) + .expect("should configure diagnostics"); + let settings = Arc::new(raw); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + let mut diagnostics = navigation_request(); + diagnostics.headers_mut().insert( + header::COOKIE, + HeaderValue::from_static("__Host-ts-console=1"), + ); + let response = run(&settings, &services, diagnostics).await; + assert_eq!( + response + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()), + Some("bypass-request") + ); + let document = String::from_utf8(body_of(response).await) + .expect("diagnostics document should be UTF-8"); + + assert_eq!(stub.recorded_request_uris().len(), 2); + assert_eq!( + cache.lookups.lock().expect("should lock lookups").len(), + 1, + "the diagnostics cookie must bypass the otherwise-eligible warm lookup" + ); + assert!(document.contains("__tsjs_gpt_diagnostics_active")); + } + + #[tokio::test] + async fn a_cache_backend_failure_falls_back_to_origin_and_is_observable() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + cache.fail_lookup.store(true, Ordering::Relaxed); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let response = run(&settings, &services, navigation_request()).await; + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response + .headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()), + Some("backend-error") + ); + let document = String::from_utf8(body_of(response).await) + .expect("fallback document should be UTF-8"); + assert!(document.contains("origin")); + assert_eq!(stub.recorded_request_uris().len(), 1); + } + + #[tokio::test] + async fn a_cache_hit_streams_rather_than_buffering() { + // The property that makes the cache worth having, and the one that regressed + // silently. Buffered assembly held the first byte until the auction resolved + // — measured at ~100x worse TTFB than shipping nothing at all, because the + // inline path already streams and waits only at ``. + // + // Asserted on the body *shape* rather than on timing: a timing test would be + // flaky, and `EdgeBody::Stream` is the structural fact that produces the + // timing. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + // Warm the cache, then take the hit. + let _ = run(&settings, &services, navigation_request()).await; + let warm = run(&settings, &services, navigation_request()).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "the second request must be a hit, or this asserts nothing" + ); + assert!( + warm.headers().get(header::CONTENT_LENGTH).is_none(), + "a streamed assembly has no length until bids resolve, and headers \ + commit before the first byte" + ); + + // The invariant that actually matters, and the one a body-shape assertion + // misses: a stream that awaited the auction before its first yield would + // still be an `EdgeBody::Stream` and would still hold the first byte. + // + // So pull exactly one chunk and prove the auction has not been collected + // yet — `ad_bids_state` is only written by the collector. No timing + // involved, so nothing to be flaky about. + let EdgeBody::Stream(mut stream) = warm.into_body() else { + panic!("a cache hit must stream, not buffer"); + }; + let first = stream + .next() + .await + .expect("the stream should yield a first chunk") + .expect("the first chunk should read"); + + assert!( + first.starts_with(b"") || first.starts_with(b"` seam emitted + // the marker because the mode was `Esi`, while assembly was skipped because + // the gate had refused a key — a fallback at one seam and not the other. + // + // Bypassing is the *normal* case against a real origin, so this path runs + // far more often than the shared one. It has to produce a working page. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + + // A `Set-Cookie` from the origin disqualifies the response, exactly as a + // real origin does. + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ("set-cookie", "sess=1"), + ], + ); + + let document = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("response should be utf-8"); + + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "the gate refused, so nothing may be stored" + ); + assert!( + !document.contains(AD_ASSEMBLY_SEAM), + "a marker must never be emitted when nothing will resolve it: {document}" + ); + assert!( + document.contains("window.tsjs"), + "and the reader must still get their bids: {document}" + ); + } + + #[tokio::test] + async fn the_seam_carries_slot_definitions_or_the_page_serves_no_ads() { + // Shared modes suppress the head slot script, so the seam is the only place + // `tsjs.adSlots` can come from. Sending only bids left it at its `[]` default, + // `adInit` defined nothing, and the page rendered perfectly with zero TS ads — + // green tests, healthy-looking page, no revenue. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let miss = body_of(run(&settings, &services, navigation_request()).await).await; + let hit = body_of(run(&settings, &services, navigation_request()).await).await; + + for (label, body) in [("miss", miss), ("hit", hit)] { + let text = String::from_utf8(body).expect("utf-8"); + assert!( + text.contains("var a=JSON.parse(") && text.contains("s(b,a)"), + "{label}: the seam must carry slot definitions and hand them to the \ + scheduler: {text}" + ); + assert!( + text.contains("test-slot"), + "{label}: the slot definitions must be populated, not `[]`" + ); + } + } + + #[tokio::test] + async fn an_assembled_response_is_private_even_when_the_ad_stack_is_off() { + // The private stamp was gated on `should_run_ad_stack`. A bot, prefetch, + // kill-switched or consent-denied request can still assemble an empty-bids + // document, and would have kept the origin's public caching directives — so a + // downstream cache could serve that to a later eligible reader. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + + let bot = HttpRequest::builder() + .method(Method::GET) + .uri("https://ts.example.com/article") + .header(header::HOST, "ts.example.com") + .header("sec-fetch-dest", "document") + .header("sec-fetch-mode", "navigate") + .header( + header::USER_AGENT, + "Googlebot/2.1 (+http://www.google.com/bot.html)", + ) + .body(EdgeBody::empty()) + .expect("should build bot request"); + let response = run(&settings, &services, bot).await; + + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|v| v.to_str().ok()), + Some("private, no-store"), + "an assembled response must be private whatever the ad stack decided" + ); + } + + #[tokio::test] + async fn a_cache_hit_keeps_the_origin_security_headers() { + // Reconstructing headers keeps origin `Set-Cookie` and caching directives out + // of a shared cache, and also silently dropped Content-Security-Policy — + // a weaker page, served faster. Policy headers are per-URL, so they belong + // with the template. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ("content-security-policy", "default-src 'self'"), + ("x-frame-options", "SAMEORIGIN"), + ], + ); + + let _ = run(&settings, &services, navigation_request()).await; + let warm = run(&settings, &services, navigation_request()).await; + + assert_eq!( + warm.headers() + .get(header::CONTENT_SECURITY_POLICY) + .and_then(|v| v.to_str().ok()), + Some("default-src 'self'"), + "a hit must not drop the origin's CSP" + ); + assert_eq!( + warm.headers() + .get("x-frame-options") + .and_then(|v| v.to_str().ok()), + Some("SAMEORIGIN") + ); + } + + #[tokio::test] + async fn a_cache_hit_preserves_every_repeated_policy_header_in_order() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ("content-security-policy", "default-src 'self'"), + ("content-security-policy", "script-src 'self'"), + ("link", "; rel=preload; as=script"), + ("link", "; rel=preload; as=style"), + ("cross-origin-opener-policy", "same-origin"), + ("cross-origin-embedder-policy", "require-corp"), + ], + ); + + let _ = run(&settings, &services, navigation_request()).await; + let warm = run(&settings, &services, navigation_request()).await; + + let values = |name: &'static str| { + warm.headers() + .get_all(name) + .iter() + .map(|value| value.to_str().expect("policy header should be text")) + .collect::>() + }; + assert_eq!( + values("content-security-policy"), + ["default-src 'self'", "script-src 'self'"] + ); + assert_eq!( + values("link"), + [ + "; rel=preload; as=script", + "; rel=preload; as=style" + ] + ); + assert_eq!(values("cross-origin-opener-policy"), ["same-origin"]); + assert_eq!(values("cross-origin-embedder-policy"), ["require-corp"]); + } + + #[tokio::test] + async fn nonce_bearing_csp_bypasses_the_shared_template_cache() { + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + for _ in 0..2 { + stub.push_response_with_headers( + 200, + b"origin" + .to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ( + "content-security-policy", + "default-src 'self'; script-src 'nonce-reader-nonce'", + ), + ], + ); + } + + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "a response-bound CSP nonce and its HTML must never be reused from template cache" + ); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty() + ); + } + + #[tokio::test] + async fn a_meta_delivered_nonce_policy_bypasses_the_shared_template_cache() { + // The response-header gate cannot see this policy at all. Storing the document + // would replay one response's nonce to every later reader — the exact thing the + // header check exists to prevent. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + for _ in 0..2 { + stub.push_response_with_headers( + 200, + br#"origin"# + .to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ], + ); + } + + let cold = run(&settings, &services, navigation_request()).await; + assert_eq!( + cold.headers() + .get(HEADER_X_TS_TEMPLATE_CACHE) + .and_then(|value| value.to_str().ok()), + Some("bypass-response"), + "the cold response must refuse to store a nonce-bearing document" + ); + let _ = body_of(cold).await; + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "the warm request must reach the origin, not a replayed nonce" + ); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty(), + "nothing nonce-bearing may reach the shared cache" + ); + } + + #[tokio::test] + async fn a_nonce_attribute_without_a_policy_bypasses_the_shared_template_cache() { + // Fail closed: the attributes say the document was written for a per-response + // policy, whether or not the policy itself survived to this scan. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + for _ in 0..2 { + stub.push_response_with_headers( + 200, + br#"origin"# + .to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ], + ); + } + + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + let _ = body_of(run(&settings, &services, navigation_request()).await).await; + + assert_eq!(stub.recorded_request_uris().len(), 2); + assert!( + cache + .entries + .lock() + .expect("should lock entries") + .is_empty() + ); + } + + #[tokio::test] + async fn a_body_close_in_script_data_corrupts_neither_the_cold_nor_the_warm_document() { + // No structural `` anywhere: a reverse byte search takes the string + // literal, and the payload's `` then terminates the publisher's script + // — in the response served cold and in the template every warm reader gets. + const PUBLISHER_SCRIPT: &str = r#""#; + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + stub.push_response_with_headers( + 200, + format!("{PUBLISHER_SCRIPT}

article

").into_bytes(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ], + ); + + let cold = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("the cold document should be UTF-8"); + let warm = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("the warm document should be UTF-8"); + + assert_eq!( + stub.recorded_request_uris().len(), + 1, + "the document is otherwise shareable and must still be stored and reused" + ); + for (label, document) in [("cold", &cold), ("warm", &warm)] { + assert!( + document.contains(PUBLISHER_SCRIPT), + "the {label} document must carry the publisher's script byte for byte: {document}" + ); + assert!( + !document.contains(AD_ASSEMBLY_SEAM), + "the {label} document must not ship an unresolved seam: {document}" + ); + } + } + + #[tokio::test] + async fn a_body_close_in_trailing_comment_data_does_not_attract_the_seam() { + // The reverse search took the *last* `` sequence, so the seam landed + // inside this comment and the assembled bids never executed. + const TRAILING_COMMENT: &str = ""; + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + stub.push_response_with_headers( + 200, + format!("

article

{TRAILING_COMMENT}") + .into_bytes(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("cache-control", "public, max-age=300"), + ], + ); + + let cold = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("the cold document should be UTF-8"); + let warm = String::from_utf8( + body_of(run(&settings, &services, navigation_request()).await).await, + ) + .expect("the warm document should be UTF-8"); + + assert_eq!(stub.recorded_request_uris().len(), 1); + for (label, document) in [("cold", &cold), ("warm", &warm)] { + assert!( + document.contains(TRAILING_COMMENT), + "the {label} document must leave the publisher comment intact: {document}" + ); + assert!( + !document.contains(AD_ASSEMBLY_SEAM), + "the {label} document must not ship an unresolved seam: {document}" + ); + assert!( + !document.contains(TEMPLATE_SEAM_PLACEHOLDER), + "the transform-owned placeholder must never reach a reader: {document}" + ); + } + } + + #[tokio::test] + async fn a_post_is_never_answered_from_a_cached_get() { + // `handle_publisher_request` is the `*`-method fallback route, so a publisher + // path that renders a page on GET and accepts a form or webhook on POST reaches + // here for both. Serving the cached GET to the POST swallows the mutating + // request entirely: the origin never sees it, the caller gets 200 and a page, + // and nothing anywhere reports a problem. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + // Warm the cache with a GET. + let _ = run(&settings, &services, navigation_request()).await; + assert_eq!(stub.recorded_request_uris().len(), 1); + + let post = HttpRequest::builder() + .method(Method::POST) + .uri("https://ts.example.com/article") + .header(header::HOST, "ts.example.com") + .body(EdgeBody::from("field=value")) + .expect("should build post request"); + let _ = run(&settings, &services, post).await; + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "the POST must reach the origin rather than being answered from the \ + cached GET" + ); + assert_eq!( + cache.entries.lock().expect("should lock entries").len(), + 1, + "and it must not store a template of its own" + ); + } + + #[tokio::test] + async fn an_authenticated_request_is_not_served_a_shared_template() { + // The stored template is perfectly cacheable; this request is not entitled + // to it. The store gate cannot express that, because it is a property of + // the reader rather than of the bytes — which is why the lookup re-checks. + let stub = Arc::new(StubHttpClient::new()); + let cache = Arc::new(MemoryTemplateCache::default()); + let settings = Arc::new(settings_with_mode("esi")); + let services = services(Arc::clone(&stub), Arc::clone(&cache)); + queue_shareable_html(&stub); + queue_shareable_html(&stub); + + let _ = run(&settings, &services, navigation_request()).await; + assert_eq!( + cache.entries.lock().expect("should lock entries").len(), + 1, + "the cold request should have populated the cache" + ); + + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + let consent = crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + ..Default::default() + }; + let mut ec_context = EcContext::new_for_test(None, consent); + let authenticated = HttpRequest::builder() + .method(Method::GET) + .uri("https://ts.example.com/article") + .header(header::HOST, "ts.example.com") + .header("sec-fetch-dest", "document") + .header(header::AUTHORIZATION, "Basic dXNlcjpwYXNz") + .body(EdgeBody::empty()) + .expect("should build authenticated request"); + let _ = handle_publisher_request( + &settings, + &services, + None, + &mut ec_context, + AuctionDispatch { + orchestrator: &orchestrator, + slots: &[article_slot()], + registry: None, + }, + authenticated, + EdgeCacheHeader::SMaxageFallback, + ) + .await + .expect("should proxy publisher request"); + + assert_eq!( + stub.recorded_request_uris().len(), + 2, + "an authenticated request must reach the origin rather than read a \ + shared template" + ); + } + } + + mod template_cache_gate_tests { + //! `cache::core` stores whatever it is handed and rejects nothing, so every + //! one of these conditions is the caller's to enforce. Each is a leak vector + //! or an eligibility rule, not a preference. + + use super::*; + use crate::creative_opportunities::AssemblyMode; + use edgezero_core::http::HeaderName; + + fn headers(pairs: &[(HeaderName, &str)]) -> edgezero_core::http::HeaderMap { + let mut map = edgezero_core::http::HeaderMap::new(); + for (name, value) in pairs { + map.insert( + name.clone(), + HeaderValue::from_str(value).expect("should build header value"), + ); + } + map + } + + fn shareable() -> edgezero_core::http::HeaderMap { + headers(&[(header::CACHE_CONTROL, "max-age=60")]) + } + + /// The shipped default: no operator has stated what the origin varies on, so the + /// key covers nothing. Responses without a `Vary` are unaffected; any `Vary` at + /// all disqualifies. + fn nothing_covered() -> VarySpec { + VarySpec::new([]) + } + + #[test] + fn an_unconfigured_deployment_never_caches_a_varying_response() { + // The fail-closed default. An operator who has not stated the origin's Vary + // must not acquire a shared cache by omission — and a real origin varies on + // something, so this is the common path, not an edge case. + // Deliberately not `Accept-Encoding`: the shared path normalizes supported + // content codings to one identity template, so that header is covered + // whatever the operator configured. Using it here would test the + // structural-coverage carve-out rather than the drift guard. + let mut varying = shareable(); + varying.insert(header::VARY, HeaderValue::from_static("rsc")); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &varying, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::VaryNotCovered(VaryGap(vec![ + "rsc".to_string() + ]))), + "an unstated Vary must disqualify rather than silently under-key" + ); + } + + #[test] + fn an_origin_that_varies_on_cookie_is_refused_even_when_declared_independent() { + // The backstop that makes `origin_is_cookie_independent` safe to offer. The + // operator asserts their origin ignores cookies; if the origin then says + // otherwise, the assertion loses. Without this, a wrong assertion would + // silently cross-serve personalized HTML. + let mut varying = shareable(); + varying.insert(header::VARY, HeaderValue::from_static("Cookie")); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + // The operator's assertion has already been applied here: this is + // `false` precisely because they declared independence. + false, + StatusCode::OK, + "text/html", + &varying, + &VarySpec::new(["cookie".to_string()]), + ), + Some(TemplateCacheBypassReason::VaryCookie), + "the origin's declaration must override both cookie independence and an \ + accidentally configured per-cookie key" + ); + } + + #[test] + fn a_private_directive_on_a_second_cache_control_line_is_refused() { + // `HeaderMap::get` returns the first value only. An origin that sends + // `Cache-Control: public, max-age=300` and then `Cache-Control: private` on a + // separate line means exactly what one comma-joined line would mean, but the + // second line was invisible — so a response the origin marked private was + // written to a cache shared between readers. The `Vary` reads a few lines up + // already use `get_all` for the same reason. + let mut split = edgezero_core::http::HeaderMap::new(); + split.append( + header::CACHE_CONTROL, + HeaderValue::from_static("public, max-age=300"), + ); + split.append(header::CACHE_CONTROL, HeaderValue::from_static("private")); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &split, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable), + "a directive on any Cache-Control line must disqualify the response" + ); + } + + #[test] + fn a_no_store_directive_on_a_second_cache_control_line_is_refused() { + // Same defect, the other directive that matters — `no-store` is the one an + // origin uses for a response that must not be written down anywhere. + let mut split = edgezero_core::http::HeaderMap::new(); + split.append( + header::CACHE_CONTROL, + HeaderValue::from_static("max-age=60"), + ); + split.append(header::CACHE_CONTROL, HeaderValue::from_static("no-store")); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &split, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable) + ); + } + + #[test] + fn cdn_specific_cache_policy_cannot_be_overridden_by_public_cache_control() { + for name in crate::response_privacy::CDN_CACHE_HEADERS { + let mut split = shareable(); + split.insert( + header::HeaderName::from_static(name), + HeaderValue::from_static("no-store"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &split, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable), + "template cache must fail closed on the CDN-specific policy header {name}" + ); + } + } + + #[test] + fn unsupported_vendor_freshness_does_not_authorize_template_cache() { + for name in crate::response_privacy::CDN_CACHE_HEADERS + .iter() + .filter(|name| **name != "surrogate-control") + { + let mut split = shareable(); + split.insert( + header::HeaderName::from_static(name), + HeaderValue::from_static("max-age=60"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &split, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable), + "the Fastly exception must not authorize the vendor policy {name}" + ); + } + } + + #[test] + fn observed_fastly_surrogate_policy_uses_edge_freshness_capped_by_configuration() { + let publisher_headers = headers(&[ + (header::CACHE_CONTROL, "public, max-age=60"), + ( + header::HeaderName::from_static("surrogate-control"), + "max-age=1200, stale-while-revalidate=21600, stale-if-error=604800", + ), + ]); + + assert_eq!( + template_cache_ttl( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + &TemplateCachePolicy::for_test(¬hing_covered(), Duration::from_secs(300),), + ), + Ok(Duration::from_secs(300)), + "Fastly's edge freshness should take precedence over the shorter browser \ + lifetime, while the configured safety ceiling remains authoritative" + ); + } + + #[test] + fn fastly_surrogate_freshness_takes_precedence_over_standard_freshness() { + for (cache_control, surrogate_control, expected) in [ + ("public, max-age=300", "max-age=30", 30), + ("public, max-age=30", "max-age=300", 300), + ] { + let publisher_headers = headers(&[ + (header::CACHE_CONTROL, cache_control), + ( + header::HeaderName::from_static("surrogate-control"), + surrogate_control, + ), + ]); + + assert_eq!( + template_cache_ttl( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + &TemplateCachePolicy::for_test( + ¬hing_covered(), + Duration::from_secs(600), + ), + ), + Ok(Duration::from_secs(expected)) + ); + } + } + + #[test] + fn surrogate_stale_windows_do_not_extend_fresh_reuse() { + let publisher_headers = headers(&[ + (header::CACHE_CONTROL, "public, max-age=300"), + ( + header::HeaderName::from_static("surrogate-control"), + "max-age=20, stale-while-revalidate=600, stale-if-error=1200", + ), + (header::AGE, "10"), + ]); + + assert_eq!( + template_cache_ttl( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + &TemplateCachePolicy::for_test(¬hing_covered(), Duration::from_secs(600),), + ), + Ok(Duration::from_secs(10)), + "stale windows are validated metadata, not fresh template cache lifetime" + ); + } + + #[test] + fn ambiguous_or_unsupported_surrogate_policy_fails_closed() { + for (policy, expected) in [ + ("max-age", TemplateCacheBypassReason::MalformedCachePolicy), + ( + "max-age=30, max-age=60", + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ( + "max-age=tomorrow", + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ( + "max-age=30, public", + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ( + "stale-if-error=60", + TemplateCacheBypassReason::NoPositiveFreshness, + ), + ("max-age=0", TemplateCacheBypassReason::NoPositiveFreshness), + ( + "max-age=30,", + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ] { + let mut publisher_headers = shareable(); + publisher_headers.insert( + header::HeaderName::from_static("surrogate-control"), + HeaderValue::from_str(policy).expect("should build Surrogate-Control"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + ¬hing_covered(), + ), + Some(expected), + "`{policy}` must fail closed" + ); + } + } + + #[test] + fn restrictive_surrogate_policy_is_never_overridden_by_standard_freshness() { + for directive in ["private", "no-store", "no-cache"] { + let mut publisher_headers = shareable(); + publisher_headers.insert( + header::HeaderName::from_static("surrogate-control"), + HeaderValue::from_str(directive).expect("should build Surrogate-Control"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable), + "`{directive}` must remain authoritative" + ); + } + } + + #[test] + fn restrictive_standard_policy_is_never_overridden_by_surrogate_freshness() { + for directive in ["private", "no-store", "no-cache"] { + let publisher_headers = headers(&[ + ( + header::CACHE_CONTROL, + &format!("public, max-age=60, {directive}"), + ), + ( + header::HeaderName::from_static("surrogate-control"), + "max-age=1200", + ), + ]); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable), + "standard `{directive}` must refuse template cache even with positive edge freshness" + ); + } + } + + #[test] + fn surrogate_control_can_authorize_fastly_edge_freshness_without_browser_freshness() { + let publisher_headers = headers(&[( + header::HeaderName::from_static("surrogate-control"), + "max-age=1200", + )]); + + assert_eq!( + template_cache_ttl( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &publisher_headers, + &TemplateCachePolicy::for_test(¬hing_covered(), Duration::from_secs(300),), + ), + Ok(Duration::from_secs(300)), + "Fastly edge freshness should not require browser freshness" + ); + } + + #[test] + fn repeated_cache_control_lines_without_a_disqualifier_still_cache() { + // The other direction: reading every value must not turn an ordinary + // multi-line `Cache-Control` into a bypass, or the fix would disable the + // cache instead of tightening it. + let mut split = edgezero_core::http::HeaderMap::new(); + split.append(header::CACHE_CONTROL, HeaderValue::from_static("public")); + split.append( + header::CACHE_CONTROL, + HeaderValue::from_static("max-age=60"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &split, + ¬hing_covered(), + ), + None + ); + } + + #[test] + fn origin_freshness_is_positive_age_adjusted_and_capped() { + let fresh_headers = headers(&[(header::CACHE_CONTROL, "public, max-age=300")]); + assert_eq!( + template_cache_ttl( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &fresh_headers, + &TemplateCachePolicy::for_test(¬hing_covered(), Duration::from_secs(60),), + ), + Ok(Duration::from_secs(60)) + ); + + let aged = headers(&[ + (header::CACHE_CONTROL, "s-maxage=50, max-age=300"), + (header::AGE, "35"), + ]); + assert_eq!( + template_cache_ttl( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &aged, + &TemplateCachePolicy::for_test(¬hing_covered(), Duration::from_secs(60),), + ), + Ok(Duration::from_secs(15)) + ); + + let old_date_without_age = headers(&[ + (header::CACHE_CONTROL, "public, max-age=60"), + (header::DATE, "Wed, 12 Aug 2026 08:00:00 GMT"), + ]); + let one_minute_later = httpdate::parse_http_date("Wed, 12 Aug 2026 08:01:00 GMT") + .expect("should parse fixture time"); + assert_eq!( + origin_shared_ttl_at( + &old_date_without_age, + one_minute_later, + Duration::from_secs(60), + ), + Err(TemplateCacheBypassReason::NoPositiveFreshness), + "an old Date is apparent age even when an upstream omitted Age" + ); + } + + #[test] + fn zero_exhausted_missing_and_malformed_freshness_are_refused() { + for (map, expected) in [ + ( + headers(&[(header::CACHE_CONTROL, "max-age=0")]), + TemplateCacheBypassReason::NoPositiveFreshness, + ), + ( + headers(&[(header::CACHE_CONTROL, "max-age=60"), (header::AGE, "60")]), + TemplateCacheBypassReason::NoPositiveFreshness, + ), + ( + headers(&[(header::CACHE_CONTROL, "public")]), + TemplateCacheBypassReason::NoPositiveFreshness, + ), + ( + headers(&[(header::CACHE_CONTROL, "max-age=tomorrow")]), + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ( + headers(&[(header::CACHE_CONTROL, "max-age=\"60")]), + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ( + headers(&[(header::CACHE_CONTROL, "max-age=+60")]), + TemplateCacheBypassReason::MalformedCachePolicy, + ), + ] { + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &map, + ¬hing_covered(), + ), + Some(expected) + ); + } + } + + #[test] + fn expires_can_authorize_but_never_extend_an_expired_response() { + let now = httpdate::parse_http_date("Wed, 12 Aug 2026 08:00:00 GMT") + .expect("should parse fixture time"); + let fresh = headers(&[ + (header::DATE, "Wed, 12 Aug 2026 08:00:00 GMT"), + (header::EXPIRES, "Wed, 12 Aug 2026 08:00:30 GMT"), + ]); + assert_eq!( + origin_shared_ttl_at(&fresh, now, Duration::from_secs(60)), + Ok(Duration::from_secs(30)) + ); + + let expired = headers(&[ + (header::DATE, "Wed, 12 Aug 2026 08:01:00 GMT"), + (header::EXPIRES, "Wed, 12 Aug 2026 08:00:30 GMT"), + ]); + assert_eq!( + origin_shared_ttl_at(&expired, now, Duration::from_secs(60)), + Err(TemplateCacheBypassReason::NoPositiveFreshness) + ); + } + + #[test] + fn request_semantics_bypass_template_cache_except_for_a_max_age_zero_reload() { + for (name, value) in [ + (header::CACHE_CONTROL, "no-cache"), + (header::CACHE_CONTROL, "max-age=30"), + (header::CACHE_CONTROL, "max-age=\"0"), + (header::CACHE_CONTROL, "min-fresh=10"), + (header::CACHE_CONTROL, "no-store"), + (header::PRAGMA, "no-cache"), + (header::PRAGMA, "legacy-extension, no-cache"), + (header::RANGE, "bytes=0-99"), + (header::IF_NONE_MATCH, "\"etag\""), + (header::IF_MODIFIED_SINCE, "Wed, 12 Aug 2026 08:00:00 GMT"), + ] { + let map = headers(&[(name.clone(), value)]); + assert!( + request_bypasses_template_cache(&map), + "{name}: {value} must bypass" + ); + } + assert!( + !request_bypasses_template_cache(&headers(&[(header::CACHE_CONTROL, "max-age=0")])), + "a browser reload may reuse template cache because the assembled response and auction \ + are still rebuilt for this reader" + ); + assert!(!request_bypasses_template_cache(&headers(&[( + header::CACHE_CONTROL, + "public" + )]))); + } + + #[test] + fn a_wildcard_vary_is_refused() { + // `VarySpec::uncovered_by` filters `*` out, with a comment saying the + // eligibility gate handles it. It did not — nothing rejected the wildcard, so + // a response the origin said no key can select was shareable. + let mut varying = shareable(); + varying.insert(header::VARY, HeaderValue::from_static("*")); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &varying, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::VaryWildcard) + ); + } + + #[test] + fn a_fully_covered_vary_is_cacheable() { + let mut varying = shareable(); + varying.insert( + header::VARY, + HeaderValue::from_static("rsc, Accept-Encoding"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &varying, + &VarySpec::new(["rsc".to_string(), "accept-encoding".to_string()]), + ), + None, + "a key covering everything the origin varies on is safe to store" + ); + } + + #[test] + fn config_drift_names_the_missing_header() { + // The failure this guards: the origin adds a header to its Vary, nobody + // updates config, and requests differing only in that header start sharing a + // template. The reason must name it, or diagnosing means a bisect. + let mut varying = shareable(); + varying.insert( + header::VARY, + HeaderValue::from_static("rsc, next-router-prefetch"), + ); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &varying, + &VarySpec::new(["rsc".to_string()]), + ), + Some(TemplateCacheBypassReason::VaryNotCovered(VaryGap(vec![ + "next-router-prefetch".to_string() + ]))), + "the uncovered header must be named" + ); + } + + #[test] + fn a_vary_split_across_repeated_headers_is_still_checked() { + // Vary is a list header, so an origin may send it once or many times. Reading + // only the first would let the rest through unkeyed. + let mut varying = shareable(); + varying.append(header::VARY, HeaderValue::from_static("rsc")); + varying.append(header::VARY, HeaderValue::from_static("cookie")); + + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &varying, + &VarySpec::new(["rsc".to_string()]), + ), + Some(TemplateCacheBypassReason::VaryCookie), + "a repeated Vary header must not hide names behind the first value" + ); + } + + #[test] + fn a_plain_shareable_html_200_is_cacheable() { + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &shareable(), + ¬hing_covered(), + ), + None, + "ESI shareable HTML 200 should be eligible" + ); + } + + #[test] + fn inline_mode_never_writes_a_template() { + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Inline, + false, + false, + StatusCode::OK, + "text/html", + &shareable(), + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::InlineMode), + "inline has no shared template to write" + ); + } + + #[test] + fn an_authorized_request_is_never_cached() { + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + true, + false, + StatusCode::OK, + "text/html", + &shareable(), + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::AuthorizedRequest), + "an authenticated response must not enter a shared cache" + ); + } + + #[test] + fn a_forwarded_request_cookie_disqualifies_even_without_set_cookie() { + // The dangerous case: session established on an earlier request, so this + // response carries no Set-Cookie, has no Cache-Control at all, is a 200, + // and is HTML — yet is personalized because TS forwarded the Cookie to + // origin unchanged. Every other condition reports it cacheable. + let no_cache_control = edgezero_core::http::HeaderMap::new(); + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + true, + StatusCode::OK, + "text/html", + &no_cache_control, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::CookieForwarded), + "cookie-personalized HTML must not become a shared template" + ); + } + + #[test] + fn an_origin_set_cookie_is_never_cached() { + let with_cookie = headers(&[ + (header::CACHE_CONTROL, "max-age=60"), + (header::SET_COOKIE, "sid=abc; Path=/"), + ]); + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &with_cookie, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginSetCookie), + "caching this would replay one visitor's cookie to the next" + ); + } + + #[test] + fn non_shareable_cache_control_is_refused_case_insensitively() { + for directive in [ + "private", + "no-store", + "no-cache", + "Private, max-age=60", + "NO-STORE", + "public, No-Cache", + ] { + let map = headers(&[(header::CACHE_CONTROL, directive)]); + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &map, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::OriginNotShareable), + "`{directive}` should disqualify the response" + ); + } + } + + #[test] + fn a_datadome_block_is_refused_by_the_status_check() { + // DataDome replaces the document with a 403 + // (`integrations/datadome/protection.rs:778`). There is no separate + // marker to detect, and none is needed. + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::FORBIDDEN, + "text/html", + &shareable(), + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::NonOkStatus), + "a blocked document must not become the shared template" + ); + } + + #[test] + fn non_html_is_refused() { + for content_type in ["text/x-component", "application/json", ""] { + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + content_type, + &shareable(), + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::NotHtml), + "`{content_type}` has no HTML template to transform" + ); + } + } + + #[test] + fn unsupported_content_encoding_is_refused_before_representation_headers_change() { + let map = headers(&[ + (header::CACHE_CONTROL, "public, max-age=60"), + (header::CONTENT_ENCODING, "zstd"), + ]); + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &map, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::UnsupportedContentEncoding) + ); + + let mut repeated = headers(&[(header::CACHE_CONTROL, "public, max-age=60")]); + repeated.append(header::CONTENT_TYPE, HeaderValue::from_static("text/html")); + repeated.append( + header::CONTENT_TYPE, + HeaderValue::from_static("application/json"), + ); + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + false, + false, + StatusCode::OK, + "text/html", + &repeated, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::MalformedRepresentationHeaders) + ); + } + + #[test] + fn leak_vectors_are_reported_before_mere_ineligibility() { + // A response that fails several conditions should name the most serious + // one, so an operator reading the log sees the security reason rather + // than a content-type quibble. + let map = headers(&[ + (header::CACHE_CONTROL, "private"), + (header::SET_COOKIE, "sid=abc"), + ]); + assert_eq!( + template_cache_bypass_reason( + AssemblyMode::Esi, + true, + false, + StatusCode::FORBIDDEN, + "application/json", + &map, + ¬hing_covered(), + ), + Some(TemplateCacheBypassReason::AuthorizedRequest), + "authorization is the most serious disqualifier and should win" + ); + } + } + + mod template_neutrality_tests { + //! The gate for #1009's shared-template design. + //! + //! An "absence of per-user values" scan is not sufficient here: the bug + //! that nearly shipped was a *conditionally present* element whose own + //! content was per-URL. These tests assert byte-identity across requests + //! that differ only in the gating decision. + + use super::*; + use crate::creative_opportunities::{ + AssemblyMode, CreativeOpportunityFormat, CreativeOpportunitySlot, + }; + + pub(super) fn slot() -> CreativeOpportunitySlot { + CreativeOpportunitySlot { + id: "atf".to_string(), + gam_unit_path: Some("/99999/example/home".to_string()), + div_id: Some("ad-atf".to_string()), + page_patterns: vec!["/**".to_string()], + formats: vec![CreativeOpportunityFormat { + width: 300, + height: 250, + media_type: MediaType::Banner, + }], + floor_price: None, + targeting: Default::default(), + providers: Default::default(), + compiled_patterns: Vec::new(), + compiled_unit: None, + } + } + + pub(super) fn settings_with_slots() -> Settings { + let mut settings = crate::test_support::tests::create_test_settings(); + // Construct the section rather than mutating it if present: the shared + // fixture does not carry `[creative_opportunities]`, and an `if let + // Some(..)` here would silently no-op and make the inline assertion + // below vacuous. + settings.creative_opportunities = Some(CreativeOpportunitiesConfig { + enabled: true, + gam_network_id: "99999".to_string(), + auction_timeout_ms: Some(500), + price_granularity: Default::default(), + section_root: None, + assembly_mode: None, + template_cache_vary: None, + template_cache_max_age_seconds: None, + origin_is_cookie_independent: None, + section_segment: None, + slot: vec![slot()], + }); + settings + } + + #[test] + fn shared_modes_emit_no_head_script_regardless_of_the_gating_decision() { + let settings = settings_with_slots(); + let slots = [slot()]; + let mode = AssemblyMode::Esi; + let ran = template_ad_slots_script(mode, true, &settings, &slots, "/"); + let did_not_run = template_ad_slots_script(mode, false, &settings, &slots, "/"); + + assert_eq!( + ran, did_not_run, + "{mode:?}: the template must be byte-identical whether or not the ad \ + stack ran; a cached object cannot carry one request's consent, bot, \ + prefetch or kill-switch decision" + ); + assert_eq!( + ran, None, + "{mode:?}: adSlots belongs in the per-request seam, not the template" + ); + } + + #[test] + fn inline_mode_keeps_its_request_dependent_behaviour() { + // Inline responses are per-navigation and never shared, so gating is + // correct there. This guards against "fixing" the shared-mode bug by + // breaking the shipped path. + let settings = settings_with_slots(); + let slots = [slot()]; + + assert!( + template_ad_slots_script(AssemblyMode::Inline, true, &settings, &slots, "/") + .is_some(), + "inline should emit adSlots when the ad stack runs" + ); + assert_eq!( + template_ad_slots_script(AssemblyMode::Inline, false, &settings, &slots, "/"), + None, + "inline should emit nothing when the ad stack does not run" + ); + } + + #[test] + fn shared_modes_are_neutral_across_differing_slot_matches() { + // Slot matching folds in the request path. Under a shared mode even + // that must not reach the template. + let settings = settings_with_slots(); + + let matched = template_ad_slots_script( + AssemblyMode::Esi, + true, + &settings, + &[slot()], + "/news/article", + ); + let unmatched = + template_ad_slots_script(AssemblyMode::Esi, true, &settings, &[], "/other"); + + assert_eq!( + matched, unmatched, + "the template must not vary with slot matching under a shared mode" + ); + } + } + + mod ssat_cache_policy_tests { + use super::*; + use crate::auction::provider::{AuctionProvider, ProviderRequestOutcome}; + use crate::auction::telemetry::{AuctionEventBatch, AuctionTelemetrySink}; + use crate::creative_opportunities::{CreativeOpportunityFormat, CreativeOpportunitySlot}; + use crate::platform::test_support::{ + NoopConfigStore, NoopGeo, NoopSecretStore, StubBackend, + }; + use crate::platform::{ + ClientInfo, PlatformError, PlatformHttpClient, PlatformPendingRequest, + PlatformResponse, PlatformSelectResult, + }; + use crate::test_support::tests::crate_test_settings_str; + + const ORIGIN_ETAG: &str = "\"origin-tag\""; + const ORIGIN_LAST_MODIFIED: &str = "Wed, 21 Oct 2015 07:28:00 GMT"; + const UNEXPECTED_304_PROVIDER: &str = "example_navigation_bidder"; + const UNEXPECTED_304_BACKEND: &str = "example-navigation-bidder-backend"; + + struct DispatchingTestProvider; + + struct RangeAwareHttpClient { + stub: StubHttpClient, + } + + impl RangeAwareHttpClient { + fn new() -> Self { + Self { + stub: StubHttpClient::new(), + } + } + } + + #[async_trait::async_trait(?Send)] + impl PlatformHttpClient for RangeAwareHttpClient { + async fn send( + &self, + request: PlatformHttpRequest, + ) -> Result> { + if request.request.headers().contains_key(header::RANGE) { + self.stub.push_response_with_headers( + 206, + b"partial".to_vec(), + vec![ + ("content-type", "text/html; charset=utf-8"), + ("content-range", "bytes 0-18/39"), + ], + ); + } else { + self.stub.push_response_with_headers( + 200, + b"origin".to_vec(), + vec![("content-type", "text/html; charset=utf-8")], + ); + } + self.stub.send(request).await + } + + async fn send_async( + &self, + request: PlatformHttpRequest, + ) -> Result> { self.stub.send_async(request).await } @@ -4849,6 +12562,19 @@ mod tests { .expect("should parse settings with auction and creative opportunities enabled") } + fn settings_with_disabled_ad_templates() -> Settings { + let toml = format!( + "{}\n[auction]\nenabled = true\n\n\ + [creative_opportunities]\nenabled = false\ngam_network_id = \"12345\"\n", + crate_test_settings_str() + ); + Settings::from_toml(&toml).expect("should parse settings with disabled ad templates") + } + + fn settings_without_creative_opportunities() -> Settings { + Settings::from_toml(&crate_test_settings_str()) + .expect("should parse settings without creative opportunities") + } fn settings_with_dispatching_provider() -> Settings { let toml = format!( "{}\n[auction]\nenabled = true\nproviders = [\"{UNEXPECTED_304_PROVIDER}\"]\n\n\ @@ -4907,13 +12633,28 @@ mod tests { .expect("should build conditional navigation request") } - fn queue_cacheable_html_response(stub: &StubHttpClient) { + fn queue_cacheable_html_response(stub: &StubHttpClient) { + queue_html_response_with_cache_control(stub, "public, max-age=300"); + } + + fn queue_html_response_with_cache_control( + stub: &StubHttpClient, + cache_control: &'static str, + ) { + queue_html_response_with_status_and_cache_control(stub, 200, cache_control); + } + + fn queue_html_response_with_status_and_cache_control( + stub: &StubHttpClient, + status: u16, + cache_control: &'static str, + ) { stub.push_response_with_headers( - 200, + status, b"origin".to_vec(), vec![ ("content-type", "text/html; charset=utf-8"), - ("cache-control", "public, max-age=300"), + ("cache-control", cache_control), ("etag", ORIGIN_ETAG), ("last-modified", ORIGIN_LAST_MODIFIED), ("surrogate-control", "max-age=300"), @@ -4924,14 +12665,40 @@ mod tests { ); } + fn non_regulated_consent() -> crate::consent::ConsentContext { + crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + ..Default::default() + } + } + async fn run_with_slots( settings: &Settings, services: &RuntimeServices, slots: &[CreativeOpportunitySlot], req: Request, + ) -> PublisherResponse { + run_with_slots_and_consent(settings, services, slots, req, non_regulated_consent()) + .await + } + + async fn run_with_slots_and_consent( + settings: &Settings, + services: &RuntimeServices, + slots: &[CreativeOpportunitySlot], + req: Request, + consent: crate::consent::ConsentContext, ) -> PublisherResponse { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); - run_with_orchestrator(settings, services, &orchestrator, slots, req).await + run_with_orchestrator_and_consent( + settings, + services, + &orchestrator, + slots, + req, + consent, + ) + .await } async fn run_with_orchestrator( @@ -4941,10 +12708,25 @@ mod tests { slots: &[CreativeOpportunitySlot], req: Request, ) -> PublisherResponse { - let consent = crate::consent::ConsentContext { - jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, - ..Default::default() - }; + run_with_orchestrator_and_consent( + settings, + services, + orchestrator, + slots, + req, + non_regulated_consent(), + ) + .await + } + + async fn run_with_orchestrator_and_consent( + settings: &Settings, + services: &RuntimeServices, + orchestrator: &AuctionOrchestrator, + slots: &[CreativeOpportunitySlot], + req: Request, + consent: crate::consent::ConsentContext, + ) -> PublisherResponse { let mut ec_context = EcContext::new_for_test(None, consent); handle_publisher_request( @@ -4958,6 +12740,7 @@ mod tests { registry: None, }, req, + EdgeCacheHeader::SMaxageFallback, ) .await .expect("should proxy publisher request") @@ -4967,6 +12750,7 @@ mod tests { match response { PublisherResponse::Buffered(response) | PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } | PublisherResponse::PassThrough { response, .. } => response.into_parts().0, } } @@ -5126,7 +12910,73 @@ mod tests { ); for (header_name, expected) in [ - (header::CACHE_CONTROL, "public, max-age=300"), + (header::CACHE_CONTROL, "max-age=60"), + (header::ETAG, ORIGIN_ETAG), + (header::LAST_MODIFIED, ORIGIN_LAST_MODIFIED), + ( + header::HeaderName::from_static("surrogate-control"), + "max-age=300", + ), + ( + header::HeaderName::from_static("fastly-surrogate-control"), + "max-age=300", + ), + ( + header::HeaderName::from_static("cdn-cache-control"), + "max-age=300", + ), + ( + header::HeaderName::from_static("cloudflare-cdn-cache-control"), + "max-age=300", + ), + ] { + assert_eq!( + response_head + .headers + .get(&header_name) + .and_then(|value| value.to_str().ok()), + Some(expected), + "publisher response without matched slots should preserve {header_name}" + ); + } + } + + #[tokio::test] + async fn disabled_ad_templates_use_short_browser_cache_policy() { + // Arrange + let settings = settings_with_disabled_ad_templates(); + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, "public, max-age=300"); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let slots = [article_slot()]; + + // Act + let response = run_with_slots( + &settings, + &services, + &slots, + conditional_navigation_request(), + ) + .await; + let response_head = response_head(response); + + // Assert + assert_eq!( + stub.recorded_cache_bypass_flags(), + vec![false], + "disabled server-side ad templates should not bypass the origin cache" + ); + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("max-age=60"), + "disabled server-side ad templates should use the short browser cache policy" + ); + for (header_name, expected) in [ (header::ETAG, ORIGIN_ETAG), (header::LAST_MODIFIED, ORIGIN_LAST_MODIFIED), ( @@ -5149,10 +12999,290 @@ mod tests { assert_eq!( response_head .headers - .get(&header_name) + .get(&header_name) + .and_then(|value| value.to_str().ok()), + Some(expected), + "disabled server-side ad templates should preserve {header_name}" + ); + } + } + + #[tokio::test] + async fn navigation_without_matched_slots_replaces_origin_cache_policy() { + let settings = settings_with_enabled_auction_and_creative_opportunities(); + + for cache_control in ["no-cache", "max-age=0", "must-revalidate", "s-maxage=0"] { + // Arrange + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, cache_control); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + // Act + let response = + run_with_slots(&settings, &services, &[], conditional_navigation_request()) + .await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("max-age=60"), + "inactive server-side ad templates should replace origin {cache_control} policy" + ); + } + } + + #[tokio::test] + async fn navigation_without_matched_slots_preserves_private_origin_cache_policy() { + let settings = settings_with_enabled_auction_and_creative_opportunities(); + + for cache_control in ["private, max-age=0", "No-Store"] { + // Arrange + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, cache_control); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + // Act + let response = + run_with_slots(&settings, &services, &[], conditional_navigation_request()) + .await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some(cache_control), + "inactive server-side ad templates should preserve private origin {cache_control} policy" + ); + for (header_name, expected) in [ + (header::ETAG, ORIGIN_ETAG), + (header::LAST_MODIFIED, ORIGIN_LAST_MODIFIED), + ( + header::HeaderName::from_static("surrogate-control"), + "max-age=300", + ), + ( + header::HeaderName::from_static("fastly-surrogate-control"), + "max-age=300", + ), + ( + header::HeaderName::from_static("cdn-cache-control"), + "max-age=300", + ), + ( + header::HeaderName::from_static("cloudflare-cdn-cache-control"), + "max-age=300", + ), + ] { + assert_eq!( + response_head + .headers + .get(&header_name) + .and_then(|value| value.to_str().ok()), + Some(expected), + "inactive server-side ad templates should preserve {header_name}" + ); + } + } + } + + #[tokio::test] + async fn request_scoped_ad_stack_suppression_preserves_origin_cache_policy() { + let settings = settings_with_enabled_auction_and_creative_opportunities(); + let slots = [article_slot()]; + let mut bot_request = conditional_navigation_request(); + bot_request.headers_mut().insert( + "user-agent", + HeaderValue::from_static("Mozilla/5.0 (compatible; Googlebot/2.1)"), + ); + let mut prefetch_request = conditional_navigation_request(); + prefetch_request + .headers_mut() + .insert("sec-purpose", HeaderValue::from_static("prefetch")); + + for (skip_reason, request, consent) in [ + ("bot", bot_request, non_regulated_consent()), + ("prefetch", prefetch_request, non_regulated_consent()), + ( + "consent denied", + conditional_navigation_request(), + crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::Gdpr, + ..Default::default() + }, + ), + ] { + // Arrange + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, "no-cache"); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + // Act + let response = + run_with_slots_and_consent(&settings, &services, &slots, request, consent) + .await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("no-cache"), + "{skip_reason} should retain the origin cache policy" + ); + } + } + + #[tokio::test] + async fn absent_creative_opportunities_use_short_browser_cache_policy() { + // Arrange + let settings = settings_without_creative_opportunities(); + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, "no-cache"); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + // Act + let response = + run_with_slots(&settings, &services, &[], conditional_navigation_request()).await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("max-age=60"), + "absent creative opportunities should be treated as an inactive server-side ad stack" + ); + } + + #[tokio::test] + async fn inactive_ad_stack_preserves_non_ok_response_cache_policy() { + let settings = settings_with_disabled_ad_templates(); + + for status in [206, 404, 500, 503] { + // Arrange + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_status_and_cache_control(&stub, status, "no-cache"); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + // Act + let response = run_with_slots( + &settings, + &services, + &[article_slot()], + conditional_navigation_request(), + ) + .await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("no-cache"), + "inactive server-side ad templates should preserve origin policy on {status}" + ); + } + } + + #[tokio::test] + async fn inactive_ad_stack_preserves_gpt_diagnostics_cache_privacy() { + // Arrange + let mut settings = settings_with_disabled_ad_templates(); + settings + .integrations + .insert_config("gpt_diagnostics", &serde_json::json!({ "enabled": true })) + .expect("should enable GPT diagnostics"); + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, "no-cache"); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let request = HttpRequest::builder() + .method(Method::GET) + .uri("https://ts.example.com/article?ts_console=1") + .header(header::HOST, "ts.example.com") + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build GPT diagnostics request"); + + // Act + let response = run_with_slots(&settings, &services, &[article_slot()], request).await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("private, no-store"), + "active GPT diagnostics should retain cache privacy when server-side ad templates are inactive" + ); + } + + #[tokio::test] + async fn inactive_ad_stack_preserves_non_get_and_non_document_cache_policy() { + let settings = settings_with_disabled_ad_templates(); + + for request in [ + HttpRequest::builder() + .method(Method::POST) + .uri("https://ts.example.com/article") + .header(header::HOST, "ts.example.com") + .header("sec-fetch-dest", "document") + .body(EdgeBody::empty()) + .expect("should build non-GET document request"), + HttpRequest::builder() + .method(Method::GET) + .uri("https://ts.example.com/article") + .header(header::HOST, "ts.example.com") + .header("sec-fetch-dest", "empty") + .body(EdgeBody::empty()) + .expect("should build non-document request"), + ] { + // Arrange + let stub = Arc::new(StubHttpClient::new()); + queue_html_response_with_cache_control(&stub, "no-cache"); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + + // Act + let response = + run_with_slots(&settings, &services, &[article_slot()], request).await; + let response_head = response_head(response); + + // Assert + assert_eq!( + response_head + .headers + .get(header::CACHE_CONTROL) .and_then(|value| value.to_str().ok()), - Some(expected), - "publisher response without matched slots should preserve {header_name}" + Some("no-cache"), + "inactive server-side ad templates should preserve non-document request policy" ); } } @@ -5200,7 +13330,9 @@ mod tests { // Assert let response = match response { PublisherResponse::Buffered(response) => response, - PublisherResponse::PassThrough { .. } | PublisherResponse::Stream { .. } => { + PublisherResponse::PassThrough { .. } + | PublisherResponse::Stream { .. } + | PublisherResponse::AssembleTemplate { .. } => { panic!("unexpected origin 304 should return a buffered response") } }; @@ -5217,6 +13349,13 @@ mod tests { Some("private, no-store"), "eligible origin 304 should return an explicitly non-storable response" ); + assert!( + response + .extensions() + .get::() + .is_some(), + "invalid origin 304 response should remain private after late response effects" + ); for header_name in [ header::ETAG, header::LAST_MODIFIED, @@ -5283,7 +13422,9 @@ mod tests { // Assert let response = match response { PublisherResponse::Buffered(response) => response, - PublisherResponse::PassThrough { .. } | PublisherResponse::Stream { .. } => { + PublisherResponse::PassThrough { .. } + | PublisherResponse::Stream { .. } + | PublisherResponse::AssembleTemplate { .. } => { panic!("noneligible origin 304 should remain buffered") } }; @@ -5336,6 +13477,188 @@ mod tests { } } + #[tokio::test] + async fn publisher_asset_cache_policy_applies_to_non_html_response() { + let settings = Settings::from_toml(&format!( + r#"{} + + [[cache.asset_rules]] + id = "publisher-fingerprinted-assets" + enabled = true + path_globs = ["/assets/**/*.png"] + fingerprint_style = "hex" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + )) + .expect("should parse settings with cache rule"); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"png".to_vec(), + vec![ + (header::CONTENT_TYPE.as_str(), "image/png"), + (header::CACHE_CONTROL.as_str(), "public, max-age=60"), + ], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/assets/logo.0123abcd.png") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + + let response = run_publisher_proxy(&settings, &services, request).await; + let PublisherResponse::PassThrough { response, .. } = response else { + panic!("should pass through non-HTML asset response"); + }; + + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("public, max-age=31536000, s-maxage=31536000, immutable"), + "matched publisher asset should receive immutable browser and edge policy" + ); + assert_eq!( + response + .headers() + .get("surrogate-control") + .and_then(|value| value.to_str().ok()), + None, + "S-maxage fallback should keep the edge TTL in Cache-Control" + ); + } + + #[tokio::test] + async fn publisher_asset_policy_response_with_cookie_is_private_after_finalization() { + let settings = Settings::from_toml(&format!( + r#"{} + + [[cache.asset_rules]] + id = "publisher-fingerprinted-assets" + enabled = true + path_globs = ["/assets/**/*.png"] + fingerprint_style = "hex" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + )) + .expect("should parse settings with cache rule"); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"png".to_vec(), + vec![ + (header::CONTENT_TYPE.as_str(), "image/png"), + (header::CACHE_CONTROL.as_str(), "public, max-age=60"), + (header::SET_COOKIE.as_str(), "viewer=example; Path=/"), + ], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/assets/logo.0123abcd.png") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + + let response = run_publisher_proxy(&settings, &services, request).await; + let PublisherResponse::PassThrough { mut response, .. } = response else { + panic!("should pass through non-HTML asset response"); + }; + crate::response_privacy::apply_response_headers_with_cache_privacy( + &settings, + &mut response, + ); + + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("private, max-age=0"), + "publisher asset with Set-Cookie must become private after finalization" + ); + assert!( + response.headers().get("surrogate-control").is_none(), + "publisher asset with Set-Cookie must not retain a shared-cache header" + ); + } + + #[tokio::test] + async fn publisher_asset_cache_policy_skips_html_response() { + let settings = Settings::from_toml(&format!( + r#"{} + + [[cache.asset_rules]] + id = "broad-publisher-path" + enabled = true + path_glob = "/news/*.html" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + fingerprint_style = "hex" + "#, + crate_test_settings_str() + )) + .expect("should parse settings with cache rule"); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response_with_headers( + 200, + b"news".to_vec(), + vec![ + (header::CONTENT_TYPE.as_str(), "text/html; charset=utf-8"), + (header::CACHE_CONTROL.as_str(), "public, max-age=60"), + ], + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let request = HttpRequest::builder() + .method(Method::GET) + .uri("https://publisher.example/news/story.0123abcd.html") + .header(header::HOST, "publisher.example") + .body(EdgeBody::empty()) + .expect("should build request"); + + let response = run_publisher_proxy(&settings, &services, request).await; + let response = match response { + PublisherResponse::Stream { response, .. } | PublisherResponse::Buffered(response) => { + response + } + PublisherResponse::PassThrough { .. } | PublisherResponse::AssembleTemplate { .. } => { + panic!("should classify HTML response for processing") + } + }; + + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("public, max-age=60"), + "asset policy must not apply shared asset caching to publisher HTML" + ); + assert!( + response.headers().get("surrogate-control").is_none(), + "HTML response must not receive a shared-cache header" + ); + } + #[tokio::test] async fn publisher_request_uses_platform_http_client_with_http_types() { let settings = create_test_settings(); @@ -5357,7 +13680,8 @@ mod tests { *response.body_mut() = body; response } - PublisherResponse::Stream { response, .. } => response, + PublisherResponse::Stream { response, .. } + | PublisherResponse::AssembleTemplate { response, .. } => response, }; assert_eq!(response.status(), StatusCode::OK); @@ -5619,6 +13943,7 @@ mod tests { registry: None, }, req, + EdgeCacheHeader::SMaxageFallback, ) .await .expect("should proxy publisher request"); @@ -5984,41 +14309,70 @@ mod tests { #[test] fn server_side_ad_stack_runs_only_when_all_auction_gates_pass() { + let enabled_config = ServerSideAdStackConfig { + ad_templates_enabled: true, + auction_enabled: true, + }; assert!( - should_run_server_side_ad_stack(true, true, false, false, true, true, true), - "GET, real navigation, matched slots, and consent should run TS ad stack" + should_run_server_side_ad_stack(true, true, false, false, true, true, enabled_config,), + "GET, real navigation, enabled templates, matched slots, and consent should run TS ad stack" ); assert!( - !should_run_server_side_ad_stack(false, true, false, false, true, true, true), + !should_run_server_side_ad_stack(false, true, false, false, true, true, enabled_config,), "non-GET requests should skip TS ad stack" ); assert!( - !should_run_server_side_ad_stack(true, false, false, false, true, true, true), + !should_run_server_side_ad_stack(true, false, false, false, true, true, enabled_config,), "non-document requests should skip TS ad stack" ); assert!( - !should_run_server_side_ad_stack(true, true, true, false, true, true, true), + !should_run_server_side_ad_stack(true, true, true, false, true, true, enabled_config,), "prefetch requests should skip TS ad stack and injection" ); assert!( - !should_run_server_side_ad_stack(true, true, false, true, true, true, true), + !should_run_server_side_ad_stack(true, true, false, true, true, true, enabled_config,), "bot requests should skip TS ad stack and injection" ); assert!( - !should_run_server_side_ad_stack(true, true, false, false, false, true, true), + !should_run_server_side_ad_stack(true, true, false, false, false, true, enabled_config,), "requests with no matching slots should skip TS ad stack" ); assert!( - !should_run_server_side_ad_stack(true, true, false, false, true, false, true), + !should_run_server_side_ad_stack(true, true, false, false, true, false, enabled_config,), "requests without required consent should skip TS ad stack and injection" ); assert!( - !should_run_server_side_ad_stack(true, true, false, false, true, true, false), + !should_run_server_side_ad_stack( + true, + true, + false, + false, + true, + true, + ServerSideAdStackConfig { + ad_templates_enabled: true, + auction_enabled: false, + }, + ), "disabled [auction].enabled kill switch should skip TS ad stack and injection" ); + assert!( + !should_run_server_side_ad_stack( + true, + true, + false, + false, + true, + true, + ServerSideAdStackConfig { + ad_templates_enabled: false, + auction_enabled: true, + }, + ), + "disabled [creative_opportunities].enabled switch should skip TS ad stack and injection" + ); } - #[tokio::test] async fn body_close_hold_loop_processes_close_tail_before_reading_post_body_chunks() { let settings = create_test_settings(); @@ -6039,7 +14393,7 @@ mod tests { read_count: Arc::clone(&read_count), body_close_processed_at: Arc::clone(&body_close_processed_at), }; - let ad_bids_state = Arc::new(Mutex::new(None)); + let ad_bids_state = AdBidsState::default(); let ctx = AuctionCollectCtx { dispatched, telemetry: AuctionTelemetryCarry { @@ -6084,7 +14438,7 @@ mod tests { let settings = create_test_settings(); let services = noop_services(); let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); - let ad_bids_state = Arc::new(Mutex::new(None)); + let ad_bids_state = AdBidsState::default(); let mut state = AuctionHoldState::new( DispatchedAuctionGuard::new(DispatchedAuction::empty_for_test( test_auction_request(), @@ -6133,6 +14487,7 @@ mod tests { ); assert!( ad_bids_state + .script_cell() .lock() .expect("should lock bid state") .is_none(), @@ -6150,6 +14505,7 @@ mod tests { ); assert!( ad_bids_state + .script_cell() .lock() .expect("should lock bid state") .is_some(), @@ -6581,6 +14937,59 @@ mod tests { ); } + #[test] + fn esi_reader_encoding_negotiation_honours_quality_identity_and_repeated_fields() { + let headers = |values: &[&str]| { + let mut headers = edgezero_core::http::HeaderMap::new(); + for value in values { + headers.append( + header::ACCEPT_ENCODING, + HeaderValue::from_str(value).expect("should build accept-encoding"), + ); + } + headers + }; + + assert_eq!( + negotiate_reader_compression(&headers(&[])), + Ok(Compression::None) + ); + assert_eq!( + negotiate_reader_compression(&headers(&["gzip;q=0.8", "br;q=0.4, identity;q=0.1"])), + Ok(Compression::Gzip) + ); + assert_eq!( + negotiate_reader_compression(&headers(&["gzip, br"])), + Ok(Compression::Brotli), + "server preference breaks an equal-quality tie" + ); + assert_eq!( + negotiate_reader_compression(&headers(&["gzip;q=0.5"])), + Ok(Compression::None), + "implicit identity has q=1" + ); + assert_eq!( + negotiate_reader_compression(&headers(&["zstd, identity;q=0"])), + Err(ReaderEncodingError::NoAcceptableEncoding) + ); + assert_eq!( + negotiate_reader_compression(&headers(&["gzip;q=invalid"])), + Err(ReaderEncodingError::Malformed) + ); + for malformed in [ + "gzip;q=1e-1", + "gzip;q=0.1234", + "gzip;q=1.001", + "not a coding;q=1", + ] { + assert_eq!( + negotiate_reader_compression(&headers(&[malformed])), + Err(ReaderEncodingError::Malformed), + "{malformed} is not valid Accept-Encoding syntax" + ); + } + } + #[test] fn tsjs_dynamic_returns_not_found_for_unknown_filename() { let settings = create_test_settings(); @@ -6591,7 +15000,8 @@ mod tests { "https://publisher.example/static/tsjs=unknown.js", ); - let response = handle_tsjs_dynamic(&req, ®istry).expect("should handle tsjs request"); + let response = handle_tsjs_dynamic(&req, ®istry, EdgeCacheHeader::SMaxageFallback) + .expect("should handle tsjs request"); assert_eq!(response.status(), StatusCode::NOT_FOUND); } @@ -6605,7 +15015,8 @@ mod tests { "https://publisher.example/static/tsjs=tsjs-unified.min.js", ); - let response = handle_tsjs_dynamic(&req, ®istry).expect("should handle tsjs request"); + let response = handle_tsjs_dynamic(&req, ®istry, EdgeCacheHeader::SMaxageFallback) + .expect("should handle tsjs request"); assert_eq!(response.status(), StatusCode::OK); } @@ -6627,7 +15038,8 @@ mod tests { HeaderValue::from_static("__Host-ts-console=1"), ); - let response = handle_tsjs_dynamic(&req, ®istry).expect("should handle tsjs request"); + let response = handle_tsjs_dynamic(&req, ®istry, EdgeCacheHeader::SMaxageFallback) + .expect("should handle tsjs request"); assert_eq!(response.status(), StatusCode::OK); assert!(!response.headers().contains_key(header::SET_COOKIE)); @@ -6689,7 +15101,8 @@ mod tests { "https://publisher.example/static/tsjs=tsjs-prebid.min.js", ); - let response = handle_tsjs_dynamic(&req, ®istry).expect("should handle tsjs request"); + let response = handle_tsjs_dynamic(&req, ®istry, EdgeCacheHeader::SMaxageFallback) + .expect("should handle tsjs request"); assert_eq!( response.status(), StatusCode::OK, @@ -6718,7 +15131,8 @@ mod tests { "https://publisher.example/static/tsjs=tsjs-prebid.min.js", ); - let response = handle_tsjs_dynamic(&req, ®istry).expect("should handle tsjs request"); + let response = handle_tsjs_dynamic(&req, ®istry, EdgeCacheHeader::SMaxageFallback) + .expect("should handle tsjs request"); assert_eq!( response.status(), StatusCode::NOT_FOUND, @@ -6736,7 +15150,8 @@ mod tests { "https://publisher.example/static/tsjs=tsjs-evil.min.js", ); - let response = handle_tsjs_dynamic(&req, ®istry).expect("should handle tsjs request"); + let response = handle_tsjs_dynamic(&req, ®istry, EdgeCacheHeader::SMaxageFallback) + .expect("should handle tsjs request"); assert_eq!( response.status(), StatusCode::NOT_FOUND, @@ -6744,6 +15159,106 @@ mod tests { ); } + #[test] + fn tsjs_dynamic_uses_immutable_cache_for_matching_hash() { + let settings = create_test_settings(); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let module_ids = registry.js_module_ids_immediate(); + let hash = trusted_server_js::concatenated_hash(&module_ids); + let request = build_request( + Method::GET, + &format!("https://publisher.example/static/tsjs=tsjs-unified.min.js?v={hash}"), + ); + + let response = handle_tsjs_dynamic(&request, ®istry, EdgeCacheHeader::SurrogateControl) + .expect("should handle tsjs request"); + + assert_eq!(response.status(), StatusCode::OK); + assert_eq!( + response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()), + Some("public, max-age=31536000, immutable"), + "matching content-versioned bundle should be immutable" + ); + assert_eq!( + response + .headers() + .get("surrogate-control") + .and_then(|value| value.to_str().ok()), + Some("max-age=31536000"), + "matching content-versioned bundle should set the Fastly edge TTL" + ); + } + + #[test] + fn tsjs_dynamic_uses_cloudflare_edge_header_when_selected() { + let settings = create_test_settings(); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let module_ids = registry.js_module_ids_immediate(); + let hash = trusted_server_js::concatenated_hash(&module_ids); + let request = build_request( + Method::GET, + &format!("https://publisher.example/static/tsjs=tsjs-unified.min.js?v={hash}"), + ); + + let response = handle_tsjs_dynamic( + &request, + ®istry, + EdgeCacheHeader::CloudflareCdnCacheControl, + ) + .expect("should handle tsjs request"); + + assert_eq!( + response + .headers() + .get("cloudflare-cdn-cache-control") + .and_then(|value| value.to_str().ok()), + Some("max-age=31536000"), + "Cloudflare requests should use its edge cache header" + ); + assert!( + response.headers().get("surrogate-control").is_none(), + "Cloudflare requests should not emit Fastly's edge cache header" + ); + } + + #[test] + fn tsjs_dynamic_keeps_short_cache_for_mismatched_hash() { + let settings = create_test_settings(); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let request = build_request( + Method::GET, + "https://publisher.example/static/tsjs=tsjs-unified.min.js?v=not-the-hash", + ); + + let response = handle_tsjs_dynamic(&request, ®istry, EdgeCacheHeader::SurrogateControl) + .expect("should handle tsjs request"); + let cache_control = response + .headers() + .get(header::CACHE_CONTROL) + .and_then(|value| value.to_str().ok()) + .expect("should set cache-control"); + + assert_eq!(response.status(), StatusCode::OK); + assert!( + cache_control.contains("max-age=300") && !cache_control.contains("immutable"), + "mismatched hash should retain the short, mutable cache policy" + ); + assert_eq!( + response + .headers() + .get("surrogate-control") + .and_then(|value| value.to_str().ok()), + Some("max-age=300, stale-while-revalidate=60, stale-if-error=86400"), + "mismatched hash should retain the short Fastly edge TTL" + ); + } + #[tokio::test] async fn publisher_request_sends_configured_host_header_override() { let mut settings = create_test_settings(); @@ -6799,6 +15314,10 @@ mod tests { let body = EdgeBody::from(compressed); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "gzip".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -6806,7 +15325,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -6848,6 +15367,10 @@ mod tests { IntegrationRegistry::new(&settings).expect("should create integration registry"); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -6855,7 +15378,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/html; charset=utf-8".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -6886,6 +15409,10 @@ mod tests { let registry = IntegrationRegistry::new(&settings).expect("should create integration registry"); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -6893,7 +15420,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/html; charset=utf-8".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7002,6 +15529,10 @@ mod tests { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7009,7 +15540,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7056,6 +15587,10 @@ mod tests { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "gzip".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7063,7 +15598,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7113,6 +15648,10 @@ mod tests { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "deflate".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7120,7 +15659,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7170,6 +15709,10 @@ mod tests { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "br".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7177,7 +15720,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7227,6 +15770,10 @@ mod tests { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "br".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7234,7 +15781,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7272,6 +15819,10 @@ mod tests { fn non_html_stream_params(content_encoding: &str) -> OwnedProcessResponseParams { OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: content_encoding.to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7279,7 +15830,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7459,8 +16010,12 @@ mod tests { IntegrationRegistry::new(&settings).expect("should create integration registry"); let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); - let state = Arc::new(Mutex::new(None)); + let state = AdBidsState::default(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7524,8 +16079,12 @@ mod tests { IntegrationRegistry::new(&settings).expect("should create integration registry"); let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); - let state = Arc::new(Mutex::new(None)); + let state = AdBidsState::default(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "gzip".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7593,6 +16152,10 @@ mod tests { let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); let services = noop_services(); let mut params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7600,7 +16163,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: Some(test_auction_request()), dispatched_auction: Some(DispatchedAuction::empty_for_test( @@ -7653,6 +16216,10 @@ mod tests { .body(EdgeBody::empty()) .expect("should build response"); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: content_encoding.to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7660,7 +16227,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/css".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -7795,6 +16362,10 @@ mod tests { dispatched_auction: Option, ) -> OwnedProcessResponseParams { OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: content_encoding.to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -7805,7 +16376,7 @@ mod tests { r#""# .to_string(), ), - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: dispatched_auction.as_ref().map(|_| test_auction_request()), dispatched_auction, @@ -8175,6 +16746,10 @@ mod tests { let ec_context = EcContext::new_for_test(None, crate::consent::types::ConsentContext::default()); OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -8182,7 +16757,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/html; charset=utf-8".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: Some(AuctionObservationContext::from_parts( AuctionSource::SpaNavigation, "proxy.example.com", @@ -8358,6 +16933,10 @@ mod tests { .map(bytes::Bytes::copy_from_slice) .collect(); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "gzip".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -8368,7 +16947,7 @@ mod tests { r#""# .to_string(), ), - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: Some(test_auction_request()), dispatched_auction: Some(DispatchedAuction::empty_for_test( @@ -8427,8 +17006,12 @@ mod tests { IntegrationRegistry::new(&settings).expect("should create integration registry"); let bids_script = r#""#; - let state = Arc::new(Mutex::new(Some(bids_script.to_string()))); + let state = AdBidsState::with_script(bids_script); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -8483,6 +17066,10 @@ mod tests { // Claim gzip encoding but feed non-gzip bytes. The GzDecoder will // error as soon as it tries to read the gzip header. let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: "gzip".to_string(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -8490,7 +17077,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/html".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -8592,6 +17179,10 @@ mod tests { let body = EdgeBody::from(html.to_vec()); let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -8599,7 +17190,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/html; charset=utf-8".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -8650,6 +17241,10 @@ mod tests { // Small, single-fragment RSC script — placeholder path (not fallback). let html = br#""#; let params = OwnedProcessResponseParams { + csp_nonce_observed: None, + template_cache_key: None, + seam_ad_slots: None, + policy_headers: Vec::new(), content_encoding: String::new(), origin_host: "origin.example.com".to_string(), origin_url: "https://origin.example.com".to_string(), @@ -8657,7 +17252,7 @@ mod tests { request_scheme: "https".to_string(), content_type: "text/html".to_string(), ad_slots_script: None, - ad_bids_state: Arc::new(Mutex::new(None)), + ad_bids_state: AdBidsState::default(), auction_observation: None, auction_request: None, dispatched_auction: None, @@ -8694,8 +17289,9 @@ mod tests { #[cfg(test)] mod creative_opportunities_tests { use super::super::{ - MatchedSlotsContext, build_ad_slots_script, build_auction_request, build_bid_map, - build_bids_script, diagnostics_auction_id, html_escape_for_script, write_bids_to_state, + AdBidsState, MatchedSlotsContext, build_ad_slots_script, build_auction_request, + build_bid_map, build_bids_script, diagnostics_auction_id, html_escape_for_script, + write_bids_to_state, }; use crate::auction::types::{ApsRendererV1, ApsTagType, Bid, BidRenderer, MediaType}; use crate::consent::ConsentContext; @@ -8716,10 +17312,15 @@ mod tests { fn make_config() -> CreativeOpportunitiesConfig { CreativeOpportunitiesConfig { + enabled: true, gam_network_id: "21765378893".to_string(), auction_timeout_ms: Some(500), price_granularity: PriceGranularity::Dense, section_root: None, + assembly_mode: None, + template_cache_vary: None, + template_cache_max_age_seconds: None, + origin_is_cookie_independent: None, section_segment: None, slot: Vec::new(), } @@ -9014,7 +17615,7 @@ mod tests { ), ); - let state = std::sync::Arc::new(std::sync::Mutex::new(None)); + let state = AdBidsState::default(); write_bids_to_state( &winning_bids, PriceGranularity::Dense, @@ -9025,6 +17626,7 @@ mod tests { Some(&auction_request.id), ); let script = state + .script_cell() .lock() .expect("should lock initial bid state") .clone() @@ -9059,6 +17661,7 @@ mod tests { Some(&auction_request.id), ); let empty_script = state + .script_cell() .lock() .expect("should lock empty initial bid state") .clone() @@ -10467,6 +19070,58 @@ mod tests { make_page_bids_request_on(PAGE_BIDS_PATH, path) } + #[tokio::test] + async fn page_bids_format_absent_or_json_returns_json() { + let settings = settings_with_co(); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + for path_and_format in ["/2024/article", "/2024/article&format=json"] { + let response = run_page_bids_response( + &settings, + &orchestrator, + &[], + make_page_bids_request(path_and_format), + ) + .await; + assert_eq!( + response.status(), + StatusCode::OK, + "should accept page-bids format in `{path_and_format}`" + ); + assert_eq!( + response.headers().get(header::CONTENT_TYPE), + Some(&HeaderValue::from_static("application/json")), + "should return JSON for `{path_and_format}`" + ); + assert!( + response + .extensions() + .get::() + .is_some(), + "successful per-user page-bids JSON should remain terminal-private" + ); + } + } + + #[tokio::test] + async fn page_bids_format_rejects_removed_unknown_and_empty_values() { + let settings = settings_with_co(); + let orchestrator = AuctionOrchestrator::new(settings.auction.clone()); + for format in ["fragment", "scrpit", ""] { + let response = run_page_bids_response( + &settings, + &orchestrator, + &[], + make_page_bids_request(&format!("/2024/article&format={format}")), + ) + .await; + assert_eq!( + response.status(), + StatusCode::BAD_REQUEST, + "should reject page-bids format `{format}`" + ); + } + } + /// Builds a page-bids request against an explicit endpoint path, so the /// canonical route and its deprecated alias can be compared directly. fn make_page_bids_request_on(endpoint: &str, path: &str) -> Request { @@ -11504,6 +20159,7 @@ mod tests { registry: None, }, req, + EdgeCacheHeader::SMaxageFallback, ) .await .expect("should proxy publisher request"); @@ -11587,6 +20243,7 @@ mod tests { registry: None, }, req, + EdgeCacheHeader::SMaxageFallback, ) .await .expect("should proxy publisher request"); diff --git a/crates/trusted-server-core/src/response_privacy.rs b/crates/trusted-server-core/src/response_privacy.rs index 21ba9f20b..7aeb89876 100644 --- a/crates/trusted-server-core/src/response_privacy.rs +++ b/crates/trusted-server-core/src/response_privacy.rs @@ -9,10 +9,21 @@ //! cache such as Cloudflare would otherwise serve an operator/origin //! `Cache-Control: public` on a cookie-bearing response as-is. -use edgezero_core::http::{HeaderName, HeaderValue, Response, header}; +use edgezero_core::http::{HeaderMap, HeaderName, HeaderValue, Response, header}; +use crate::cache_policy::{ + cache_control_headers_are_private_or_no_store, is_edge_cache_header_name, + remove_edge_cache_headers, +}; use crate::settings::Settings; +/// Marks a response whose `private, no-store` policy belongs to Trusted Server. +/// +/// Platform terminal hooks use this out-of-band marker to re-enforce the policy after +/// late integrations run without rewriting unrelated origin-private responses. +#[derive(Clone, Copy, Debug)] +pub struct TerminalPrivateResponse; + /// CDN-targeted cache headers stripped from every cookie-bearing response. /// /// A single source of truth so the adapter copies of the privacy downgrade @@ -30,21 +41,72 @@ fn strip_cdn_cache_headers(response: &mut Response) { } } -/// Forces synthesized HTML to be private and non-storable. +/// Whether `Cache-Control` already forbids shared caching. /// -/// Use this exact policy whenever Trusted Server changes an origin HTML -/// representation with request-specific content: force `private, no-store`, -/// remove origin validators, and remove all CDN-targeted cache directives. -pub(crate) fn enforce_synthesized_html_cache_privacy(response: &mut Response) { +/// Extracted because both arms of the cookie-privacy net below need it. +/// +/// `publisher::template_cache_bypass_reason` deliberately does **not** call this and keeps its own +/// copy: it additionally treats `no-cache` as non-shareable, because "revalidate before +/// reuse" is correct for an HTTP cache and too permissive for a spike-owned one. The +/// duplicate is the stricter of the two, so consolidating them would loosen the shared +/// template-cache gate rather than tidy it. +/// +/// Directives are case-insensitive (RFC 9111 §5.2), so `No-Store` and `Private` +/// count. `no-cache` deliberately does **not**: it requires revalidation before +/// reuse, not a refusal to store, so a `no-cache` response is still shareable. +/// Callers needing the stricter reading must check it themselves. +#[must_use] +pub fn is_private_or_no_store(headers: &HeaderMap) -> bool { + cache_control_headers_are_private_or_no_store(headers) +} + +/// Reassert the terminal privacy invariant for a synthesized per-reader response. +/// +/// Call this after every configurable response mutation. It deliberately overwrites +/// `Cache-Control` and strips validators, expiry metadata, and CDN-specific cache +/// directives so a later integration cannot turn an assembled document into C3. +pub fn enforce_private_no_store(response: &mut Response) { response.headers_mut().insert( header::CACHE_CONTROL, HeaderValue::from_static("private, no-store"), ); - response.headers_mut().remove(header::ETAG); - response.headers_mut().remove(header::LAST_MODIFIED); + for name in [ + header::ETAG.as_str(), + header::LAST_MODIFIED.as_str(), + header::EXPIRES.as_str(), + header::AGE.as_str(), + ] { + response.headers_mut().remove(name); + } strip_cdn_cache_headers(response); } +/// Marks a Trusted Server response as terminal-private and applies its cache policy. +pub(crate) fn enforce_terminal_private_cache_privacy(response: &mut Response) { + enforce_private_no_store(response); + response.extensions_mut().insert(TerminalPrivateResponse); +} + +/// Forces synthesized HTML to be private and non-storable. +/// +/// Use this exact policy whenever Trusted Server changes an origin HTML +/// representation with request-specific content: force `private, no-store`, +/// remove origin validators, and remove all CDN-targeted cache directives. +pub(crate) fn enforce_synthesized_html_cache_privacy(response: &mut Response) { + enforce_terminal_private_cache_privacy(response); +} + +/// Removes runtime edge-cache headers from a response finalized as uncacheable. +/// +/// Call this after any late response-header mutations so a final `private` or +/// `no-store` directive cannot coexist with an independently authoritative edge +/// cache header. +pub fn enforce_uncacheable_cache_privacy(response: &mut Response) { + if is_private_or_no_store(response.headers()) { + remove_edge_cache_headers(response.headers_mut()); + } +} + /// Forces cookie-bearing responses to stay private to shared caches. /// /// Any response that sets a per-user cookie (notably the EC identity cookie) @@ -63,14 +125,7 @@ pub fn enforce_set_cookie_cache_privacy(response: &mut Response) { // independent of Cache-Control and would otherwise let a shared cache store // and replay one visitor's Set-Cookie. strip_cdn_cache_headers(response); - // Cache-Control directives are case-insensitive (RFC 9111 §5.2), so match - // against a lowercased copy — `No-Store` / `Private` must count. - let already_uncacheable = response - .headers() - .get(header::CACHE_CONTROL) - .and_then(|v| v.to_str().ok()) - .map(str::to_ascii_lowercase) - .is_some_and(|v| v.contains("private") || v.contains("no-store")); + let already_uncacheable = is_private_or_no_store(response.headers()); if !already_uncacheable { response.headers_mut().insert( header::CACHE_CONTROL, @@ -96,19 +151,13 @@ pub fn enforce_set_cookie_cache_privacy(response: &mut Response) { pub fn apply_response_headers_with_cache_privacy(settings: &Settings, response: &mut Response) { enforce_set_cookie_cache_privacy(response); - let response_is_uncacheable = response - .headers() - .get(header::CACHE_CONTROL) - .and_then(|v| v.to_str().ok()) - .map(str::to_ascii_lowercase) - .is_some_and(|v| v.contains("private") || v.contains("no-store")); + let response_is_uncacheable = is_private_or_no_store(response.headers()); + enforce_uncacheable_cache_privacy(response); for (key, value) in &settings.response_headers { if response_is_uncacheable && (key.eq_ignore_ascii_case(header::CACHE_CONTROL.as_str()) - || CDN_CACHE_HEADERS - .iter() - .any(|name| key.eq_ignore_ascii_case(name))) + || is_edge_cache_header_name(key)) { continue; } @@ -129,6 +178,8 @@ pub fn apply_response_headers_with_cache_privacy(settings: &Settings, response: response.headers_mut().insert(header_name, header_value); } + enforce_uncacheable_cache_privacy(response); + // Operator headers can themselves introduce Set-Cookie (alongside public // or surrogate cache headers) onto a previously cookieless response, which // the pre-apply pass could not see. Re-run the downgrade so the final @@ -188,6 +239,13 @@ mod tests { "private, no-store", "synthesized HTML should always be non-storable" ); + assert!( + response + .extensions() + .get::() + .is_some(), + "should mark synthesized HTML for terminal privacy re-enforcement" + ); for header_name in [header::ETAG.as_str(), header::LAST_MODIFIED.as_str()] .into_iter() .chain(CDN_CACHE_HEADERS.iter().copied()) @@ -296,6 +354,40 @@ mod tests { } } + #[test] + fn terminal_private_stamp_removes_every_cache_and_validator_header() { + let mut response = response_builder() + .header(header::CACHE_CONTROL, "public, s-maxage=600") + .header(header::ETAG, "\"origin\"") + .header(header::LAST_MODIFIED, "Wed, 12 Aug 2026 00:00:00 GMT") + .header(header::EXPIRES, "Wed, 12 Aug 2026 01:00:00 GMT") + .header(header::AGE, "30") + .header("surrogate-control", "max-age=600") + .header("cdn-cache-control", "public, max-age=600") + .body(edgezero_core::body::Body::empty()) + .expect("should build response"); + + enforce_private_no_store(&mut response); + + assert_eq!( + response.headers()[header::CACHE_CONTROL], + "private, no-store" + ); + for name in [ + header::ETAG.as_str(), + header::LAST_MODIFIED.as_str(), + header::EXPIRES.as_str(), + header::AGE.as_str(), + "surrogate-control", + "cdn-cache-control", + ] { + assert!( + !response.headers().contains_key(name), + "terminal private stamp must strip {name}" + ); + } + } + #[test] fn applies_operator_headers_on_cookieless_response() { let settings = settings_with_response_headers(&[("x-operator", "value")]); diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 598c00056..38dba78e9 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -1,6 +1,7 @@ #[cfg(test)] use config::{Config, Environment, File, FileFormat}; use error_stack::{Report, ResultExt}; +use glob::{MatchOptions, Pattern}; use regex::Regex; use serde::{Deserialize, Deserializer, Serialize, de::DeserializeOwned}; use serde_json::Value as JsonValue; @@ -8,10 +9,12 @@ use std::collections::{HashMap, HashSet}; use std::ops::{Deref, DerefMut}; use std::str::FromStr; use std::sync::OnceLock; +use std::time::Duration; use url::Url; use validator::{Validate, ValidationError}; use crate::auction_config_types::AuctionConfig; +use crate::cache_policy::{CachePolicy, CacheVisibility}; use crate::consent_config::ConsentConfig; use crate::creative_opportunities::CreativeOpportunitiesConfig; use crate::error::TrustedServerError; @@ -1866,866 +1869,2228 @@ fn validate_tinybird_secret(value: &str, setting: &str) -> Result<(), Report, +} - /// Inject a `` HTML comment before `` dumping a - /// redacted per-provider auction result: pipeline stats (SSP count, mediator - /// status, winning bid count) plus every provider response — each bid's - /// creative previewed (not the full `adm` markup) and provider metadata - /// filtered to a fail-closed allowlist that drops identity-bearing keys. - /// Never enable in production — visible in page source and injects (bounded) - /// raw HTML from SSPs. - #[serde(default)] - pub auction_html_comment: bool, +impl CacheSettings { + fn normalize(&mut self) { + for rule in &mut self.asset_rules { + rule.normalize(); + } + } - /// Enable the testing-only direct GAM-replace path and the verbose per-bid - /// `debug_bid` blob in `window.tsjs.bids`. + /// Eagerly validate runtime-only cache settings artifacts. /// - /// Note: the sanitized winning `adm` is now injected **unconditionally** for - /// production inline rendering through the pbRender bridge (see - /// [`crate::publisher::build_bid_map`]); this flag no longer gates `adm`. - /// What it still gates is the client-side `debug_bid` signal that turns on - /// the direct GAM-creative replacement (`injectAdmIntoSlot`), which bypasses - /// GAM entirely — useful for validating the auction→creative pipeline while - /// PBS Cache is unavailable. The `debug_bid` blob also carries the raw, - /// un-sanitized creative for diagnostics, so never enable in production. - #[serde(default)] - pub inject_adm_for_testing: bool, + /// # Errors + /// + /// Returns a configuration error if any rule ID is duplicate, or if an + /// enabled rule has an invalid policy/matcher or cannot compile its regex/glob. + pub fn prepare_runtime(&self) -> Result<(), Report> { + let mut seen_ids = HashSet::new(); + for rule in &self.asset_rules { + if rule.id.is_empty() { + return Err(Report::new(TrustedServerError::Configuration { + message: "cache.asset_rules id must not be empty".to_string(), + })); + } + if !seen_ids.insert(rule.id.clone()) { + return Err(Report::new(TrustedServerError::Configuration { + message: format!("cache.asset_rules contains duplicate id `{}`", rule.id), + })); + } + } + for rule in &self.asset_rules { + rule.prepare_runtime()?; + } + Ok(()) + } + + /// Resolve the first enabled asset cache rule that matches `path`. + /// + /// # Errors + /// + /// Returns a configuration error if a lazily prepared matcher unexpectedly + /// fails to compile. + pub fn asset_policy_for_path( + &self, + path: &str, + ) -> Result, Report> { + for rule in &self.asset_rules { + if rule.matches_path(path)? { + return Ok(Some(rule.cache_policy())); + } + } + Ok(None) + } } -/// Tester-cookie endpoint configuration. +/// A configurable cache rule for publisher-origin or rehosted static assets. #[derive(Debug, Default, Clone, Deserialize, Serialize)] -pub struct TesterCookieConfig { - /// Enable tester-cookie endpoints that set and clear `ts-tester`. +#[serde(deny_unknown_fields)] +pub struct CacheAssetRule { + /// Stable operator-facing identifier for logs/tests/config errors. + pub id: String, + /// Whether this rule participates in matching. #[serde(default)] pub enabled: bool, -} - -#[derive(Debug, Default, Clone, Deserialize, Serialize, Validate)] -#[serde(deny_unknown_fields)] -pub struct Settings { - #[validate(nested)] - pub publisher: Publisher, + /// Built-in framework/static preset matcher. #[serde(default)] - pub tester_cookie: TesterCookieConfig, + pub preset: Option, + /// Raw path prefix matcher. #[serde(default)] - #[validate(nested)] - pub ec: Ec, + pub path_prefix: Option, + /// Single glob matcher retained for concise configs. #[serde(default)] - pub integrations: IntegrationSettings, - #[serde(default, deserialize_with = "vec_from_seq_or_map")] - #[validate(nested)] - pub handlers: Vec, - #[serde(default, deserialize_with = "map_from_obj_or_str")] - pub response_headers: HashMap, - pub request_signing: Option, + pub path_glob: Option, + /// Multiple glob matchers. #[serde(default)] - #[validate(nested)] - pub rewrite: Rewrite, + pub path_globs: Vec, + /// Regex matcher applied to the request path. #[serde(default)] - pub auction: AuctionConfig, + pub path_regex: Option, + /// File extensions matched against the request path, case-insensitively. #[serde(default)] - pub consent: ConsentConfig, + pub extensions: Vec, + /// Bundler fingerprint style required in the filename before matching. #[serde(default)] - pub proxy: Proxy, + pub fingerprint_style: Option, + /// Browser-facing cache visibility. #[serde(default)] - pub creative_opportunities: Option, + pub visibility: CachePolicyVisibility, + /// Browser cache TTL rendered as `max-age`. #[serde(default)] - pub image_optimizer: ImageOptimizerSettings, + pub browser_ttl_seconds: Option, + /// Shared edge cache TTL rendered as runtime-specific edge control. #[serde(default)] - pub tinybird: TinybirdSettings, + pub edge_ttl_seconds: Option, + /// Optional stale-while-revalidate duration. #[serde(default)] - pub debug: DebugConfig, + pub stale_while_revalidate_seconds: Option, + /// Optional stale-if-error duration. + #[serde(default)] + pub stale_if_error_seconds: Option, + /// Whether browser caches may treat the response as immutable. + #[serde(default)] + pub immutable: bool, + #[serde(skip)] + compiled_regex: OnceLock>, + #[serde(skip)] + compiled_globs: OnceLock, String>>, } -impl Settings { - /// Creates a new [`Settings`] instance from a TOML string. - /// - /// # Errors - /// - /// - [`TrustedServerError::Configuration`] if the TOML is invalid or missing required fields - pub fn from_toml(toml_str: &str) -> Result> { - let settings: Self = - toml::from_str(toml_str).change_context(TrustedServerError::Configuration { - message: "Failed to deserialize TOML configuration".to_string(), - })?; - - Self::finalize_deserialized(settings, "Configuration") +impl CacheAssetRule { + fn normalize(&mut self) { + self.id = self.id.trim().to_string(); + self.path_prefix = self + .path_prefix + .take() + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()); + self.path_glob = self + .path_glob + .take() + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()); + self.path_globs = self + .path_globs + .iter() + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()) + .collect(); + self.path_regex = self + .path_regex + .take() + .map(|value| value.trim().to_string()) + .filter(|value| !value.is_empty()); + self.extensions = self + .extensions + .iter() + .map(|value| value.trim().trim_start_matches('.').to_ascii_lowercase()) + .filter(|value| !value.is_empty()) + .collect(); } - /// Creates a new [`Settings`] instance from a JSON value. - /// - /// Runtime config-store loading uses this after verifying the `app_config` - /// blob envelope and extracting the same typed settings shape. - /// - /// # Errors - /// - /// - [`TrustedServerError::Configuration`] if the JSON value is invalid or missing required fields - pub fn from_json_value(value: JsonValue) -> Result> { - let settings: Self = - serde_json::from_value(value).change_context(TrustedServerError::Configuration { - message: "Failed to deserialize JSON configuration".to_string(), - })?; + fn prepare_runtime(&self) -> Result<(), Report> { + if !self.enabled { + return Ok(()); + } - Self::finalize_deserialized(settings, "Configuration") + self.validate_matcher_shape()?; + self.compiled_regex().map(|_| ())?; + self.compiled_globs().map(|_| ())?; + self.validate_policy_shape()?; + Ok(()) } - /// Creates a new [`Settings`] instance from a TOML string with legacy - /// test-only `TRUSTED_SERVER__` environment variable overrides. - /// - /// Runtime loading does not use this legacy helper; `EdgeZero` CLI app-config - /// overlays are applied before deserializing [`crate::config::TrustedServerAppConfig`]. - /// This helper remains available to existing tests that exercise legacy - /// parsing behavior. - /// - /// # Errors - /// - /// - [`TrustedServerError::Configuration`] if the TOML is invalid or missing required fields - #[cfg(test)] - pub fn from_toml_and_env(toml_str: &str) -> Result> { - let environment = Environment::default() - .prefix(ENVIRONMENT_VARIABLE_PREFIX) - .separator(ENVIRONMENT_VARIABLE_SEPARATOR); + fn validate_matcher_shape(&self) -> Result<(), Report> { + if self.path_glob.is_some() && !self.path_globs.is_empty() { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` must use path_glob or path_globs, not both", + self.id + ), + })); + } - let toml = File::from_str(toml_str, FileFormat::Toml); - let config = Config::builder() - .add_source(toml) - .add_source(environment) - .build() - .change_context(TrustedServerError::Configuration { - message: "Failed to build configuration".to_string(), - })?; - let settings: Self = - config - .try_deserialize() - .change_context(TrustedServerError::Configuration { - message: "Failed to deserialize configuration".to_string(), - })?; + let matcher_count = usize::from(self.preset.is_some()) + + usize::from(self.path_prefix.is_some()) + + usize::from(self.path_glob.is_some() || !self.path_globs.is_empty()) + + usize::from(self.path_regex.is_some()) + + usize::from(!self.extensions.is_empty()); - Self::finalize_deserialized(settings, "Build-time configuration") + if matcher_count != 1 { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` must configure exactly one matcher", + self.id + ), + })); + } + Ok(()) } - pub(crate) fn finalize_deserialized( - mut settings: Self, - validation_label: &str, - ) -> Result> { - settings.proxy.normalize(); - settings.image_optimizer.normalize(); - settings.consent.validate(); - - settings.prepare_runtime()?; + fn validate_policy_shape(&self) -> Result<(), Report> { + if self.visibility == CachePolicyVisibility::Private { + if self.edge_ttl_seconds.is_some() { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` sets edge_ttl_seconds with private visibility; private rules must use browser_ttl_seconds", + self.id + ), + })); + } + if self.browser_ttl_seconds.is_none() { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` with private visibility must configure browser_ttl_seconds", + self.id + ), + })); + } + } else if self.browser_ttl_seconds.is_none() && self.edge_ttl_seconds.is_none() { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` must configure browser_ttl_seconds or edge_ttl_seconds", + self.id + ), + })); + } - settings.validate().map_err(|err| { - Report::new(TrustedServerError::Configuration { - message: format!("{validation_label} validation failed: {err}"), - }) - })?; + if !self.immutable { + return Ok(()); + } - settings.validate_admin_coverage()?; - settings.validate_admin_handler_passwords()?; + if self + .browser_ttl_seconds + .is_none_or(|browser_ttl| browser_ttl == 0) + { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` sets immutable without a positive browser_ttl_seconds", + self.id + ), + })); + } - if settings.auction.enabled && !settings.auction.rewrite_creatives { - log::warn!( - "Auction creative rewriting disabled; creative assets and clicks may contact third-party hosts directly" - ); + let preset_is_content_addressed = + matches!(self.preset, Some(CacheAssetPreset::NextJsStatic)); + if !preset_is_content_addressed { + match self.fingerprint_style { + None => { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` sets immutable without fingerprint_style or a content-addressed preset", + self.id + ), + })); + } + Some(CacheAssetFingerprintStyle::ViteBase64Url) => { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` cannot set immutable with vite-base64-url; use a content-addressed preset or an unambiguous fingerprint_style", + self.id + ), + })); + } + Some(_) => {} + } } - Ok(settings) + Ok(()) } - /// Eagerly prepare runtime-only settings artifacts. - /// - /// # Errors - /// - /// Returns a configuration error if any cached runtime artifact cannot be - /// prepared, if any handler path regex does not compile, or if a creative - /// opportunity slot is invalid. - pub fn prepare_runtime(&mut self) -> Result<(), Report> { - self.image_optimizer.prepare_runtime()?; - self.proxy.prepare_runtime()?; - self.tinybird.prepare_runtime()?; - self.validate_asset_image_optimizer_profile_sets()?; + fn compiled_regex(&self) -> Result, Report> { + let Some(pattern) = self.path_regex.as_deref() else { + return Ok(None); + }; + match self + .compiled_regex + .get_or_init(|| Regex::new(pattern).map_err(|err| err.to_string())) + { + Ok(regex) => Ok(Some(regex)), + Err(message) => Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` path_regex `{pattern}` failed to compile: {message}", + self.id + ), + })), + } + } - for handler in &self.handlers { - handler.prepare_runtime()?; + fn compiled_globs(&self) -> Result, Report> { + if self.path_glob.is_none() && self.path_globs.is_empty() { + return Ok(None); } - if let Some(co) = &mut self.creative_opportunities { - co.compile_slots(); - // Parse `gam_unit_path` templates once here (mirrors the compiled - // glob cache) so request-time rendering is substitution-only. - co.compile_unit_templates().map_err(|err| { - Report::new(TrustedServerError::Configuration { - message: format!("Invalid creative opportunity gam_unit_path template: {err}"), - }) - })?; - // Slots flow into injected HTML/JS, provider payloads, and GPT - // calls. Env/private config can bypass static review, so validate - // the full runtime shape on every load path. - co.validate_runtime().map_err(|err| { - Report::new(TrustedServerError::Configuration { - message: format!("Invalid creative opportunity slot config: {err}"), - }) - })?; + match self.compiled_globs.get_or_init(|| { + let mut compiled = Vec::new(); + let source_patterns = self + .path_glob + .iter() + .chain(self.path_globs.iter()) + .map(String::as_str); + for pattern in source_patterns { + compile_cache_asset_glob_patterns(pattern, &mut compiled)?; + } + Ok(compiled) + }) { + Ok(patterns) => Ok(Some(patterns.as_slice())), + Err(message) => Err(Report::new(TrustedServerError::Configuration { + message: format!( + "cache.asset_rules `{}` glob matcher failed to compile: {message}", + self.id + ), + })), } + } - for (name, value) in &self.response_headers { - http::header::HeaderName::from_bytes(name.as_bytes()).map_err(|_| { - Report::new(TrustedServerError::Configuration { - message: format!("Invalid response header name: {name}"), - }) - })?; - http::header::HeaderValue::from_str(value).map_err(|_| { - Report::new(TrustedServerError::Configuration { - message: format!("Invalid response header value for {name}"), - }) - })?; + fn matches_path(&self, path: &str) -> Result> { + if !self.enabled || !self.matcher_matches_path(path)? { + return Ok(false); } - Ok(()) - } + if let Some(style) = self.fingerprint_style + && !filename_contains_fingerprint(path, style) + { + log::debug!( + "cache asset rule `{}` rejects path `{path}` because the filename has no {style:?} fingerprint", + self.id + ); + return Ok(false); + } - /// Returns compiled creative opportunity slots, or empty slice if feature is disabled. - #[must_use] - pub fn creative_opportunity_slots( - &self, - ) -> &[crate::creative_opportunities::CreativeOpportunitySlot] { - self.creative_opportunities - .as_ref() - .map(|co| co.slot.as_slice()) - .unwrap_or(&[]) + Ok(true) } - /// Rejects known placeholder secret values. - /// - /// # Errors - /// - /// Returns [`TrustedServerError::InsecureDefault`] when one or more secret - /// fields still contain a placeholder value. - pub fn reject_placeholder_secrets(&self) -> Result<(), Report> { - let mut insecure_fields: Vec = Vec::new(); - - if Ec::is_placeholder_passphrase(self.ec.passphrase.expose()) { - insecure_fields.push("ec.passphrase".to_owned()); + fn matcher_matches_path(&self, path: &str) -> Result> { + if let Some(preset) = self.preset { + return Ok(preset.matches_path(path)); } - if Publisher::is_placeholder_proxy_secret(self.publisher.proxy_secret.expose()) { - insecure_fields.push("publisher.proxy_secret".to_owned()); + if let Some(prefix) = self.path_prefix.as_deref() { + return Ok(path.starts_with(prefix)); } - for partner in &self.ec.partners { - if EcPartner::is_placeholder_api_token(partner.api_token.expose()) { - insecure_fields.push(format!("ec.partners[{}].api_token", partner.source_domain)); - } + if let Some(patterns) = self.compiled_globs()? { + return Ok(patterns + .iter() + .any(|pattern| pattern.matches_with(path, CACHE_ASSET_GLOB_MATCH_OPTIONS))); } - for handler in &self.handlers { - if Handler::is_placeholder_password(handler.password.expose()) { - insecure_fields.push(format!("handlers[{}].password", handler.path)); + if let Some(regex) = self.compiled_regex()? { + return Ok(regex.is_match(path)); + } + if !self.extensions.is_empty() { + return Ok(path_extension(path).is_some_and(|extension| { + self.extensions + .iter() + .any(|candidate| candidate == &extension) + })); + } + Ok(false) + } + + fn cache_policy(&self) -> CachePolicy { + CachePolicy { + visibility: self.visibility.into(), + browser_ttl: self.browser_ttl_seconds.map(Duration::from_secs), + edge_ttl: self.edge_ttl_seconds.map(Duration::from_secs), + stale_while_revalidate: self.stale_while_revalidate_seconds.map(Duration::from_secs), + stale_if_error: self.stale_if_error_seconds.map(Duration::from_secs), + immutable: self.immutable, + } + } +} + +const CACHE_ASSET_GLOB_MATCH_OPTIONS: MatchOptions = MatchOptions { + case_sensitive: true, + require_literal_separator: true, + require_literal_leading_dot: false, +}; + +fn compile_cache_asset_glob_patterns( + pattern: &str, + compiled: &mut Vec, +) -> Result<(), String> { + let mut variants = vec![pattern.to_string()]; + let mut variant_index = 0; + + while variant_index < variants.len() { + let variant = variants[variant_index].clone(); + let optional_segments = variant + .match_indices("**/") + .map(|(index, _)| index) + .collect::>(); + for segment_start in optional_segments { + let without_segment = format!( + "{}{}", + &variant[..segment_start], + &variant[segment_start + "**/".len()..] + ); + if !variants.contains(&without_segment) { + variants.push(without_segment); } } + variant_index += 1; + } - if insecure_fields.is_empty() { - return Ok(()); + for variant in variants { + compiled.push(Pattern::new(&variant).map_err(|err| err.to_string())?); + } + + Ok(()) +} + +/// Built-in cache-rule presets that operators can enable explicitly. +#[derive(Debug, Clone, Copy, Deserialize, Serialize, Eq, PartialEq)] +#[serde(rename_all = "kebab-case")] +pub enum CacheAssetPreset { + /// Next.js build output under `/_next/static/`. + #[serde(rename = "nextjs-static")] + NextJsStatic, +} + +impl CacheAssetPreset { + fn matches_path(self, path: &str) -> bool { + match self { + Self::NextJsStatic => path.starts_with("/_next/static/"), } + } +} - Err(Report::new(TrustedServerError::InsecureDefault { - field: insecure_fields.join(", "), - })) +/// Cache visibility parsed from operator configuration. +#[derive(Debug, Default, Clone, Copy, Deserialize, Serialize, Eq, PartialEq)] +#[serde(rename_all = "kebab-case")] +pub enum CachePolicyVisibility { + /// Public browser/cache visibility. + #[default] + Public, + /// Private browser visibility. + Private, +} + +impl From for CacheVisibility { + fn from(value: CachePolicyVisibility) -> Self { + match value { + CachePolicyVisibility::Public => Self::Public, + CachePolicyVisibility::Private => Self::Private, + } } +} - fn validate_asset_image_optimizer_profile_sets( - &self, - ) -> Result<(), Report> { - for route in &self.proxy.asset_routes { - let Some(config) = &route.image_optimizer else { - continue; - }; - if !config.enabled { - continue; +fn path_extension(path: &str) -> Option { + let filename = path.rsplit('/').next()?; + let (_, extension) = filename.rsplit_once('.')?; + (!extension.is_empty()).then(|| extension.to_ascii_lowercase()) +} + +/// Operator-selected filename fingerprint convention for a cache rule. +#[derive(Debug, Clone, Copy, Deserialize, Serialize, Eq, PartialEq)] +#[serde(rename_all = "kebab-case")] +pub enum CacheAssetFingerprintStyle { + /// A hexadecimal suffix, such as `app.0123abcd.js`. + Hex, + /// An eight-character uppercase Base32 suffix, such as `app-VRTVD5R5.js`. + EsbuildBase32, + /// An eight-character `Base64URL` suffix for non-immutable rules, such as `index-BsELY24f.js`. + ViteBase64Url, +} + +impl CacheAssetFingerprintStyle { + fn matches_candidate(self, candidate: &str) -> bool { + match self { + Self::Hex => { + candidate.len() >= 8 + && candidate.chars().all(|ch| ch.is_ascii_hexdigit()) + && candidate.chars().any(|ch| ch.is_ascii_alphabetic()) } - if !self - .image_optimizer - .profile_sets - .contains_key(&config.profile_set) - { - return Err(Report::new(TrustedServerError::Configuration { - message: format!( - "proxy.asset_routes prefix `{}` references unknown image_optimizer profile_set `{}`", - route.prefix, config.profile_set - ), - })); + Self::EsbuildBase32 => { + candidate.len() == 8 + && candidate + .chars() + .all(|ch| ch.is_ascii_uppercase() || matches!(ch, '2'..='7')) + && candidate.chars().any(|ch| ch.is_ascii_alphabetic()) + } + Self::ViteBase64Url => { + candidate.len() == 8 + && candidate + .chars() + .all(|ch| ch.is_ascii_alphanumeric() || matches!(ch, '-' | '_')) + && candidate.chars().any(|ch| ch.is_ascii_uppercase()) + && candidate.chars().any(|ch| { + ch.is_ascii_lowercase() || ch.is_ascii_digit() || matches!(ch, '-' | '_') + }) } } - Ok(()) } +} - /// Resolve the longest matching asset route for the request path. - #[must_use] - pub fn asset_route_for_path(&self, path: &str) -> Option<&ProxyAssetRoute> { - self.proxy.asset_route_for_path(path) +fn filename_contains_fingerprint(path: &str, style: CacheAssetFingerprintStyle) -> bool { + let filename = path.rsplit('/').next().unwrap_or(path); + let Some((stem, extension)) = filename.rsplit_once('.') else { + return false; + }; + if stem.is_empty() || extension.is_empty() { + return false; } - /// Resolve the first handler whose regex matches the request path. - /// - /// # Errors + stem.char_indices() + .filter(|(_, ch)| matches!(ch, '.' | '-' | '_' | '~')) + .any(|(separator_index, separator)| { + let candidate_start = separator_index + separator.len_utf8(); + let prefix = &stem[..separator_index]; + let candidate = &stem[candidate_start..]; + !prefix.is_empty() && style.matches_candidate(candidate) + }) +} + +/// Debug-only features. All flags default to `false` (off in production). +#[derive(Debug, Default, Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct DebugConfig { + /// Expose the JA4/TLS probabilistic identifier debug endpoint at `GET /_ts/debug/ja4`. /// - /// Returns a configuration error if any handler regex does not compile. - pub fn handler_for_path( - &self, - path: &str, - ) -> Result, Report> { - for handler in &self.handlers { - if handler.matches_path(path)? { - return Ok(Some(handler)); - } - } + /// When `false` (the default), the endpoint returns 404. Enable only for + /// intentional Fastly/browser TLS investigation. The endpoint reflects + /// Fastly-observed TLS details that browser JS cannot normally read. + #[serde(default)] + pub ja4_endpoint_enabled: bool, - Ok(None) - } + /// Inject a `` HTML comment before `` dumping + /// per-provider auction diagnostics. The default validates response-level + /// metadata, but bid fields and bounded creative previews remain visible; + /// this is not a fully anonymized dump. Never enable in production. + #[serde(default)] + pub auction_html_comment: bool, - /// Known admin endpoint paths that must be covered by a handler. + /// Content and verbosity of the `auction_html_comment` dump. Ignored + /// when `auction_html_comment` is false. /// - /// [`from_toml`](Self::from_toml) rejects configurations - /// where any of these paths lack a matching handler, ensuring admin - /// endpoints are always protected by authentication. - /// Update [`ADMIN_ENDPOINTS`](Self::ADMIN_ENDPOINTS) when adding new - /// admin routes to `crates/trusted-server-adapter-fastly/src/main.rs`. - pub(crate) const ADMIN_ENDPOINTS: &[&str] = - &["/_ts/admin/keys/rotate", "/_ts/admin/keys/deactivate"]; + /// The default table must stay omitted from serialized config blobs: + /// [`DebugConfig`] denies unknown fields, so an older binary rejects a blob + /// carrying this table during a mixed-version deployment or rollback. Any + /// non-default table still serializes and requires restoring a compatible + /// blob before rolling back. + #[serde( + default, + skip_serializing_if = "is_default_auction_debug_comment_options" + )] + pub auction_html_comment_options: AuctionDebugCommentOptions, - /// Returns admin endpoint paths that no configured handler covers. + /// Enable the testing-only direct GAM-replace path and the verbose per-bid + /// `debug_bid` blob in `window.tsjs.bids`. /// - /// Called during settings finalization to enforce that every admin endpoint - /// has a handler. An empty return - /// value means all admin endpoints are properly covered. + /// Note: the sanitized winning `adm` is now injected **unconditionally** for + /// production inline rendering through the pbRender bridge (see + /// [`crate::publisher::build_bid_map`]); this flag no longer gates `adm`. + /// What it still gates is the client-side `debug_bid` signal that turns on + /// the direct GAM-creative replacement (`injectAdmIntoSlot`), which bypasses + /// GAM entirely — useful for validating the auction→creative pipeline while + /// PBS Cache is unavailable. The `debug_bid` blob also carries the raw, + /// un-sanitized creative for diagnostics, so never enable in production. + #[serde(default)] + pub inject_adm_for_testing: bool, +} + +/// Metadata keys safe to surface in the `ts-debug` auction comment. +/// +/// Fail-closed superset: any key not listed here — notably `debug`, which +/// carries the resolved `OpenRTB` request (EC ID, `user.ext.eids`, the TC +/// consent string, `device.ip`, `device.geo`) plus per-bidder `httpcalls` — +/// is dropped in [`AuctionDebugCommentVerbosity::Redacted`] mode regardless +/// of what an operator lists in [`AuctionDebugCommentOptions::metadata_keys`]. +/// `metadata_keys` is a subset selector against this const, never a way to +/// add new keys. +pub(crate) const AUCTION_DEBUG_METADATA_ALLOWLIST: &[&str] = + &["error_type", "http_status", "message"]; + +/// Provider-controlled diagnostic keys exposed only by `Upstream` or `Full`. +/// +/// Values remain untyped upstream JSON and may contain request or identity +/// data. Keeping this list separate prevents [`AuctionDebugCommentOptions::metadata_keys`] +/// from widening the default response-metadata boundary. +pub(crate) const AUCTION_DEBUG_UPSTREAM_METADATA_KEYS: &[&str] = &[ + "errors", + "warnings", + "responsetimemillis", + "bidstatus", + "upstream_message", + "upstream_message_truncated", +]; + +fn default_true() -> bool { + true +} + +fn default_auction_debug_metadata_keys() -> Vec { + AUCTION_DEBUG_METADATA_ALLOWLIST + .iter() + .map(std::string::ToString::to_string) + .collect() +} + +// This predicate preserves rollback compatibility by omitting the default table. +fn is_default_auction_debug_comment_options(value: &AuctionDebugCommentOptions) -> bool { + *value == AuctionDebugCommentOptions::default() +} + +/// Behavior of the `` auction dump. Only consulted when +/// [`DebugConfig::auction_html_comment`] is true. +/// +/// `deny_unknown_fields` matches the convention used by sibling config +/// structs in this file, including the `DebugConfig` this struct nests +/// under: an operator typo (e.g. `metadata_key` instead of `metadata_keys`) +/// must fail config load loudly, not be silently ignored. +#[derive(Debug, Clone, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct AuctionDebugCommentOptions { + /// Include the `provider_responses` section at all. + #[serde(default = "default_true")] + pub include_provider_responses: bool, + + /// Include `mediator_response` when a mediator ran. + #[serde(default = "default_true")] + pub include_mediator_response: bool, + + /// Include each provider's `bids` array (vs. status/metadata only). + #[serde(default = "default_true")] + pub include_bids: bool, + + /// Subset of [`AUCTION_DEBUG_METADATA_ALLOWLIST`] to surface in + /// [`AuctionDebugCommentVerbosity::Redacted`] mode. This selector cannot + /// unlock provider diagnostics, and entries outside the fixed allowlist are + /// rejected at config load by + /// [`validate_metadata_keys`](Self::validate_metadata_keys). /// - /// # Errors + /// [`AuctionDebugCommentVerbosity::Upstream`] builds on the redacted + /// metadata, so this subset still gates those three keys there; the six + /// upstream diagnostics are unlocked by `verbosity` alone. Ignored entirely + /// when `verbosity` is [`AuctionDebugCommentVerbosity::Full`]. + #[serde(default = "default_auction_debug_metadata_keys")] + pub metadata_keys: Vec, + + /// `Redacted` (default): validated `metadata_keys` subset only, with + /// creative previews truncated to `MAX_BID_CREATIVE_DUMP_BYTES`. + /// `Upstream`: redacted fields plus six untyped provider diagnostics; + /// creative previews remain truncated. + /// `Full`: raw `response.metadata` verbatim, including the `debug` + /// subtree (httpcalls/resolvedrequest) when present, and no creative + /// truncation. The total dump byte cap and comment-terminator + /// neutralization still apply unconditionally. /// - /// Returns [`TrustedServerError::Configuration`] if any handler has an invalid path regex. - pub(crate) fn uncovered_admin_endpoints( - &self, - ) -> Result, Report> { - let mut uncovered = Vec::new(); - for &path in Self::ADMIN_ENDPOINTS { - let mut covered = false; - for h in &self.handlers { - if h.matches_path(path)? { - covered = true; - break; - } - } - if !covered { - uncovered.push(path); - } + /// NEVER enable `Upstream` or `Full` in production — identity-bearing + /// request/response data may become visible via view-source. + #[serde(default)] + pub verbosity: AuctionDebugCommentVerbosity, + + /// JSON representation used for the outer auction dump. + #[serde(default)] + pub format: AuctionDebugCommentFormat, +} + +impl Default for AuctionDebugCommentOptions { + fn default() -> Self { + Self { + include_provider_responses: true, + include_mediator_response: true, + include_bids: true, + metadata_keys: default_auction_debug_metadata_keys(), + verbosity: AuctionDebugCommentVerbosity::Redacted, + format: AuctionDebugCommentFormat::Compact, } - Ok(uncovered) } +} - /// Validates that every admin endpoint is covered by at least one handler. +impl AuctionDebugCommentOptions { + pub(crate) fn normalize(&mut self) { + self.metadata_keys = self + .metadata_keys + .drain(..) + .map(|key| key.trim().to_string()) + .filter(|key| !key.is_empty()) + .collect(); + } + + /// Reject [`Self::metadata_keys`] entries outside + /// [`AUCTION_DEBUG_METADATA_ALLOWLIST`]. + /// + /// Render time intersects the configured list with the allowlist, so an + /// entry outside it is dead config that silently renders `metadata: {}`. + /// Fail the load loudly instead, matching the `deny_unknown_fields` + /// contract on this struct. The render-time intersection stays as + /// defense-in-depth for config paths that bypass this check. /// /// # Errors /// - /// Returns [`TrustedServerError::Configuration`] listing any uncovered - /// admin endpoints. - fn validate_admin_coverage(&self) -> Result<(), Report> { - let uncovered = self.uncovered_admin_endpoints()?; - if uncovered.is_empty() { + /// Returns [`TrustedServerError::Configuration`] naming every unknown key. + pub(crate) fn validate_metadata_keys(&self) -> Result<(), Report> { + let unknown: Vec<&str> = self + .metadata_keys + .iter() + .map(String::as_str) + .filter(|key| !AUCTION_DEBUG_METADATA_ALLOWLIST.contains(key)) + .collect(); + + if unknown.is_empty() { return Ok(()); } + Err(Report::new(TrustedServerError::Configuration { message: format!( - "No handler covers admin endpoint(s): {}. \ - Add a [[handlers]] entry with a path regex matching /_ts/admin/ \ - to protect admin access.", - uncovered.join(", ") + "debug.auction_html_comment_options.metadata_keys contains unsupported keys [{}]; supported keys are [{}]", + unknown.join(", "), + AUCTION_DEBUG_METADATA_ALLOWLIST.join(", ") ), })) } +} - fn validate_admin_handler_passwords(&self) -> Result<(), Report> { - for handler in &self.handlers { - let covers_admin = Self::ADMIN_ENDPOINTS - .iter() - .try_fold(false, |covered, path| { - handler.matches_path(path).map(|matches| covered || matches) - })?; +/// Verbosity of the `ts-debug` auction comment. See +/// [`AuctionDebugCommentOptions::verbosity`]. +#[derive(Debug, Clone, Copy, Default, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AuctionDebugCommentVerbosity { + #[default] + Redacted, + Upstream, + Full, +} - if covers_admin && is_admin_placeholder_password(handler.password.expose()) { - return Err(Report::new(TrustedServerError::Configuration { - message: format!( - "Admin handler `{}` uses a placeholder password; configure a strong secret", - handler.path - ), - })); - } - } +/// JSON representation used for the outer `ts-debug` auction dump. +#[derive(Debug, Clone, Copy, Default, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AuctionDebugCommentFormat { + #[default] + Compact, + Pretty, +} - Ok(()) - } +/// Tester-cookie endpoint configuration. +#[derive(Debug, Default, Clone, Deserialize, Serialize)] +pub struct TesterCookieConfig { + /// Enable tester-cookie endpoints that set and clear `ts-tester`. + #[serde(default)] + pub enabled: bool, +} - /// Retrieves the integration configuration of a specific type. +#[derive(Debug, Default, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] +pub struct Settings { + #[validate(nested)] + pub publisher: Publisher, + #[serde(default)] + pub tester_cookie: TesterCookieConfig, + #[serde(default)] + #[validate(nested)] + pub ec: Ec, + #[serde(default)] + pub integrations: IntegrationSettings, + #[serde(default, deserialize_with = "vec_from_seq_or_map")] + #[validate(nested)] + pub handlers: Vec, + #[serde(default, deserialize_with = "map_from_obj_or_str")] + pub response_headers: HashMap, + pub request_signing: Option, + #[serde(default)] + #[validate(nested)] + pub rewrite: Rewrite, + #[serde(default)] + pub auction: AuctionConfig, + #[serde(default)] + pub consent: ConsentConfig, + #[serde(default)] + pub cache: CacheSettings, + #[serde(default)] + pub proxy: Proxy, + #[serde(default)] + pub creative_opportunities: Option, + #[serde(default)] + pub image_optimizer: ImageOptimizerSettings, + #[serde(default)] + pub tinybird: TinybirdSettings, + #[serde(default)] + pub debug: DebugConfig, +} + +impl Settings { + /// Creates a new [`Settings`] instance from a TOML string. /// /// # Errors /// - /// Returns an error if the integration configuration exists but cannot be deserialized as the requested type. - pub fn integration_config( - &self, - integration_id: &str, - ) -> Result, Report> - where - T: IntegrationConfig, - { - self.integrations.get_typed(integration_id) - } -} - -fn validate_publisher_domain(value: &str) -> Result<(), ValidationError> { - if value.trim() != value || value.is_empty() || value.len() > 253 { - return Err(ValidationError::new("invalid_publisher_domain")); - } - if value.starts_with('.') || value.ends_with('.') || value.contains(['/', ':']) { - return Err(ValidationError::new("invalid_publisher_domain")); - } + /// - [`TrustedServerError::Configuration`] if the TOML is invalid or missing required fields + pub fn from_toml(toml_str: &str) -> Result> { + let settings: Self = + toml::from_str(toml_str).change_context(TrustedServerError::Configuration { + message: "Failed to deserialize TOML configuration".to_string(), + })?; - for label in value.split('.') { - if label.is_empty() || label.len() > 63 { - return Err(ValidationError::new("invalid_publisher_domain")); - } - let bytes = label.as_bytes(); - if bytes.first() == Some(&b'-') || bytes.last() == Some(&b'-') { - return Err(ValidationError::new("invalid_publisher_domain")); - } - if !bytes - .iter() - .all(|byte| byte.is_ascii_alphanumeric() || *byte == b'-') - { - return Err(ValidationError::new("invalid_publisher_domain")); - } + Self::finalize_deserialized(settings, "Configuration") } - Ok(()) -} - -fn validate_cookie_domain(value: &str) -> Result<(), ValidationError> { - // `=` is excluded: it only has special meaning in the name=value pair, - // not within the Domain attribute value. - if value.contains([';', '\n', '\r']) { - let mut err = ValidationError::new("cookie_metacharacters"); - err.message = - Some("cookie_domain must not contain cookie metacharacters (;, \\n, \\r)".into()); - return Err(err); - } - Ok(()) -} + /// Creates a new [`Settings`] instance from a JSON value. + /// + /// Runtime config-store loading uses this after verifying the `app_config` + /// blob envelope and extracting the same typed settings shape. + /// + /// # Errors + /// + /// - [`TrustedServerError::Configuration`] if the JSON value is invalid or missing required fields + pub fn from_json_value(value: JsonValue) -> Result> { + let settings: Self = + serde_json::from_value(value).change_context(TrustedServerError::Configuration { + message: "Failed to deserialize JSON configuration".to_string(), + })?; -fn validate_no_trailing_slash(value: &str) -> Result<(), ValidationError> { - if value.ends_with('/') { - let mut err = ValidationError::new("trailing_slash"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include a trailing slash".into()); - return Err(err); + Self::finalize_deserialized(settings, "Configuration") } - Ok(()) -} -fn validate_host_header_override(value: &str) -> Result<(), ValidationError> { - if let Err(reason) = validate_host_header_override_value(value) { - let mut err = ValidationError::new("invalid_host_header_override"); - err.add_param("value".into(), &value); - err.add_param("reason".into(), &reason); - err.message = Some( - "origin_host_header_override must be a valid host or host:port without scheme, path, query, or fragment" - .into(), - ); - return Err(err); - } + /// Creates a new [`Settings`] instance from a TOML string with legacy + /// test-only `TRUSTED_SERVER__` environment variable overrides. + /// + /// Runtime loading does not use this legacy helper; `EdgeZero` CLI app-config + /// overlays are applied before deserializing [`crate::config::TrustedServerAppConfig`]. + /// This helper remains available to existing tests that exercise legacy + /// parsing behavior. + /// + /// # Errors + /// + /// - [`TrustedServerError::Configuration`] if the TOML is invalid or missing required fields + #[cfg(test)] + pub fn from_toml_and_env(toml_str: &str) -> Result> { + let environment = Environment::default() + .prefix(ENVIRONMENT_VARIABLE_PREFIX) + .separator(ENVIRONMENT_VARIABLE_SEPARATOR); - Ok(()) -} + let toml = File::from_str(toml_str, FileFormat::Toml); + let config = Config::builder() + .add_source(toml) + .add_source(environment) + .build() + .change_context(TrustedServerError::Configuration { + message: "Failed to build configuration".to_string(), + })?; + let settings: Self = + config + .try_deserialize() + .change_context(TrustedServerError::Configuration { + message: "Failed to deserialize configuration".to_string(), + })?; -fn validate_redacted_not_empty(value: &Redacted) -> Result<(), ValidationError> { - if value.expose().is_empty() { - return Err(ValidationError::new("empty_value")); + Self::finalize_deserialized(settings, "Build-time configuration") } - Ok(()) -} -fn validate_asset_route_prefix(value: &str) -> Result<(), ValidationError> { - if !value.starts_with('/') { - let mut err = ValidationError::new("invalid_prefix"); - err.add_param("value".into(), &value); - err.message = Some("asset-route prefix must start with '/'".into()); - return Err(err); - } + pub(crate) fn finalize_deserialized( + mut settings: Self, + validation_label: &str, + ) -> Result> { + settings.cache.normalize(); + settings.proxy.normalize(); + settings.image_optimizer.normalize(); + settings.debug.auction_html_comment_options.normalize(); + settings.consent.validate(); - Ok(()) -} + settings.prepare_runtime()?; -fn validate_proxy_origin_url(value: &str) -> Result<(), ValidationError> { - validate_no_trailing_slash(value)?; + settings.validate().map_err(|err| { + Report::new(TrustedServerError::Configuration { + message: format!("{validation_label} validation failed: {err}"), + }) + })?; - let parsed = Url::parse(value).map_err(|parse_error| { - let mut err = ValidationError::new("invalid_origin_url"); - err.add_param("value".into(), &value); - err.add_param("message".into(), &parse_error.to_string()); - err.message = Some("origin_url must be an absolute http or https URL".into()); - err - })?; + settings.validate_admin_coverage()?; + settings.validate_admin_handler_passwords()?; - if !matches!(parsed.scheme(), "http" | "https") { - let mut err = ValidationError::new("invalid_origin_url_scheme"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must use http or https".into()); - return Err(err); - } + if settings.auction.enabled && !settings.auction.rewrite_creatives { + log::warn!( + "Auction creative rewriting disabled; creative assets and clicks may contact third-party hosts directly" + ); + } - if parsed.host_str().is_none() { - let mut err = ValidationError::new("missing_origin_host"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must include a host".into()); - return Err(err); + Ok(settings) } - if !parsed.username().is_empty() || parsed.password().is_some() { - let mut err = ValidationError::new("origin_url_has_userinfo"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include username or password".into()); - return Err(err); - } + /// Eagerly prepare runtime-only settings artifacts. + /// + /// # Errors + /// + /// Returns a configuration error if any cached runtime artifact cannot be + /// prepared, if any handler path regex does not compile, if a creative + /// opportunity slot is invalid, or if + /// [`AuctionDebugCommentOptions::metadata_keys`] names an unsupported key. + pub fn prepare_runtime(&mut self) -> Result<(), Report> { + self.image_optimizer.prepare_runtime()?; + self.cache.prepare_runtime()?; + self.proxy.prepare_runtime()?; + self.tinybird.prepare_runtime()?; + self.debug + .auction_html_comment_options + .validate_metadata_keys()?; + self.validate_asset_image_optimizer_profile_sets()?; - if parsed.fragment().is_some() { - let mut err = ValidationError::new("origin_url_has_fragment"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include a fragment".into()); - return Err(err); - } + for handler in &self.handlers { + handler.prepare_runtime()?; + } - if !matches!(parsed.path(), "" | "/") { - let mut err = ValidationError::new("origin_url_has_path"); - err.add_param("value".into(), &value); - err.message = - Some("origin_url must not include a path; only scheme/host/port are used".into()); - return Err(err); + if let Some(co) = &mut self.creative_opportunities { + co.compile_slots(); + // Parse `gam_unit_path` templates once here (mirrors the compiled + // glob cache) so request-time rendering is substitution-only. + co.compile_unit_templates().map_err(|err| { + Report::new(TrustedServerError::Configuration { + message: format!("Invalid creative opportunity gam_unit_path template: {err}"), + }) + })?; + // Slots flow into injected HTML/JS, provider payloads, and GPT + // calls. Env/private config can bypass static review, so validate + // the full runtime shape on every load path. + co.validate_runtime().map_err(|err| { + Report::new(TrustedServerError::Configuration { + message: format!("Invalid creative opportunity slot config: {err}"), + }) + })?; + } + + for (name, value) in &self.response_headers { + http::header::HeaderName::from_bytes(name.as_bytes()).map_err(|_| { + Report::new(TrustedServerError::Configuration { + message: format!("Invalid response header name: {name}"), + }) + })?; + http::header::HeaderValue::from_str(value).map_err(|_| { + Report::new(TrustedServerError::Configuration { + message: format!("Invalid response header value for {name}"), + }) + })?; + } + + Ok(()) + } + + /// Returns compiled creative opportunity slots when template delivery is enabled. + #[must_use] + pub fn creative_opportunity_slots( + &self, + ) -> &[crate::creative_opportunities::CreativeOpportunitySlot] { + self.creative_opportunities + .as_ref() + .filter(|co| co.enabled) + .map(|co| co.slot.as_slice()) + .unwrap_or(&[]) + } + + /// Rejects known placeholder secret values. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::InsecureDefault`] when one or more secret + /// fields still contain a placeholder value. + pub fn reject_placeholder_secrets(&self) -> Result<(), Report> { + let mut insecure_fields: Vec = Vec::new(); + + if Ec::is_placeholder_passphrase(self.ec.passphrase.expose()) { + insecure_fields.push("ec.passphrase".to_owned()); + } + if Publisher::is_placeholder_proxy_secret(self.publisher.proxy_secret.expose()) { + insecure_fields.push("publisher.proxy_secret".to_owned()); + } + for partner in &self.ec.partners { + if EcPartner::is_placeholder_api_token(partner.api_token.expose()) { + insecure_fields.push(format!("ec.partners[{}].api_token", partner.source_domain)); + } + } + for handler in &self.handlers { + if Handler::is_placeholder_password(handler.password.expose()) { + insecure_fields.push(format!("handlers[{}].password", handler.path)); + } + } + + if insecure_fields.is_empty() { + return Ok(()); + } + + Err(Report::new(TrustedServerError::InsecureDefault { + field: insecure_fields.join(", "), + })) + } + + fn validate_asset_image_optimizer_profile_sets( + &self, + ) -> Result<(), Report> { + for route in &self.proxy.asset_routes { + let Some(config) = &route.image_optimizer else { + continue; + }; + if !config.enabled { + continue; + } + if !self + .image_optimizer + .profile_sets + .contains_key(&config.profile_set) + { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "proxy.asset_routes prefix `{}` references unknown image_optimizer profile_set `{}`", + route.prefix, config.profile_set + ), + })); + } + } + Ok(()) + } + + /// Resolve the first matching configured asset cache policy for the request path. + /// + /// # Errors + /// + /// Returns a configuration error if matcher preparation unexpectedly fails. + pub fn asset_cache_policy_for_path( + &self, + path: &str, + ) -> Result, Report> { + self.cache.asset_policy_for_path(path) + } + + /// Resolve the longest matching asset route for the request path. + #[must_use] + pub fn asset_route_for_path(&self, path: &str) -> Option<&ProxyAssetRoute> { + self.proxy.asset_route_for_path(path) + } + + /// Resolve the first handler whose regex matches the request path. + /// + /// # Errors + /// + /// Returns a configuration error if any handler regex does not compile. + pub fn handler_for_path( + &self, + path: &str, + ) -> Result, Report> { + for handler in &self.handlers { + if handler.matches_path(path)? { + return Ok(Some(handler)); + } + } + + Ok(None) + } + + /// Returns whether `path` is within the reserved Trusted Server admin + /// namespace. + #[must_use] + pub(crate) fn is_admin_path(path: &str) -> bool { + path == "/_ts/admin" || path.starts_with("/_ts/admin/") + } + + /// Known admin endpoint paths that must be covered by a handler. + /// + /// [`from_toml`](Self::from_toml) rejects configurations + /// where any of these paths lack a matching handler, ensuring admin + /// endpoints are always protected by authentication. + /// Update [`ADMIN_ENDPOINTS`](Self::ADMIN_ENDPOINTS) when adding new + /// admin routes to `crates/trusted-server-adapter-fastly/src/app.rs`. + /// + /// The `/_ts/admin/ec/{id}` entry is the canonical router pattern. Its + /// coverage is checked via [`admin_auth_probes`](Self::admin_auth_probes), + /// while validation errors continue to report this operator-facing route + /// template. + pub(crate) const ADMIN_ENDPOINTS: &[&str] = &[ + "/_ts/admin/keys/rotate", + "/_ts/admin/keys/deactivate", + "/_ts/admin/ec", + "/_ts/admin/ec/{id}", + "/_ts/admin/eids", + ]; + + /// Probes that establish handler coverage for the dynamic + /// `/_ts/admin/ec/{id}` route. + /// + /// Coverage cannot be sampled: the router accepts any single segment after + /// `/_ts/admin/ec/` and basic auth runs on the raw path before routing, so + /// a handler that matches only some ID shapes leaves the rest of the route + /// surface — including malformed IDs, which still reach the admin handler — + /// unauthenticated at configuration time and fail-closed at runtime. + /// + /// Both probes must match the same configuration for the route to count as + /// covered. The bare prefix rejects handlers anchored to specific ID + /// shapes; the concrete ID rejects handlers anchored to the prefix itself + /// (`^/_ts/admin/ec/$`). Together they admit only prefix-level matchers + /// such as `^/_ts/admin` or `^/_ts/admin/ec/`. + const ADMIN_EC_ID_AUTH_PROBES: [&str; 2] = [ + "/_ts/admin/ec/", + concat!( + "/_ts/admin/ec/", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ".Ab12Z9", + ), + ]; + + fn admin_auth_probes(path: &'static str) -> [&'static str; 2] { + match path { + "/_ts/admin/ec/{id}" => Self::ADMIN_EC_ID_AUTH_PROBES, + path => [path, path], + } + } + + /// Returns admin endpoint paths that no configured handler covers. + /// + /// Called during settings finalization to enforce that every admin endpoint + /// has a handler. An empty return + /// value means all admin endpoints are properly covered. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::Configuration`] if any handler has an invalid path regex. + pub(crate) fn uncovered_admin_endpoints( + &self, + ) -> Result, Report> { + let mut uncovered = Vec::new(); + for &path in Self::ADMIN_ENDPOINTS { + let mut covered = true; + for probe in Self::admin_auth_probes(path) { + let mut probe_covered = false; + for handler in &self.handlers { + if handler.matches_path(probe)? { + probe_covered = true; + break; + } + } + covered &= probe_covered; + } + if !covered { + uncovered.push(path); + } + } + Ok(uncovered) + } + + /// Validates that every admin endpoint is covered by at least one handler. + /// + /// # Errors + /// + /// Returns [`TrustedServerError::Configuration`] listing any uncovered + /// admin endpoints. + fn validate_admin_coverage(&self) -> Result<(), Report> { + let uncovered = self.uncovered_admin_endpoints()?; + if uncovered.is_empty() { + return Ok(()); + } + Err(Report::new(TrustedServerError::Configuration { + message: format!( + "No handler covers admin endpoint(s): {}. \ + Add a [[handlers]] entry with a path regex matching /_ts/admin/ \ + to protect admin access.", + uncovered.join(", ") + ), + })) + } + + /// Rejects placeholder and well-known weak handler passwords. + /// + /// Applies to every handler rather than to handlers inferred to cover an + /// admin endpoint: handler selection is first-match-wins over operator + /// regexes, so a narrow handler can shadow the admin namespace for paths no + /// probe enumerates. Handlers are Trusted Server's own basic-auth gates, so + /// a placeholder password is never valid on any of them. + fn validate_admin_handler_passwords(&self) -> Result<(), Report> { + for handler in &self.handlers { + if is_admin_placeholder_password(handler.password.expose()) { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "Handler `{}` uses a placeholder password; configure a strong secret", + handler.path + ), + })); + } + } + + Ok(()) + } + + /// Retrieves the integration configuration of a specific type. + /// + /// # Errors + /// + /// Returns an error if the integration configuration exists but cannot be deserialized as the requested type. + pub fn integration_config( + &self, + integration_id: &str, + ) -> Result, Report> + where + T: IntegrationConfig, + { + self.integrations.get_typed(integration_id) + } +} + +fn validate_publisher_domain(value: &str) -> Result<(), ValidationError> { + if value.trim() != value || value.is_empty() || value.len() > 253 { + return Err(ValidationError::new("invalid_publisher_domain")); + } + if value.starts_with('.') || value.ends_with('.') || value.contains(['/', ':']) { + return Err(ValidationError::new("invalid_publisher_domain")); + } + + for label in value.split('.') { + if label.is_empty() || label.len() > 63 { + return Err(ValidationError::new("invalid_publisher_domain")); + } + let bytes = label.as_bytes(); + if bytes.first() == Some(&b'-') || bytes.last() == Some(&b'-') { + return Err(ValidationError::new("invalid_publisher_domain")); + } + if !bytes + .iter() + .all(|byte| byte.is_ascii_alphanumeric() || *byte == b'-') + { + return Err(ValidationError::new("invalid_publisher_domain")); + } + } + + Ok(()) +} + +fn validate_cookie_domain(value: &str) -> Result<(), ValidationError> { + // `=` is excluded: it only has special meaning in the name=value pair, + // not within the Domain attribute value. + if value.contains([';', '\n', '\r']) { + let mut err = ValidationError::new("cookie_metacharacters"); + err.message = + Some("cookie_domain must not contain cookie metacharacters (;, \\n, \\r)".into()); + return Err(err); + } + Ok(()) +} + +fn validate_no_trailing_slash(value: &str) -> Result<(), ValidationError> { + if value.ends_with('/') { + let mut err = ValidationError::new("trailing_slash"); + err.add_param("value".into(), &value); + err.message = Some("origin_url must not include a trailing slash".into()); + return Err(err); + } + Ok(()) +} + +fn validate_host_header_override(value: &str) -> Result<(), ValidationError> { + if let Err(reason) = validate_host_header_override_value(value) { + let mut err = ValidationError::new("invalid_host_header_override"); + err.add_param("value".into(), &value); + err.add_param("reason".into(), &reason); + err.message = Some( + "origin_host_header_override must be a valid host or host:port without scheme, path, query, or fragment" + .into(), + ); + return Err(err); + } + + Ok(()) +} + +fn validate_redacted_not_empty(value: &Redacted) -> Result<(), ValidationError> { + if value.expose().is_empty() { + return Err(ValidationError::new("empty_value")); + } + Ok(()) +} + +fn validate_asset_route_prefix(value: &str) -> Result<(), ValidationError> { + if !value.starts_with('/') { + let mut err = ValidationError::new("invalid_prefix"); + err.add_param("value".into(), &value); + err.message = Some("asset-route prefix must start with '/'".into()); + return Err(err); + } + + Ok(()) +} + +fn validate_proxy_origin_url(value: &str) -> Result<(), ValidationError> { + validate_no_trailing_slash(value)?; + + let parsed = Url::parse(value).map_err(|parse_error| { + let mut err = ValidationError::new("invalid_origin_url"); + err.add_param("value".into(), &value); + err.add_param("message".into(), &parse_error.to_string()); + err.message = Some("origin_url must be an absolute http or https URL".into()); + err + })?; + + if !matches!(parsed.scheme(), "http" | "https") { + let mut err = ValidationError::new("invalid_origin_url_scheme"); + err.add_param("value".into(), &value); + err.message = Some("origin_url must use http or https".into()); + return Err(err); + } + + if parsed.host_str().is_none() { + let mut err = ValidationError::new("missing_origin_host"); + err.add_param("value".into(), &value); + err.message = Some("origin_url must include a host".into()); + return Err(err); + } + + if !parsed.username().is_empty() || parsed.password().is_some() { + let mut err = ValidationError::new("origin_url_has_userinfo"); + err.add_param("value".into(), &value); + err.message = Some("origin_url must not include username or password".into()); + return Err(err); + } + + if parsed.fragment().is_some() { + let mut err = ValidationError::new("origin_url_has_fragment"); + err.add_param("value".into(), &value); + err.message = Some("origin_url must not include a fragment".into()); + return Err(err); + } + + if !matches!(parsed.path(), "" | "/") { + let mut err = ValidationError::new("origin_url_has_path"); + err.add_param("value".into(), &value); + err.message = + Some("origin_url must not include a path; only scheme/host/port are used".into()); + return Err(err); + } + + if parsed.query().is_some() { + let mut err = ValidationError::new("origin_url_has_query"); + err.add_param("value".into(), &value); + err.message = Some("origin_url must not include a query string".into()); + return Err(err); + } + + Ok(()) +} + +fn validate_path(value: &str) -> Result<(), ValidationError> { + Regex::new(value).map(|_| ()).map_err(|err| { + let mut validation_error = ValidationError::new("invalid_regex"); + validation_error.add_param("value".into(), &value); + validation_error.add_param("message".into(), &err.to_string()); + validation_error + }) +} +fn from_value_or_str<'de, D, T>(deserializer: D) -> Result +where + D: Deserializer<'de>, + T: DeserializeOwned + FromStr, + T::Err: std::fmt::Display, +{ + let value = JsonValue::deserialize(deserializer)?; + match value { + JsonValue::String(value) => T::from_str(&value).map_err(serde::de::Error::custom), + other => serde_json::from_value(other).map_err(serde::de::Error::custom), + } +} + +// Helper: allow Vec fields to deserialize from either a JSON array or a map of numeric indices. +// This lets env vars like TRUSTED_SERVER__INTEGRATIONS__PREBID__BIDDERS__0=smartadserver work, which the config env source +// represents as an object {"0": "value"} rather than a sequence. Also supports string inputs that are +// JSON arrays or comma-separated values. +/// Deserializes a `HashMap` from either: +/// - A TOML table / JSON object (standard deserialization) +/// - A JSON string (e.g. from env var: `'{"Key": "value"}'`) +/// +/// This allows setting map fields via environment variables while +/// preserving key casing and special characters like hyphens. +pub(crate) fn map_from_obj_or_str<'de, D>( + deserializer: D, +) -> Result, D::Error> +where + D: Deserializer<'de>, +{ + let v = JsonValue::deserialize(deserializer)?; + match v { + JsonValue::Object(map) => map + .into_iter() + .map(|(k, v)| { + let val = match v { + JsonValue::String(s) => s, + other => other.to_string(), + }; + Ok((k, val)) + }) + .collect(), + JsonValue::String(s) => { + let txt = s.trim(); + if txt.starts_with('{') { + serde_json::from_str::>(txt) + .map_err(serde::de::Error::custom) + } else { + Err(serde::de::Error::custom( + "expected JSON object string, e.g. '{\"Key\": \"value\"}'", + )) + } + } + JsonValue::Null => Ok(HashMap::new()), + other => Err(serde::de::Error::custom(format!( + "expected object or JSON string, got {other}", + ))), + } +} + +pub(crate) fn bool_from_bool_or_str<'de, D>(deserializer: D) -> Result +where + D: Deserializer<'de>, +{ + let value = JsonValue::deserialize(deserializer)?; + match value { + JsonValue::Bool(value) => Ok(value), + JsonValue::String(value) => value + .trim() + .parse::() + .map_err(serde::de::Error::custom), + other => Err(serde::de::Error::custom(format!( + "expected bool or parseable bool string, got {other}" + ))), + } +} + +pub(crate) fn vec_from_seq_or_map<'de, D, T>(deserializer: D) -> Result, D::Error> +where + D: Deserializer<'de>, + T: DeserializeOwned, +{ + let v = JsonValue::deserialize(deserializer)?; + match v { + JsonValue::Array(arr) => arr + .into_iter() + .map(|item| serde_json::from_value(item).map_err(serde::de::Error::custom)) + .collect(), + JsonValue::Object(map) => { + let mut items: Vec<(usize, T)> = Vec::with_capacity(map.len()); + for (k, val) in map.into_iter() { + let idx = k.parse::().map_err(|_| { + serde::de::Error::custom(format!("Invalid index '{}' in map for Vec field", k)) + })?; + let parsed: T = serde_json::from_value(val).map_err(serde::de::Error::custom)?; + items.push((idx, parsed)); + } + items.sort_by_key(|(idx, _)| *idx); + Ok(items.into_iter().map(|(_, v)| v).collect()) + } + JsonValue::String(s) => { + let txt = s.trim(); + if txt.starts_with('[') && txt.ends_with(']') { + if let Ok(vec) = serde_json::from_str::>(txt) { + return Ok(vec); + } + // Not valid JSON array — strip brackets and split on commas + let inner = txt[1..txt.len() - 1].trim(); + let parts: Vec<&str> = inner + .split(',') + .map(str::trim) + .filter(|p| !p.is_empty()) + .collect(); + let mut out: Vec = Vec::with_capacity(parts.len()); + for p in parts { + let json = format!("\"{}\"", p.replace('"', "\\\"")); + let parsed: T = + serde_json::from_str(&json).map_err(serde::de::Error::custom)?; + out.push(parsed); + } + Ok(out) + } else { + let parts = if txt.contains(',') { + txt.split(',') + .map(str::trim) + .filter(|p| !p.is_empty()) + .collect::>() + } else { + vec![txt] + }; + let mut out: Vec = Vec::with_capacity(parts.len()); + for p in parts { + let json = format!("\"{}\"", p.replace('"', "\\\"")); + let parsed: T = + serde_json::from_str(&json).map_err(serde::de::Error::custom)?; + out.push(parsed); + } + Ok(out) + } + } + other => Err(serde::de::Error::custom(format!( + "expected array, map of indices, or parseable string, got {}", + other + ))), + } +} + +#[cfg(test)] +mod tests { + use super::*; + use regex::Regex; + use serde_json::json; + use std::collections::HashSet; + + use crate::auction::build_orchestrator; + use crate::integrations::{ + IntegrationRegistry, gpt::GptConfig, nextjs::NextJsIntegrationConfig, + prebid::PrebidIntegrationConfig, + }; + use crate::redacted::Redacted; + use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + + #[test] + fn auction_debug_comment_options_default_matches_serde_defaults() { + let opts = AuctionDebugCommentOptions::default(); + assert!(opts.include_provider_responses, "should default to true"); + assert!(opts.include_mediator_response, "should default to true"); + assert!(opts.include_bids, "should default to true"); + assert_eq!( + opts.metadata_keys, + vec![ + "error_type".to_string(), + "http_status".to_string(), + "message".to_string(), + ], + "should default to only schema-validated response metadata" + ); + assert_eq!( + opts.verbosity, + AuctionDebugCommentVerbosity::Redacted, + "should default to Redacted" + ); + assert_eq!( + opts.format, + AuctionDebugCommentFormat::Compact, + "should default to compact output" + ); + } + + #[test] + fn auction_debug_comment_options_normalize_trims_and_drops_empty_keys() { + let mut opts = AuctionDebugCommentOptions { + metadata_keys: vec![ + " http_status ".to_string(), + "".to_string(), + "debug".to_string(), + ], + ..AuctionDebugCommentOptions::default() + }; + opts.normalize(); + assert_eq!( + opts.metadata_keys, + vec!["http_status".to_string(), "debug".to_string()] + ); + } + + #[test] + fn auction_debug_comment_options_deserializes_upstream_verbosity() { + let options: AuctionDebugCommentOptions = toml::from_str(r#"verbosity = "upstream""#) + .expect("should deserialize upstream verbosity"); + assert_eq!(options.verbosity, AuctionDebugCommentVerbosity::Upstream); + } + + #[test] + fn auction_debug_comment_options_deserializes_pretty_format() { + let options: AuctionDebugCommentOptions = + toml::from_str(r#"format = "pretty""#).expect("should deserialize pretty format"); + assert_eq!(options.format, AuctionDebugCommentFormat::Pretty); + } + + #[test] + fn auction_debug_comment_options_bad_format_fails_config_load() { + let result: Result = + toml::from_str(r#"format = "expanded""#); + assert!( + result.is_err(), + "unrecognized format must fail to deserialize, not silently fall back" + ); + } + + #[test] + fn bad_verbosity_string_fails_config_load() { + // Deserialize AuctionDebugCommentOptions directly, not a full Settings — + // Settings has required fields with no #[serde(default)] (e.g. + // `publisher`), so a full-Settings fixture missing them would fail with + // "missing field `publisher`" regardless of whether `verbosity` itself + // deserialized correctly, testing the wrong thing. + let result: Result = + toml::from_str(r#"verbosity = "everything""#); + assert!( + result.is_err(), + "unrecognized verbosity must fail to deserialize, not silently fall back" + ); + } + + #[test] + fn auction_debug_comment_options_unknown_metadata_key_fails_config_load() { + let toml = format!( + "{}\n[debug]\nauction_html_comment = true\n\n[debug.auction_html_comment_options]\nmetadata_keys = [\"http_staus\", \"errors\"]\n", + crate_test_settings_str() + ); + let error = Settings::from_toml(&toml) + .expect_err("should reject metadata keys outside the fixed allowlist"); + let rendered = format!("{error:?}"); + assert!( + rendered.contains("http_staus") && rendered.contains("errors"), + "error should name every unsupported key, got {rendered}" + ); + } + + #[test] + fn auction_debug_comment_options_allowlisted_metadata_keys_load() { + let toml = format!( + "{}\n[debug]\nauction_html_comment = true\n\n[debug.auction_html_comment_options]\nmetadata_keys = [\" message \"]\n", + crate_test_settings_str() + ); + let settings = Settings::from_toml(&toml).expect("should accept an allowlisted key"); + assert_eq!( + settings.debug.auction_html_comment_options.metadata_keys, + vec!["message".to_string()], + "normalize should trim before validation runs" + ); + } + + #[test] + fn auction_debug_comment_options_unknown_field_fails_config_load() { + let result: Result = + toml::from_str(r#"metadata_key = ["message"]"#); + assert!( + result.is_err(), + "a misspelled field must fail config load, not be silently ignored" + ); + } + + #[test] + fn default_auction_debug_comment_options_stay_out_of_serialized_config() { + // Rollback contract: `DebugConfig` denies unknown fields, so the + // previous binary rejects a config blob carrying a table it does not + // know. Defaults must therefore serialize to nothing. + #[derive(Deserialize)] + #[serde(deny_unknown_fields)] + struct LegacyDebugConfig { + #[serde(default)] + ja4_endpoint_enabled: bool, + #[serde(default)] + auction_html_comment: bool, + #[serde(default)] + inject_adm_for_testing: bool, + } + + let value = serde_json::to_value(DebugConfig::default()) + .expect("should serialize the default debug config"); + assert!( + value.get("auction_html_comment_options").is_none(), + "default options table should not be serialized, got {value}" + ); + + let legacy: LegacyDebugConfig = serde_json::from_value(value) + .expect("legacy schema should accept the default debug payload"); + assert!(!legacy.ja4_endpoint_enabled); + assert!(!legacy.auction_html_comment); + assert!(!legacy.inject_adm_for_testing); + + let configured = DebugConfig { + auction_html_comment: true, + auction_html_comment_options: AuctionDebugCommentOptions { + include_bids: false, + ..AuctionDebugCommentOptions::default() + }, + ..DebugConfig::default() + }; + let value = + serde_json::to_value(&configured).expect("should serialize a configured debug config"); + assert!( + value.get("auction_html_comment_options").is_some(), + "non-default options must still serialize, got {value}" + ); + } + + #[test] + fn tinybird_defaults_to_disabled_placeholders() { + let settings = Settings::from_toml(&crate_test_settings_str()) + .expect("should parse settings without tinybird block"); + + assert!( + !settings.tinybird.enabled, + "Tinybird should default disabled" + ); + assert_eq!(settings.tinybird.secret_store, "ts_secrets"); + assert_eq!(settings.tinybird.auction_dataset, "auction_events_raw"); + assert_eq!( + settings.tinybird.auction_token_secret, + "tinybird_auction_append_token" + ); + } + + #[test] + fn tinybird_enabled_requires_host_dataset_and_token() { + let toml = format!( + "{}\n[tinybird]\nenabled = true\napi_host = \"https://api.example.com/path\"\n", + crate_test_settings_str() + ); + + let err = Settings::from_toml(&toml).expect_err("should reject invalid api host"); + assert!( + format!("{err:?}").contains("tinybird.api_host"), + "should report tinybird.api_host validation error: {err:?}" + ); + } + + #[test] + fn tinybird_accepts_region_host_without_scheme() { + let toml = format!( + "{}\n[tinybird]\nenabled = true\napi_host = \"api.us-east.aws.tinybird.co\"\n", + crate_test_settings_str() + ); + + let settings = Settings::from_toml(&toml).expect("should accept Tinybird region host"); + assert!(settings.tinybird.enabled); + assert_eq!(settings.tinybird.api_host, "api.us-east.aws.tinybird.co"); + } + + #[test] + fn tinybird_access_enabled_is_rejected_until_emitter_is_wired() { + let toml = format!( + "{}\n[tinybird]\naccess_enabled = true\n", + crate_test_settings_str() + ); + + let err = Settings::from_toml(&toml) + .expect_err("should reject access telemetry before emitter exists"); + assert!( + format!("{err:?}").contains("tinybird.access_enabled"), + "should report unsupported tinybird.access_enabled setting: {err:?}" + ); + } + + #[test] + fn test_settings_from_valid_toml() { + let toml_str = crate_test_settings_str(); + let settings = Settings::from_toml(&toml_str); + + assert!(settings.is_ok()); + + let settings = settings.expect("should parse valid TOML"); + let prebid_cfg = settings + .integration_config::("prebid") + .expect("Prebid config query should succeed") + .expect("Prebid config should load from test settings"); + assert_eq!( + prebid_cfg.server_url, + "https://test-prebid.com/openrtb2/auction" + ); + assert!( + settings + .integration_config::("nextjs") + .expect("Next.js config query should succeed") + .is_none(), + "Next.js integration should default to disabled" + ); + let raw_nextjs = settings + .integrations + .get("nextjs") + .expect("test settings should include nextjs block"); + assert_eq!(raw_nextjs["enabled"], json!(false)); + assert_eq!( + raw_nextjs["rewrite_attributes"], + json!(["href", "link", "url"]), + "Next.js rewrite attributes should default to href/link/url" + ); + assert_eq!(settings.publisher.domain, "test-publisher.com"); + assert_eq!(settings.publisher.cookie_domain, ".test-publisher.com"); + assert!( + !settings.tester_cookie.enabled, + "tester-cookie route should default to disabled" + ); + assert_eq!( + settings.publisher.ec_cookie_domain(), + ".test-publisher.com", + "EC cookie domain should be computed as .{{domain}}" + ); + assert_eq!( + settings.publisher.origin_url, + "https://origin.test-publisher.com" + ); + assert_eq!(settings.publisher.origin_host_header_override, None); + assert_eq!( + settings.ec.passphrase.expose(), + "test-secret-key-32-bytes-minimum" + ); + + settings.validate().expect("Failed to validate settings"); } - if parsed.query().is_some() { - let mut err = ValidationError::new("origin_url_has_query"); - err.add_param("value".into(), &value); - err.message = Some("origin_url must not include a query string".into()); - return Err(err); + #[test] + fn tester_cookie_enabled_parses_from_toml() { + let toml_str = format!( + r#"{} + + [tester_cookie] + enabled = true + "#, + crate_test_settings_str() + ); + + let settings = Settings::from_toml(&toml_str).expect("should parse tester-cookie config"); + + assert!( + settings.tester_cookie.enabled, + "tester-cookie config should enable the route" + ); } - Ok(()) -} + #[test] + fn cache_asset_rule_nextjs_preset_is_operator_controlled() { + let toml_str = format!( + r#"{} -fn validate_path(value: &str) -> Result<(), ValidationError> { - Regex::new(value).map(|_| ()).map_err(|err| { - let mut validation_error = ValidationError::new("invalid_regex"); - validation_error.add_param("value".into(), &value); - validation_error.add_param("message".into(), &err.to_string()); - validation_error - }) -} -fn from_value_or_str<'de, D, T>(deserializer: D) -> Result -where - D: Deserializer<'de>, - T: DeserializeOwned + FromStr, - T::Err: std::fmt::Display, -{ - let value = JsonValue::deserialize(deserializer)?; - match value { - JsonValue::String(value) => T::from_str(&value).map_err(serde::de::Error::custom), - other => serde_json::from_value(other).map_err(serde::de::Error::custom), + [[cache.asset_rules]] + id = "nextjs-static" + enabled = true + preset = "nextjs-static" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + ); + let settings = Settings::from_toml(&toml_str).expect("should parse cache asset rule"); + + let policy = settings + .asset_cache_policy_for_path("/_next/static/chunks/app.js") + .expect("should evaluate cache rules") + .expect("should match enabled Next.js preset"); + assert_eq!( + policy, + CachePolicy::public_immutable(Duration::from_secs(31_536_000)), + "enabled preset should produce immutable static policy" + ); + + let disabled_toml = toml_str.replace("enabled = true", "enabled = false"); + let disabled_settings = + Settings::from_toml(&disabled_toml).expect("should parse disabled cache asset rule"); + assert!( + disabled_settings + .asset_cache_policy_for_path("/_next/static/chunks/app.js") + .expect("should evaluate disabled cache rules") + .is_none(), + "disabled preset must not mark framework paths immutable" + ); } -} -// Helper: allow Vec fields to deserialize from either a JSON array or a map of numeric indices. -// This lets env vars like TRUSTED_SERVER__INTEGRATIONS__PREBID__BIDDERS__0=smartadserver work, which the config env source -// represents as an object {"0": "value"} rather than a sequence. Also supports string inputs that are -// JSON arrays or comma-separated values. -/// Deserializes a `HashMap` from either: -/// - A TOML table / JSON object (standard deserialization) -/// - A JSON string (e.g. from env var: `'{"Key": "value"}'`) -/// -/// This allows setting map fields via environment variables while -/// preserving key casing and special characters like hyphens. -pub(crate) fn map_from_obj_or_str<'de, D>( - deserializer: D, -) -> Result, D::Error> -where - D: Deserializer<'de>, -{ - let v = JsonValue::deserialize(deserializer)?; - match v { - JsonValue::Object(map) => map - .into_iter() - .map(|(k, v)| { - let val = match v { - JsonValue::String(s) => s, - other => other.to_string(), - }; - Ok((k, val)) - }) - .collect(), - JsonValue::String(s) => { - let txt = s.trim(); - if txt.starts_with('{') { - serde_json::from_str::>(txt) - .map_err(serde::de::Error::custom) - } else { - Err(serde::de::Error::custom( - "expected JSON object string, e.g. '{\"Key\": \"value\"}'", - )) - } + #[test] + fn cache_asset_rule_requires_selected_fingerprint_style() { + let expected_policy = CachePolicy::public_immutable(Duration::from_secs(31_536_000)); + for (style, matching_path, non_matching_path) in [ + ("hex", "/assets/app.0123abcd.js", "/assets/app-VRTVD5R5.js"), + ( + "esbuild-base32", + "/assets/app-VRTVD5R5.js", + "/assets/index-BsELY24f.js", + ), + ] { + let toml_str = format!( + r#"{} + + [[cache.asset_rules]] + id = "publisher-assets" + enabled = true + path_globs = ["/assets/**/*.js"] + fingerprint_style = "{style}" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + ); + let settings = Settings::from_toml(&toml_str).expect("should parse cache asset rule"); + + assert_eq!( + settings + .asset_cache_policy_for_path(matching_path) + .expect("should evaluate cache rules"), + Some(expected_policy), + "{style} should match its configured fingerprint convention" + ); + assert!( + settings + .asset_cache_policy_for_path(non_matching_path) + .expect("should evaluate cache rules") + .is_none(), + "{style} should not fall through to another fingerprint convention" + ); } - JsonValue::Null => Ok(HashMap::new()), - other => Err(serde::de::Error::custom(format!( - "expected object or JSON string, got {other}", - ))), } -} -pub(crate) fn bool_from_bool_or_str<'de, D>(deserializer: D) -> Result -where - D: Deserializer<'de>, -{ - let value = JsonValue::deserialize(deserializer)?; - match value { - JsonValue::Bool(value) => Ok(value), - JsonValue::String(value) => value - .trim() - .parse::() - .map_err(serde::de::Error::custom), - other => Err(serde::de::Error::custom(format!( - "expected bool or parseable bool string, got {other}" - ))), - } -} + #[test] + fn immutable_vite_style_cannot_cache_human_named_assets() { + let rule = format!( + r#"{} -pub(crate) fn vec_from_seq_or_map<'de, D, T>(deserializer: D) -> Result, D::Error> -where - D: Deserializer<'de>, - T: DeserializeOwned, -{ - let v = JsonValue::deserialize(deserializer)?; - match v { - JsonValue::Array(arr) => arr - .into_iter() - .map(|item| serde_json::from_value(item).map_err(serde::de::Error::custom)) - .collect(), - JsonValue::Object(map) => { - let mut items: Vec<(usize, T)> = Vec::with_capacity(map.len()); - for (k, val) in map.into_iter() { - let idx = k.parse::().map_err(|_| { - serde::de::Error::custom(format!("Invalid index '{}' in map for Vec field", k)) - })?; - let parsed: T = serde_json::from_value(val).map_err(serde::de::Error::custom)?; - items.push((idx, parsed)); - } - items.sort_by_key(|(idx, _)| *idx); - Ok(items.into_iter().map(|(_, v)| v).collect()) - } - JsonValue::String(s) => { - let txt = s.trim(); - if txt.starts_with('[') && txt.ends_with(']') { - if let Ok(vec) = serde_json::from_str::>(txt) { - return Ok(vec); - } - // Not valid JSON array — strip brackets and split on commas - let inner = txt[1..txt.len() - 1].trim(); - let parts: Vec<&str> = inner - .split(',') - .map(str::trim) - .filter(|p| !p.is_empty()) - .collect(); - let mut out: Vec = Vec::with_capacity(parts.len()); - for p in parts { - let json = format!("\"{}\"", p.replace('"', "\\\"")); - let parsed: T = - serde_json::from_str(&json).map_err(serde::de::Error::custom)?; - out.push(parsed); - } - Ok(out) - } else { - let parts = if txt.contains(',') { - txt.split(',') - .map(str::trim) - .filter(|p| !p.is_empty()) - .collect::>() - } else { - vec![txt] - }; - let mut out: Vec = Vec::with_capacity(parts.len()); - for p in parts { - let json = format!("\"{}\"", p.replace('"', "\\\"")); - let parsed: T = - serde_json::from_str(&json).map_err(serde::de::Error::custom)?; - out.push(parsed); - } - Ok(out) - } + [[cache.asset_rules]] + id = "vite-assets" + enabled = true + path_globs = ["/assets/**/*.js", "/assets/**/*.jpg", "/assets/**/*.png", "/assets/**/*.svg"] + fingerprint_style = "vite-base64-url" + visibility = "public" + browser_ttl_seconds = 31536000 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + ); + + for path in [ + "/assets/hero-Portrait.jpg", + "/assets/logo-DarkMode.svg", + "/assets/banner-Summer24.png", + ] { + let error = Settings::from_toml(&rule) + .expect_err("should reject immutable Vite-style cache rule"); + assert!( + format!("{error:?}").contains("cannot set immutable with vite-base64-url"), + "{path} must not receive an immutable policy through a Vite-style rule" + ); } - other => Err(serde::de::Error::custom(format!( - "expected array, map of indices, or parseable string, got {}", - other - ))), } -} -#[cfg(test)] -mod tests { - use super::*; - use regex::Regex; - use serde_json::json; - use std::collections::HashSet; + #[test] + fn non_immutable_vite_style_remains_available_for_cache_matching() { + let toml = format!( + r#"{} - use crate::auction::build_orchestrator; - use crate::integrations::{ - IntegrationRegistry, gpt::GptConfig, nextjs::NextJsIntegrationConfig, - prebid::PrebidIntegrationConfig, - }; - use crate::redacted::Redacted; - use crate::test_support::tests::{crate_test_settings_str, create_test_settings}; + [[cache.asset_rules]] + id = "vite-assets" + enabled = true + path_glob = "/assets/*.js" + fingerprint_style = "vite-base64-url" + browser_ttl_seconds = 300 + "#, + crate_test_settings_str() + ); + let settings = + Settings::from_toml(&toml).expect("should allow Vite-style matching without immutable"); + + assert!( + settings + .asset_cache_policy_for_path("/assets/index-BsELY24f.js") + .expect("should evaluate Vite-style cache rule") + .is_some(), + "non-immutable Vite-style rule should still match a Vite output filename" + ); + } #[test] - fn tinybird_defaults_to_disabled_placeholders() { - let settings = Settings::from_toml(&crate_test_settings_str()) - .expect("should parse settings without tinybird block"); + fn cache_asset_rule_globs_respect_path_separators() { + let toml_str = format!( + r#"{} + + [[cache.asset_rules]] + id = "direct-assets" + enabled = true + path_glob = "/assets/*.js" + browser_ttl_seconds = 300 + "#, + crate_test_settings_str() + ); + let settings = Settings::from_toml(&toml_str).expect("should parse cache asset rule"); assert!( - !settings.tinybird.enabled, - "Tinybird should default disabled" + settings + .asset_cache_policy_for_path("/assets/app.js") + .expect("should evaluate direct asset rule") + .is_some(), + "single-star glob should match a direct child" ); - assert_eq!(settings.tinybird.secret_store, "ts_secrets"); - assert_eq!(settings.tinybird.auction_dataset, "auction_events_raw"); - assert_eq!( - settings.tinybird.auction_token_secret, - "tinybird_auction_append_token" + for path in ["/assets/vendor/app.js", "/assets/app.JS"] { + assert!( + settings + .asset_cache_policy_for_path(path) + .expect("should evaluate direct asset rule") + .is_none(), + "single-star glob should not match {path}" + ); + } + + let recursive_toml = toml_str.replace("/assets/*.js", "/assets/**/*.js"); + let recursive_settings = + Settings::from_toml(&recursive_toml).expect("should parse recursive cache asset rule"); + for path in ["/assets/app.js", "/assets/vendor/app.js"] { + assert!( + recursive_settings + .asset_cache_policy_for_path(path) + .expect("should evaluate recursive asset rule") + .is_some(), + "double-star glob should match {path}" + ); + } + } + + #[test] + fn cache_asset_rule_globs_expand_each_optional_recursive_segment() { + let toml = format!( + r#"{} + + [[cache.asset_rules]] + id = "nested-assets" + enabled = true + path_glob = "/a/**/b/**/c.js" + browser_ttl_seconds = 300 + "#, + crate_test_settings_str() ); + let settings = Settings::from_toml(&toml).expect("should parse recursive cache rule"); + + for path in ["/a/x/b/y/c.js", "/a/b/y/c.js", "/a/x/b/c.js", "/a/b/c.js"] { + assert!( + settings + .asset_cache_policy_for_path(path) + .expect("should evaluate recursive cache rule") + .is_some(), + "recursive pattern should match {path}" + ); + } } #[test] - fn tinybird_enabled_requires_host_dataset_and_token() { - let toml = format!( - "{}\n[tinybird]\nenabled = true\napi_host = \"https://api.example.com/path\"\n", + fn disabled_cache_asset_rules_defer_matcher_and_policy_validation() { + let toml_str = format!( + r#"{} + + [[cache.asset_rules]] + id = "disabled-invalid-regex" + enabled = false + path_regex = "[" + + [[cache.asset_rules]] + id = "disabled-placeholder" + enabled = false + + [[cache.asset_rules]] + id = "disabled-unsafe-immutable" + enabled = false + path_prefix = "/assets/" + immutable = true + "#, crate_test_settings_str() ); - let err = Settings::from_toml(&toml).expect_err("should reject invalid api host"); + let settings = + Settings::from_toml(&toml_str).expect("should defer disabled rule validation"); assert!( - format!("{err:?}").contains("tinybird.api_host"), - "should report tinybird.api_host validation error: {err:?}" + settings + .asset_cache_policy_for_path("/assets/app-DA15JTLU.js") + .expect("should evaluate disabled cache rules") + .is_none(), + "disabled rules should never match" ); } #[test] - fn tinybird_accepts_region_host_without_scheme() { - let toml = format!( - "{}\n[tinybird]\nenabled = true\napi_host = \"api.us-east.aws.tinybird.co\"\n", + fn cache_asset_rule_policy_validation_rejects_unsafe_config() { + let missing_ttl = format!( + r#"{} + + [[cache.asset_rules]] + id = "missing-ttl" + enabled = true + path_prefix = "/assets/" + "#, crate_test_settings_str() ); + let missing_ttl_err = + Settings::from_toml(&missing_ttl).expect_err("should reject rule without a TTL"); + assert!( + format!("{missing_ttl_err:?}").contains("browser_ttl_seconds or edge_ttl_seconds"), + "should explain missing TTL: {missing_ttl_err:?}" + ); - let settings = Settings::from_toml(&toml).expect("should accept Tinybird region host"); - assert!(settings.tinybird.enabled); - assert_eq!(settings.tinybird.api_host, "api.us-east.aws.tinybird.co"); - } + let immutable_without_fingerprint_style = format!( + r#"{} - #[test] - fn tinybird_access_enabled_is_rejected_until_emitter_is_wired() { - let toml = format!( - "{}\n[tinybird]\naccess_enabled = true\n", + [[cache.asset_rules]] + id = "unsafe-immutable" + enabled = true + path_prefix = "/assets/" + browser_ttl_seconds = 31536000 + immutable = true + "#, crate_test_settings_str() ); - - let err = Settings::from_toml(&toml) - .expect_err("should reject access telemetry before emitter exists"); + let fingerprint_style_err = Settings::from_toml(&immutable_without_fingerprint_style) + .expect_err("should reject immutable rule without a fingerprint style"); assert!( - format!("{err:?}").contains("tinybird.access_enabled"), - "should report unsupported tinybird.access_enabled setting: {err:?}" + format!("{fingerprint_style_err:?}").contains("fingerprint_style"), + "should explain immutable fingerprint-style requirement: {fingerprint_style_err:?}" ); - } - #[test] - fn test_settings_from_valid_toml() { - let toml_str = crate_test_settings_str(); - let settings = Settings::from_toml(&toml_str); + let immutable_without_browser_ttl = format!( + r#"{} - assert!(settings.is_ok()); + [[cache.asset_rules]] + id = "immutable-without-browser-ttl" + enabled = true + path_prefix = "/assets/" + fingerprint_style = "hex" + browser_ttl_seconds = 0 + edge_ttl_seconds = 31536000 + immutable = true + "#, + crate_test_settings_str() + ); + let browser_ttl_err = Settings::from_toml(&immutable_without_browser_ttl) + .expect_err("should reject immutable rule without positive browser TTL"); + assert!( + format!("{browser_ttl_err:?}").contains("positive browser_ttl_seconds"), + "should explain immutable browser TTL requirement: {browser_ttl_err:?}" + ); - let settings = settings.expect("should parse valid TOML"); - let prebid_cfg = settings - .integration_config::("prebid") - .expect("Prebid config query should succeed") - .expect("Prebid config should load from test settings"); - assert_eq!( - prebid_cfg.server_url, - "https://test-prebid.com/openrtb2/auction" + let private_edge_only = format!( + r#"{} + + [[cache.asset_rules]] + id = "private-edge-only" + enabled = true + path_prefix = "/assets/" + visibility = "private" + edge_ttl_seconds = 300 + "#, + crate_test_settings_str() ); + let private_edge_only_err = Settings::from_toml(&private_edge_only) + .expect_err("should reject private rule with only an edge TTL"); assert!( - settings - .integration_config::("nextjs") - .expect("Next.js config query should succeed") - .is_none(), - "Next.js integration should default to disabled" + format!("{private_edge_only_err:?}").contains("edge_ttl_seconds"), + "should explain that private rules cannot use an edge TTL: {private_edge_only_err:?}" ); - let raw_nextjs = settings - .integrations - .get("nextjs") - .expect("test settings should include nextjs block"); - assert_eq!(raw_nextjs["enabled"], json!(false)); - assert_eq!( - raw_nextjs["rewrite_attributes"], - json!(["href", "link", "url"]), - "Next.js rewrite attributes should default to href/link/url" + + let private_dual_ttl = private_edge_only.replace( + "id = \"private-edge-only\"", + "id = \"private-dual-ttl\"\n browser_ttl_seconds = 300", ); - assert_eq!(settings.publisher.domain, "test-publisher.com"); - assert_eq!(settings.publisher.cookie_domain, ".test-publisher.com"); + let private_dual_ttl_err = Settings::from_toml(&private_dual_ttl) + .expect_err("should reject private rule with browser and edge TTLs"); assert!( - !settings.tester_cookie.enabled, - "tester-cookie route should default to disabled" + format!("{private_dual_ttl_err:?}").contains("edge_ttl_seconds"), + "should reject edge TTL even when a private rule has a browser TTL: {private_dual_ttl_err:?}" ); - assert_eq!( - settings.publisher.ec_cookie_domain(), - ".test-publisher.com", - "EC cookie domain should be computed as .{{domain}}" + + let private_browser_ttl = private_edge_only.replace( + "id = \"private-edge-only\"\n enabled = true\n path_prefix = \"/assets/\"\n visibility = \"private\"\n edge_ttl_seconds = 300", + "id = \"private-browser-ttl\"\n enabled = true\n path_prefix = \"/assets/\"\n visibility = \"private\"\n browser_ttl_seconds = 300", ); + let private_settings = Settings::from_toml(&private_browser_ttl) + .expect("should accept a private rule with a browser TTL"); + let private_policy = private_settings + .asset_cache_policy_for_path("/assets/app.js") + .expect("should evaluate private cache rule") + .expect("should match private cache rule"); assert_eq!( - settings.publisher.origin_url, - "https://origin.test-publisher.com" + private_policy + .cache_control_value(crate::cache_policy::EdgeCacheHeader::SurrogateControl), + "private, max-age=300", + "private rules should render their browser TTL" ); - assert_eq!(settings.publisher.origin_host_header_override, None); assert_eq!( - settings.ec.passphrase.expose(), - "test-secret-key-32-bytes-minimum" + private_policy + .edge_header_value(crate::cache_policy::EdgeCacheHeader::SurrogateControl), + None, + "private rules should not render an edge cache TTL" ); - - settings.validate().expect("Failed to validate settings"); } #[test] - fn tester_cookie_enabled_parses_from_toml() { - let toml_str = format!( + fn cache_asset_rule_validation_rejects_invalid_config() { + let duplicate_ids = format!( r#"{} - [tester_cookie] + [[cache.asset_rules]] + id = "duplicate" + enabled = true + path_prefix = "/assets/" + + [[cache.asset_rules]] + id = "duplicate" enabled = true + path_prefix = "/static/" "#, crate_test_settings_str() ); + let duplicate_err = + Settings::from_toml(&duplicate_ids).expect_err("should reject duplicate rule ids"); + assert!( + format!("{duplicate_err:?}").contains("duplicate id"), + "should explain duplicate rule id: {duplicate_err:?}" + ); - let settings = Settings::from_toml(&toml_str).expect("should parse tester-cookie config"); + let invalid_regex = format!( + r#"{} + [[cache.asset_rules]] + id = "bad-regex" + enabled = true + path_regex = "[" + "#, + crate_test_settings_str() + ); + let regex_err = + Settings::from_toml(&invalid_regex).expect_err("should reject invalid regex"); assert!( - settings.tester_cookie.enabled, - "tester-cookie config should enable the route" + format!("{regex_err:?}").contains("path_regex"), + "should explain invalid regex: {regex_err:?}" + ); + + let invalid_shape = format!( + r#"{} + + [[cache.asset_rules]] + id = "too-many-matchers" + enabled = true + path_prefix = "/assets/" + extensions = ["js"] + "#, + crate_test_settings_str() + ); + let shape_err = + Settings::from_toml(&invalid_shape).expect_err("should reject invalid matcher shape"); + assert!( + format!("{shape_err:?}").contains("exactly one matcher"), + "should explain invalid matcher shape: {shape_err:?}" + ); + + let missing_matcher = format!( + r#"{} + + [[cache.asset_rules]] + id = "missing-matcher" + enabled = true + browser_ttl_seconds = 60 + "#, + crate_test_settings_str() + ); + let missing_matcher_err = + Settings::from_toml(&missing_matcher).expect_err("should reject missing matcher"); + assert!( + format!("{missing_matcher_err:?}").contains("exactly one matcher"), + "should explain missing matcher: {missing_matcher_err:?}" ); } @@ -4874,7 +6239,13 @@ origin_host_header_overide = "www.example.com""#, .expect("should check admin coverage"); assert_eq!( uncovered, - vec!["/_ts/admin/keys/rotate", "/_ts/admin/keys/deactivate"], + vec![ + "/_ts/admin/keys/rotate", + "/_ts/admin/keys/deactivate", + "/_ts/admin/ec", + "/_ts/admin/ec/{id}", + "/_ts/admin/eids", + ], "should report every admin endpoint as uncovered" ); } @@ -4908,11 +6279,195 @@ origin_host_header_overide = "www.example.com""#, .expect("should check admin coverage"); assert_eq!( uncovered, - vec!["/_ts/admin/keys/deactivate"], + vec![ + "/_ts/admin/keys/deactivate", + "/_ts/admin/ec", + "/_ts/admin/ec/{id}", + "/_ts/admin/eids", + ], "should detect the admin endpoints not covered by the narrow handler" ); } + #[test] + fn from_toml_rejects_literal_parameter_template_auth_coverage() { + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin/(keys/rotate|keys/deactivate|ec|eids)$" + username = "admin" + password = "strong-test-password" + + [[handlers]] + path = "^/_ts/admin/ec/[{]id[}]$" + username = "admin" + password = "strong-test-password""#, + ); + + let error = Settings::from_toml(&toml_str) + .expect_err("should reject literal parameter-template auth coverage"); + let message = format!("{error:?}"); + assert!( + message.contains("/_ts/admin/ec/{id}"), + "should identify the concrete EC route as uncovered, got: {message}" + ); + } + + #[test] + fn from_toml_rejects_lowercase_only_dynamic_admin_ec_auth_coverage() { + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin/(keys/rotate|keys/deactivate|ec|eids)$" + username = "admin" + password = "strong-test-password" + + [[handlers]] + path = "^/_ts/admin/ec/[a-f0-9]{64}[.][a-z0-9]{6}$" + username = "admin" + password = "strong-test-password""#, + ); + + let error = Settings::from_toml(&toml_str) + .expect_err("should reject lowercase-only dynamic EC auth coverage"); + let message = format!("{error:?}"); + assert!( + message.contains("/_ts/admin/ec/{id}"), + "should identify the mixed-case EC route as uncovered, got: {message}" + ); + } + + #[test] + fn from_toml_rejects_placeholder_password_on_shadowing_admin_handler() { + // Handler selection is first-match-wins, so a narrow handler placed + // ahead of the admin matcher governs the EC IDs it matches. No probe + // enumerates those IDs, so the placeholder check cannot be limited to + // handlers inferred to cover an admin endpoint. + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin/ec/[a-f0-9]{64}[.]zzzzzz$" + username = "admin" + password = "change-me-admin-password" + + [[handlers]] + path = "^/_ts/admin" + username = "admin" + password = "strong-test-password""#, + ); + + let error = Settings::from_toml(&toml_str) + .expect_err("should reject placeholder password on shadowing admin handler"); + let message = format!("{error:?}"); + assert!( + message.contains("placeholder password"), + "should identify the placeholder handler password, got: {message}" + ); + } + + #[test] + fn from_toml_rejects_weak_password_on_non_admin_handler() { + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin" + username = "admin" + password = "strong-test-password" + + [[handlers]] + path = "^/private" + username = "admin" + password = "changeme""#, + ); + + let error = Settings::from_toml(&toml_str) + .expect_err("should reject a weak password on any handler"); + let message = format!("{error:?}"); + assert!( + message.contains("placeholder password"), + "should identify the weak handler password, got: {message}" + ); + } + + #[test] + fn from_toml_rejects_sampled_id_only_dynamic_admin_ec_auth_coverage() { + // A handler anchored to the full EC ID grammar still leaves the rest of + // the route surface (malformed IDs, which the router accepts and the + // admin handler rejects with 400) unauthenticated, so coverage must not + // be inferred from ID-shaped samples. + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin/(keys/rotate|keys/deactivate|ec|eids)$" + username = "admin" + password = "strong-test-password" + + [[handlers]] + path = "^/_ts/admin/ec/[a-f0-9]{64}[.][A-Za-z0-9]{6}$" + username = "admin" + password = "strong-test-password""#, + ); + + let error = Settings::from_toml(&toml_str) + .expect_err("should reject ID-sampled dynamic EC auth coverage"); + let message = format!("{error:?}"); + assert!( + message.contains("/_ts/admin/ec/{id}"), + "should identify the dynamic EC route as uncovered, got: {message}" + ); + } + + #[test] + fn from_toml_rejects_prefix_anchored_admin_ec_auth_coverage() { + // `^/_ts/admin/ec/$` matches the prefix probe but no actual lookup. + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin/(keys/rotate|keys/deactivate|ec|eids)$" + username = "admin" + password = "strong-test-password" + + [[handlers]] + path = "^/_ts/admin/ec/$" + username = "admin" + password = "strong-test-password""#, + ); + + let error = Settings::from_toml(&toml_str) + .expect_err("should reject prefix-anchored dynamic EC auth coverage"); + let message = format!("{error:?}"); + assert!( + message.contains("/_ts/admin/ec/{id}"), + "should identify the dynamic EC route as uncovered, got: {message}" + ); + } + + #[test] + fn from_toml_accepts_prefix_matcher_admin_ec_auth_coverage() { + let toml_str = crate_test_settings_str().replace( + r#"path = "^/_ts/admin" + username = "admin" + password = "admin-pass""#, + r#"path = "^/_ts/admin/(keys/rotate|keys/deactivate|ec|eids)$" + username = "admin" + password = "strong-test-password" + + [[handlers]] + path = "^/_ts/admin/ec/" + username = "admin" + password = "strong-test-password""#, + ); + + Settings::from_toml(&toml_str) + .expect("should accept a prefix-level matcher for the dynamic EC route"); + } + #[test] fn from_toml_and_env_rejects_config_without_admin_handler() { let origin_key = format!( @@ -5014,6 +6569,10 @@ formats = [{ width = 300, height = 250 }] let co = settings .creative_opportunities .expect("should have creative_opportunities"); + assert!( + co.enabled, + "creative-opportunity templates should default to enabled" + ); assert_eq!(co.gam_network_id, "21765378893"); assert_eq!(co.auction_timeout_ms, Some(500)); assert_eq!( @@ -5023,6 +6582,45 @@ formats = [{ width = 300, height = 250 }] ); } + #[test] + fn settings_disables_creative_opportunity_slots_when_configured_off() { + let toml = format!( + "{}\n[creative_opportunities]\nenabled = false\ngam_network_id = \"21765378893\"\n\n[[creative_opportunities.slot]]\nid = \"atf\"\npage_patterns = [\"/\"]\nformats = [{{ width = 300, height = 250 }}]\n", + crate_test_settings_str() + ); + let settings = Settings::from_toml(&toml).expect("should parse disabled templates"); + assert!( + settings.creative_opportunity_slots().is_empty(), + "disabled template delivery should expose no runtime slots" + ); + } + + #[test] + fn settings_creative_opportunity_enabled_flag_supports_environment_override() { + let toml = format!( + "{}\n[creative_opportunities]\nenabled = true\ngam_network_id = \"21765378893\"\n", + crate_test_settings_str() + ); + let env_key = format!( + "{}{}CREATIVE_OPPORTUNITIES{}ENABLED", + ENVIRONMENT_VARIABLE_PREFIX, + ENVIRONMENT_VARIABLE_SEPARATOR, + ENVIRONMENT_VARIABLE_SEPARATOR + ); + + temp_env::with_var(env_key, Some("false"), || { + let settings = Settings::from_toml_and_env(&toml) + .expect("should parse template enabled environment override"); + assert!( + !settings + .creative_opportunities + .expect("should have creative opportunities") + .enabled, + "environment override should disable template delivery" + ); + }); + } + #[test] fn settings_rejects_invalid_creative_opportunity_slot_id() { let toml = r#" diff --git a/crates/trusted-server-core/src/tsjs.rs b/crates/trusted-server-core/src/tsjs.rs index 20e7d3ded..e8275b814 100644 --- a/crates/trusted-server-core/src/tsjs.rs +++ b/crates/trusted-server-core/src/tsjs.rs @@ -1,4 +1,4 @@ -use trusted_server_js::{all_module_ids, concatenated_hash, single_module_hash}; +use trusted_server_js::{concatenated_hash, single_module_hash}; /// `/static` URL for the tsjs bundle with cache-busting hash based on /// the concatenated content of the given module set. @@ -46,25 +46,28 @@ pub fn tsjs_script_tag_with_attributes( ) } -/// `/static` URL for the unified bundle with a conservative cache-busting hash. +/// `/static` URL for the unified bundle when exact module IDs are unavailable. /// -/// Hashes all compiled module IDs so the cache invalidates whenever any module -/// changes. Over-invalidates slightly (includes deferred modules in the hash) -/// but never serves stale content. Use [`tsjs_script_src`] with exact module -/// IDs when `IntegrationRegistry` is available. +/// This intentionally omits `?v=` because the serving path can only mark a URL +/// immutable when the hash matches the exact enabled module set. Use +/// [`tsjs_script_src`] with exact module IDs when [`IntegrationRegistry`] is +/// available. +/// +/// [`IntegrationRegistry`]: crate::integrations::IntegrationRegistry #[must_use] pub fn tsjs_unified_script_src() -> String { - let ids = all_module_ids(); - tsjs_script_src(&ids) + "/static/tsjs=tsjs-unified.min.js".to_string() } -/// `", + tsjs_unified_script_src() + ) } /// `/static` URL for one module with its own cache-busting hash. @@ -259,19 +262,17 @@ mod tests { } #[test] - fn tsjs_unified_helpers_use_all_module_ids() { - let ids = all_module_ids(); + fn tsjs_unified_helpers_use_unversioned_fallback_without_registry() { let src = tsjs_unified_script_src(); assert_eq!( - src, - tsjs_script_src(&ids), - "should hash all module IDs for the unified script source" + src, "/static/tsjs=tsjs-unified.min.js", + "registry-free unified helper should not emit an unverifiable hash" ); assert_eq!( tsjs_unified_script_tag(), - format!(""), - "should keep the all-module generic tag byte-for-byte unmarked" + format!(r#""#), + "should wrap the registry-free unified source" ); } @@ -335,14 +336,13 @@ mod tests { } #[test] - fn tsjs_unified_script_src_and_tag_include_cache_busting_hash() { + fn tsjs_unified_script_src_and_tag_omit_unverifiable_cache_busting_hash() { let src = tsjs_unified_script_src(); - assert!( - src.starts_with("/static/tsjs=tsjs-unified.min.js?v="), - "should include unified script URL prefix" + assert_eq!( + src, "/static/tsjs=tsjs-unified.min.js", + "should use the unified script URL without an unverifiable hash" ); - assert_sha256_hex_hash(hash_query_value(&src)); assert_eq!( tsjs_unified_script_tag(), format!(r#""#), diff --git a/crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts b/crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts index 6c0301e2c..f201e8f27 100644 --- a/crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts +++ b/crates/trusted-server-integration-tests/browser/tests/shared/aps-renderer.spec.ts @@ -155,6 +155,9 @@ const FAKE_RUNNER = `(function(){ if (bid.ext.tagtype === 'iframe') { var frame = document.createElement('iframe'); frame.setAttribute('sandbox', 'allow-scripts allow-same-origin'); + frame.width = String(bid.w); + frame.height = String(bid.h); + frame.style.border = '0'; frame.src = bid.ext.creativeurl; document.body.appendChild(frame); } else { @@ -192,7 +195,7 @@ const SCRIPT_CREATIVE = `(function(){ }, '*'); })();`; -test.describe("APS opaque renderer", () => { +test.describe("APS rendering", () => { test("renders a trustedServer adapter bid using Prebid's generated GAM ad ID", async ({ page, }) => { @@ -956,4 +959,136 @@ parent.postMessage(JSON.stringify({ 1, ); }); + + test("renders publisher-native mode through the injected friendly-frame runner", async ({ + page, + }) => { + const publisherOrigin = "https://publisher.example"; + const auctionUrl = `${publisherOrigin}/auction`; + const testUrl = `${publisherOrigin}/aps-publisher-native-test`; + const renderer = descriptor("iframe"); + let runnerRequests = 0; + + await page.route(RUNNER_URL, async (route) => { + runnerRequests += 1; + await route.fulfill({ + status: 200, + contentType: "application/javascript", + body: FAKE_RUNNER, + }); + }); + await page.route(IFRAME_CREATIVE_URL, async (route) => { + await route.fulfill({ + status: 200, + contentType: "text/html", + body: IFRAME_CREATIVE, + }); + }); + await page.route(auctionUrl, async (route) => { + await route.fulfill({ + status: 200, + contentType: "application/json", + body: JSON.stringify({ + id: "fictional-native-auction", + seatbid: [ + { + seat: "aps", + bid: [ + { + id: renderer.bidId, + impid: "publisher-native-slot", + price: 1.23, + w: renderer.width, + h: renderer.height, + ext: { trusted_server: { renderer } }, + }, + ], + }, + ], + ext: {}, + }), + }); + }); + await page.route(testUrl, async (route) => { + await route.fulfill({ + status: 200, + contentType: "text/html", + headers: { + "Content-Security-Policy": + "default-src 'none'; script-src 'unsafe-inline' https://client.aps.amazon-adsystem.com https://creative.example; connect-src 'self'; frame-src https://creative.example", + }, + body: ` + +
existing publisher content
`, + }); + }); + + await page.goto(testUrl); + await page.addScriptTag({ path: clientAuctionBundlePaths().core }); + await page.evaluate(() => { + const tsjs = ( + window as unknown as { + tsjs: { + addAdUnits(units: Array>): void; + requestAds(): void; + }; + } + ).tsjs; + tsjs.addAdUnits([ + { + code: "publisher-native-slot", + mediaTypes: { banner: { sizes: [[300, 250]] } }, + bids: [], + }, + ]); + tsjs.requestAds(); + }); + + await expect.poll(() => runnerRequests).toBe(1); + const frame = page.locator("#publisher-native-slot > iframe"); + await expect(frame).toHaveCount(1); + await expect(frame).toBeVisible(); + expect(await frame.getAttribute("sandbox")).toBeNull(); + await expect( + frame + .contentFrame() + .locator(`iframe[src="${IFRAME_CREATIVE_URL}"]`), + ).toHaveCount(1); + await expect( + page.locator("#publisher-native-slot .existing"), + ).toHaveCount(0); + expect( + await frame.evaluate((element: HTMLIFrameElement) => { + const document = element.contentDocument!; + const creative = document.body.querySelector("iframe")!; + return { + bodyMargin: getComputedStyle(document.body).margin, + bodyPadding: getComputedStyle(document.body).padding, + creativeDisplay: getComputedStyle(creative).display, + clientWidth: document.documentElement.clientWidth, + clientHeight: document.documentElement.clientHeight, + scrollWidth: document.documentElement.scrollWidth, + scrollHeight: document.documentElement.scrollHeight, + }; + }), + ).toEqual({ + bodyMargin: "0px", + bodyPadding: "0px", + creativeDisplay: "block", + clientWidth: 300, + clientHeight: 250, + scrollWidth: 300, + scrollHeight: 250, + }); + expect( + await page + .locator("#publisher-native-slot") + .evaluate( + (slot) => + slot.querySelectorAll( + 'iframe[src*="/integrations/aps/renderer"]', + ).length, + ), + ).toBe(0); + }); }); diff --git a/crates/trusted-server-integration-tests/tests/common/ec.rs b/crates/trusted-server-integration-tests/tests/common/ec.rs index cde6ad1c4..0a1f149c3 100644 --- a/crates/trusted-server-integration-tests/tests/common/ec.rs +++ b/crates/trusted-server-integration-tests/tests/common/ec.rs @@ -403,3 +403,79 @@ impl Drop for MinimalOrigin { } } } + +/// A minimal HTTP origin that reflects the request's Cookie header in a +/// cacheable HTML response. +/// +/// This makes it possible to assert that an edge runtime does not reuse a +/// cookie-influenced publisher response for another visitor. +pub struct CookieVaryingOrigin { + shutdown_tx: mpsc::Sender<()>, + handle: Option>, +} + +impl CookieVaryingOrigin { + /// Starts the cookie-varying origin on `127.0.0.1:{port}`. + /// + /// # Panics + /// + /// Panics if the port is already in use. + pub fn start(port: u16) -> Self { + let listener = + TcpListener::bind(format!("127.0.0.1:{port}")).expect("should bind origin port"); + listener + .set_nonblocking(true) + .expect("should set listener nonblocking"); + let (shutdown_tx, shutdown_rx) = mpsc::channel::<()>(); + + let handle = thread::spawn(move || { + loop { + if shutdown_rx.try_recv().is_ok() { + break; + } + + match listener.accept() { + Ok((mut stream, _addr)) => { + let mut buf = [0u8; 4096]; + let Ok(bytes_read) = stream.read(&mut buf) else { + continue; + }; + let request = String::from_utf8_lossy(&buf[..bytes_read]); + let cookie = request + .lines() + .find_map(|line| { + let (name, value) = line.split_once(':')?; + name.eq_ignore_ascii_case("cookie").then(|| value.trim()) + }) + .unwrap_or("viewer=missing"); + let body = format!("{cookie}"); + let response = format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + ); + let _ = stream.write_all(response.as_bytes()); + let _ = stream.flush(); + } + Err(err) if err.kind() == std::io::ErrorKind::WouldBlock => { + thread::sleep(Duration::from_millis(10)); + } + Err(_) => break, + } + } + }); + + Self { + shutdown_tx, + handle: Some(handle), + } + } +} + +impl Drop for CookieVaryingOrigin { + fn drop(&mut self) { + let _ = self.shutdown_tx.send(()); + if let Some(handle) = self.handle.take() { + let _ = handle.join(); + } + } +} diff --git a/crates/trusted-server-integration-tests/tests/integration.rs b/crates/trusted-server-integration-tests/tests/integration.rs index 76a267d1f..ebd026410 100644 --- a/crates/trusted-server-integration-tests/tests/integration.rs +++ b/crates/trusted-server-integration-tests/tests/integration.rs @@ -2,6 +2,7 @@ mod common; mod environments; mod frameworks; +use common::ec::CookieVaryingOrigin; use common::runtime::{RuntimeEnvironment, TestError, origin_port, wasm_binary_path}; use environments::{RUNTIME_ENVIRONMENTS, ReadyCheckOptions, wait_for_http_ready}; use error_stack::ResultExt as _; @@ -164,6 +165,55 @@ fn test_nextjs_cloudflare() { test_combination(&runtime, &framework).expect("should pass Next.js on Cloudflare Workers"); } +#[test] +#[ignore = "requires the `wrangler` CLI in $PATH and a prebuilt Cloudflare Workers bundle (run build.sh first); the test starts wrangler dev automatically"] +fn test_cloudflare_dynamic_publisher_response_does_not_cross_cookie_boundaries() { + init_logger(); + let _origin = CookieVaryingOrigin::start(origin_port()); + let runtime = environments::cloudflare::CloudflareWorkers; + let process = runtime + .spawn(&wasm_binary_path()) + .expect("should start Cloudflare Worker"); + let client = reqwest::blocking::Client::new(); + + let first_response = client + .get(format!("{}/cache-regression", process.base_url)) + .header("cookie", "viewer=first") + .send() + .expect("should request first dynamic publisher response"); + assert_eq!( + first_response.status().as_u16(), + 200, + "first dynamic publisher response should succeed" + ); + let first_body = first_response + .text() + .expect("should read first dynamic publisher response"); + + let second_response = client + .get(format!("{}/cache-regression", process.base_url)) + .header("cookie", "viewer=second") + .send() + .expect("should request second dynamic publisher response"); + assert_eq!( + second_response.status().as_u16(), + 200, + "second dynamic publisher response should succeed" + ); + let second_body = second_response + .text() + .expect("should read second dynamic publisher response"); + + assert!( + first_body.contains("viewer=first"), + "first response must preserve its cookie-specific origin body: {first_body}" + ); + assert!( + second_body.contains("viewer=second"), + "second response must not reuse the first visitor's body: {second_body}" + ); +} + #[test] #[ignore = "requires Docker and pre-built trusted-server-axum binary"] fn test_wordpress_axum() { diff --git a/crates/trusted-server-js/Cargo.toml b/crates/trusted-server-js/Cargo.toml index f3af9bfcf..67a4ac698 100644 --- a/crates/trusted-server-js/Cargo.toml +++ b/crates/trusted-server-js/Cargo.toml @@ -15,10 +15,11 @@ workspace = true doctest = false name = "trusted_server_js" path = "src/lib.rs" -test = false [build-dependencies] build-print = { workspace = true } +hex = { workspace = true } +sha2 = { workspace = true } which = { workspace = true } [dependencies] diff --git a/crates/trusted-server-js/build.rs b/crates/trusted-server-js/build.rs index 6d6bdde9f..ba6cd88f2 100644 --- a/crates/trusted-server-js/build.rs +++ b/crates/trusted-server-js/build.rs @@ -12,6 +12,7 @@ use std::path::{Path, PathBuf}; use std::process::{Command, ExitStatus}; use build_print::{info, warn}; +use sha2::{Digest as _, Sha256}; fn main() { // Rebuild if TS sources change (belt-and-suspenders): enumerate every file under lib/ @@ -125,7 +126,7 @@ fn main() { // Copy each module file to OUT_DIR for (_, filename) in &modules { - copy_bundle(filename, true, &crate_dir, &dist_dir, &out_dir); + copy_bundle(filename, true, &dist_dir, &out_dir); } // Generate tsjs_modules.rs with include_str!() for each module @@ -139,9 +140,10 @@ fn main() { ) .expect("should write generated module header"); for (id, filename) in &modules { + let sha256 = bundle_sha256(&out_dir.join(filename)); writeln!( codegen, - " TsjsModuleMeta {{\n bundle: include_str!(concat!(env!(\"OUT_DIR\"), \"/{filename}\")),\n id: \"{id}\",\n }},\n" + " TsjsModuleMeta {{\n bundle: include_str!(concat!(env!(\"OUT_DIR\"), \"/{filename}\")),\n id: \"{id}\",\n sha256: \"{sha256}\",\n }},\n" ) .expect("should write generated module entry"); } @@ -149,6 +151,7 @@ fn main() { codegen.push_str("\npub(crate) struct TsjsModuleMeta {\n"); codegen.push_str(" pub bundle: &'static str,\n"); codegen.push_str(" pub id: &'static str,\n"); + codegen.push_str(" pub sha256: &'static str,\n"); codegen.push_str("}\n"); let generated_path = out_dir.join("tsjs_modules.rs"); @@ -160,30 +163,36 @@ fn main() { }); } -fn copy_bundle(filename: &str, required: bool, crate_dir: &Path, dist_dir: &Path, out_dir: &Path) { - let primary = dist_dir.join(filename); - let fallback = crate_dir.join("dist").join(filename); +fn bundle_sha256(path: &Path) -> String { + let content = fs::read(path).unwrap_or_else(|err| { + panic!( + "tsjs: failed to read copied bundle {} for hashing: {err}", + path.display() + ); + }); + hex::encode(Sha256::digest(&content)) +} + +fn copy_bundle(filename: &str, required: bool, dist_dir: &Path, out_dir: &Path) { + let source = dist_dir.join(filename); let target = out_dir.join(filename); - for source in [&primary, &fallback] { - if source.exists() { - if let Err(err) = fs::copy(source, &target) { - assert!( - !required, - "tsjs: failed to copy {} to {}: {err}", - source.display(), - target.display() - ); - } - return; + if source.exists() { + if let Err(err) = fs::copy(&source, &target) { + assert!( + !required, + "tsjs: failed to copy {} to {}: {err}", + source.display(), + target.display() + ); } + return; } assert!( !required, - "tsjs: bundle {filename} not found: {} (and fallback {}). Ensure Node is installed and `npm run build` succeeds, or commit dist/{filename}.", - primary.display(), - fallback.display() + "tsjs: bundle {filename} not found: {}. Ensure Node is installed and `npm run build` succeeds, or commit dist/{filename}.", + source.display() ); fs::write(&target, "").expect("should write optional empty bundle placeholder"); diff --git a/crates/trusted-server-js/lib/src/core/index.ts b/crates/trusted-server-js/lib/src/core/index.ts index 5d8e41971..b2c4e41e1 100644 --- a/crates/trusted-server-js/lib/src/core/index.ts +++ b/crates/trusted-server-js/lib/src/core/index.ts @@ -36,10 +36,10 @@ api.getConfig = getConfig; // Provide core requestAds API api.requestAds = requestAds; // Defensive defaults: the edge injects adSlots (head-open) and bids (before -// ) only when the server-side ad stack runs for the request. When it -// is gated off (kill switch, consent fail-closed, bots, prefetch), page code -// reading window.tsjs.bids / window.tsjs.adSlots must still see defined -// values instead of throwing. Injected scripts overwrite these wholesale. +// ) only when server-side ad templates run for the request. When template +// delivery is disabled or gated off (auction/consent, bots, prefetch), page code +// reading window.tsjs.bids / window.tsjs.adSlots must still see defined values +// instead of throwing. Injected scripts overwrite these wholesale. api.adSlots ??= []; api.bids ??= {}; // Point global tsjs diff --git a/crates/trusted-server-js/lib/src/core/request.ts b/crates/trusted-server-js/lib/src/core/request.ts index df3fee6a6..c05070a9d 100644 --- a/crates/trusted-server-js/lib/src/core/request.ts +++ b/crates/trusted-server-js/lib/src/core/request.ts @@ -1,5 +1,5 @@ // Request orchestration for tsjs: unified auction endpoint with iframe-based creative rendering. -import { renderApsCreative } from '../integrations/aps/render'; +import { dispatchApsRendering, renderApsCreative } from '../integrations/aps/render'; import { buildAdRequest, sendAuction } from './auction'; import { collectContext } from './context'; @@ -52,7 +52,13 @@ export function requestAds( for (const bid of bids) { if (!bid.impid) continue; if (bid.renderer) { - renderApsCreative({ slotId: bid.impid, renderer: bid.renderer }); + void Promise.resolve( + dispatchApsRendering({ + slotId: bid.impid, + renderer: bid.renderer, + trustedServer: (renderer) => renderApsCreative({ slotId: bid.impid, renderer }), + }) + ); continue; } if (!bid.adm) { diff --git a/crates/trusted-server-js/lib/src/core/types.ts b/crates/trusted-server-js/lib/src/core/types.ts index 0c68d43fe..03ff0aca2 100644 --- a/crates/trusted-server-js/lib/src/core/types.ts +++ b/crates/trusted-server-js/lib/src/core/types.ts @@ -207,7 +207,15 @@ export interface GptDiagnosticsRequestCycle { viewableAtMs?: number; durations: GptDiagnosticsDurations; isEmpty?: boolean; + /** Configured sizes Trusted Server supplied to GPT for this request. */ + requestedSlotSizes?: ReadonlyArray; + /** Exact fill size fact GPT reported in its `slotRenderEnded` callback. */ size?: Size; + /** + * Outer CSS box observed on the uniquely bound, connected slot element after + * a filled GPT render. This is not an assertion about internal creative pixels. + */ + observedSlotSize?: Size; isBackfill?: boolean; slotContentChanged?: boolean; incompleteSequence: boolean; @@ -318,12 +326,13 @@ export interface GptDiagnosticsApi { * and stops the writers from becoming part of the public contract. */ export interface GptDiagnosticsRecorder { - /** Record Trusted Server's creative opportunity for an associated GPT slot. */ + /** Record Trusted Server's creative opportunity and configured sizes for an associated GPT slot. */ recordTrustedServerOpportunity( slot: GptDiagnosticsSlotHandle, auctionSlotId: string, opportunity: GptDiagnosticsTrustedServerOpportunity, - trustedServerAuctionId?: string + trustedServerAuctionId?: string, + requestedSlotSizes?: ReadonlyArray ): void; /** Mark slots whose next observed GPT request follows the Prebid refresh path. */ recordPrebidRefresh(slots: GptDiagnosticsSlotHandle[]): void; @@ -429,6 +438,8 @@ export interface TsjsApi { gptSlotHandoffInternal?: boolean; /** Guards SPA pushState hook installation. */ spaHookInstalled?: boolean; + /** Internal one-shot state shared by bootstrap and bundle scheduler installs. */ + initialAdInitScheduled?: boolean; /** * Monotonic count of committed SPA navigations, incremented synchronously by * the SPA auction hook the moment it accepts a route change. The deferred @@ -452,8 +463,18 @@ export interface TsjsApi { * Lives in the bundle so the lifecycle is executable under test and shares * [`navGeneration`] with the SPA auction hook; `gpt_bootstrap.js` installs * a minimal fallback for pages where the bundle fails to load. + * + * `initialSlots` exists for the shared-template `` seam, which is the + * only place slot definitions arrive with the bids rather than from the head + * script. Passing them here rather than assigning `tsjs.adSlots` before the + * call puts them behind the same generation guard: an assignment made ahead + * of the guard would clobber a committed SPA navigation's slots with the SSR + * document's, and then be read by that route's `adInit()`. */ - scheduleInitialAdInit?: (initialBids?: Record) => void; + scheduleInitialAdInit?: ( + initialBids?: Record, + initialSlots?: AuctionSlot[] + ) => void; /** Read-only GPT lifecycle diagnostics API, present only in an activated tab. */ gptDiagnostics?: GptDiagnosticsApi; /** diff --git a/crates/trusted-server-js/lib/src/integrations/aps/render.ts b/crates/trusted-server-js/lib/src/integrations/aps/render.ts index 85f17adf9..9a773beb3 100644 --- a/crates/trusted-server-js/lib/src/integrations/aps/render.ts +++ b/crates/trusted-server-js/lib/src/integrations/aps/render.ts @@ -1,7 +1,12 @@ import { log } from '../../core/log'; +import { findSlot } from '../../core/render'; import type { ApsPrebidRendererEntry, ApsRendererV1, TsjsApi } from '../../core/types'; export const APS_RENDERER_PATH = '/integrations/aps/renderer'; +export const APS_RENDERING_MODE_META_NAME = 'trusted-server-aps-rendering-mode'; +export const APS_PREBID_CREATIVE_RUNNER_URL = + 'https://client.aps.amazon-adsystem.com/prebid-creative.js'; +export const APS_NATIVE_RENDERER_TIMEOUT_MS = 10_000; export const APS_RENDERER_SANDBOX = 'allow-forms allow-pointer-lock allow-popups allow-popups-to-escape-sandbox allow-scripts allow-top-navigation-by-user-activation'; export const APS_UNIVERSAL_CREATIVE_RENDERER_VERSION = 4; @@ -38,6 +43,36 @@ type ValidatedRendererCacheEntry = { renderer: ApsRendererV1; }; const validatedRendererCache = new WeakMap(); +const nativeDispatches = new Map(); + +function releaseNativeDispatch(slotId: string, dispatch: symbol): boolean { + if (nativeDispatches.get(slotId) !== dispatch) return false; + nativeDispatches.delete(slotId); + return true; +} + +function findApsContainer(slotId: string): HTMLElement | null { + const direct = findSlot(slotId); + if (direct) return direct; + + try { + for (const [divId, mappedSlotId] of Object.entries(window.tsjs?.divToSlotId ?? {})) { + if (mappedSlotId !== slotId) continue; + const mapped = findSlot(divId); + if (mapped) return mapped; + } + + const configuredDivId = window.tsjs?.adSlots?.find((slot) => slot.id === slotId)?.div_id; + return configuredDivId ? findSlot(configuredDivId) : null; + } catch { + return null; + } +} + +function cancelPendingApsRendering(slotId: string): void { + const container = findApsContainer(slotId); + if (container) pendingFrameCancels.get(container)?.(); +} function isRecord(value: unknown): value is Record { return typeof value === 'object' && value !== null && !Array.isArray(value); @@ -277,6 +312,207 @@ export function consumeApsPrebidRenderer(adId: string, expected: ApsPrebidRender return true; } +/** Whether the server explicitly selected the opt-in publisher-native runner mode. */ +export function isPublisherNativeApsRendering(): boolean { + return ( + document.head.querySelector( + `meta[name="${APS_RENDERING_MODE_META_NAME}"][content="publisher_native"]` + ) !== null + ); +} + +export interface DispatchApsRenderingOptions { + slotId: string; + renderer: unknown; + /** Existing Trusted Server owner, invoked only in the default mode. */ + trustedServer: (renderer: ApsRendererV1) => boolean; +} + +/** + * Dispatch a validated APS descriptor to exactly one configured rendering owner. + * + * Native mode loads APS's fixed Prebid creative runner in a publisher-origin friendly + * frame. Superseded attempts are cancelled and never fall back to the opaque renderer. + */ +export function dispatchApsRendering({ + slotId, + renderer: input, + trustedServer, +}: DispatchApsRenderingOptions): boolean | Promise { + // Every attempt supersedes a pending frame for this slot, including an invalid + // replacement that fails before a new frame can be created. + cancelPendingApsRendering(slotId); + const dispatch = Symbol(slotId); + nativeDispatches.set(slotId, dispatch); + + const renderer = validateApsRenderer(input); + if (!renderer) { + releaseNativeDispatch(slotId, dispatch); + log.warn('APS renderer: rejected descriptor'); + return false; + } + if (!isPublisherNativeApsRendering()) { + try { + return trustedServer(renderer); + } finally { + releaseNativeDispatch(slotId, dispatch); + } + } + + let rendering: Promise; + try { + rendering = renderApsPublisherNative({ slotId, renderer }); + } catch { + releaseNativeDispatch(slotId, dispatch); + log.warn('APS native renderer: failed to start publisher-origin frame'); + return Promise.resolve(false); + } + + return rendering.then((accepted) => { + if (!releaseNativeDispatch(slotId, dispatch)) { + log.warn('APS native renderer: ignored stale completion'); + return false; + } + return accepted; + }); +} + +export interface RenderApsPublisherNativeOptions { + slotId: string; + renderer: unknown; +} + +function prepareApsRunnerDocument( + frameWindow: Window & typeof globalThis, + frameDocument: Document +): void { + for (const element of [frameDocument.documentElement, frameDocument.body]) { + element.style.margin = '0px'; + element.style.padding = '0px'; + } + + const normalizeFrame = (node: Node): void => { + if ( + node instanceof frameWindow.HTMLIFrameElement && + node.parentElement === frameDocument.body + ) { + node.style.display = 'block'; + } + }; + Array.from(frameDocument.body.children).forEach(normalizeFrame); + new frameWindow.MutationObserver((records) => { + for (const record of records) record.addedNodes.forEach(normalizeFrame); + }).observe(frameDocument.body, { childList: true }); +} + +/** Render the exact selected response through APS's fixed runner in a friendly iframe. */ +export function renderApsPublisherNative({ + slotId, + renderer: input, +}: RenderApsPublisherNativeOptions): Promise { + const renderer = validateApsRenderer(input); + const container = findApsContainer(slotId); + if (!renderer || !container) { + log.warn( + renderer ? 'APS native renderer: slot not found' : 'APS renderer: rejected descriptor' + ); + return Promise.resolve(false); + } + + // Keep an already committed creative visible until the replacement runner loads. + pendingFrameCancels.get(container)?.(); + const iframe = document.createElement('iframe'); + iframe.title = 'Ad content'; + iframe.width = String(renderer.width); + iframe.height = String(renderer.height); + iframe.style.border = '0'; + iframe.style.display = 'none'; + activeFrames.set(container, iframe); + + return new Promise((resolve) => { + let settled = false; + let runner: HTMLScriptElement | undefined; + + const cleanup = (): void => { + window.clearTimeout(timeoutId); + runner?.removeEventListener('load', commit); + runner?.removeEventListener('error', fail); + }; + const finish = (accepted: boolean, warning?: string): void => { + if (settled) return; + settled = true; + cleanup(); + if (pendingFrameCancels.get(container) === cancel) pendingFrameCancels.delete(container); + + if (!accepted || activeFrames.get(container) !== iframe || !iframe.isConnected) { + if (activeFrames.get(container) === iframe) activeFrames.delete(container); + iframe.remove(); + if (warning) log.warn(warning); + resolve(false); + return; + } + + for (const child of Array.from(container.children)) { + if (child !== iframe) child.remove(); + } + iframe.style.display = ''; + resolve(true); + }; + const cancel = (): void => finish(false); + function fail(): void { + finish(false, 'APS native renderer: creative runner failed'); + } + function commit(): void { + finish(true); + } + + const timeoutId = window.setTimeout( + () => finish(false, 'APS native renderer: creative runner timed out'), + APS_NATIVE_RENDERER_TIMEOUT_MS + ); + pendingFrameCancels.set(container, cancel); + container.appendChild(iframe); + + try { + const frameWindow = iframe.contentWindow as + | (Window & + typeof globalThis & { + _aps: Map> }>; + }) + | null; + const frameDocument = iframe.contentDocument; + if (!frameWindow || !frameDocument) { + fail(); + return; + } + + frameDocument.open(); + frameDocument.write( + '' + ); + frameDocument.close(); + prepareApsRunnerDocument(frameWindow, frameDocument); + frameWindow._aps = new Map(); + frameWindow._aps.set(renderer.accountId, { + queue: [ + new frameWindow.CustomEvent('prebid/creative/render', { + detail: { aaxResponse: renderer.aaxResponse, seatBidId: renderer.bidId }, + }), + ], + store: new Map([['listeners', new Map()]]), + }); + + runner = frameDocument.createElement('script'); + runner.src = APS_PREBID_CREATIVE_RUNNER_URL; + runner.addEventListener('load', commit, { once: true }); + runner.addEventListener('error', fail, { once: true }); + frameDocument.head.appendChild(runner); + } catch { + fail(); + } + }); +} + function createNonce(): string | undefined { if (typeof crypto === 'undefined' || typeof crypto.getRandomValues !== 'function') return undefined; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts index 46acd1b0c..345e970d5 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt/index.ts @@ -11,6 +11,7 @@ import { APS_UNIVERSAL_CREATIVE_RENDERER, APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, apsRendererUrl, + dispatchApsRendering, consumeApsPrebidRenderer, getApsPrebidRenderer, validateApsRenderer, @@ -686,15 +687,18 @@ function installInitialLoadDetector(ts: TsjsApi): void { * SSR bootstrap as current. For the same reason the initial bids payload is * passed in and applied here, generation-guarded — assigning it * unconditionally at body end would clobber the live bids a faster SPA - * navigation already applied. When a navigation has committed since — or - * commits while the deferred callback is pending — the SSR payload is - * dropped and `adInit()` is not run: running anyway would re-run the newer - * route's live slots/bids, destroying and redefining that route's TS slots - * and double-refreshing it. The generation counter (not a URL comparison) - * keeps this guard aligned with the SPA auction hook's own navigation - * identity: a query-only history change the hook ignores must not cancel the - * initial call, while an `/a → /b → /a` round trip — where the URL compares - * equal again — must. + * navigation already applied. + * + * Shared-template seams pass `initialSlots`; inline documents omit them because + * their head script already installed the slots. An explicit empty array clears + * that state, while omission preserves it. The scheduler accepts only its first + * generation-0 call so duplicate public API calls cannot define and display the + * initial slots twice. The latch lives on `tsjs` so a bootstrap fallback that + * claims the initial pass keeps that claim when the bundle replaces its + * scheduler. If a navigation commits before scheduling or before the deferred + * callback, the SSR payload and `adInit()` are both dropped. The generation + * counter (not a URL comparison) keeps this aligned with the SPA auction hook's + * navigation identity. * * Hidden documents: browsers do not service `requestAnimationFrame` while a * document is hidden, so a background-tab load (Cmd+click, open-in-new-tab) @@ -706,9 +710,14 @@ function installInitialLoadDetector(ts: TsjsApi): void { * holds whenever the request is actually issued. */ function installScheduleInitialAdInit(ts: TsjsApi): void { - ts.scheduleInitialAdInit = function (initialBids?: Record) { - if ((ts.navGeneration ?? 0) !== 0) return; - if (initialBids) ts.bids = initialBids; + ts.scheduleInitialAdInit = function ( + initialBids?: Record, + initialSlots?: AuctionSlot[] + ) { + if ((ts.navGeneration ?? 0) !== 0 || ts.initialAdInitScheduled) return; + ts.initialAdInitScheduled = true; + if (initialSlots !== undefined) ts.adSlots = initialSlots; + if (initialBids !== undefined) ts.bids = initialBids; const runUnlessNavigated = (): void => { if ((ts.navGeneration ?? 0) !== 0) return; ts.adInit?.(); @@ -1070,24 +1079,18 @@ export function installTsAdInit(): void { if (bid[key]) gptSlot.setTargeting(key, String(bid[key]!)); }); gptSlot.setTargeting(TS_INITIAL_TARGETING_KEY, '1'); + const requestedSlotSizes = ts.gptSlotHandoffs?.[slotDivId2]?.formats; // Diagnostics are observational only. A missing or malformed debug // implementation must never interrupt slot mapping or delivery. try { const opportunity = trustedServerOpportunity(bid); - if (bid.hb_auction_id !== undefined) { - ts.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( - gptSlot, - slot.id, - opportunity, - bid.hb_auction_id - ); - } else { - ts.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( - gptSlot, - slot.id, - opportunity - ); - } + ts.gptDiagnosticsRecorder?.recordTrustedServerOpportunity( + gptSlot, + slot.id, + opportunity, + bid.hb_auction_id, + requestedSlotSizes + ); } catch { // Diagnostics must not alter ad delivery. } @@ -1115,8 +1118,8 @@ export function installTsAdInit(): void { ts.prevSlotTargetingKeys = nextSlotTargetingKeys; // Whether this call produced any TS slot to render. A gated page-bids - // response (auction kill switch or consent denial) returns no slots, so - // the loops above leave these empty. + // response (template switch, auction gate, or consent denial) returns no + // slots, so the loops above leave these empty. const hasRenderableWork = slotsToDisplay.length > 0 || slotsToRefresh.length > 0; // enableSingleRequest and enableServices must only be called once per page @@ -1424,10 +1427,10 @@ export function installSpaAuctionHook(): void { // This route is now the committed, loaded state — a later failed // navigation rolls back here, and a return trip no-ops correctly. lastAppliedPath = path; - // An empty page-bids response (auction kill switch or consent gate) carries - // no TS slots. Only run adInit() when there are slots to apply or prior TS - // state to sweep — otherwise a consent-denied or kill-switched navigation - // must not enter the GPT command queue and risk activating services. + // An empty page-bids response (template switch, auction, or consent gate) + // carries no TS slots. Only run adInit() when there are slots to apply or + // prior TS state to sweep — otherwise a gated navigation must not enter + // the GPT command queue and risk activating services. const hasPriorTsState = (ts.prevGptSlots?.length ?? 0) > 0 || Object.keys(ts.prevSlotTargetingKeys ?? {}).length > 0 || @@ -1697,29 +1700,49 @@ export function installTsRenderBridge(): void { e.stopImmediatePropagation(); if (!messageSourceBelongsToAdUnit(e.source, prebidRendererEntry.adUnitCode)) return; const renderer = validateApsRenderer(prebidRendererEntry.renderer); - const rendererUrl = apsRendererUrl(); - if (!renderer || !rendererUrl) return; - if (!hasConsumedPrebidApsIdCapacity(consumedPrebidApsIds, adId)) return; + if (!renderer || !hasConsumedPrebidApsIdCapacity(consumedPrebidApsIds, adId)) return; if (!consumeApsPrebidRenderer(adId, prebidRendererEntry)) return; recordConsumedPrebidApsId(consumedPrebidApsIds, adId, prebidRendererEntry.expiresAt); - port.postMessage( - JSON.stringify({ - message: 'Prebid Response', - adId, - renderer: APS_UNIVERSAL_CREATIVE_RENDERER, - rendererVersion: APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, - rendererUrl, - apsRenderer: renderer, - width: renderer.width, - height: renderer.height, - }) - ); - - try { - prebidRendererEntry.markUsed(); - } catch (err) { - log.warn(`[tsjs-gpt] APS Prebid markUsed callback threw for '${adId}'`, err); + const markUsed = (): void => { + try { + prebidRendererEntry.markUsed(); + } catch (err) { + log.warn(`[tsjs-gpt] APS Prebid markUsed callback threw for '${adId}'`, err); + } + }; + const dispatched = dispatchApsRendering({ + slotId: prebidRendererEntry.adUnitCode, + renderer, + trustedServer: (validatedRenderer) => { + const rendererUrl = apsRendererUrl(); + if (!rendererUrl) return false; + try { + port.postMessage( + JSON.stringify({ + message: 'Prebid Response', + adId, + renderer: APS_UNIVERSAL_CREATIVE_RENDERER, + rendererVersion: APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, + rendererUrl, + apsRenderer: validatedRenderer, + width: validatedRenderer.width, + height: validatedRenderer.height, + }) + ); + return true; + } catch (err) { + log.warn(`[tsjs-gpt] APS Prebid response post failed for '${adId}'`, err); + return false; + } + }, + }); + if (typeof dispatched === 'boolean') { + if (dispatched) markUsed(); + } else { + void dispatched.then((accepted) => { + if (accepted) markUsed(); + }); } return; } @@ -1748,19 +1771,34 @@ export function installTsRenderBridge(): void { e.stopImmediatePropagation(); if (consumedServerApsBySlot.get(slotId) === adId) return; const renderer = validateApsRenderer(matchedBid.renderer); - const rendererUrl = apsRendererUrl(); - if (!renderer || !rendererUrl) return; + if (!renderer) return; consumedServerApsBySlot.set(slotId, adId); - port.postMessage( - JSON.stringify({ - message: 'Prebid Response', - adId, - renderer: APS_UNIVERSAL_CREATIVE_RENDERER, - rendererVersion: APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, - rendererUrl, - apsRenderer: renderer, - width: renderer.width, - height: renderer.height, + void Promise.resolve( + dispatchApsRendering({ + slotId, + renderer, + trustedServer: (validatedRenderer) => { + const rendererUrl = apsRendererUrl(); + if (!rendererUrl) return false; + try { + port.postMessage( + JSON.stringify({ + message: 'Prebid Response', + adId, + renderer: APS_UNIVERSAL_CREATIVE_RENDERER, + rendererVersion: APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, + rendererUrl, + apsRenderer: validatedRenderer, + width: validatedRenderer.width, + height: validatedRenderer.height, + }) + ); + return true; + } catch (err) { + log.warn(`[tsjs-gpt] APS server response post failed for '${slotId}'`, err); + return false; + } + }, }) ); return; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts index 99f876b3f..475bc7f93 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/api.ts @@ -17,7 +17,8 @@ interface ApiStore { slot: GptDiagnosticsSlotHandle, auctionSlotId: string, opportunity: GptDiagnosticsTrustedServerOpportunity, - trustedServerAuctionId?: string + trustedServerAuctionId?: string, + requestedSlotSizes?: ReadonlyArray ): void; recordPrebidRefresh(slots: GptDiagnosticsSlotHandle[]): void; recordTrustedServerCreativeRequest(auctionSlotId: string): number | undefined; @@ -63,7 +64,9 @@ function cloneExportSnapshot(snapshot: GptDiagnosticsExportV1): GptDiagnosticsEx requests: slot.requests.map((cycle) => ({ ...cycle, durations: { ...cycle.durations }, + requestedSlotSizes: cycle.requestedSlotSizes?.map((size) => [...size]), size: cycle.size ? [...cycle.size] : undefined, + observedSlotSize: cycle.observedSlotSize ? [...cycle.observedSlotSize] : undefined, adManager: cycle.adManager ? { ...cycle.adManager, @@ -149,18 +152,21 @@ export class GptDiagnosticsApiController { }; this.recorder = { - recordTrustedServerOpportunity: (slot, auctionSlotId, opportunity, trustedServerAuctionId) => + recordTrustedServerOpportunity: ( + slot, + auctionSlotId, + opportunity, + trustedServerAuctionId, + requestedSlotSizes + ) => safelyRecord(() => { - if (trustedServerAuctionId === undefined) { - this.store.recordTrustedServerOpportunity(slot, auctionSlotId, opportunity); - } else { - this.store.recordTrustedServerOpportunity( - slot, - auctionSlotId, - opportunity, - trustedServerAuctionId - ); - } + this.store.recordTrustedServerOpportunity( + slot, + auctionSlotId, + opportunity, + trustedServerAuctionId, + requestedSlotSizes + ); }), recordPrebidRefresh: (slots) => safelyRecord(() => this.store.recordPrebidRefresh(slots)), recordTrustedServerCreativeRequest: (auctionSlotId) => @@ -191,7 +197,9 @@ export class GptDiagnosticsApiController { requests: slot.requests.map((cycle) => ({ ...cycle, durations: { ...cycle.durations }, + requestedSlotSizes: cycle.requestedSlotSizes?.map((size) => [...size]), size: cycle.size ? [...cycle.size] : undefined, + observedSlotSize: cycle.observedSlotSize ? [...cycle.observedSlotSize] : undefined, adManager: cycle.adManager ? { ...cycle.adManager, diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/badges.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/badges.ts index 57fce3d85..4dc1250a2 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/badges.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/badges.ts @@ -2,6 +2,7 @@ import type { GptDiagnosticsRequestCycle } from '../../core/types'; import type { GptDiagnosticsBindingManager } from './binding'; import { unhandledCase } from './exhaustive'; +import { formatSizes, scheduleFrame } from './presentation_helpers'; import type { GptDiagnosticsBindingInput, GptDiagnosticsStoreSlotSnapshot, @@ -26,6 +27,7 @@ type BadgeWindow = Window & { const BADGE_MAX_WIDTH_PX = 260; const BADGE_EDGE_GUTTER_PX = 4; +const MAX_BADGE_REQUESTED_SLOT_SIZES = 3; interface BadgeOptions { window?: BadgeWindow; @@ -33,14 +35,6 @@ interface BadgeOptions { scheduleFrame?: (callback: () => void) => void; } -function defaultScheduleFrame(callback: () => void): void { - if (typeof requestAnimationFrame === 'function') { - requestAnimationFrame(() => callback()); - } else { - queueMicrotask(callback); - } -} - function intersectsViewport(rectangle: DOMRect, window: Window): boolean { return ( rectangle.width > 0 && @@ -115,7 +109,19 @@ function badgeText(cycle: GptDiagnosticsRequestCycle): string { const delivery = deliveryLabel(cycle); if (delivery) firstLine.push(delivery); if (cycle.requestPath === 'competing') firstLine.push('Competing paths'); - if (cycle.size) firstLine.push(`${cycle.size[0]}×${cycle.size[1]}`); + if (cycle.requestedSlotSizes) { + const displayedSizes = cycle.requestedSlotSizes.slice(0, MAX_BADGE_REQUESTED_SLOT_SIZES); + const remainingSizeCount = cycle.requestedSlotSizes.length - displayedSizes.length; + firstLine.push( + `Requested ${formatSizes(displayedSizes)}${ + remainingSizeCount > 0 ? ` +${remainingSizeCount}` : '' + }` + ); + } + if (cycle.size) firstLine.push(`GPT fill ${cycle.size[0]}×${cycle.size[1]}`); + if (cycle.observedSlotSize) { + firstLine.push(`Outer box ${cycle.observedSlotSize[0]}×${cycle.observedSlotSize[1]}`); + } const timingLine: string[] = []; const response = formatMilliseconds(cycle.durations.requestToResponseMs); @@ -159,7 +165,8 @@ export class GptDiagnosticsBadgeManager { this.bindings = bindings; this.window = options.window ?? (window as unknown as BadgeWindow); this.document = options.document ?? document; - this.scheduleFrame = options.scheduleFrame ?? defaultScheduleFrame; + this.scheduleFrame = + options.scheduleFrame ?? ((callback) => scheduleFrame(this.window, callback)); this.refreshSlotElementIds(); this.unsubscribeStore = this.store.subscribe(() => { this.refreshSlotElementIds(); diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/binding.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/binding.ts index 489e7beb1..c6c46a727 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/binding.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/binding.ts @@ -1,5 +1,6 @@ import type { GptDiagnosticsBinding, GptDiagnosticsSlotExport } from '../../core/types'; +import { scheduleFrame } from './presentation_helpers'; import type { GptDiagnosticsBindingInput } from './store'; interface BindingStore { @@ -27,14 +28,6 @@ export interface GptDiagnosticsBindingView { type BindingListener = () => void; -function defaultScheduleFrame(callback: () => void): void { - if (typeof requestAnimationFrame === 'function') { - requestAnimationFrame(() => callback()); - } else { - queueMicrotask(callback); - } -} - function isVisibleInViewport(element: HTMLElement, window: BindingWindow): boolean { const rectangle = element.getBoundingClientRect(); if (rectangle.width <= 0 || rectangle.height <= 0) return false; @@ -97,7 +90,8 @@ export class GptDiagnosticsBindingManager { this.store = store; this.document = options.document ?? document; this.window = options.window ?? (window as unknown as BindingWindow); - this.scheduleFrame = options.scheduleFrame ?? defaultScheduleFrame; + this.scheduleFrame = + options.scheduleFrame ?? ((callback) => scheduleFrame(this.window, callback)); this.unsubscribeStore = this.store.subscribe(() => this.scheduleRefresh()); this.window.addEventListener('scroll', this.scheduleRefresh, { passive: true }); diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts index 75bf97823..d7271710c 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/index.ts @@ -7,6 +7,7 @@ import { GptDiagnosticsBindingManager } from './binding'; import { GptDiagnosticsObserver } from './observer'; import type { GptObserverWindow } from './observer'; import { GptDiagnosticsOverlay } from './overlay'; +import { GptDiagnosticsSlotSizeObserver } from './slot_size_observer'; import { GptDiagnosticsStore } from './store'; interface GptDiagnosticsRuntime { @@ -44,6 +45,7 @@ export function installGptDiagnosticsRuntime( let bindings: GptDiagnosticsBindingManager | undefined; let badges: GptDiagnosticsBadgeManager | undefined; let overlay: GptDiagnosticsOverlay | undefined; + let slotSizeObserver: GptDiagnosticsSlotSizeObserver | undefined; let apiController: GptDiagnosticsApiController | undefined; try { @@ -59,6 +61,7 @@ export function installGptDiagnosticsRuntime( window: target, document: target.document, }); + slotSizeObserver = new GptDiagnosticsSlotSizeObserver(store, bindings, { window: target }); overlay = new GptDiagnosticsOverlay(store, bindings, { window: target, document: target.document, @@ -83,6 +86,7 @@ export function installGptDiagnosticsRuntime( apiController?.destroy(); overlay?.destroy(); badges?.destroy(); + slotSizeObserver?.destroy(); bindings?.destroy(); delete target.__tsjs_gpt_diagnostics_runtime; }, @@ -95,6 +99,7 @@ export function installGptDiagnosticsRuntime( apiController?.destroy(); overlay?.destroy(); badges?.destroy(); + slotSizeObserver?.destroy(); bindings?.destroy(); log.warn('gpt diagnostics: runtime installation failed', error); return undefined; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts index e99f1345b..63c0b6fb3 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/overlay.ts @@ -3,6 +3,7 @@ import type { GptDiagnosticsRequestCycle } from '../../core/types'; import type { GptDiagnosticsBindingManager } from './binding'; import { unhandledCase } from './exhaustive'; +import { formatSizes, scheduleFrame } from './presentation_helpers'; import type { GptDiagnosticsStoreSlotSnapshot, GptDiagnosticsStoreSnapshot } from './store'; export const GPT_DIAGNOSTICS_HOST_ID = 'trusted-server-gpt-diagnostics'; @@ -99,14 +100,6 @@ const PANEL_STYLES = ` } `; -function defaultScheduleFrame(callback: () => void): void { - if (typeof requestAnimationFrame === 'function') { - requestAnimationFrame(() => callback()); - } else { - queueMicrotask(callback); - } -} - function latestCycle( slot: GptDiagnosticsStoreSlotSnapshot ): GptDiagnosticsRequestCycle | undefined { @@ -277,7 +270,13 @@ function cycleFacts(cycle: GptDiagnosticsRequestCycle): string[] { if (cycle.loadAtMs !== undefined) facts.push('GPT slot onload observed'); if (cycle.viewableAtMs !== undefined) facts.push('GPT impressionViewable observed'); if (cycle.incompleteSequence) facts.push('Incomplete sequence'); - if (cycle.size) facts.push(`Rendered size ${cycle.size[0]}×${cycle.size[1]}`); + if (cycle.requestedSlotSizes) { + facts.push(`Requested slot sizes ${formatSizes(cycle.requestedSlotSizes)}`); + } + if (cycle.size) facts.push(`GPT-reported fill size ${cycle.size[0]}×${cycle.size[1]}`); + if (cycle.observedSlotSize) { + facts.push(`Observed outer slot box ${cycle.observedSlotSize[0]}×${cycle.observedSlotSize[1]}`); + } if (cycle.isBackfill !== undefined) facts.push(`Backfill ${cycle.isBackfill ? 'yes' : 'no'}`); if (cycle.slotContentChanged !== undefined) { facts.push(`Slot content changed ${cycle.slotContentChanged ? 'yes' : 'no'}`); @@ -358,7 +357,8 @@ export class GptDiagnosticsOverlay { this.bindings = bindings; this.window = options.window ?? (window as unknown as OverlayWindow); this.document = options.document ?? document; - this.scheduleFrame = options.scheduleFrame ?? defaultScheduleFrame; + this.scheduleFrame = + options.scheduleFrame ?? ((callback) => scheduleFrame(this.window, callback)); this.onExport = options.onExport ?? (() => undefined); this.onShadowRoot = options.onShadowRoot; this.onBadgeLayerChange = options.onBadgeLayerChange; diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/presentation_helpers.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/presentation_helpers.ts new file mode 100644 index 000000000..b601833b8 --- /dev/null +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/presentation_helpers.ts @@ -0,0 +1,18 @@ +import type { Size } from '../../core/types'; + +/** Formats CSS sizes consistently across diagnostics presentation surfaces. */ +export function formatSizes(sizes: ReadonlyArray): string { + return sizes.map((size) => `${size[0]}×${size[1]}`).join(', '); +} + +/** Schedules presentation work in the target window's next animation frame. */ +export function scheduleFrame( + window: Pick, + callback: () => void +): void { + if (typeof window.requestAnimationFrame === 'function') { + window.requestAnimationFrame(() => callback()); + } else { + queueMicrotask(callback); + } +} diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/slot_size_observer.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/slot_size_observer.ts new file mode 100644 index 000000000..c3328f992 --- /dev/null +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/slot_size_observer.ts @@ -0,0 +1,143 @@ +import type { Size } from '../../core/types'; + +import type { GptDiagnosticsBindingManager } from './binding'; +import { scheduleFrame } from './presentation_helpers'; +import type { GptDiagnosticsStoreSnapshot } from './store'; + +interface SlotSizeStore { + snapshot(): GptDiagnosticsStoreSnapshot; + recordObservedSlotSize(runtimeSlotNumber: number, requestNumber: number, size: Size): void; + subscribe(listener: () => void): () => void; +} + +interface SlotSizeBindings { + get: GptDiagnosticsBindingManager['get']; + subscribe(listener: () => void): () => void; +} + +type SlotSizeWindow = Window & { + HTMLElement: typeof HTMLElement; + ResizeObserver?: typeof ResizeObserver; +}; + +interface SlotSizeObserverOptions { + window?: SlotSizeWindow; + scheduleFrame?: (callback: () => void) => void; +} + +interface ObservedCycle { + runtimeSlotNumber: number; + requestNumber: number; +} + +function latestFilledCycle( + slot: GptDiagnosticsStoreSnapshot['slots'][number] +): ObservedCycle | undefined { + const cycle = slot.requests[slot.requests.length - 1]; + if (!cycle || cycle.isEmpty !== false || cycle.renderAtMs === undefined) return undefined; + return { runtimeSlotNumber: slot.runtimeSlotNumber, requestNumber: cycle.requestNumber }; +} + +/** + * Observes the outer CSS boxes of uniquely bound elements after filled GPT renders. + * + * Measurements remain separately labelled from GPT's reported creative size and + * are conditionally written with the runtime-slot and request-cycle identity that + * was current when the measurement was scheduled. + */ +export class GptDiagnosticsSlotSizeObserver { + private readonly store: SlotSizeStore; + private readonly bindings: SlotSizeBindings; + private readonly window: SlotSizeWindow; + private readonly scheduleFrame: (callback: () => void) => void; + private readonly unsubscribeStore: () => void; + private readonly unsubscribeBindings: () => void; + private resizeObserver?: ResizeObserver; + private refreshScheduled = false; + private destroyed = false; + + constructor( + store: SlotSizeStore, + bindings: SlotSizeBindings, + options: SlotSizeObserverOptions = {} + ) { + this.store = store; + this.bindings = bindings; + this.window = options.window ?? (window as unknown as SlotSizeWindow); + this.scheduleFrame = + options.scheduleFrame ?? ((callback) => scheduleFrame(this.window, callback)); + this.unsubscribeStore = this.store.subscribe(this.scheduleRefresh); + this.unsubscribeBindings = this.bindings.subscribe(this.scheduleRefresh); + this.refresh(); + } + + destroy(): void { + if (this.destroyed) return; + this.destroyed = true; + this.unsubscribeStore(); + this.unsubscribeBindings(); + this.resizeObserver?.disconnect(); + } + + private readonly scheduleRefresh = (): void => { + if (this.destroyed || this.refreshScheduled) return; + this.refreshScheduled = true; + this.scheduleFrame(() => { + this.refreshScheduled = false; + this.refresh(); + }); + }; + + private refresh(): void { + if (this.destroyed) return; + this.resizeObserver?.disconnect(); + const observations = new Map(); + const ResizeObserverConstructor = this.window.ResizeObserver; + if (typeof ResizeObserverConstructor === 'function') { + this.resizeObserver = new ResizeObserverConstructor((entries) => { + for (const entry of entries) { + const element = entry.target; + if (!(element instanceof this.window.HTMLElement)) continue; + const cycle = observations.get(element); + if (cycle) this.scheduleMeasure(element, cycle); + } + }); + } + + for (const slot of this.store.snapshot().slots) { + const cycle = latestFilledCycle(slot); + const binding = this.bindings.get(slot.runtimeSlotNumber); + if (!cycle || binding.binding.status !== 'bound' || !binding.element?.isConnected) continue; + observations.set(binding.element, cycle); + this.resizeObserver?.observe(binding.element); + this.scheduleMeasure(binding.element, cycle); + } + } + + private scheduleMeasure(element: HTMLElement, cycle: ObservedCycle): void { + this.scheduleFrame(() => this.measure(element, cycle)); + } + + private measure(element: HTMLElement, cycle: ObservedCycle): void { + if (this.destroyed) return; + + const binding = this.bindings.get(cycle.runtimeSlotNumber); + if (binding.binding.status !== 'bound' || binding.element !== element || !element.isConnected) { + return; + } + + const rectangle = element.getBoundingClientRect(); + if ( + !Number.isFinite(rectangle.width) || + !Number.isFinite(rectangle.height) || + rectangle.width < 0 || + rectangle.height < 0 + ) { + return; + } + this.store.recordObservedSlotSize(cycle.runtimeSlotNumber, cycle.requestNumber, [ + Math.round(rectangle.width), + Math.round(rectangle.height), + ]); + } +} diff --git a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts index 0324a56cc..ca3348ae9 100644 --- a/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts +++ b/crates/trusted-server-js/lib/src/integrations/gpt_diagnostics/store.ts @@ -21,6 +21,7 @@ export const MAX_DIAGNOSTIC_SLOTS = 64; export const MAX_REQUEST_CYCLES_PER_SLOT = 10; export const MAX_CALLBACK_ISSUES = 128; export const MAX_TRUSTED_SERVER_ASSOCIATIONS = 64; +export const MAX_REQUESTED_SLOT_SIZES = 16; export const CREATIVE_ATTEMPT_WINDOW_MS = 30_000; export const MAX_CREATIVE_ATTEMPTS = 128; export const MAX_ATTRIBUTION_ISSUES = 128; @@ -106,6 +107,7 @@ interface PendingSourceEvidence { observedAtMs: number; trustedServerOpportunity?: GptDiagnosticsTrustedServerOpportunity; trustedServerAuctionId?: string; + requestedSlotSizes?: ReadonlyArray; } interface PendingRequestIntent { @@ -205,6 +207,29 @@ function normalizedAuctionId(value: unknown): string | undefined { return new TextEncoder().encode(trimmed).length <= 256 ? trimmed : undefined; } +function normalizedRequestedSlotSizes(value: unknown): ReadonlyArray | undefined { + if (!Array.isArray(value)) return undefined; + + const requestedSlotSizes: Size[] = []; + for (const candidate of value.slice(0, MAX_REQUESTED_SLOT_SIZES)) { + if ( + !Array.isArray(candidate) || + candidate.length !== 2 || + typeof candidate[0] !== 'number' || + typeof candidate[1] !== 'number' || + !Number.isFinite(candidate[0]) || + !Number.isFinite(candidate[1]) || + candidate[0] <= 0 || + candidate[1] <= 0 + ) { + continue; + } + requestedSlotSizes.push(Object.freeze([candidate[0], candidate[1]] as [number, number])); + } + + return requestedSlotSizes.length > 0 ? Object.freeze(requestedSlotSizes) : undefined; +} + function responseClass(cycle: MutableRequestCycle): GptDiagnosticsResponseClass | undefined { if (cycle.renderAtMs === undefined) return undefined; if (cycle.isEmpty === true) return 'empty'; @@ -248,7 +273,9 @@ function copyCycle(cycle: MutableRequestCycle, nowMs: number): GptDiagnosticsReq return { ...cycle, durations: derivedDurations(cycle), + requestedSlotSizes: cycle.requestedSlotSizes?.map((size) => [...size] as Size), size: cycle.size ? ([...cycle.size] as Size) : undefined, + observedSlotSize: cycle.observedSlotSize ? ([...cycle.observedSlotSize] as Size) : undefined, adManager: cycle.adManager ? { ...cycle.adManager, @@ -309,7 +336,8 @@ export class GptDiagnosticsStore { slot: GptDiagnosticsSlotLike, auctionSlotId: string, opportunity: GptDiagnosticsTrustedServerOpportunity, - trustedServerAuctionId?: string + trustedServerAuctionId?: string, + requestedSlotSizes?: ReadonlyArray ): void { if ( !isSlotObject(slot) || @@ -331,6 +359,7 @@ export class GptDiagnosticsStore { this.recordRequestIntentSource(slot, 'trusted_server_direct', { trustedServerOpportunity: opportunity, trustedServerAuctionId: normalizedAuctionId(trustedServerAuctionId), + requestedSlotSizes: normalizedRequestedSlotSizes(requestedSlotSizes), }); } @@ -578,6 +607,9 @@ export class GptDiagnosticsStore { ...(trustedServerEvidence?.trustedServerAuctionId !== undefined ? { trustedServerAuctionId: trustedServerEvidence.trustedServerAuctionId } : {}), + ...(trustedServerEvidence?.requestedSlotSizes !== undefined + ? { requestedSlotSizes: trustedServerEvidence.requestedSlotSizes } + : {}), ...(trustedServerEvidence ? { opportunityToRequestMs: validDuration(trustedServerEvidence.observedAtMs, timestampMs), @@ -666,6 +698,47 @@ export class GptDiagnosticsStore { ); } + /** + * Retain an outer CSS box only when this exact slot and request cycle still + * identify a filled render. Async DOM measurements use this guard so a prior + * render cannot alter a later refresh cycle. + */ + recordObservedSlotSize(runtimeSlotNumber: number, requestNumber: number, size: Size): void { + if ( + !Number.isSafeInteger(requestNumber) || + requestNumber <= 0 || + !Number.isFinite(size[0]) || + !Number.isFinite(size[1]) || + size[0] < 0 || + size[1] < 0 + ) { + return; + } + + const record = this.slots.get(runtimeSlotNumber); + if (!record) return; + + const cycle = record.requests.find((candidate) => candidate.requestNumber === requestNumber); + if ( + !cycle || + record.requests[record.requests.length - 1] !== cycle || + cycle.isEmpty !== false || + cycle.renderAtMs === undefined + ) { + return; + } + + const observedSlotSize: Size = [size[0], size[1]]; + if ( + cycle.observedSlotSize?.[0] === observedSlotSize[0] && + cycle.observedSlotSize[1] === observedSlotSize[1] + ) { + return; + } + cycle.observedSlotSize = observedSlotSize; + this.notify(); + } + recordSlotOnload(slot: GptDiagnosticsSlotLike): void { const timestampMs = this.timestamp(); this.matchCycle( @@ -861,7 +934,10 @@ export class GptDiagnosticsStore { private recordRequestIntentSource( slot: object, source: RequestIntentSource, - facts: Pick = {} + facts: Pick< + PendingSourceEvidence, + 'trustedServerOpportunity' | 'trustedServerAuctionId' | 'requestedSlotSizes' + > = {} ): void { const observedAtMs = this.now(); let intent = this.pendingRequestIntents.get(slot); diff --git a/crates/trusted-server-js/lib/test/core/request.test.ts b/crates/trusted-server-js/lib/test/core/request.test.ts index dc17c9e87..bfc6c88f6 100644 --- a/crates/trusted-server-js/lib/test/core/request.test.ts +++ b/crates/trusted-server-js/lib/test/core/request.test.ts @@ -1,6 +1,10 @@ import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; import type { AdUnit } from '../../src/core/types'; +import { + APS_PREBID_CREATIVE_RUNNER_URL, + APS_RENDERING_MODE_META_NAME, +} from '../../src/integrations/aps/render'; import envelope from '../fixtures/aps-renderer-v1.json'; async function flushRequestAds(): Promise { @@ -137,6 +141,71 @@ describe('request.requestAds', () => { expect(document.querySelector('#slot1 span')).toBeNull(); }); + it('contract test: renders a direct APS bid through the injected native runner', async () => { + const apsBid = envelope.seatbid[0].bid[0]; + const renderer = { + type: 'aps' as const, + version: 1 as const, + accountId: 'example-account-id', + bidId: apsBid.id, + tagType: apsBid.ext.tagtype as 'iframe', + creativeUrl: apsBid.ext.creativeurl, + aaxResponse: btoa(JSON.stringify(envelope)), + width: apsBid.w, + height: apsBid.h, + }; + const marker = document.createElement('meta'); + marker.name = APS_RENDERING_MODE_META_NAME; + marker.content = 'publisher_native'; + document.head.appendChild(marker); + globalThis.fetch = vi.fn().mockResolvedValue({ + ok: true, + status: 200, + headers: { get: () => 'application/json' }, + json: async () => ({ + seatbid: [ + { + seat: 'aps', + bid: [{ impid: 'slot1', ext: { trusted_server: { renderer } } }], + }, + ], + }), + }); + + try { + const { addAdUnits } = await import('../../src/core/registry'); + const { requestAds } = await import('../../src/core/request'); + document.body.innerHTML = '
existing
'; + addAdUnits({ code: 'slot1', mediaTypes: { banner: { sizes: [[300, 250]] } } }); + + requestAds(); + await flushRequestAds(); + const frame = document.querySelector('#slot1 iframe')!; + const runner = frame.contentDocument?.querySelector('script'); + expect(runner).not.toBeNull(); + const frameWindow = frame.contentWindow as unknown as { + _aps: Map>> }>; + }; + const queued = frameWindow._aps.get(renderer.accountId)?.queue[0]; + + expect(frame.getAttribute('sandbox')).toBeNull(); + expect(runner!.src).toBe(APS_PREBID_CREATIVE_RUNNER_URL); + expect(queued?.type).toBe('prebid/creative/render'); + expect(queued?.detail).toEqual({ + aaxResponse: renderer.aaxResponse, + seatBidId: renderer.bidId, + }); + expect(document.querySelector('#slot1 span')).not.toBeNull(); + + runner!.dispatchEvent(new Event('load')); + await Promise.resolve(); + expect(document.querySelector('#slot1 span')).toBeNull(); + expect(frame.style.display).toBe(''); + } finally { + marker.remove(); + } + }); + it('does not mutate the slot for an invalid APS descriptor', async () => { globalThis.fetch = vi.fn().mockResolvedValue({ ok: true, diff --git a/crates/trusted-server-js/lib/test/integrations/aps/render.test.ts b/crates/trusted-server-js/lib/test/integrations/aps/render.test.ts index eae60c90e..6d3edc40b 100644 --- a/crates/trusted-server-js/lib/test/integrations/aps/render.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/aps/render.test.ts @@ -4,11 +4,15 @@ import envelope from '../../fixtures/aps-renderer-v1.json'; import type { ApsRendererV1 } from '../../../src/core/types'; import { log } from '../../../src/core/log'; import { + APS_NATIVE_RENDERER_TIMEOUT_MS, + APS_PREBID_CREATIVE_RUNNER_URL, APS_RENDERER_PATH, APS_RENDERER_SANDBOX, + APS_RENDERING_MODE_META_NAME, APS_UNIVERSAL_CREATIVE_RENDERER, APS_UNIVERSAL_CREATIVE_RENDERER_VERSION, apsRendererUrl, + dispatchApsRendering, getApsPrebidRenderer, parseApsRendererDescriptor, registerApsPrebidRenderer, @@ -16,6 +20,33 @@ import { validateApsRenderer, } from '../../../src/integrations/aps/render'; +function enablePublisherNativeMode(): void { + const marker = document.createElement('meta'); + marker.name = APS_RENDERING_MODE_META_NAME; + marker.content = 'publisher_native'; + document.head.appendChild(marker); +} + +function disablePublisherNativeMode(): void { + document.head + .querySelectorAll(`meta[name="${APS_RENDERING_MODE_META_NAME}"]`) + .forEach((marker) => marker.remove()); +} + +function nativeRunnerState(frame: HTMLIFrameElement): { + runner: HTMLScriptElement; + event: CustomEvent<{ aaxResponse: string; seatBidId: string }>; +} { + const runner = frame.contentDocument?.querySelector('script'); + const frameWindow = frame.contentWindow as unknown as { + _aps: Map> }>; + }; + const account = frameWindow._aps.get('example-account-id'); + expect(runner).not.toBeNull(); + expect(account?.queue).toHaveLength(1); + return { runner: runner!, event: account!.queue[0] }; +} + function encodeBytes(bytes: Uint8Array): string { let binary = ''; for (const byte of bytes) binary += String.fromCharCode(byte); @@ -261,6 +292,197 @@ describe('Prebid APS renderer registry', () => { }); }); +describe('publisher-native APS runner contract tests', () => { + beforeEach(() => { + document.body.innerHTML = '
existing
'; + enablePublisherNativeMode(); + }); + + afterEach(() => { + disablePublisherNativeMode(); + delete window.tsjs; + vi.restoreAllMocks(); + document.body.innerHTML = ''; + }); + + it('queues the exact selected response for the fixed APS runner and commits on load', async () => { + const trustedServer = vi.fn(() => true); + const unrelatedMarker = document.createElement('meta'); + unrelatedMarker.name = APS_RENDERING_MODE_META_NAME; + unrelatedMarker.content = 'trusted_server'; + document.head.appendChild(unrelatedMarker); + + const accepted = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer, + }); + const slot = document.getElementById('fictional-slot')!; + const frame = slot.querySelector('iframe')!; + const { runner, event } = nativeRunnerState(frame); + + expect(frame.getAttribute('sandbox')).toBeNull(); + expect(frame.style.display).toBe('none'); + expect(runner.src).toBe(APS_PREBID_CREATIVE_RUNNER_URL); + expect(event.type).toBe('prebid/creative/render'); + expect(event.detail).toEqual({ + aaxResponse: descriptor().aaxResponse, + seatBidId: descriptor().bidId, + }); + expect(slot.querySelector('span')).not.toBeNull(); + expect(trustedServer).not.toHaveBeenCalled(); + + const runnerDocument = frame.contentDocument!; + expect(runnerDocument.documentElement.style.margin).toBe('0px'); + expect(runnerDocument.documentElement.style.padding).toBe('0px'); + expect(runnerDocument.body.style.margin).toBe('0px'); + expect(runnerDocument.body.style.padding).toBe('0px'); + const creativeFrame = runnerDocument.createElement('iframe'); + runnerDocument.body.appendChild(creativeFrame); + await vi.waitFor(() => expect(creativeFrame.style.display).toBe('block')); + + runner.dispatchEvent(new Event('load')); + await expect(accepted).resolves.toBe(true); + expect(slot.querySelector('span')).toBeNull(); + expect(frame.style.display).toBe(''); + }); + + it('fails closed when the runner fails without clearing publisher content', async () => { + const trustedServer = vi.fn(() => true); + const accepted = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer, + }); + const frame = document.querySelector('#fictional-slot iframe')!; + const { runner } = nativeRunnerState(frame); + + runner.dispatchEvent(new Event('error')); + + await expect(accepted).resolves.toBe(false); + expect(trustedServer).not.toHaveBeenCalled(); + expect(document.querySelector('#fictional-slot iframe')).toBeNull(); + expect(document.querySelector('#fictional-slot span')).not.toBeNull(); + }); + + it('cancels a pending runner when a newer dispatch replaces it', async () => { + const first = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer: () => true, + }); + const firstFrame = document.querySelector('#fictional-slot iframe')!; + + const second = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer: () => true, + }); + const secondFrame = document.querySelector('#fictional-slot iframe')!; + + expect(firstFrame.isConnected).toBe(false); + expect(secondFrame).not.toBe(firstFrame); + await expect(first).resolves.toBe(false); + nativeRunnerState(secondFrame).runner.dispatchEvent(new Event('load')); + await expect(second).resolves.toBe(true); + }); + + it('lets an invalid replacement cancel an older pending runner', async () => { + const first = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer: () => true, + }); + const second = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor({ aaxResponse: 'invalid' }), + trustedServer: () => true, + }); + + expect(second).toBe(false); + await expect(first).resolves.toBe(false); + expect(document.querySelector('#fictional-slot iframe')).toBeNull(); + expect(document.querySelector('#fictional-slot span')).not.toBeNull(); + }); + + it('lets a trusted-server dispatch supersede an older native frame', async () => { + const first = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer: () => true, + }); + + disablePublisherNativeMode(); + const trustedServer = vi.fn(() => true); + const second = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer, + }); + + expect(second).toBe(true); + expect(trustedServer).toHaveBeenCalledOnce(); + await expect(first).resolves.toBe(false); + expect(document.querySelector('#fictional-slot iframe')).toBeNull(); + }); + + it('resolves a logical GPT slot through the injected div mapping', async () => { + document.body.innerHTML = '
existing
'; + window.tsjs = { divToSlotId: { 'div-header': 'homepage_header' } } as typeof window.tsjs; + + const accepted = dispatchApsRendering({ + slotId: 'homepage_header', + renderer: descriptor(), + trustedServer: () => true, + }); + const frame = document.querySelector('#div-header iframe')!; + nativeRunnerState(frame).runner.dispatchEvent(new Event('load')); + + await expect(accepted).resolves.toBe(true); + expect(document.querySelector('#div-header span')).toBeNull(); + }); + + it('contains throwing publisher slot mappings without falling back', async () => { + const tsjs = {} as NonNullable; + Object.defineProperty(tsjs, 'divToSlotId', { + get: () => { + throw new Error('fictional mapping lookup failure'); + }, + }); + window.tsjs = tsjs; + const trustedServer = vi.fn(() => true); + + await expect( + dispatchApsRendering({ + slotId: 'logical-slot', + renderer: descriptor(), + trustedServer, + }) + ).resolves.toBe(false); + expect(trustedServer).not.toHaveBeenCalled(); + expect(document.querySelector('iframe')).toBeNull(); + }); + + it('times out an unacknowledged runner without clearing publisher content', async () => { + vi.useFakeTimers(); + try { + const result = dispatchApsRendering({ + slotId: 'fictional-slot', + renderer: descriptor(), + trustedServer: () => true, + }); + await vi.advanceTimersByTimeAsync(APS_NATIVE_RENDERER_TIMEOUT_MS); + + await expect(result).resolves.toBe(false); + expect(vi.getTimerCount()).toBe(0); + expect(document.querySelector('#fictional-slot iframe')).toBeNull(); + expect(document.querySelector('#fictional-slot span')).not.toBeNull(); + } finally { + vi.useRealTimers(); + } + }); +}); + describe('direct APS rendering', () => { beforeEach(() => { document.body.innerHTML = '
existing
'; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts index 7e7c6d10a..cac8a08a3 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/ad_init.test.ts @@ -7,6 +7,38 @@ import { describe, it, expect, vi, beforeEach, afterEach, afterAll } from 'vites import envelope from '../../fixtures/aps-renderer-v1.json'; import type { AuctionBidData, TsjsApi } from '../../../src/core/types'; +import { + APS_PREBID_CREATIVE_RUNNER_URL, + APS_RENDERING_MODE_META_NAME, +} from '../../../src/integrations/aps/render'; + +function enablePublisherNativeMode(): HTMLMetaElement { + const marker = document.createElement('meta'); + marker.name = APS_RENDERING_MODE_META_NAME; + marker.content = 'publisher_native'; + document.head.appendChild(marker); + return marker; +} + +function nativeRunnerIn(divId: string): { + frame: HTMLIFrameElement; + runner: HTMLScriptElement; + event: CustomEvent<{ aaxResponse: string; seatBidId: string }>; +} { + const container = document.getElementById(divId)!; + const frame = Array.from(container.querySelectorAll('iframe')).find( + (candidate) => candidate.title === 'Ad content' + ); + expect(frame).not.toBeUndefined(); + const runner = frame!.contentDocument?.querySelector('script'); + const frameWindow = frame!.contentWindow as unknown as { + _aps: Map> }>; + }; + const event = Array.from(frameWindow._aps.values())[0]?.queue[0]; + expect(runner?.src).toBe(APS_PREBID_CREATIVE_RUNNER_URL); + expect(event).not.toBeUndefined(); + return { frame: frame!, runner: runner!, event }; +} function apsRenderer() { const bid = envelope.seatbid[0].bid[0]; @@ -206,7 +238,8 @@ describe('installTsAdInit', () => { function configureOpportunityDiagnostics( bid: AuctionBidData | undefined, - recordTrustedServerOpportunity: ReturnType + recordTrustedServerOpportunity: ReturnType, + formats: Array<[number, number]> = [[300, 250]] ) { const mockSlot = { addService: vi.fn().mockReturnThis(), @@ -232,7 +265,7 @@ describe('installTsAdInit', () => { id: 'atf_sidebar_ad', gam_unit_path: '/123/atf', div_id: 'div-atf-sidebar', - formats: [[300, 250]], + formats, targeting: {}, }, ], @@ -293,7 +326,9 @@ describe('installTsAdInit', () => { expect(recordTrustedServerOpportunity).toHaveBeenCalledWith( mockSlot, 'atf_sidebar_ad', - expectedOpportunity + expectedOpportunity, + undefined, + undefined ); } ); @@ -318,7 +353,45 @@ describe('installTsAdInit', () => { mockSlot, 'atf_sidebar_ad', 'unrenderable_candidate', - 'auction-123' + 'auction-123', + undefined + ); + }); + + it('retains handoff formats when reusing a Trusted Server-defined GPT slot', async () => { + const recordTrustedServerOpportunity = vi.fn(); + const formats: Array<[number, number]> = [ + [300, 250], + [728, 90], + [320, 50], + ]; + const { mockSlot } = configureOpportunityDiagnostics( + undefined, + recordTrustedServerOpportunity, + formats + ); + (window as TestWindow).tsjs!.gptSlotHandoffs = { + 'div-atf-sidebar': { + gamUnitPath: '/123/atf', + formats, + divIdPrefix: 'div-atf-sidebar', + slotElementId: 'div-atf-sidebar', + publisherClaimed: true, + suppressPublisherDisplay: false, + suppressPublisherRefresh: false, + }, + }; + + const { installTsAdInit } = await import('../../../src/integrations/gpt/index'); + installTsAdInit(); + (window as TestWindow).tsjs!.adInit!(); + + expect(recordTrustedServerOpportunity).toHaveBeenCalledWith( + mockSlot, + 'atf_sidebar_ad', + 'no_candidate', + undefined, + formats ); }); @@ -334,7 +407,9 @@ describe('installTsAdInit', () => { expect(recordTrustedServerOpportunity).toHaveBeenCalledWith( mockSlot, 'atf_sidebar_ad', - 'no_candidate' + 'no_candidate', + undefined, + undefined ); }); @@ -3198,6 +3273,78 @@ describe('installTsRenderBridge', () => { beaconSpy.mockRestore(); }); + it('contract test: renders a server APS owner with the injected runner and no Universal Creative response', async () => { + const renderer = apsRenderer(); + (window as TestWindow).tsjs.bids.homepage_header = { + hb_adid: renderer.bidId, + renderer, + }; + const marker = enablePublisherNativeMode(); + + try { + const bridgeListener = await captureBridgeListener(); + const source = createTrustedSlotIframe(); + const portMessages: string[] = []; + const request = Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: renderer.bidId }), + ports: [{ postMessage: (message: string) => portMessages.push(message) }], + source, + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent; + + bridgeListener(request); + bridgeListener(request); + const native = nativeRunnerIn('div-header'); + expect(native.event.type).toBe('prebid/creative/render'); + expect(native.event.detail).toEqual({ + aaxResponse: renderer.aaxResponse, + seatBidId: renderer.bidId, + }); + native.runner.dispatchEvent(new Event('load')); + await Promise.resolve(); + await Promise.resolve(); + + expect(native.frame.style.display).toBe(''); + expect(portMessages).toEqual([]); + expect(document.querySelector('iframe[src*="/integrations/aps/renderer"]')).toBeNull(); + } finally { + marker.remove(); + } + }); + + it('contract test: fails a server APS runner without a Universal Creative response or fallback', async () => { + const renderer = apsRenderer(); + (window as TestWindow).tsjs.bids.homepage_header = { + hb_adid: renderer.bidId, + renderer, + }; + const marker = enablePublisherNativeMode(); + + try { + const bridgeListener = await captureBridgeListener(); + const source = createTrustedSlotIframe(); + const portMessages: string[] = []; + const request = Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: renderer.bidId }), + ports: [{ postMessage: (message: string) => portMessages.push(message) }], + source, + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent; + + bridgeListener(request); + nativeRunnerIn('div-header').runner.dispatchEvent(new Event('error')); + await Promise.resolve(); + await Promise.resolve(); + bridgeListener(request); + + expect(portMessages).toEqual([]); + expect(document.querySelector('iframe[title="Ad content"]')).toBeNull(); + expect(document.querySelector('iframe[src*="/integrations/aps/renderer"]')).toBeNull(); + } finally { + marker.remove(); + } + }); + it('serves a registered Prebid APS renderer when its generated ad ID differs from the APS bid ID', async () => { const renderer = apsRenderer(); const prebidAdId = 'prebid-generated-ad-id'; @@ -3253,6 +3400,91 @@ describe('installTsRenderBridge', () => { foreignIframe.remove(); }); + it('contract test: fails a registered APS runner without a Universal Creative response or markUsed', async () => { + const renderer = apsRenderer(); + const prebidAdId = 'native-prebid-decline-ad-id'; + const markUsed = vi.fn(); + (window as TestWindow).tsjs.apsPrebidRenderers = { + [prebidAdId]: { + adUnitCode: 'div-header', + renderer, + registeredAt: Date.now(), + expiresAt: Date.now() + 60_000, + markUsed, + }, + }; + const marker = enablePublisherNativeMode(); + + try { + const bridgeListener = await captureBridgeListener(); + const source = createTrustedSlotIframe(); + const portMessages: string[] = []; + const request = Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: prebidAdId }), + ports: [{ postMessage: (message: string) => portMessages.push(message) }], + source, + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent; + + bridgeListener(request); + nativeRunnerIn('div-header').runner.dispatchEvent(new Event('error')); + await Promise.resolve(); + await Promise.resolve(); + bridgeListener(request); + + expect(markUsed).not.toHaveBeenCalled(); + expect(portMessages).toEqual([]); + expect((window as TestWindow).tsjs.apsPrebidRenderers[prebidAdId]).toBeUndefined(); + expect(document.querySelector('iframe[title="Ad content"]')).toBeNull(); + expect(document.querySelector('iframe[src*="/integrations/aps/renderer"]')).toBeNull(); + } finally { + marker.remove(); + } + }); + + it('contract test: consumes a registered APS capability and marks it used only after runner load', async () => { + const renderer = apsRenderer(); + const prebidAdId = 'native-prebid-ad-id'; + const markUsed = vi.fn(); + (window as TestWindow).tsjs.apsPrebidRenderers = { + [prebidAdId]: { + adUnitCode: 'div-header', + renderer, + registeredAt: Date.now(), + expiresAt: Date.now() + 60_000, + markUsed, + }, + }; + const marker = enablePublisherNativeMode(); + + try { + const bridgeListener = await captureBridgeListener(); + const source = createTrustedSlotIframe(); + const portMessages: string[] = []; + const request = Object.assign(new Event('message'), { + data: JSON.stringify({ message: 'Prebid Request', adId: prebidAdId }), + ports: [{ postMessage: (message: string) => portMessages.push(message) }], + source, + stopImmediatePropagation: vi.fn(), + }) as unknown as MessageEvent; + + bridgeListener(request); + expect(markUsed).not.toHaveBeenCalled(); + const native = nativeRunnerIn('div-header'); + native.runner.dispatchEvent(new Event('load')); + await Promise.resolve(); + await Promise.resolve(); + bridgeListener(request); + + expect(native.frame.style.display).toBe(''); + expect(markUsed).toHaveBeenCalledOnce(); + expect(portMessages).toEqual([]); + expect((window as TestWindow).tsjs.apsPrebidRenderers[prebidAdId]).toBeUndefined(); + } finally { + marker.remove(); + } + }); + it('still serves the APS renderer when markUsed throws', async () => { const renderer = apsRenderer(); const prebidAdId = 'throwing-mark-used-ad-id'; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts index e7b513fdb..77dc6faca 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts @@ -33,6 +33,7 @@ interface MockGoogleTag { pubads: () => unknown; enableServices: () => void; display: (divId: string) => void; + getConfig?: (key: string) => Record; setConfig?: (config: Record) => void; } @@ -268,6 +269,57 @@ describe('gpt_bootstrap.js fallback', () => { expect(adInit).toHaveBeenCalledTimes(1); }); + it('fallback scheduler accepts only the first schedule call', () => { + runBootstrap(); + const ts = (window as TestWindow).tsjs!; + const adInit = vi.fn(); + ts.adInit = adInit; + + ts.scheduleInitialAdInit!({ first: { hb_pb: '1.00' } }); + ts.scheduleInitialAdInit!({ second: { hb_pb: '2.00' } }); + expect(ts.bids).toEqual({ first: { hb_pb: '1.00' } }); + + window.dispatchEvent(new Event('load')); + flushFrame(); + flushFrame(); + expect(adInit).toHaveBeenCalledTimes(1); + }); + + it('fallback scheduler preserves head-injected slots when initialSlots is omitted', () => { + runBootstrap(); + const ts = (window as TestWindow).tsjs!; + ts.adInit = vi.fn(); + const headSlot = { + id: 'head_slot', + gam_unit_path: '/123/head', + div_id: 'div-head', + formats: [[300, 250]] as Array<[number, number]>, + }; + ts.adSlots = [headSlot]; + + ts.scheduleInitialAdInit!({ head_slot: { hb_pb: '1.00' } }); + + expect(ts.adSlots).toEqual([headSlot]); + }); + + it('fallback scheduler replaces existing slots when initialSlots is explicitly empty', () => { + runBootstrap(); + const ts = (window as TestWindow).tsjs!; + ts.adInit = vi.fn(); + ts.adSlots = [ + { + id: 'stale_slot', + gam_unit_path: '/123/stale', + div_id: 'div-stale', + formats: [[300, 250]], + }, + ]; + + ts.scheduleInitialAdInit!({}, []); + + expect(ts.adSlots).toEqual([]); + }); + it('fallback scheduler rides animation frames in a hidden document, holding adInit until first view', () => { // Mirrors the bundle scheduler's intended hidden-tab behavior: rAF is not // serviced while hidden, so a background-tab load holds the initial @@ -308,6 +360,36 @@ describe('gpt_bootstrap.js fallback', () => { expect(adInit).not.toHaveBeenCalled(); }); + it('fallback scheduler guards the SSR slot definitions with the same generation check', () => { + // The shared-template seam hands slots to the scheduler rather than assigning + // them itself, so the fallback has to honour the same guard as the bundle. If it + // applied them unconditionally, a page whose bundle failed to load would take the + // stale SSR slots over a committed navigation's. + runBootstrap(); + const ts = (window as TestWindow).tsjs!; + ts.adInit = vi.fn(); + const liveSlot = { + id: 'live_slot', + gam_unit_path: '/123/live', + div_id: 'div-live', + formats: [[300, 250]] as Array<[number, number]>, + }; + const ssrSlot = { + id: 'ssr_slot', + gam_unit_path: '/123/ssr', + div_id: 'div-ssr', + formats: [[728, 90]] as Array<[number, number]>, + }; + + ts.scheduleInitialAdInit!({ ssr_slot: { hb_pb: '1.00' } }, [ssrSlot]); + expect(ts.adSlots).toEqual([ssrSlot]); + + ts.adSlots = [liveSlot]; + ts.navGeneration = 1; + ts.scheduleInitialAdInit!({ ssr_slot: { hb_pb: '1.00' } }, [ssrSlot]); + expect(ts.adSlots).toEqual([liveSlot]); + }); + it('fallback adInit defines, targets, and displays a TS slot through the command queue', () => { const mockSlot = { addService: vi.fn().mockReturnThis(), diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts index 999c60c55..727300aa1 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts @@ -1,3 +1,6 @@ +import { readFileSync } from 'node:fs'; +import path from 'node:path'; + import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; import type { TsjsApi } from '../../../src/core/types'; @@ -9,6 +12,14 @@ type TestWindow = Window & { const originalPushState = history.pushState.bind(history); const originalReplaceState = history.replaceState.bind(history); +const BOOTSTRAP_SOURCE = readFileSync( + path.resolve(process.cwd(), '../../trusted-server-core/src/integrations/gpt_bootstrap.js'), + 'utf8' +); + +function runBootstrap(): void { + new Function(BOOTSTRAP_SOURCE)(); +} /** * Executable lifecycle coverage for `tsjs.scheduleInitialAdInit` — the @@ -148,6 +159,52 @@ describe('scheduleInitialAdInit', () => { expect(adInit).toHaveBeenCalledTimes(1); }); + it('accepts only the first schedule call', async () => { + await importGptModule(); + const ts = (window as TestWindow).tsjs!; + const adInit = vi.fn(); + ts.adInit = adInit; + + ts.scheduleInitialAdInit!({ first: { hb_pb: '1.00' } }); + ts.scheduleInitialAdInit!({ second: { hb_pb: '2.00' } }); + expect(ts.bids).toEqual({ first: { hb_pb: '1.00' } }); + + window.dispatchEvent(new Event('load')); + flushFrame(); + flushFrame(); + expect(adInit).toHaveBeenCalledTimes(1); + }); + + it('keeps the first schedule claim across bootstrap-to-bundle handoff', async () => { + runBootstrap(); + const ts = (window as TestWindow).tsjs!; + const firstSlot = { + id: 'first_slot', + gam_unit_path: '/123/first', + div_id: 'div-first', + formats: [[300, 250]] as Array<[number, number]>, + }; + const secondSlot = { + id: 'second_slot', + gam_unit_path: '/123/second', + div_id: 'div-second', + formats: [[728, 90]] as Array<[number, number]>, + }; + + ts.scheduleInitialAdInit!({ first_slot: { hb_pb: '1.00' } }, [firstSlot]); + await importGptModule(); + const adInit = vi.fn(); + ts.adInit = adInit; + ts.scheduleInitialAdInit!({ second_slot: { hb_pb: '2.00' } }, [secondSlot]); + + expect(ts.bids).toEqual({ first_slot: { hb_pb: '1.00' } }); + expect(ts.adSlots).toEqual([firstSlot]); + window.dispatchEvent(new Event('load')); + flushFrame(); + flushFrame(); + expect(adInit).toHaveBeenCalledTimes(1); + }); + it('still runs after a query-only history change before load', async () => { // The SPA auction hook identifies routes by pathname only, so a query-only // replaceState is not a navigation: it must neither trigger an auction nor @@ -310,6 +367,104 @@ describe('scheduleInitialAdInit', () => { expect(adInit).toHaveBeenCalledTimes(1); }); + it('applies the SSR slot definitions on the initial document', async () => { + // Under a shared-template mode the head script emits no `tsjs.adSlots`, so the + // `` seam is the only source of slot definitions. They must arrive, or + // `adInit()` iterates an empty list and the page defines no TS slots at all. + await importGptModule(); + const ts = (window as TestWindow).tsjs!; + ts.adInit = vi.fn(); + const ssrSlot = { + id: 'ssr_slot', + gam_unit_path: '/123/ssr', + div_id: 'div-ssr', + formats: [[728, 90]] as Array<[number, number]>, + }; + + ts.scheduleInitialAdInit!({ ssr_slot: { hb_pb: '1.00' } }, [ssrSlot]); + + expect(ts.adSlots).toEqual([ssrSlot]); + expect(ts.bids).toEqual({ ssr_slot: { hb_pb: '1.00' } }); + }); + + it('preserves head-injected slots when initialSlots is omitted', async () => { + await importGptModule(); + const ts = (window as TestWindow).tsjs!; + ts.adInit = vi.fn(); + const headSlot = { + id: 'head_slot', + gam_unit_path: '/123/head', + div_id: 'div-head', + formats: [[300, 250]] as Array<[number, number]>, + }; + ts.adSlots = [headSlot]; + + ts.scheduleInitialAdInit!({ head_slot: { hb_pb: '1.00' } }); + + expect(ts.adSlots).toEqual([headSlot]); + }); + + it('replaces existing slots when initialSlots is explicitly empty', async () => { + await importGptModule(); + const ts = (window as TestWindow).tsjs!; + ts.adInit = vi.fn(); + ts.adSlots = [ + { + id: 'stale_slot', + gam_unit_path: '/123/stale', + div_id: 'div-stale', + formats: [[300, 250]], + }, + ]; + + ts.scheduleInitialAdInit!({}, []); + + expect(ts.adSlots).toEqual([]); + }); + + it('drops the SSR slot definitions when a navigation has already committed', async () => { + // The guard covered the bids and the adInit call, but the shared-template seam + // assigned `tsjs.adSlots` on the line *before* calling the scheduler — outside the + // guard entirely. A navigation that committed while the SSR document was still + // streaming therefore kept its own bids and silently lost its slots to the stale + // SSR payload, and the next `adInit()` for that route defined the wrong slots. + fetchStub.mockResolvedValue({ + ok: true, + json: async () => ({ slots: [], bids: {} }), + }); + await importGptModule(); + const ts = (window as TestWindow).tsjs!; + const adInit = vi.fn(); + ts.adInit = adInit; + + history.pushState({}, '', '/b'); + await flushAsync(); + expect(ts.navGeneration).toBe(1); + const liveSlot = { + id: 'live_slot', + gam_unit_path: '/123/live', + div_id: 'div-live', + formats: [[300, 250]] as Array<[number, number]>, + }; + ts.adSlots = [liveSlot]; + + ts.scheduleInitialAdInit!({ ssr_slot: { hb_pb: '1.00' } }, [ + { + id: 'ssr_slot', + gam_unit_path: '/123/ssr', + div_id: 'div-ssr', + formats: [[728, 90]], + }, + ]); + + expect(ts.adSlots).toEqual([liveSlot]); + + window.dispatchEvent(new Event('load')); + flushFrame(); + flushFrame(); + expect(adInit).not.toHaveBeenCalled(); + }); + it('cancels queued GPT work when a navigation commits before the command queue drains', async () => { // adInit() only queues its slot work on googletag.cmd, which drains when // GPT itself loads — possibly long after the generation check that diff --git a/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts index 7127efcb4..314348fa8 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt/spa_hook.test.ts @@ -216,9 +216,9 @@ describe('installSpaAuctionHook', () => { }); it('skips adInit on an empty page-bids response with no prior TS state', async () => { - // A gated page-bids response (auction kill switch or consent denial) returns - // no slots. With no prior TS state to sweep, the hook must not call adInit() - // so a consent-denied navigation cannot activate the publisher's GPT setup. + // A gated page-bids response (template switch, auction gate, or consent + // denial) returns no slots. With no prior TS state to sweep, the hook must + // not call adInit() so a gated navigation cannot activate publisher GPT. fetchStub.mockResolvedValue({ ok: true, json: async () => ({ slots: [], bids: {} }), diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/api.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/api.test.ts index 2e2ae2d2b..2e3a63b4a 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/api.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/api.test.ts @@ -124,7 +124,9 @@ describe('GptDiagnosticsApiController', () => { expect(store.recordTrustedServerOpportunity).toHaveBeenCalledWith( slot, 'auction-slot-example', - 'renderable_candidate' + 'renderable_candidate', + undefined, + undefined ); expect(store.recordPrebidRefresh).toHaveBeenCalledTimes(1); expect(store.recordPrebidRefresh).toHaveBeenCalledWith(slots); @@ -156,7 +158,8 @@ describe('GptDiagnosticsApiController', () => { slot, 'auction-slot-example', 'renderable_candidate', - 'auction-123' + 'auction-123', + undefined ); }); @@ -214,6 +217,10 @@ describe('GptDiagnosticsApiController', () => { requestNumber: 1, durations: {}, incompleteSequence: false, + requestedSlotSizes: [ + [300, 250], + [728, 90], + ], adManager: { yieldGroupIds: [10], companyIds: [20], @@ -253,6 +260,14 @@ describe('GptDiagnosticsApiController', () => { expect(snapshot.attributionIssues).toEqual(source.attributionIssues); expect(snapshot.attributionIssues).not.toBe(source.attributionIssues); expect(snapshot.attributionIssues?.[0]).not.toBe(source.attributionIssues[0]); + expect(cycle?.requestedSlotSizes).toEqual([ + [300, 250], + [728, 90], + ]); + expect(cycle?.requestedSlotSizes).not.toBe(source.slots[0]?.requests[0]?.requestedSlotSizes); + expect(cycle?.requestedSlotSizes?.[0]).not.toBe( + source.slots[0]?.requests[0]?.requestedSlotSizes?.[0] + ); expect(cycle?.trustedServerCreativeFailures).toEqual(['cache_fetch_failed']); expect(cycle?.trustedServerCreativeFailures).not.toBe( source.slots[0]?.requests[0]?.trustedServerCreativeFailures @@ -348,10 +363,12 @@ describe('GptDiagnosticsApiController', () => { 'incompleteSequence', 'isBackfill', 'isEmpty', + 'observedSlotSize', 'renderAtMs', 'requestNumber', 'requestPath', 'requestedAtMs', + 'requestedSlotSizes', 'responseAtMs', 'responseClass', 'size', @@ -428,6 +445,10 @@ describe('GptDiagnosticsApiController', () => { requestNumber: 1, durations: { requestToResponseMs: 10 }, incompleteSequence: false, + requestedSlotSizes: [ + [300, 250], + [728, 90], + ], adManager: { yieldGroupIds: [10], companyIds: [20] }, trustedServerCreativeFailures: ['cache_fetch_failed' as const], }, @@ -463,6 +484,9 @@ describe('GptDiagnosticsApiController', () => { controller.api.subscribe((snapshot) => { const cycle = snapshot.slots[0]!.requests[0]!; cycle.durations.requestToResponseMs = 999; + const requestedSlotSizes = cycle.requestedSlotSizes as unknown as Array<[number, number]>; + requestedSlotSizes[0]![0] = 1; + requestedSlotSizes.push([970, 250]); cycle.adManager!.yieldGroupIds!.push(99); cycle.trustedServerCreativeFailures!.push('response_post_failed'); snapshot.attributionIssues?.push({ @@ -484,6 +508,10 @@ describe('GptDiagnosticsApiController', () => { expect(observedSnapshot?.capturedAt).toBe('2026-08-10T00:00:00.000Z'); const observedCycle = observedSnapshot?.slots[0]?.requests[0]; expect(observedCycle?.durations.requestToResponseMs).toBe(10); + expect(observedCycle?.requestedSlotSizes).toEqual([ + [300, 250], + [728, 90], + ]); expect(observedCycle?.adManager?.yieldGroupIds).toEqual([10]); expect(observedCycle?.trustedServerCreativeFailures).toEqual(['cache_fetch_failed']); expect(observedSnapshot?.attributionIssues).toHaveLength(1); diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/badges.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/badges.test.ts index 675e4f442..80c240a3e 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/badges.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/badges.test.ts @@ -252,7 +252,12 @@ describe('GptDiagnosticsBadgeManager', () => { renderAtMs: 318, viewableAtMs: 1318, isEmpty: false, + requestedSlotSizes: [ + [728, 90], + [970, 250], + ], size: [728, 90], + observedSlotSize: [980, 270], incompleteSequence: false, durations: { requestToResponseMs: 276, @@ -260,7 +265,23 @@ describe('GptDiagnosticsBadgeManager', () => { renderToViewableMs: 1000, }, }) - ).toBe('Filled · 728×90\nResponse 276 ms · Render 42 ms\nViewable after 1 s'); + ).toBe( + 'Filled · Requested 728×90, 970×250 · GPT fill 728×90 · Outer box 980×270\nResponse 276 ms · Render 42 ms\nViewable after 1 s' + ); + expect( + gptDiagnosticsBadgeTextForTest({ + requestNumber: 1, + isEmpty: false, + requestedSlotSizes: [ + [300, 250], + [320, 50], + [728, 90], + [970, 250], + ], + incompleteSequence: false, + durations: {}, + }) + ).toBe('Filled · Requested 300×250, 320×50, 728×90 +1'); expect( gptDiagnosticsBadgeTextForTest({ requestNumber: 1, diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts index b89a8f507..41cad667a 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/overlay.test.ts @@ -405,6 +405,18 @@ describe('GptDiagnosticsOverlay', () => { const element = document.createElement('div'); element.id = 'filled-slot'; document.body.append(element); + store.recordTrustedServerOpportunity( + filledSlot, + 'filled-slot-auction', + 'renderable_candidate', + undefined, + [ + [300, 250], + [728, 90], + [320, 50], + [970, 250], + ] + ); store.recordSlotRequested(filledSlot); now = 20; store.recordSlotResponseReceived(filledSlot); @@ -414,6 +426,7 @@ describe('GptDiagnosticsOverlay', () => { size: [300, 250], isBackfill: true, }); + store.recordObservedSlotSize(1, 1, [320, 270]); now = 30; store.recordSlotOnload(filledSlot); now = 35; @@ -457,7 +470,9 @@ describe('GptDiagnosticsOverlay', () => { expect(root!.textContent).toContain('/example/site/filled-slot'); expect(root!.textContent).toContain('Empty'); expect(root!.textContent).toContain('Previous requests (1)'); - expect(root!.textContent).toContain('Rendered size 300×250'); + expect(root!.textContent).toContain('Requested slot sizes 300×250, 728×90, 320×50, 970×250'); + expect(root!.textContent).toContain('GPT-reported fill size 300×250'); + expect(root!.textContent).toContain('Observed outer slot box 320×270'); expect(root!.textContent).toContain('Backfill yes'); expect(root!.textContent).toContain('GPT slot onload observed'); expect(root!.textContent).toContain('GPT impressionViewable observed'); diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/slot_size_observer.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/slot_size_observer.test.ts new file mode 100644 index 000000000..3709221ab --- /dev/null +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/slot_size_observer.test.ts @@ -0,0 +1,263 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; + +import type { GptDiagnosticsRequestCycle } from '../../../src/core/types'; +import { GptDiagnosticsSlotSizeObserver } from '../../../src/integrations/gpt_diagnostics/slot_size_observer'; +import type { GptDiagnosticsStoreSnapshot } from '../../../src/integrations/gpt_diagnostics/store'; + +type SlotSizeTestWindow = Window & { + HTMLElement: typeof HTMLElement; + ResizeObserver?: typeof ResizeObserver; +}; + +class ResizeObserverMock { + static instances: ResizeObserverMock[] = []; + readonly observe = vi.fn(); + readonly disconnect = vi.fn(); + + constructor(readonly callback: ResizeObserverCallback) { + ResizeObserverMock.instances.push(this); + } + + emit(element: Element): void { + this.callback([{ target: element } as ResizeObserverEntry], this as unknown as ResizeObserver); + } +} + +function cycle(requestNumber: number, isEmpty: boolean | undefined): GptDiagnosticsRequestCycle { + return { + requestNumber, + isEmpty, + renderAtMs: 1, + durations: {}, + incompleteSequence: false, + }; +} + +function snapshot(requests: GptDiagnosticsRequestCycle[]): GptDiagnosticsStoreSnapshot { + return { + gptObserved: true, + slots: [ + { + runtimeSlotNumber: 1, + slotElementId: 'ad-slot-example', + requests, + }, + ], + callbackIssues: [], + attributionIssues: [], + coverage: { + slotRequested: { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 }, + slotResponseReceived: { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 }, + slotRenderEnded: { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 }, + slotOnload: { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 }, + impressionViewable: { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 }, + slotVisibilityChanged: { observed: 0, matched: 0, unmatched: 0, ambiguous: 0 }, + }, + metadata: { + droppedCallbacks: 0, + droppedAttributionIssues: 0, + evictedSlots: 0, + evictedRequestCycles: 0, + }, + }; +} + +describe('GptDiagnosticsSlotSizeObserver', () => { + afterEach(() => { + ResizeObserverMock.instances = []; + document.body.replaceChildren(); + }); + + it('keeps GPT 1×1 distinct from the observed outer box and updates it on resize', () => { + const element = document.createElement('div'); + document.body.append(element); + const getBoundingClientRect = vi.spyOn(element, 'getBoundingClientRect'); + getBoundingClientRect.mockReturnValue({ width: 728.4, height: 90.5 } as DOMRect); + const requests = [cycle(1, false)]; + requests[0].size = [1, 1]; + const store = { + snapshot: () => snapshot(requests), + recordObservedSlotSize: vi.fn(), + subscribe: () => () => undefined, + }; + const bindings = { + get: () => ({ binding: { status: 'bound' as const }, element, visible: true }), + subscribe: () => () => undefined, + }; + + const observer = new GptDiagnosticsSlotSizeObserver(store, bindings, { + window: { HTMLElement, ResizeObserver: ResizeObserverMock } as unknown as SlotSizeTestWindow, + scheduleFrame: (callback) => callback(), + }); + + expect(store.recordObservedSlotSize).toHaveBeenCalledWith(1, 1, [728, 91]); + expect(requests[0].size).toEqual([1, 1]); + + getBoundingClientRect.mockReturnValue({ width: 969.6, height: 250.2 } as DOMRect); + ResizeObserverMock.instances[ResizeObserverMock.instances.length - 1]!.emit(element); + expect(store.recordObservedSlotSize).toHaveBeenLastCalledWith(1, 1, [970, 250]); + observer.destroy(); + }); + + it('uses the provided window to schedule the initial measurement', () => { + const element = document.createElement('div'); + document.body.append(element); + vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ + width: 300, + height: 250, + } as DOMRect); + const requestAnimationFrame = vi.fn((callback: FrameRequestCallback) => { + callback(0); + return 1; + }); + const store = { + snapshot: () => snapshot([cycle(1, false)]), + recordObservedSlotSize: vi.fn(), + subscribe: () => () => undefined, + }; + const bindings = { + get: () => ({ binding: { status: 'bound' as const }, element, visible: true }), + subscribe: () => () => undefined, + }; + + const observer = new GptDiagnosticsSlotSizeObserver(store, bindings, { + window: { + HTMLElement, + ResizeObserver: ResizeObserverMock, + requestAnimationFrame, + } as unknown as SlotSizeTestWindow, + }); + + expect(requestAnimationFrame).toHaveBeenCalledTimes(1); + expect(store.recordObservedSlotSize).toHaveBeenCalledWith(1, 1, [300, 250]); + observer.destroy(); + }); + + it('records one initial measurement when ResizeObserver is unavailable', () => { + const element = document.createElement('div'); + document.body.append(element); + const getBoundingClientRect = vi.spyOn(element, 'getBoundingClientRect'); + getBoundingClientRect.mockReturnValue({ width: 300, height: 250 } as DOMRect); + const store = { + snapshot: () => snapshot([cycle(1, false)]), + recordObservedSlotSize: vi.fn(), + subscribe: () => () => undefined, + }; + const bindings = { + get: () => ({ binding: { status: 'bound' as const }, element, visible: true }), + subscribe: () => () => undefined, + }; + + const observer = new GptDiagnosticsSlotSizeObserver(store, bindings, { + window: { HTMLElement } as unknown as SlotSizeTestWindow, + scheduleFrame: (callback) => callback(), + }); + + expect(store.recordObservedSlotSize).toHaveBeenCalledWith(1, 1, [300, 250]); + expect(ResizeObserverMock.instances).toHaveLength(0); + getBoundingClientRect.mockReturnValue({ width: 728, height: 90 } as DOMRect); + expect(store.recordObservedSlotSize).toHaveBeenCalledTimes(1); + observer.destroy(); + }); + + it('does not measure after destruction when a frame is pending', () => { + const element = document.createElement('div'); + document.body.append(element); + vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ + width: 300, + height: 250, + } as DOMRect); + const frames: Array<() => void> = []; + const store = { + snapshot: () => snapshot([cycle(1, false)]), + recordObservedSlotSize: vi.fn(), + subscribe: () => () => undefined, + }; + const bindings = { + get: () => ({ binding: { status: 'bound' as const }, element, visible: true }), + subscribe: () => () => undefined, + }; + + const observer = new GptDiagnosticsSlotSizeObserver(store, bindings, { + window: { + HTMLElement, + ResizeObserver: ResizeObserverMock, + } as unknown as SlotSizeTestWindow, + scheduleFrame: (callback) => frames.push(callback), + }); + observer.destroy(); + frames.shift()!(); + + expect(store.recordObservedSlotSize).not.toHaveBeenCalled(); + }); + + it.each(['unbound', 'ambiguous'] as const)('does not observe %s slots', (status) => { + const element = document.createElement('div'); + document.body.append(element); + const store = { + snapshot: () => snapshot([cycle(1, false)]), + recordObservedSlotSize: vi.fn(), + subscribe: () => () => undefined, + }; + const bindings = { + get: () => ({ binding: { status }, element, visible: false }), + subscribe: () => () => undefined, + }; + + const observer = new GptDiagnosticsSlotSizeObserver(store, bindings, { + window: { + HTMLElement, + ResizeObserver: ResizeObserverMock, + } as unknown as SlotSizeTestWindow, + scheduleFrame: (callback) => callback(), + }); + + expect(store.recordObservedSlotSize).not.toHaveBeenCalled(); + expect( + ResizeObserverMock.instances[ResizeObserverMock.instances.length - 1]!.observe + ).not.toHaveBeenCalled(); + observer.destroy(); + }); + + it('cannot apply a delayed prior-cycle measurement to a later refresh', () => { + const element = document.createElement('div'); + document.body.append(element); + vi.spyOn(element, 'getBoundingClientRect').mockReturnValue({ + width: 300, + height: 250, + } as DOMRect); + const requests = [cycle(1, false)]; + const listeners: Array<() => void> = []; + const store = { + snapshot: () => snapshot(requests), + recordObservedSlotSize: vi.fn(), + subscribe: (listener: () => void) => { + listeners.push(listener); + return () => undefined; + }, + }; + const bindings = { + get: () => ({ binding: { status: 'bound' as const }, element, visible: true }), + subscribe: () => () => undefined, + }; + const frames: Array<() => void> = []; + const observer = new GptDiagnosticsSlotSizeObserver(store, bindings, { + window: { + HTMLElement, + ResizeObserver: ResizeObserverMock, + } as unknown as SlotSizeTestWindow, + scheduleFrame: (callback) => frames.push(callback), + }); + const firstObserver = ResizeObserverMock.instances[0]; + + requests.push(cycle(2, false)); + listeners[0](); + frames.shift()!(); + firstObserver.emit(element); + while (frames.length > 0) frames.shift()!(); + + expect(store.recordObservedSlotSize).toHaveBeenCalledWith(1, 1, [300, 250]); + expect(store.recordObservedSlotSize).toHaveBeenCalledWith(1, 2, [300, 250]); + observer.destroy(); + }); +}); diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts index 4c6721f3a..52aef6a7f 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/store.test.ts @@ -7,6 +7,7 @@ import { MAX_CALLBACK_ISSUES, MAX_CREATIVE_ATTEMPTS, MAX_DIAGNOSTIC_SLOTS, + MAX_REQUESTED_SLOT_SIZES, MAX_REQUEST_CYCLES_PER_SLOT, MAX_TRUSTED_SERVER_ASSOCIATIONS, REQUEST_PATH_ATTRIBUTION_WINDOW_MS, @@ -519,6 +520,38 @@ describe('GptDiagnosticsStore', () => { expect(cycle.responseClass).toBe('reservation'); }); + it('retains an observed outer slot box separately from GPT reported size', () => { + const store = new GptDiagnosticsStore({ now: () => 10 }); + const slot = fakeSlot('ad-slot-outer-box'); + + store.recordSlotRequested(slot); + store.recordSlotResponseReceived(slot); + store.recordSlotRenderEnded(slot, { isEmpty: false, size: [1, 1] }); + store.recordObservedSlotSize(1, 1, [728, 90]); + + const cycle = store.snapshot().slots[0].requests[0]; + expect(cycle.size).toEqual([1, 1]); + expect(cycle.observedSlotSize).toEqual([728, 90]); + }); + + it('rejects a stale prior-cycle outer-box measurement after a refresh', () => { + const store = new GptDiagnosticsStore({ now: () => 10 }); + const slot = fakeSlot('ad-slot-stale-outer-box'); + + store.recordSlotRequested(slot); + store.recordSlotResponseReceived(slot); + store.recordSlotRenderEnded(slot, { isEmpty: false }); + store.recordSlotRequested(slot); + store.recordSlotResponseReceived(slot); + store.recordSlotRenderEnded(slot, { isEmpty: false }); + store.recordObservedSlotSize(1, 1, [300, 250]); + store.recordObservedSlotSize(1, 2, [970, 250]); + + const requests = store.snapshot().slots[0].requests; + expect(requests[0].observedSlotSize).toBeUndefined(); + expect(requests[1].observedSlotSize).toEqual([970, 250]); + }); + it('separates a fill without Ad Manager identifiers from a reservation', () => { const store = new GptDiagnosticsStore({ now: () => 10 }); const slot = fakeSlot('ad-slot-default'); @@ -645,6 +678,62 @@ describe('GptDiagnosticsStore', () => { expect(cycles[1].trustedServerOpportunity).toBeUndefined(); }); + it('retains all configured requested slot sizes on only the correlated next request', () => { + const store = new GptDiagnosticsStore({ now: () => 10, defer: () => undefined }); + const slot = fakeSlot('requested-sizes'); + const formats: Array<[number, number]> = [ + [300, 250], + [728, 90], + [320, 50], + ]; + + store.recordTrustedServerOpportunity( + slot, + 'auction-slot', + 'renderable_candidate', + undefined, + formats + ); + formats[0]![0] = 1; + formats.push([970, 250]); + store.recordSlotRequested(slot); + store.recordSlotRequested(slot); + + const cycles = store.snapshot().slots[0]!.requests; + expect(cycles[0]?.requestedSlotSizes).toEqual([ + [300, 250], + [728, 90], + [320, 50], + ]); + expect(cycles[1]?.requestedSlotSizes).toBeUndefined(); + }); + + it('bounds and validates configured requested slot sizes before retaining them', () => { + const store = new GptDiagnosticsStore({ now: () => 10, defer: () => undefined }); + const slot = fakeSlot('validated-requested-sizes'); + const formats: Array<[number, number]> = Array.from( + { length: MAX_REQUESTED_SLOT_SIZES + 2 }, + (_, index) => [index + 1, 250] + ); + formats[0] = [0, 250]; + formats[1] = [300, Number.NaN]; + + store.recordTrustedServerOpportunity( + slot, + 'auction-slot', + 'renderable_candidate', + undefined, + formats + ); + store.recordSlotRequested(slot); + + const requested = store.snapshot().slots[0]!.requests[0]!.requestedSlotSizes; + expect(requested).toHaveLength(MAX_REQUESTED_SLOT_SIZES - 2); + expect(requested).not.toContainEqual([0, 250]); + expect(requested).not.toContainEqual([300, Number.NaN]); + expect(requested).not.toContainEqual([MAX_REQUESTED_SLOT_SIZES + 1, 250]); + }); + it('consumes a combined request intent with independent source facts', () => { let now = 10; const deferred: Array<() => void> = []; diff --git a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/types.test.ts b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/types.test.ts index 6b3103060..f4f4c7486 100644 --- a/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/types.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/gpt_diagnostics/types.test.ts @@ -11,6 +11,7 @@ import type { GptDiagnosticsRequestPath, GptDiagnosticsResponseClass, GptDiagnosticsSlotExport, + Size, TsjsApi, } from '../../../src/core/types'; @@ -138,6 +139,8 @@ describe('GPT diagnostics public types', () => { requestPath: 'publisher_refresh', requestIntentId: 7, trustedServerAuctionId: 'ts-auc-example', + requestedSlotSizes: [[300, 250]], + observedSlotSize: [728, 90], opportunityToRequestMs: 24, replacedRequestNumber: 1, previousRenderToRequestMs: 6048, @@ -174,6 +177,8 @@ describe('GPT diagnostics public types', () => { expectTypeOf(evidenceCycle.requestPath).toEqualTypeOf(); expectTypeOf(evidenceCycle.requestIntentId).toEqualTypeOf(); expectTypeOf(evidenceCycle.trustedServerAuctionId).toEqualTypeOf(); + expectTypeOf(evidenceCycle.requestedSlotSizes).toEqualTypeOf | undefined>(); + expectTypeOf(evidenceCycle.observedSlotSize).toEqualTypeOf(); expectTypeOf(evidenceSnapshot.attributionIssues).toEqualTypeOf< GptDiagnosticsAttributionIssue[] | undefined >(); diff --git a/crates/trusted-server-js/src/bundle.rs b/crates/trusted-server-js/src/bundle.rs index 7ddc060ab..be5aa35cc 100644 --- a/crates/trusted-server-js/src/bundle.rs +++ b/crates/trusted-server-js/src/bundle.rs @@ -1,5 +1,5 @@ use std::collections::HashMap; -use std::sync::OnceLock; +use std::sync::{Mutex, MutexGuard, OnceLock}; use hex::encode; use sha2::{Digest as _, Sha256}; @@ -10,7 +10,7 @@ include!(concat!(env!("OUT_DIR"), "/tsjs_modules.rs")); #[must_use] #[inline] pub fn module_bundle(id: &str) -> Option<&'static str> { - module_map().get(id).copied() + module_meta_map().get(id).map(|module| module.bundle) } /// Return all available module IDs, in discovery order (core first). @@ -27,56 +27,160 @@ pub fn all_module_ids() -> Vec<&'static str> { #[must_use] #[inline] pub fn concatenate_modules(ids: &[&str]) -> String { - let map = module_map(); - let mut parts: Vec<&str> = Vec::new(); + let ordered_ids = concatenated_module_ids(ids); + let mut body = String::new(); + visit_concatenated_module_parts(&ordered_ids, |part| body.push_str(part)); + body +} + +/// SHA-256 hash of the concatenated modules, for cache-busting URLs. +/// +/// The hash is computed over the same byte sequence as [`concatenate_modules`] +/// without allocating that concatenated body. Results are memoized by ordered +/// module ID list for reused processes or isolates. Fastly creates a fresh Wasm +/// instance per request, but still benefits from hashing without materializing +/// the concatenated body. +#[must_use] +#[inline] +pub fn concatenated_hash(ids: &[&str]) -> String { + let key = concatenated_module_ids(ids); + if let Some(hash) = lock_concatenated_hash_cache().get(&key).cloned() { + return hash; + } + + let hash = hash_concatenated_modules(&key); + lock_concatenated_hash_cache().insert(key, hash.clone()); + hash +} + +/// SHA-256 hash of a single module's content (without prepending core). +/// +/// Used for cache-busting URLs of deferred modules served individually. +#[must_use] +#[inline] +pub fn single_module_hash(id: &str) -> Option<&'static str> { + module_meta_map().get(id).map(|module| module.sha256) +} + +fn concatenated_module_ids(ids: &[&str]) -> Vec<&'static str> { + let map = module_meta_map(); + let mut ordered = Vec::new(); - // Core always first if let Some(core) = map.get("core") { - parts.push(core); + ordered.push(core.id); } - // Then requested modules (excluding core, already included) for id in ids { if *id == "core" { continue; } - if let Some(bundle) = map.get(id) { - parts.push(bundle); + if let Some(module) = map.get(*id) { + ordered.push(module.id); } } - parts.join(";\n") + ordered } -/// SHA-256 hash of the concatenated modules, for cache-busting URLs. -#[must_use] -#[inline] -pub fn concatenated_hash(ids: &[&str]) -> String { - let body = concatenate_modules(ids); +fn hash_concatenated_modules(ids: &[&'static str]) -> String { let mut hasher = Sha256::new(); - hasher.update(body.as_bytes()); + visit_concatenated_module_parts(ids, |part| hasher.update(part.as_bytes())); encode(hasher.finalize()) } -/// SHA-256 hash of a single module's content (without prepending core). -/// -/// Used for cache-busting URLs of deferred modules served individually. -#[must_use] -#[inline] -pub fn single_module_hash(id: &str) -> Option { - module_bundle(id).map(|content| { - let mut hasher = Sha256::new(); - hasher.update(content.as_bytes()); - encode(hasher.finalize()) - }) +fn visit_concatenated_module_parts(ids: &[&'static str], mut visit: F) +where + F: FnMut(&'static str), +{ + let map = module_meta_map(); + let mut first = true; + + for id in ids { + let Some(module) = map.get(*id) else { + continue; + }; + if first { + first = false; + } else { + visit(";\n"); + } + visit(module.bundle); + } } -fn module_map() -> &'static HashMap<&'static str, &'static str> { - static MAP: OnceLock> = OnceLock::new(); +fn module_meta_map() -> &'static HashMap<&'static str, &'static TsjsModuleMeta> { + static MAP: OnceLock> = OnceLock::new(); MAP.get_or_init(|| { TSJS_MODULES .iter() - .map(|module| (module.id, module.bundle)) + .map(|module| (module.id, module)) .collect() }) } + +fn lock_concatenated_hash_cache() -> MutexGuard<'static, HashMap, String>> { + match concatenated_hash_cache().lock() { + Ok(guard) => guard, + Err(poisoned) => poisoned.into_inner(), + } +} + +fn concatenated_hash_cache() -> &'static Mutex, String>> { + static CACHE: OnceLock, String>>> = OnceLock::new(); + CACHE.get_or_init(|| Mutex::new(HashMap::new())) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn sha256_hex(bytes: &[u8]) -> String { + encode(Sha256::digest(bytes)) + } + + #[test] + fn generated_single_module_hashes_match_bundle_contents() { + for id in all_module_ids() { + let bundle = module_bundle(id).expect("should have module bundle"); + let generated_hash = single_module_hash(id).expect("should have generated hash"); + + assert_eq!( + generated_hash, + sha256_hex(bundle.as_bytes()), + "generated hash for module {id} should match included bundle bytes" + ); + } + } + + #[test] + fn concatenated_hash_matches_concatenated_bundle_contents() { + let available_ids = all_module_ids(); + let non_core_ids = available_ids + .iter() + .copied() + .filter(|id| *id != "core") + .take(3) + .collect::>(); + + let mut cases: Vec> = vec![Vec::new()]; + if let Some(first) = non_core_ids.first().copied() { + cases.push(vec![first]); + } + if non_core_ids.len() >= 2 { + cases.push(non_core_ids[..2].to_vec()); + cases.push(non_core_ids[..2].iter().rev().copied().collect()); + } + if non_core_ids.len() >= 3 { + cases.push(non_core_ids[..3].to_vec()); + } + + for ids in cases { + let concatenated = concatenate_modules(&ids); + assert_eq!( + concatenated_hash(&ids), + sha256_hex(concatenated.as_bytes()), + "concatenated hash should match concatenated bundle bytes for {ids:?}" + ); + } + } +} diff --git a/docs/guide/api-reference.md b/docs/guide/api-reference.md index 340d5d425..b4cb64481 100644 --- a/docs/guide/api-reference.md +++ b/docs/guide/api-reference.md @@ -7,6 +7,7 @@ Quick reference for all Trusted Server HTTP endpoints. - [First-Party Endpoints](#first-party-endpoints) - Core ad serving and proxying - [Edge Cookie Endpoints](#edge-cookie-endpoints) - Identity sync and enrichment - [Request Signing](#request-signing-endpoints) - Cryptographic signing and key management +- [Admin Diagnostics](#admin-diagnostic-endpoints) - Protected EC troubleshooting - [TSJS Library](#tsjs-library-endpoint) - JavaScript library serving - [Utility Endpoints](#utility-endpoints) - Optional operational helpers - [Integration Endpoints](#integration-endpoints) - Third-party service proxying @@ -580,6 +581,87 @@ curl -X POST https://edge.example.com/_ts/admin/keys/deactivate \ --- +## Admin Diagnostic Endpoints + +These endpoints expose sensitive identity and cookie data and require HTTP Basic Authentication. Configure a handler that covers the entire `/_ts/admin` namespace; startup rejects configurations that do not protect every admin route, including handlers that match only some `/_ts/admin/ec/{id}` values — the dynamic route needs a prefix-level matcher such as `^/_ts/admin` or `^/_ts/admin/ec/`. The whole `/_ts/admin` prefix is reserved: any admin path that reaches publisher fallback — unknown, malformed, or percent-encoded (`/_ts/admin%2Fec`) — is answered locally with `404` and is never proxied, so an admin `Authorization` header and request body never reach the publisher origin. The retired non-`/_ts` `/admin/keys` aliases are reserved the same way. Normal diagnostic-handler responses after successful authentication are JSON with `Cache-Control: no-store`. Missing or invalid credentials receive the shared plaintext `401 Unauthorized` Basic-auth challenge. Unexpected configuration or KV failures use the adapter's shared plaintext `5xx` error response. Those authentication and internal-error responses are outside the diagnostic JSON and cache-header contract. + +The examples below use fictional IDs and values only. + +### GET /\_ts/admin/ec + +### GET /\_ts/admin/ec/`{id}` + +Reads an EC identity-graph record for troubleshooting. The explicit route accepts an EC ID in `{64 lowercase hex}.{6 alphanumeric}` format. The bare route uses the request's `ts-ec` cookie. + +This lookup is implemented only by the Fastly adapter because the identity graph is stored in Fastly KV. Other adapters return `501 Not Implemented`. + +**Response fields:** + +- `ec_id`, `store`, and `generation` identify the raw KV lookup. +- `entry` preserves the stored JSON shape, including unknown and legacy fields. Derived `created_iso` and `consent.updated_iso` fields are added only when absent. +- `metadata` preserves the stored metadata JSON shape. +- `tombstone` reports whether consent has been withdrawn. It is absent when the entry body cannot be parsed as JSON or deserialized as the typed EC schema. +- `auction.eids` previews the partner EIDs the stored record can contribute; `auction.skipped` explains filtered IDs. +- `entry_error`, `metadata_error`, and `raw_body` keep malformed or schema-incompatible records inspectable. + +The auction preview validates the stored record and partner configuration, but cannot reproduce live per-request consent checks. It must not be treated as proof that a specific auction request will receive those EIDs. + +**Status codes:** + +| Status | Meaning | +| ------ | ----------------------------------------------------------- | +| `200` | Record found, including inspectable corrupt records | +| `400` | Invalid explicit EC ID | +| `401` | Missing or invalid Basic credentials | +| `404` | Record not found, or the bare route has no `ts-ec` cookie | +| `405` | Method other than `GET` (`Allow: GET`) | +| `501` | EC identity graph unavailable on this adapter or deployment | +| `5xx` | Unexpected configuration or KV failure (plaintext) | + +```bash +curl -u admin:secure-password \ + "https://edge.example.com/_ts/admin/ec/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.abc123" + +curl -u admin:secure-password \ + --cookie "ts-ec=aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.abc123" \ + "https://edge.example.com/_ts/admin/ec" +``` + +### GET /\_ts/admin/eids + +Parses the request's `ts-eids` and `sharedId` cookies and previews which configured partner IDs cookie ingestion would match or drop. It performs request inspection only: it does not read or write KV and is available on every adapter. + +After successful authentication this endpoint always returns `200 OK`; missing or malformed cookies are represented by `cookie_present`, `sharedid_present`, and `parse_error`. The `ingest.matched` and `ingest.unmatched` arrays show the ingestion preview. Each unmatched entry contains its `source` and either a `no_partner` reason when no configured partner recognizes it or `no_valid_uid` when the partner exists but every supplied UID is empty or exceeds the storage limit. + +```json +{ + "ingest": { + "matched": [ + { + "source_domain": "configured.example", + "uid": "fictional-uid" + } + ], + "unmatched": [ + { + "source": "unknown.example", + "reason": "no_partner" + } + ] + } +} +``` + +```bash +curl -u admin:secure-password \ + --cookie "sharedId=fictional-shared-id" \ + "https://edge.example.com/_ts/admin/eids" +``` + +Malformed diagnostic paths return a local `404`, and unsupported methods return a local `405`; they are never forwarded to the publisher origin. + +--- + ## TSJS Library Endpoint ### GET /static/tsjs=`` @@ -768,6 +850,9 @@ curl -u admin:secure-password https://edge.example.com/_ts/admin/keys/rotate - `/_ts/admin/keys/rotate` - `/_ts/admin/keys/deactivate` +- `/_ts/admin/ec` +- `/_ts/admin/ec/{id}` +- `/_ts/admin/eids` - Any paths matching configured `handlers` patterns --- diff --git a/docs/guide/architecture.md b/docs/guide/architecture.md index 3b20000ec..da1a58bcd 100644 --- a/docs/guide/architecture.md +++ b/docs/guide/architecture.md @@ -53,13 +53,13 @@ Native Axum dev/test adapter (native binary): **Current limitations compared to the Fastly adapter:** -| Feature | Axum dev server | -| ------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | -| KV store | Unavailable — synthetic-ID and consent routes degrade gracefully | -| Geo lookup | Always returns `None` | -| Config/secret-store writes | Return an error (read-only via env vars) | -| Admin key management (`/_ts/admin/keys/*`) | Returns 501 Not Implemented. Legacy `/admin/keys/*` aliases are denied locally with 404 and are not proxied to the publisher fallback | -| Auction fan-out ordering | Requests run concurrently via `tokio::spawn`; `select` returns first-to-complete but does not replicate Fastly's priority-queue tie-breaking | +| Feature | Axum dev server | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| KV store | Unavailable — synthetic-ID and consent routes degrade gracefully | +| Geo lookup | Always returns `None` | +| Config/secret-store writes | Return an error (read-only via env vars) | +| Admin key management (`/_ts/admin/keys/*`) | Returns 501 Not Implemented. Retired `/admin/keys` aliases, including trailing, descendant, and percent-encoded forms, are denied locally with 404 and are not proxied to the publisher fallback | +| Auction fan-out ordering | Requests run concurrently via `tokio::spawn`; `select` returns first-to-complete but does not replicate Fastly's priority-queue tie-breaking | ### trusted-server-adapter-spin diff --git a/docs/guide/auction-orchestration.md b/docs/guide/auction-orchestration.md index b4ba6b797..a47dbf155 100644 --- a/docs/guide/auction-orchestration.md +++ b/docs/guide/auction-orchestration.md @@ -823,3 +823,78 @@ Every auction response includes structured metadata in `ext.orchestrator`: "time_ms": 145 } ``` + +### SSAT HTML Debug Comment + +For local server-side auction template (SSAT) investigation, Trusted Server can +insert a `` comment before the page's bids script. Enable +it in `trusted-server.toml`, push the local configuration, restart the local +server, and search the page source for `ts-debug`: + +```toml +[debug] +auction_html_comment = true + +[debug.auction_html_comment_options] +include_provider_responses = true +include_mediator_response = false +include_bids = false +verbosity = "full" +format = "pretty" +``` + +```bash +ts config validate +ts config push --adapter fastly --local +fastly compute serve +``` + +This example is useful when investigating raw Prebid Server requests and +responses without spending the dump budget on winning creatives. Raw PBS +`debug.httpcalls` and `resolvedrequest` metadata also require +`debug = true` under `[integrations.prebid]`. + +| Option | Default | Behavior | +| ---------------------------- | -------------------------------------- | ---------------------------------------------------------------------------------------------- | +| `include_provider_responses` | `true` | Include the provider response array | +| `include_mediator_response` | `true` | Include the mediator response when a mediator ran | +| `include_bids` | `true` | Include bid objects; when `false`, provider status and metadata remain | +| `metadata_keys` | `error_type`, `http_status`, `message` | Subset of the fixed validated keys; gates them in `redacted` and `upstream`, ignored in `full` | +| `verbosity` | `redacted` | Select `redacted`, `upstream`, or `full` sensitivity | +| `format` | `compact` | Use compact outer JSON or indented outer JSON with `pretty` | + +`metadata_keys` is a subset selector against a fixed allowlist — +`error_type`, `http_status`, and `message` — never a way to add keys. Any other +entry fails config load rather than being silently ignored. + +The verbosity modes form an explicit sensitivity ladder: + +- `redacted` reconstructs only validated `error_type`, `http_status`, and a + server-generated `message`, intersected with `metadata_keys`. A successful + provider response can therefore have `metadata: {}`. +- `upstream` adds provider-controlled errors, warnings, response timings, bid + statuses, and bounded upstream-message fields. It builds on the redacted + metadata, so `metadata_keys` still gates the three validated keys, while the + provider diagnostics are unlocked by `verbosity` alone. It does not include + raw PBS `httpcalls` or `resolvedrequest`. +- `full` includes raw response metadata and untruncated creatives, ignoring + `metadata_keys` entirely. It can expose IP addresses, geo data, identifiers, + consent strings, request signatures, and complete provider request/response + bodies. + +`format = "pretty"` indents only the outer dump. JSON-looking fields such as +`requestbody` and `responsebody` remain strings exactly as captured, so their +contents still appear escaped. Use a local JSON inspection tool when those +nested values need additional formatting. + +The summary line's `winning=N` count is computed before section filtering, so +it can be nonzero while `include_bids = false` produces empty bid arrays. Every +mode and format neutralizes HTML-comment terminators and enforces a 256 KiB +total dump cap. A capped dump ends with `…(truncated N bytes)` and is no longer +valid JSON. + +::: danger Local debugging only +Do not enable the auction HTML comment in production. Even `redacted` can +contain bid-level data and creative previews, while `upstream` and `full` may +expose identity-bearing request data to anyone who can view the page source. +::: diff --git a/docs/guide/cli.md b/docs/guide/cli.md index b6829895e..14ca2af31 100644 --- a/docs/guide/cli.md +++ b/docs/guide/cli.md @@ -76,6 +76,44 @@ Trusted Server settings JSON. This blob model is intentional because full Trusted Server configs can exceed Fastly limits when split into one config-store entry per setting. +### Diagnose ad-template configuration + +The static `ts config ad-templates` commands evaluate local configuration +without launching a browser: + +| Command | Purpose | +| ---------------------------------------- | ------------------------------------------------------------------------- | +| `lint` | Summarize configuration and report invalid slot page patterns. | +| `match [--details]` | List matching slots; `--details` includes divs, paths, formats/providers. | +| `check --expected-slot ID` | Assert the exact matching slot set; repeat `--expected-slot`. | +| `check --expect-no-slots` | Assert that no slots match. | +| `explain ` | Print every runtime ad-stack gate and its final yes/no verdict. | + +`check --allow-extra-slots` permits matches beyond the repeated +`--expected-slot` values. It conflicts with `--expect-no-slots`. + +`explain` models a GET navigation with consent allowed by default. Use +`--method `, `--non-navigation`, `--prefetch`, `--bot`, or +`--consent-denied` to model another request. Provider configuration is printed +as a separate advisory; it does not change the runtime gate verdict. + +Every `ts config ad-templates ...` and `ts audit ad-templates ...` command +accepts the same config-location flags: + +| Flag | Behavior | +| --------------------- | ----------------------------------------------------------------------------- | +| `--app-config ` | Read this app config instead of deriving `.toml` from the manifest. | +| `--manifest ` | Read this manifest; defaults to `edgezero.toml`. | +| `--no-env` | Disable `TRUSTED_SERVER__...` overlays for read-only commands. | + +The mutating audit generator always edits file-backed values and never writes +environment-only overlays into TOML, including during `--dry-run`. + +For CI-oriented assertions, exit code 0 means the assertion passed, 1 means the +command ran and found drift (`config ad-templates check` or audit verification +with `--strict`), and 2 means argument parsing, configuration, browser launch, +or another tool operation failed. + ## Lifecycle commands Lifecycle commands delegate to the selected EdgeZero adapter: @@ -98,7 +136,7 @@ Chrome or Chromium must be installed locally. The command checks common PATH names and standard macOS/Linux install locations. ```bash -ts audit https://publisher.example +ts audit generate https://publisher.example ``` By default, the command writes: @@ -118,13 +156,13 @@ ts config validate If a config already exists, avoid overwriting it: ```bash -ts audit https://publisher.example --no-config +ts audit generate https://publisher.example --no-config ``` Use custom output paths when reviewing artifacts first: ```bash -ts audit https://publisher.example \ +ts audit generate https://publisher.example \ --js-assets audit/js-assets.toml \ --config audit/trusted-server.toml ``` @@ -132,9 +170,277 @@ ts audit https://publisher.example \ Use `--force` only when replacing existing output files is intentional: ```bash -ts audit https://publisher.example --force +ts audit generate https://publisher.example --force +``` + +The legacy `ts audit ` form remains a compatibility alias for artifact +generation. New automation should use `ts audit generate `. + +## Generate ad-template slots from a live site + +`ts audit ad-templates generate ` discovers the publisher's ad slots and +rewrites the `[creative_opportunities]` slot array in `trusted-server.toml` in +place, preserving every other section and comment. + +```bash +ts audit ad-templates generate https://publisher.example/ +``` + +It samples the site rather than a single page. Ad slots repeat per site +section, so the crawl is sized by the publisher's taxonomy — a dozen sections — +not its catalogue: + +1. Load the requested page and read its links and, from `robots.txt`, its + sitemap. +2. Group both into candidate sections, keeping one landing page and one article + per section. +3. Load those pages, recording each slot's div, sizes, and GAM ad-unit path. +4. Reconcile every slot across the pages it appeared on. +5. Infer a `{section}` ad-unit template if the evidence proves one. +6. Verify the result loads, then write it. + +### What it writes + +Given a site whose ad units track the section, the run produces: + +```toml +[creative_opportunities] +gam_network_id = "99999" +section_root = "homepage" +section_segment = 0 + +[[creative_opportunities.slot]] +id = "ad-header-0" +div_id = "ad-header-0" +gam_unit_path = "/{network_id}/example/{section}" +page_patterns = ["/", "/deals", "/deals/*", "/news", "/news/*"] +formats = [{ width = 728, height = 90 }] +``` + +Each section contributes **two** patterns. `*` crosses `/` in this glob +dialect, so `/news/*` matches `/news/a/b` but not the bare `/news` landing +page; emitting only the star form would drop the landing page from the slot. + +Sizes are unioned across pages, so a format that renders only on articles +survives alongside the homepage's. + +### When it keeps literal paths, and when it refuses + +A wrong ad-unit template makes the publisher bid against inventory that does not +exist, so the command prefers a narrow literal path over a plausible guess. + +| Situation | Result | +| --------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Only one page was crawled | Literal path. One observation cannot distinguish a literal from a template. | +| The ad unit never varied by section | Literal path. | +| A section's slug is not derivable from its URL (`/site-news` requesting `.../sitenews`) | The slot is omitted; the note lists the ad-unit paths it used and says none generalized. | +| No crawled page lacked a section segment, so `section_root` is unwitnessed | No template is written and the reason names the crawl gap. A slot that merely never appears on the root (a sidebar, an in-article unit) still templates, borrowing the `section_root` another slot witnessed; a note says so. | +| Two path segments could both be the section | No template; the ambiguity is reported. | +| The ad unit varies by device, geo, or anything the URL cannot supply | The refused slot is omitted and the reason is written as a note. | +| Crawled pages report different GAM network ids | The run fails; the pages are not one property. | +| More than a quarter of crawled pages return no slots | The run fails. That is the signature of bot protection serving challenge pages, and writing from it would silently narrow the slot set. | +| Several live elements normalize onto one div-id prefix | The whole group is omitted, on every page of the crawl. A prefix resolves to at most one element and the exact ids change per render; the prefix is named in a note. | +| A per-render token sits before the placement part of a div id | The slot is omitted from a single observation and the family prefix is named in a note; no stable prefix identifies one element. | + +Every run checks that the config it produced still loads before replacing the +file, and `--dry-run` runs the same check — a clean preview is evidence the +config loads, not just that it parses. Dry-run stdout is a zero-context unified +diff containing only the managed creative-opportunity fields; notes and refusal +reasons go to stderr, so unrelated config and secrets are not printed. Crawl +progress also goes to stderr, one line per phase and page — for example +`Auditing desktop [2/17]: /news`. Progress renders the path only, never the +origin, userinfo, query, or fragment, and there is no flag to suppress it. A +`--dry-run` that changes nothing says so on stderr too, leaving stdout an empty +diff. + +### Bounding and steering the crawl + +```bash +# Cover more of a large site. +ts audit ad-templates generate https://publisher.example/ --max-sections 20 --max-pages 41 + +# Audit exactly one page, as earlier releases did. +ts audit ad-templates generate https://publisher.example/ --max-pages 1 + +# Set the patterns yourself; this disables pattern inference entirely. +ts audit ad-templates generate https://publisher.example/ \ + --page-pattern '/' --page-pattern '/news' --page-pattern '/news/*' + +# Preview without writing. +ts audit ad-templates generate https://publisher.example/ --dry-run ``` +Re-running merges into the existing slots: a slot seen again keeps its +hand-tuned fields and gains this run's patterns and newly observed formats, and a hand-written +`gam_unit_path` template is preserved. `--replace` discards existing slots +instead, which also discards any template you wrote by hand. + +A merge refuses to change the section policy that preserved `{section}` slots +were written against: if the config already sets `section_root` (or +`section_segment`) and this run infers different values, the run fails and asks +for `--replace` as an explicit migration. A config whose `{section}` slots have +no `section_root` at all is a different case — the runtime rejects such a file +outright — so the first merge adopts the inferred policy and makes it loadable +instead of demanding `--replace`. + +Locale-prefixed sites are inferred at their observed section depth. Only real +ISO 639-1 language codes are read as a locale prefix, so a two-letter _section_ +root such as `/tv` or `/us` keeps sections at the first segment. For +example, `/en/news/story` can produce `section_segment = 1`; generated patterns +retain the locale prefix (`/en/news` and `/en/news/*`). The crawler never +invents an unwitnessed locale or section. + +Behind bot protection, pass a valid clearance cookie. The crawl reuses one +browser session, so clearance earned on the first page carries to the rest, and +`--page-delay-ms` spaces the requests — an unpaced crawl is both discourteous to +the origin and likelier to be challenged partway through: + +```bash +ts audit ad-templates generate https://publisher.example/ \ + --cookie '=' --page-delay-ms 1500 +``` + +Some origins refuse a headless browser outright regardless of the cookie. +`--headful` runs a visible one, which is also the quickest way to _see_ whether +a challenge is being shown: + +```bash +ts audit ad-templates generate https://publisher.example/ --headful +``` + +### Sites behind a consent platform + +Publishers gate slot definition behind their consent platform, and the audit +runs in a throwaway browser profile with no consent cookie. Left alone, such a +site defines no slots at all and looks identical to a site with no ad stack. + +The crawl therefore answers the two IAB interfaces every compliant platform +exposes — TCF v2 and US Privacy — as a consenting, out-of-scope reader, before +any page script runs. This changes only what the audit browser sees; it does not +affect the publisher's own readers. Pass `--no-assume-consent` to observe the +un-consented page instead. + +When a page still yields no slots, the run reports GPT's observable state — +whether the library reached `apiReady`, how many queued commands never drained, +how many scripts ran. An empty slot registry has several very different causes, +and that line distinguishes them. + +### Auditing a production hostname served locally + +`ts dev proxy` serves a production hostname from a local Trusted Server. +Auditing through it keeps the page's origin, cookie scope, and any origin checks +in the ad stack matching production rather than `localhost`: + +```bash +ts dev proxy --map www.publisher.example=127.0.0.1:7676 --upstream-plaintext --rewrite-host + +ts audit ad-templates generate https://www.publisher.example/ \ + --browser-proxy 127.0.0.1:18080 --danger-accept-invalid-certs +``` + +`--danger-accept-invalid-certs` covers the proxy's MITM certificate when the +throwaway browser profile does not trust its CA; installing that CA +(`ts dev proxy ca`) is preferable. Against a real origin the flag is dangerous — +the audit sends any `--cookie` session upstream and treats the response as +evidence, so an invalid certificate could mean an impersonator is both +harvesting the session and fabricating the result. + +Note that a local Trusted Server injects its own configured slots into the page, +so a run through the proxy can rediscover config it already has. Slot ids that +are absent from the current config are the publisher's own. + +### Slots that change div id on every render + +Some ad stacks build div ids from a per-render token, so one placement arrives +under a new id on every page. Those ids match nothing at runtime, so the run +declines to write them and reports the group instead: + +```text +note: skipped 3 slot(s) that look like one placement under a per-render div id + on `/123456789/publisher/overlay` (ex_slot_a1_overlay_1, …); + they share the prefix `ex_slot`. Add it once by hand with a div_id prefix + that is stable across renders +``` + +The detection is by evidence, not by recognising token shapes: candidates share +an ad-unit path and formats, and what separates a fragmented placement from two +legitimate siblings on one unit is co-occurrence — real siblings appear together +on a page, fragments never do. The suggested prefix is a starting point only, not +written as a `div_id`, because it reaches only as far as the observed tokens +happen to agree. + +### Checking for a device split + +Publishers often serve a different ad unit per device +(`/network/desktop/news` against `/network/mobile/news`). A desktop-only crawl +cannot see that — it infers a template correct for desktop and silently wrong +for every mobile impression. + +```bash +ts audit ad-templates generate https://publisher.example/ --profiles desktop,mobile +``` + +Each page is loaded once per profile. Where the profiles disagree, the slot is +omitted and the diagnostic explains the conflicting paths. The generator does +not fall back to a fabricated default ad unit. + +### Deploy ordering for templated config + +> **A config containing `section_root` or `section_segment` is not +> rollback-safe.** These keys are rejected outright by a Trusted Server binary +> that predates ad-unit templating, and the rejection fails the _entire_ +> configuration load — not just the ad-template section — so every route serves +> an error. This is a full-site outage, not a degraded ad stack. + +When a run reports that it wrote a `{section}` template: + +1. Deploy the template-aware binary **first**. +2. Then `ts config push`. +3. Do **not** roll that binary back while the config is live. + +A run that did not template writes neither key, and leaves the config exactly as +rollback-safe as it was. + +### Audit safety defaults + +Every `ts audit` browser session validates TLS certificates. This matters +because `--cookie` sends a real session to the origin and the page's own +response becomes the audit's evidence, so a certificate-invalid host could both +harvest the session and fabricate what the audit reports. Override only for a +host you control with a known self-signed certificate: + +```bash +ts audit page https://staging.publisher.example --danger-accept-invalid-certs +``` + +`ts audit ad-templates verify` matches configured slots against the +**post-redirect** path, so it refuses a redirect that leaves the requested +origin rather than accepting another site's evidence as verification. Allow it +for a known redirect between your own properties (for example apex to `www`): + +```bash +ts audit ad-templates verify https://publisher.example/ --allow-cross-origin-redirect +``` + +Verification accepts multiple URLs and reuses one browser/profile. Add +`--strict` to return exit 1 when a confirmable slot is missing or partially +confirmed, and `--json` for the stable machine-readable report. Video- and +native-only slots are reported as `unconfirmable`; that records a checker +limitation and does not fail strict mode. A live out-of-page slot with no sizes +against banner-configured formats is reported `partial` and does fail strict +mode. `--scroll` enables the optional second evidence phase and labels evidence +first seen after the deterministic scroll. + +Browser-backed ad-template generation and verification share `--chrome`, +`--headful`, `--browser-proxy`, `--no-assume-consent`, +`--settle-quiet-ms`, `--settle-max-ms`, and +`--danger-accept-invalid-certs`. Verification also accepts +`--browser-profile desktop|mobile`; generation uses +`--profiles desktop,mobile` to compare both profiles. `--cookie NAME=VALUE` is +repeatable and creates host-only, root-path cookies; HTTPS targets also mark +them Secure. Verification refuses cookies when URLs span multiple origins. The quiet settle window +must not exceed the maximum. + `ts audit` is not an EdgeZero adapter command. It has no `--adapter` option and it does not provision resources, push config, build, deploy, or contact platform APIs. diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index ddb6544ce..44f038f96 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -72,6 +72,7 @@ fail and the service will return its startup-error response. | `[ec]` | Edge Cookie (EC) ID generation | | `[tester_cookie]` | Optional tester-cookie endpoint | | `[proxy]` | Proxy SSRF allowlist and asset routes | +| `[cache]` | Static/rehosted asset cache policy rules | | `[image_optimizer]` | Reusable Image Optimizer profile sets | | `[request_signing]` | Ed25519 request signing | | `[auction]` | Auction orchestration | @@ -645,6 +646,23 @@ path = "^/api/v[0-9]+/private" # /api/v1/private, /api/v2/private **Validation**: Application startup fails if regex is invalid. +::: warning Admin coverage and passwords are validated at startup + +Startup fails when no handler covers an admin route. The dynamic +`/_ts/admin/ec/{id}` route accepts any segment after `/_ts/admin/ec/`, and +Basic Auth runs on the raw path before routing, so coverage cannot be inferred +from ID-shaped samples: a pattern such as +`^/_ts/admin/ec/[a-f0-9]{64}[.][A-Za-z0-9]{6}$` is rejected. Use a prefix-level +matcher (`^/_ts/admin`, or `^/_ts/admin/ec/` alongside the other admin +patterns). + +Startup also fails when any handler — admin or not — uses a placeholder or +well-known weak password (`changeme`, `password`, `admin`, or a +`replace-with-…` template value). Handler selection is first-match-wins, so a +narrow handler ahead of the admin pattern governs the paths it matches. + +::: + ::: warning Scope patterns to the paths you mean Handler patterns are matched against the full request path, so a broad pattern @@ -1031,6 +1049,125 @@ when_missing = "smart" See [Asset Routes](/guide/asset-routes) for request flow, S3 auth details, and Image Optimizer behavior. +## Cache Configuration + +Static and rehosted asset cache upgrades are operator-controlled. By default, +Trusted Server leaves arbitrary publisher-origin assets under origin cache +control. Add `[[cache.asset_rules]]` entries only for paths that are known to be +content-addressed or otherwise safe for the configured TTL. + +### `[[cache.asset_rules]]` + +Rules are evaluated in file order; the first enabled matching rule wins. +Disabled rules never match, and their matcher and policy validation is deferred +until they are enabled. Rule IDs are always normalized and must remain nonempty +and unique, including for disabled placeholders. + +| Field | Type | Required | Description | +| -------------------------------- | ------------- | -------- | ---------------------------------------------------------------------------------- | +| `id` | String | Yes | Unique operator-facing rule identifier | +| `enabled` | Boolean | No | Whether the rule participates in matching (default `false`) | +| `preset` | String | Matcher | Built-in preset such as `nextjs-static` | +| `path_prefix` | String | Matcher | Request path prefix | +| `path_glob` | String | Matcher | Single glob matched against the request path | +| `path_globs` | Array[String] | Matcher | Multiple globs matched against the request path | +| `path_regex` | String | Matcher | Regex matched against the request path | +| `extensions` | Array[String] | Matcher | Case-insensitive file extensions | +| `fingerprint_style` | String | No | Required bundler fingerprint convention before matching | +| `visibility` | String | No | `public` or `private` (default `public`) | +| `browser_ttl_seconds` | Integer | Policy | Browser `max-age`; required for private rules and positive with `immutable = true` | +| `edge_ttl_seconds` | Integer | Policy | Public rules only: TTL emitted through the runtime-specific shared-cache directive | +| `stale_while_revalidate_seconds` | Integer | No | Optional `stale-while-revalidate` | +| `stale_if_error_seconds` | Integer | No | Optional `stale-if-error` | +| `immutable` | Boolean | No | Add `immutable` for a validated content-addressed rule | + +An enabled rule must configure exactly one matcher. Public rules must configure +at least one of `browser_ttl_seconds` or `edge_ttl_seconds`; private rules must +configure `browser_ttl_seconds` and must not configure `edge_ttl_seconds`. +`path_glob` and `path_globs` are mutually exclusive. `immutable = true` +additionally requires a positive browser TTL and either the content-addressed +`nextjs-static` preset, `hex`, or `esbuild-base32`. + +The filename fingerprint check examines the suffix immediately before the final +extension and requires a nonempty filename prefix separated by `.`, `-`, `_`, +or `~`. The accepted immutable conventions are: + +- `hex`: hexadecimal suffixes of at least eight characters containing a letter, + such as `app.0123abcd.js`; +- `esbuild-base32`: eight-character uppercase Base32 suffixes, such as + `app-VRTVD5R5.js`. + +`vite-base64-url` remains available for non-immutable cache rules, but it cannot +prove content addressing. Ordinary names such as `hero-Portrait.jpg` can match +its eight-character Base64URL shape. A matching rule whose selected fingerprint +style fails emits a debug log with the rule ID and rejected path. + +Glob patterns are case-sensitive. `*` matches within a single path component, +while `**` matches recursively: `/assets/*.js` matches `/assets/app.js` but not +`/assets/vendor/app.js`; `/assets/**/*.js` matches both. + +**Next.js preset example** (disabled until the publisher confirms +`/_next/static/` is content-addressed): + +```toml +[[cache.asset_rules]] +id = "nextjs-static" +enabled = false +preset = "nextjs-static" +visibility = "public" +browser_ttl_seconds = 31536000 +edge_ttl_seconds = 31536000 +immutable = true +``` + +**Publisher allowlist example** (enable only for an unambiguous immutable +filename convention): + +```toml +[[cache.asset_rules]] +id = "publisher-fingerprinted-assets" +enabled = false +path_globs = [ + "/assets/**/*.js", + "/assets/**/*.css", + "/assets/**/*.png", + "/assets/**/*.webp", +] +fingerprint_style = "hex" +visibility = "public" +browser_ttl_seconds = 31536000 +edge_ttl_seconds = 31536000 +immutable = true +``` + +If `[cache]` is omitted or no enabled rule matches, Trusted Server preserves the +origin cache policy for publisher-origin assets. On the publisher pass-through +path, an origin `private` or `no-store` directive vetoes a matching rule. Other +origin cache directives, including `no-cache`, are replaced by the configured +policy. `Vary` is preserved, so do not assign a public immutable rule to paths +that vary by cookies or other user-specific request state. + +On a configured Fastly asset-rehost route, a matching rule is authoritative +over the third-party origin's cache defaults, including `no-store`, because +Trusted Server owns the rehosted copy. A later Trusted Server or operator-applied +`private` or `no-store` directive still vetoes public policy reapplication and +removes shared-cache headers. + +TS-owned validated hash URLs such as `/static/tsjs=...js?v=` use their +built-in cache policy and do not require an asset rule. Shared-cache keys for +`/static/tsjs=` must preserve `v`; otherwise a matching immutable response can +collide with the missing or mismatched version's short-TTL response. + +`edge_ttl_seconds` only emits the selected runtime's shared-cache directive for +public rules. The runtime or service must also enable and consume that +directive. The checked-in Cloudflare manifests intentionally do not enable +Workers Cache: the Worker serves the full publisher gateway, not an isolated +static-only entrypoint. Emitting `Cloudflare-CDN-Cache-Control` alone must not +be treated as permission to cache every response. Any future Workers Cache +opt-in must isolate or explicitly allowlist cacheable traffic. Fastly synthetic +and final egress responses still require explicit runtime cache integration, +tracked in [#908](https://github.com/IABTechLab/trusted-server/issues/908). + ## Integration Configurations Settings for built-in integrations (Prebid, Next.js, Osano, Permutive, Testlight). For other @@ -1347,8 +1484,31 @@ Defines the ad slots the trusted server offers on a page: which pages each slot appears on (`page_patterns`), its supported sizes (`formats`), and the GAM ad unit it maps to (`gam_unit_path`). +`enabled` is the dedicated server-side ad-template switch. It defaults to `true` +for compatibility with existing configurations. Set it to `false` to stop +publisher HTML and SPA page-bids template delivery while retaining the slot +configuration and direct `POST /auction` endpoint. + +#### Publisher document cache policy + +For a successful GET publisher document, Trusted Server applies the +browser-facing `Cache-Control: max-age=60` policy from +[#1007](https://github.com/IABTechLab/trusted-server/issues/1007) when the +server-side ad stack is structurally inactive. This includes an absent +`[creative_opportunities]` section, `enabled = false`, no slot matching the +path, or a disabled auction. The policy replaces the origin browser cache +policy except when the origin sends `private` or `no-store`, which are +preserved. Bot, prefetch, and consent-denied requests also retain the origin +policy because they can produce a request-specific representation for the same +URL. Error responses and non-document requests retain the origin policy. + +Any response that carries `Set-Cookie` is finalized as +`Cache-Control: private, max-age=0`; this privacy rule takes precedence over +the short inactive-stack policy. + ```toml [creative_opportunities] +enabled = true # set to false to disable server-side ad templates gam_network_id = "123456789" price_granularity = "dense" @@ -1367,6 +1527,146 @@ page_patterns = ["/", "/news", "/news/*", "/reviews", "/reviews/*"] formats = [{ width = 728, height = 90 }] ``` +The same switch can be overridden through the typed CLI environment overlay. +Because EdgeZero only replaces TOML leaves that already exist, first add +`enabled = true` to the `[creative_opportunities]` block in the base config +before using this override. See [Environment Variable Overrides (Typed +CLI)](#environment-variable-overrides-typed-cli) for the general overlay rules. + +```bash +TRUSTED_SERVER__CREATIVE_OPPORTUNITIES__ENABLED=false +``` + +### Shared template assembly (`assembly_mode = "esi"`) + +### Shared template assembly (`assembly_mode = "esi"`) + +This configuration is an experimental validation spike scoped to +[IABTechLab/trusted-server#1009](https://github.com/IABTechLab/trusted-server/issues/1009), +not a settled production cache interface. + +`assembly_mode` controls how initial-page slot and bid state is delivered: + +- `inline` (default) transforms every origin response and injects the current + reader's slots and bids directly. +- `esi` opts into a reader-neutral transformed-template cache on Fastly. The + cache stores identity bytes containing one inert, versioned comment. On an + authorized cold miss, Fastly replaces that comment in a private working copy + with one synthetic ESI include and resolves it from the already-built reader + state using the pinned `stackpop/esi` parser. No HTTP fragment request occurs. + Warm hits use an exact byte split instead, preserving the fast article-prefix + stream while the auction finishes. + +This is deliberately not general publisher-controlled ESI. A transformed origin +document containing any `` head marker selecting the injected friendly-frame runner below. The marker works under a publisher CSP that blocks inline scripts. Unknown values fail configuration deserialization. + +### Publisher-native runner experiment + +`publisher_native` is an opt-in browser experiment, **not** general APS compatibility proof. No public `apstag` API was found that accepts an externally selected OpenRTB `aaxResponse`. In controlled browser testing, `apstag.renderImp(document, bidId)` did not render the Trusted Server bid because that bid was absent from the SDK's browser-auction state. Trusted Server therefore does not call `apstag`, `fetchBids`, or `setDisplayBids`, mutate the publisher's APS SDK, or start a second auction. Instead, this mode reuses the same `prebid/creative/render` runner contract already used by `trusted_server` mode, but inside a publisher-origin frame; that observed vendor contract still requires APS account-team validation. + +No publisher JavaScript change is required. After validating and freezing the exact selected descriptor, Trusted Server JS: + +1. resolves the direct-auction slot or its injected GAM div mapping; +2. creates a hidden, publisher-origin friendly iframe sized to the winner; +3. initializes only that fresh frame's account-scoped `_aps` event queue; +4. queues `prebid/creative/render` with the selected `aaxResponse` and bid ID; and +5. loads the fixed `https://client.aps.amazon-adsystem.com/prebid-creative.js` runner. + +The existing publisher content remains visible until the runner script loads. A runner error, a blocked script, a missing slot, a superseding dispatch, or a load taking longer than 10 seconds removes the pending frame and visibly declines the bid. It never falls back to `/integrations/aps/renderer` or sends a Universal Creative renderer response. Trusted Server treats runner load as successful handoff; the runner owns subsequent creative completion and resource loading. + +Unlike `trusted_server` mode, this friendly frame deliberately has no opaque-origin sandbox. The fixed APS runner and its creative execute with the behavior of a publisher-origin integration, so `publisher_native` has a larger security surface—especially when `allow_script_creatives = true`. Use only a controlled cohort, and ensure publisher CSP permits the APS runner and required creative resources. + +For a client-side Prebid APS capability, Trusted Server consumes the one-shot capability before starting the runner and calls `markWinningBidAsUsed` only after the runner loads. For server/GPT ownership, it similarly claims the slot/ad ID first. This prevents native and Trusted Server rendering from both owning the same response. + +Disable or coordinate existing publisher-native APS demand for every `publisher_native` cohort. Otherwise the publisher's normal APS auction and this server-selected bid can duplicate demand. Validate the exact account, inventory, CSP, iframe/script creative behavior, impression reporting, and click-through behavior with the APS account team before any production rollout. + Set `inventory_domain` and `inventory_page_origin` together only when the public deployment hostname differs from the inventory identity authorized by APS. The domain becomes `site.domain`. The HTTPS page origin replaces the current page's scheme and host while preserving its path; query and fragment data are removed before forwarding. The origin must be the inventory domain or one of its subdomains and cannot include credentials, a port, path, query, or fragment. These values come only from operator configuration; Trusted Server never accepts APS inventory identity from the client auction payload. APS uses ordinary auction slot IDs and banner formats. Legacy creative-opportunity APS `slot_id` configuration is accepted for compatibility but ignored, and `bidders.aps.slotID` is not required. Remove both during migration. @@ -150,7 +174,7 @@ Trusted Server does not insert APS creative markup into the publisher document. Seats, `impid`, markup, notifications, user-sync data, sibling bids, losing seats, and unknown fields are not exposed. The browser decodes this envelope and cross-checks the ID, dimensions, URL, and tag type before any DOM mutation or message suppression. -Both rendering paths use `GET /integrations/aps/renderer`, a static Trusted Server document with its own restrictive CSP. The document initializes the account-keyed APS queue and then loads only the fixed runner at `https://client.aps.amazon-adsystem.com/prebid-creative.js`. +In `trusted_server` mode, both rendering paths use `GET /integrations/aps/renderer`, a static Trusted Server document with its own restrictive CSP. The document initializes the account-keyed APS queue and then loads only the fixed runner at `https://client.aps.amazon-adsystem.com/prebid-creative.js`. The outer iframe uses these sandbox permissions: @@ -167,11 +191,11 @@ It deliberately omits `allow-same-origin`, so APS and bidder execution remains b ### Direct `/auction` -The TSJS auction client validates the typed renderer descriptor, creates the opaque renderer iframe, and sends the minimized envelope after the frame loads. Ordinary non-APS `adm` continues through the existing sanitizer and generic creative iframe. +In `trusted_server` mode, the TSJS auction client validates the typed renderer descriptor, creates the opaque renderer iframe, and sends the minimized envelope after the frame loads. In `publisher_native` mode it creates the injected friendly iframe and queues the response for the fixed APS Prebid creative runner. Ordinary non-APS `adm` continues through the existing sanitizer and generic creative iframe. ### GAM and Universal Creative -For initial navigation and page-bids, Trusted Server publishes the same descriptor in `window.tsjs.bids`. The source-checked Prebid Universal Creative bridge accepts requests only from the iframe that owns the matching `hb_adid`, validates the complete envelope, and returns a static dynamic-renderer program that creates the same opaque renderer iframe. +For initial navigation and page-bids, Trusted Server publishes the same descriptor in `window.tsjs.bids`. In `publisher_native` mode the ownership-checked bridge resolves the publisher div and starts the friendly-frame runner without sending a Universal Creative renderer response; in `trusted_server` mode it uses the static dynamic renderer described below. The source-checked Prebid Universal Creative bridge accepts requests only from the iframe that owns the matching `hb_adid`, validates the complete envelope, and returns a static dynamic-renderer program that creates the same opaque renderer iframe. For client-side `trustedServer` adapter auctions, Prebid generates its own `hb_adid`. Trusted Server binds that generated ID to the validated APS descriptor in a bounded, expiring browser registry before GAM refresh. The bridge verifies that the requesting Universal Creative iframe belongs to the same ad unit, consumes the capability once, and passes the APS bid ID separately to the Amazon runner. @@ -215,11 +239,12 @@ Use fictional values in source-controlled configuration and fixtures. Supply con 1. Obtain APS account-team confirmation for edge-originated OpenRTB traffic. 2. Enable Trusted Server APS only for an isolated cohort and disable native APS demand there. -3. Keep `allow_script_creatives = false` and observe iframe bids through direct and GAM paths. +3. Keep the default `trusted_server` mode and `allow_script_creatives = false`; observe iframe bids through direct and GAM paths. 4. Confirm outbound privacy fields, aggregate diagnostics, decoded-price competition, line-item targeting, dimensions, click-throughs, and opaque-origin isolation. -5. Run the restrictive-CSP browser proof for script behavior. -6. Only then enable script creatives for the isolated cohort and validate them in a real browser. -7. Expand traffic only after APS confirmation and successful controlled validation. +5. In a still-smaller cohort, set `rendering_mode = "publisher_native"` and confirm the fixed runner request, friendly-frame dimensions, iframe creatives, impression reporting, and click-throughs without a request to `/integrations/aps/renderer`. +6. Confirm the publisher CSP permits the runner but does not need to permit inline Trusted Server scripts. +7. Only after reviewing the friendly-frame security tradeoff, enable script creatives for the isolated native cohort and validate them in a real browser. +8. Expand traffic only after APS confirmation and successful controlled validation. ## Troubleshooting @@ -235,12 +260,12 @@ Use fictional values in source-controlled configuration and fixtures. Supply con ### Winner targets but does not render -- Confirm `GET /integrations/aps/renderer` returns HTML with its CSP and `Referrer-Policy: no-referrer`. -- Confirm publisher CSP permits `frame-src 'self'`. +- In `trusted_server` mode, confirm `GET /integrations/aps/renderer` returns HTML with its CSP and `Referrer-Policy: no-referrer`, and that publisher CSP permits `frame-src 'self'`. +- In `publisher_native` mode, confirm the mode `` marker is present, the slot receives a hidden friendly iframe, and `https://client.aps.amazon-adsystem.com/prebid-creative.js` is not blocked by CSP. The static renderer route is intentionally absent in this mode. - Confirm the GAM creative uses the supported Prebid Universal Creative bridge and the winning `hb_adid`. - For client-side `trustedServer` adapter auctions, confirm Prebid's `bidResponse` contains a generated `adId` and that the corresponding capability appears briefly in `window.tsjs.apsPrebidRenderers` before rendering. -- Ensure no native APS path is trying to handle the same cohort. -- Keep script creatives disabled while diagnosing iframe rendering. +- Ensure no publisher APS auction is trying to handle the same cohort. +- Keep script creatives disabled while diagnosing either rendering mode. ## Verification diff --git a/docs/guide/integrations/datadome.md b/docs/guide/integrations/datadome.md index 0c2d8ae6c..87b3083eb 100644 --- a/docs/guide/integrations/datadome.md +++ b/docs/guide/integrations/datadome.md @@ -68,30 +68,30 @@ patterns = ["(?i)\\.(avi|flv|mka|mkv|mov|mp4|mpeg|mpg|mp3|flac|ogg|ogm|opus|wav| ### Configuration options -| Option | Type | Default | Description | -| -------------------------------------- | ------- | -------------------------------- | ----------------------------------------------------------------------- | -| `enabled` | boolean | `false` | Enable the DataDome integration | -| `sdk_origin` | string | `https://js.datadome.co` | DataDome SDK origin URL for `tags.js` | -| `api_origin` | string | `https://api-js.datadome.co` | DataDome signal collection API origin URL for `/js/*` | -| `cache_ttl_seconds` | integer | `3600` | Cache TTL for `tags.js` | -| `rewrite_sdk` | boolean | `true` | Rewrite DataDome script URLs in HTML to first-party paths | -| `enable_protection` | boolean | `false` | Call the Protection API before route matching | -| `server_side_key_secret_store` | string | `ts_secrets` | Runtime secret store containing the DataDome server-side key | -| `server_side_key_secret_name` | string | `datadome_server_side_key` | Secret name containing the DataDome server-side key | -| `protection_api_origin` | string | `https://api-fastly.datadome.co` | Protection API origin | -| `timeout_ms` | integer | `1500` | Dynamic backend first-byte timeout for Protection API calls | -| `protection_excluded_methods` | array | `["OPTIONS"]` | HTTP methods skipped before the Protection API call | -| `protection_excluded_asns` | array | `[]` | Client autonomous system numbers skipped before the Protection API call | -| `protection_excluded_ip_cidrs` | array | `[]` | Inline client IP CIDR ranges skipped before the Protection API call | -| `protection_excluded_ip_cidr_sources` | array | `[]` | Config Store sources containing dynamic client IP CIDR bypass lists | -| `protection_ip_list_cache_ttl_seconds` | integer | `300` | Process-local cache TTL for Config Store-backed IP CIDR bypass lists | -| `protection_exclusion_rules` | array | Static asset path regex | Structured method/path/query/IP/ASN exclusion rules | -| `protection_test_bypass` | object | omitted | Staging-only fixed-header bypass; secret must contain at least 32 bytes | -| `enable_graphql_support` | boolean | `false` | Reserved for future GraphQL body inspection; ignored in v1 | -| `client_side_key` | string | `""` | DataDome client-side JavaScript key used for tag injection | -| `inject_client_side_tag` | boolean | `true` | Auto-inject the browser tag when `client_side_key` is non-empty | -| `client_side_tag_url` | string | `/integrations/datadome/tags.js` | Root-relative or HTTPS script URL used by auto-injection | -| `client_side_configuration` | object | `{ ajaxListenerPath = true }` | Options assigned to `window.ddoptions` | +| Option | Type | Default | Description | +| -------------------------------------- | ------- | -------------------------------- | ------------------------------------------------------------------------------ | +| `enabled` | boolean | `false` | Enable the DataDome integration | +| `sdk_origin` | string | `https://js.datadome.co` | DataDome SDK origin URL for `tags.js` | +| `api_origin` | string | `https://api-js.datadome.co` | DataDome signal collection API origin URL for `/js/*` | +| `cache_ttl_seconds` | integer | `3600` | Cache TTL for `tags.js` | +| `rewrite_sdk` | boolean | `true` | Rewrite DataDome script URLs in HTML to first-party paths | +| `enable_protection` | boolean | `false` | Call the Protection API before route matching | +| `server_side_key_secret_store` | string | `ts_secrets` | Runtime secret store containing the DataDome server-side key | +| `server_side_key_secret_name` | string | `datadome_server_side_key` | Secret name containing the DataDome server-side key | +| `protection_api_origin` | string | `https://api-fastly.datadome.co` | Protection API origin | +| `timeout_ms` | integer | `1500` | Dynamic backend first-byte timeout for Protection API calls | +| `protection_excluded_methods` | array | `["OPTIONS"]` | HTTP methods skipped before the Protection API call | +| `protection_excluded_asns` | array | `[]` | Client autonomous system numbers skipped before the Protection API call | +| `protection_excluded_ip_cidrs` | array | `[]` | Inline client IP CIDR ranges skipped before the Protection API call | +| `protection_excluded_ip_cidr_sources` | array | `[]` | Config Store sources containing dynamic client IP CIDR bypass lists | +| `protection_ip_list_cache_ttl_seconds` | integer | `300` | Process-local cache TTL for Config Store-backed IP CIDR bypass lists | +| `protection_exclusion_rules` | array | Static asset path regex | Structured method/path/query/IP/ASN exclusion rules | +| `protection_test_bypass` | object | omitted | Configuration-gated fixed-header bypass; secret must contain at least 32 bytes | +| `enable_graphql_support` | boolean | `false` | Reserved for future GraphQL body inspection; ignored in v1 | +| `client_side_key` | string | `""` | DataDome client-side JavaScript key used for tag injection | +| `inject_client_side_tag` | boolean | `true` | Auto-inject the browser tag when `client_side_key` is non-empty | +| `client_side_tag_url` | string | `/integrations/datadome/tags.js` | Root-relative or HTTPS script URL used by auto-injection | +| `client_side_configuration` | object | `{ ajaxListenerPath = true }` | Options assigned to `window.ddoptions` | ## Client-side setup @@ -169,20 +169,19 @@ A request is protected when all of the following are true: 5. The client IP does not match `protection_excluded_ip_cidrs` or any Config Store-backed CIDR source. 6. The client ASN is not listed in `protection_excluded_asns`. 7. No `protection_exclusion_rules` match. -8. The request does not contain a matching enabled `protection_test_bypass` credential while `FASTLY_IS_STAGING=1`. +8. The request does not contain a matching enabled `protection_test_bypass` credential. Static assets are excluded by default using a case-insensitive file-extension regex. Trusted Server internal routes such as `/static/tsjs=`, `/integrations/`, `/first-party/`, admin routes, discovery routes, and signature-verification routes are also excluded by default. Auction traffic at `/auction` is protected by default. -### Staging test bypass +### Test bypass For short-lived browser automation on an access-controlled staging site, you can configure a static header credential that skips only the server-side Protection API: ```toml -# Runtime activation also requires FASTLY_IS_STAGING=1. [integrations.datadome.protection_test_bypass] enabled = true credential_secret_store = "ts_secrets" @@ -190,25 +189,20 @@ credential_secret_name = "datadome_test_bypass" ``` `protection_test_bypass` requires `enable_protection = true`; it is disabled -when omitted and is runtime-active only when `FASTLY_IS_STAGING=1`. -`FASTLY_IS_STAGING` is supplied at runtime by Fastly (`1` in staging and `0` in -production); it is not compiled into or promoted with the Wasm artifact. Verify -staging through the `X-TS-ENV: staging` response signal and the integration -activation log, and verify production omits that response signal. A retained -section cannot bypass protection in a production or other non-staging runtime. -Store a randomly generated credential containing at least 32 bytes of -high-entropy material in the configured Secret Store, configure this section -only while needed, protect the site with an outer access control such as Basic -Auth, and remove the section when testing finishes. +when omitted or when its `enabled` field is false. Store a randomly generated +credential containing at least 32 bytes of high-entropy material in the +configured Secret Store, configure this section only while needed, and protect +the site with an outer access control such as Basic Auth. Remove the section +when testing finishes. Whenever the enabled DataDome request filter runs on the Fastly adapter, the fixed `x-ts-datadome-bypass` header is removed before configuration or credential checks. It therefore cannot reach DataDome or the publisher origin -through that path when the bypass is absent, disabled, inactive, or invalid. -Active credentials are compared in constant time and never logged. Duplicate -header values fail closed. Scope the header to the staging origin; do not attach -it to every request in a browser context because that can disclose the -credential to third-party origins. With Playwright: +through that path when the bypass is absent, disabled, or invalid. Active +credentials are compared in constant time and never logged. Duplicate header +values fail closed. Scope the header to the test origin; do not attach it to +every request in a browser context because that can disclose the credential to +third-party origins. With Playwright: ```ts await context.route('https://staging.example.com/**', async (route) => { @@ -233,7 +227,7 @@ This behavior applies to: - `protection_excluded_ip_cidr_sources`; - structured `ip_cidr` rules; - structured `ip_cidr_source` rules; and -- a matching enabled `protection_test_bypass` credential in a staging runtime. +- a matching enabled `protection_test_bypass` credential. Method, ASN, path, query-parameter, static-asset, and internal-route exclusions alone do not suppress the client-side tag. However, a simultaneous matching IP diff --git a/docs/guide/integrations/gpt-diagnostics.md b/docs/guide/integrations/gpt-diagnostics.md index 8982cb5dd..c5a46ca24 100644 --- a/docs/guide/integrations/gpt-diagnostics.md +++ b/docs/guide/integrations/gpt-diagnostics.md @@ -120,7 +120,7 @@ Each request cycle can show: - GPT slot-onload, impression-viewable, and visibility observations. - Non-negative request-to-response, response-to-render, render-to-load, and render-to-viewable durations. -- Rendered size, backfill, and slot-content-change facts exposed by GPT. +- GPT-reported rendered size, a separately labelled observed outer slot box when safely bound, backfill, and slot-content-change facts. - Current DOM binding status and viewport intersection. Elapsed time alone never changes a pending GPT request to Incomplete. Incomplete @@ -337,6 +337,32 @@ because selector support is unavailable or throws, the export reports `dom_uniqueness_unverifiable`. Framework replacement of an element with a new unique element using the same exact ID is rebound automatically. +When Trusted Server associates a GPT slot with its next request, diagnostics retains +`requestedSlotSizes`: the configured `AuctionSlot.formats` list Trusted Server supplied +to GPT for that request. It is a bounded validated copy of the complete configured +list, not an inferred responsive size or a claim about the final selected size. It is +omitted for publisher and otherwise unknown request paths where Trusted Server did not +supply formats. + +For an explicitly filled render, diagnostics can also retain `observedSlotSize`: the +current outer CSS box of the uniquely bound, connected slot element. This is measured +after `slotRenderEnded`. When `ResizeObserver` is available, it remains current +while that same request cycle is latest for the GPT slot; otherwise it is the most +recently sampled box. It is displayed separately from `size`, which remains the exact +GPT-reported `slotRenderEnded.size` fill-size fact. The panel and badge label the three +separate facts as requested slot sizes, GPT-reported fill size, and observed outer slot +box. The observed box may differ from GPT's reported size (for example, a flexible APS +creative can report `1×1` while its allocated outer slot box is larger). It is a +publisher-page layout measurement, not a claim about universal internal creative-pixel +dimensions. A collapsed or hidden bound element can report `0×0`, which records the +page layout state rather than an invalid measurement. Empty, unbound, missing, or +ambiguous slots do not report an observed box; delayed measurements from an older cycle +are rejected after a refresh. + +Cross-origin and SafeFrame boundaries prevent diagnostics from inspecting iframe +content or altering the APS sandbox, so this field cannot prove the inner creative's +pixels. + Badges and the panel live in a closed Shadow DOM. Diagnostics do not add attributes, classes, or inline styles to publisher slot elements. @@ -387,6 +413,8 @@ The allowlisted export contains: - `version: 1` and an ISO `capturedAt` timestamp. - Current page origin and pathname, excluding query parameters and fragments. - Retained slots, binding facts, visibility, and request cycles. +- `requestedSlotSizes` when Trusted Server supplied configured formats for that exact + request, plus GPT-reported fill `size` and an optional observed outer `observedSlotSize`. - Request path, request intent ID, opportunity, creative-progress timestamps, and safe failure enums. - The per-auction diagnostics token (`trustedServerAuctionId`) and the @@ -426,6 +454,7 @@ inaccessible to JavaScript. - Retained request cycles per slot: 10. - Retained callback issues: 128. - Retained auction-slot-to-GPT-slot associations: 64. +- Requested slot sizes per correlated request: the first 16 configured entries, with invalid entries dropped. - Retained creative attempts, including status tombstones: 128. - Retained attribution issues: 128. diff --git a/docs/package-lock.json b/docs/package-lock.json index e31c9db5e..e6003067a 100644 --- a/docs/package-lock.json +++ b/docs/package-lock.json @@ -14,7 +14,7 @@ "eslint": "^10", "mermaid": "^11.12.3", "prettier": "^3.4.2", - "typescript-eslint": "8.57", + "typescript-eslint": "8.66", "vitepress": "^1.5.0", "vitepress-plugin-mermaid": "^2.0.17" } @@ -180,7 +180,6 @@ "integrity": "sha512-Jc360x4yqb3eEg4OY4KEIdGePBxZogivKI+OGIU8aLXgAYPTECvzeOBc90312yHA1hr3AeRlAFl0rIc8lQaIrQ==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@algolia/client-common": "5.50.0", "@algolia/requester-browser-xhr": "5.50.0", @@ -1886,7 +1885,6 @@ "integrity": "sha512-GYDxsZi3ChgmckRT9HPU0WEhKLP08ev/Yfcq2AstjrDASOYCSXeyjDsHg4v5t4jOj7cyDX3vmprafKlWIG9MXQ==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "undici-types": "~7.16.0" } @@ -1914,20 +1912,20 @@ "license": "MIT" }, "node_modules/@typescript-eslint/eslint-plugin": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.57.2.tgz", - "integrity": "sha512-NZZgp0Fm2IkD+La5PR81sd+g+8oS6JwJje+aRWsDocxHkjyRw0J5L5ZTlN3LI1LlOcGL7ph3eaIUmTXMIjLk0w==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.66.0.tgz", + "integrity": "sha512-p088eaGrzYz1s+7cov0aMOCkNGTJlVxF4jgubf28c8L0Cv9Rloj8YBHnv4hXLq6IIEE1AsjNWavO+k+8kP2Y0A==", "dev": true, "license": "MIT", "dependencies": { "@eslint-community/regexpp": "^4.12.2", - "@typescript-eslint/scope-manager": "8.57.2", - "@typescript-eslint/type-utils": "8.57.2", - "@typescript-eslint/utils": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2", + "@typescript-eslint/scope-manager": "8.66.0", + "@typescript-eslint/type-utils": "8.66.0", + "@typescript-eslint/utils": "8.66.0", + "@typescript-eslint/visitor-keys": "8.66.0", "ignore": "^7.0.5", "natural-compare": "^1.4.0", - "ts-api-utils": "^2.4.0" + "ts-api-utils": "^2.5.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -1937,15 +1935,15 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "@typescript-eslint/parser": "^8.57.2", + "@typescript-eslint/parser": "^8.66.0", "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/eslint-plugin/node_modules/ignore": { - "version": "7.0.5", - "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.5.tgz", - "integrity": "sha512-Hs59xBNfUIunMFgWAbGX5cq6893IbWg4KnrjbYwX3tx0ztorVgTDA6B2sxf8ejHJ4wz8BqGUMYlnzNBer5NvGg==", + "version": "7.0.6", + "resolved": "https://registry.npmjs.org/ignore/-/ignore-7.0.6.tgz", + "integrity": "sha512-BAg6QkE8W+TuQLrrw0Ugr7HegXduRuuj8/ti2kSOc+jz1dmx8/WNcjr6XGnq5YpDWxFwwaavqD0+jIUOKelTsw==", "dev": true, "license": "MIT", "engines": { @@ -1953,17 +1951,16 @@ } }, "node_modules/@typescript-eslint/parser": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/parser/-/parser-8.57.2.tgz", - "integrity": "sha512-30ScMRHIAD33JJQkgfGW1t8CURZtjc2JpTrq5n2HFhOefbAhb7ucc7xJwdWcrEtqUIYJ73Nybpsggii6GtAHjA==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/parser/-/parser-8.66.0.tgz", + "integrity": "sha512-X6ypGChaWYk6PBtUg2BwuTZEFFcHJAtGTVJ9/lCTOufhZ4i9fNolQNnktq+kkMCwMj7V8Svsq7+TxSDslmhE0g==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { - "@typescript-eslint/scope-manager": "8.57.2", - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2", + "@typescript-eslint/scope-manager": "8.66.0", + "@typescript-eslint/types": "8.66.0", + "@typescript-eslint/typescript-estree": "8.66.0", + "@typescript-eslint/visitor-keys": "8.66.0", "debug": "^4.4.3" }, "engines": { @@ -1975,18 +1972,18 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/project-service": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/project-service/-/project-service-8.57.2.tgz", - "integrity": "sha512-FuH0wipFywXRTHf+bTTjNyuNQQsQC3qh/dYzaM4I4W0jrCqjCVuUh99+xd9KamUfmCGPvbO8NDngo/vsnNVqgw==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/project-service/-/project-service-8.66.0.tgz", + "integrity": "sha512-7MthGPTt4BP69lSryqpqq8HQqxuzynssckL/jyDyk3+TNMQ3y2jFWkptCrktWvBrP+EH787Nl5N5Qpw7WZg+5g==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/tsconfig-utils": "^8.57.2", - "@typescript-eslint/types": "^8.57.2", + "@typescript-eslint/tsconfig-utils": "^8.66.0", + "@typescript-eslint/types": "^8.66.0", "debug": "^4.4.3" }, "engines": { @@ -1997,18 +1994,18 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/scope-manager": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/scope-manager/-/scope-manager-8.57.2.tgz", - "integrity": "sha512-snZKH+W4WbWkrBqj4gUNRIGb/jipDW3qMqVJ4C9rzdFc+wLwruxk+2a5D+uoFcKPAqyqEnSb4l2ULuZf95eSkw==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/scope-manager/-/scope-manager-8.66.0.tgz", + "integrity": "sha512-8TGcH25j9zqJ/IULB/ppyhRvxA8QYfFEZ7nfbg6/BN9spDgb8fPWQXlE5l8TWBL50EtUx007uZ1o9VOwrq2/9g==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2" + "@typescript-eslint/types": "8.66.0", + "@typescript-eslint/visitor-keys": "8.66.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2019,9 +2016,9 @@ } }, "node_modules/@typescript-eslint/tsconfig-utils": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/tsconfig-utils/-/tsconfig-utils-8.57.2.tgz", - "integrity": "sha512-3Lm5DSM+DCowsUOJC+YqHHnKEfFh5CoGkj5Z31NQSNF4l5wdOwqGn99wmwN/LImhfY3KJnmordBq/4+VDe2eKw==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/tsconfig-utils/-/tsconfig-utils-8.66.0.tgz", + "integrity": "sha512-9D5gLYZG4rOjcoag8MQ/fWI8WqA9wcPDyOGyWtWFhvM1lHRbliqUSPIY5J3zqCU1tvSwzXxnnjhQhz5Ne7mJ4g==", "dev": true, "license": "MIT", "engines": { @@ -2032,21 +2029,21 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/type-utils": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/type-utils/-/type-utils-8.57.2.tgz", - "integrity": "sha512-Co6ZCShm6kIbAM/s+oYVpKFfW7LBc6FXoPXjTRQ449PPNBY8U0KZXuevz5IFuuUj2H9ss40atTaf9dlGLzbWZg==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/type-utils/-/type-utils-8.66.0.tgz", + "integrity": "sha512-LG2dWfjZQQp0ADtAu/EWJVayefGL2UEZ3CDeI44D9v3rXB/WYUqE/jpO28KrEKul5AySrmI+Zh1v6v+xW2U9+g==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2", - "@typescript-eslint/utils": "8.57.2", + "@typescript-eslint/types": "8.66.0", + "@typescript-eslint/typescript-estree": "8.66.0", + "@typescript-eslint/utils": "8.66.0", "debug": "^4.4.3", - "ts-api-utils": "^2.4.0" + "ts-api-utils": "^2.5.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2057,13 +2054,13 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/types": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/types/-/types-8.57.2.tgz", - "integrity": "sha512-/iZM6FnM4tnx9csuTxspMW4BOSegshwX5oBDznJ7S4WggL7Vczz5d2W11ecc4vRrQMQHXRSxzrCsyG5EsPPTbA==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/types/-/types-8.66.0.tgz", + "integrity": "sha512-H6gcYaSDOyvL3AD/jHUtUFo2jqGgn/F6nuyuZSu0QTesxL+cP4dQoIMrODRofuJC09g64+WgZ6tE19Y1N2YIFQ==", "dev": true, "license": "MIT", "engines": { @@ -2075,21 +2072,21 @@ } }, "node_modules/@typescript-eslint/typescript-estree": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/typescript-estree/-/typescript-estree-8.57.2.tgz", - "integrity": "sha512-2MKM+I6g8tJxfSmFKOnHv2t8Sk3T6rF20A1Puk0svLK+uVapDZB/4pfAeB7nE83uAZrU6OxW+HmOd5wHVdXwXA==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/typescript-estree/-/typescript-estree-8.66.0.tgz", + "integrity": "sha512-8/x4INiiQb10jGgXYD7116/zQ+OL84ZIFn0za68wwFHCanT/VLbBEroWht8RV8fn0/ZCAoazHLQgwUC0UQcDfg==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/project-service": "8.57.2", - "@typescript-eslint/tsconfig-utils": "8.57.2", - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/visitor-keys": "8.57.2", + "@typescript-eslint/project-service": "8.66.0", + "@typescript-eslint/tsconfig-utils": "8.66.0", + "@typescript-eslint/types": "8.66.0", + "@typescript-eslint/visitor-keys": "8.66.0", "debug": "^4.4.3", "minimatch": "^10.2.2", "semver": "^7.7.3", "tinyglobby": "^0.2.15", - "ts-api-utils": "^2.4.0" + "ts-api-utils": "^2.5.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2099,20 +2096,20 @@ "url": "https://opencollective.com/typescript-eslint" }, "peerDependencies": { - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/utils": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/utils/-/utils-8.57.2.tgz", - "integrity": "sha512-krRIbvPK1ju1WBKIefiX+bngPs+odIQUtR7kymzPfo1POVw3jlF+nLkmexdSSd4UCbDcQn+wMBATOOmpBbqgKg==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/utils/-/utils-8.66.0.tgz", + "integrity": "sha512-jasearZPolBw5NJNYGMwxzHMF83niVWmMU1VdHzG1CyfI2VS7f7nZltnKtHcg20hW+7Uo5GfK4MeDPoU3qI8EA==", "dev": true, "license": "MIT", "dependencies": { "@eslint-community/eslint-utils": "^4.9.1", - "@typescript-eslint/scope-manager": "8.57.2", - "@typescript-eslint/types": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2" + "@typescript-eslint/scope-manager": "8.66.0", + "@typescript-eslint/types": "8.66.0", + "@typescript-eslint/typescript-estree": "8.66.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -2123,17 +2120,17 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/@typescript-eslint/visitor-keys": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/@typescript-eslint/visitor-keys/-/visitor-keys-8.57.2.tgz", - "integrity": "sha512-zhahknjobV2FiD6Ee9iLbS7OV9zi10rG26odsQdfBO/hjSzUQbkIYgda+iNKK1zNiW2ey+Lf8MU5btN17V3dUw==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/@typescript-eslint/visitor-keys/-/visitor-keys-8.66.0.tgz", + "integrity": "sha512-dkKR8q+lKciskj1Y3vthHktl+3cMLWGyVUP23bRiPZ5O9BRT++4EqDDV+TVeIKBL1VXVEqrJlz8MYbcnvJcAlg==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/types": "8.57.2", + "@typescript-eslint/types": "8.66.0", "eslint-visitor-keys": "^5.0.0" }, "engines": { @@ -2433,7 +2430,6 @@ "integrity": "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw==", "dev": true, "license": "MIT", - "peer": true, "bin": { "acorn": "bin/acorn" }, @@ -2474,7 +2470,6 @@ "integrity": "sha512-yE5I83Q2s8euVou8Y3feXK08wyZInJWLYXgWO6Xti9jBUEZAGUahyeQ7wSZWkifLWVnQVKEz5RAmBlXG5nqxog==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@algolia/abtesting": "1.16.0", "@algolia/client-abtesting": "5.50.0", @@ -2567,7 +2562,6 @@ "integrity": "sha512-opLQzEVriiH1uUQ4Kctsd49bRoFDXGGSC4GUqj7pGyxM3RehRhvTlZJc1FL/Flew2p5uwxa1tUDWKzI4wNM8pg==", "dev": true, "license": "Apache-2.0", - "peer": true, "dependencies": { "@chevrotain/cst-dts-gen": "11.1.2", "@chevrotain/gast": "11.1.2", @@ -2672,7 +2666,6 @@ "integrity": "sha512-iJc4TwyANnOGR1OmWhsS9ayRS3s+XQ185FmuHObThD+5AeJCakAAbWv8KimMTt08xCCLNgneQwFp+JRJOr9qGQ==", "dev": true, "license": "MIT", - "peer": true, "engines": { "node": ">=0.10" } @@ -3107,7 +3100,6 @@ "integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==", "dev": true, "license": "ISC", - "peer": true, "engines": { "node": ">=12" } @@ -3363,7 +3355,6 @@ "integrity": "sha512-S9jlY/ELKEUwwQnqWDO+f+m6sercqOPSqXM5Go94l7DOmxHVDgmSFGWEzeE/gwgTAr0W103BWt0QLe/7mabIvA==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@eslint-community/eslint-utils": "^4.8.0", "@eslint-community/regexpp": "^4.12.2", @@ -3613,7 +3604,6 @@ "integrity": "sha512-/yNdlIkpWbM0ptxno3ONTuf+2g318kh2ez3KSeZN5dZ8YC6AAmgeWz+GasYYiBJPFaYcSAPeu4GfhUaChzIJXA==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "tabbable": "^6.4.0" } @@ -3979,7 +3969,6 @@ "integrity": "sha512-fEnci+Immw6lKMFI8sqzjlATTyjLkRa6axrEgLV2yHTfv8r+h1wjFbV6xeRtd4rUV1cS4EpR9rwp3Rci7TRWDw==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@braintree/sanitize-url": "^7.1.1", "@iconify/utils": "^3.0.2", @@ -4300,12 +4289,11 @@ "license": "ISC" }, "node_modules/picomatch": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz", - "integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==", + "version": "4.0.5", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz", + "integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==", "dev": true, "license": "MIT", - "peer": true, "engines": { "node": ">=12" }, @@ -4552,9 +4540,9 @@ "peer": true }, "node_modules/semver": { - "version": "7.7.4", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.4.tgz", - "integrity": "sha512-vFKC2IEtQnVhpT78h1Yp8wzwrf8CM+MzKMHGJZfBtzhZNycRFnXsHk6E5TxIkkMsgNS7mdX3AGB7x2QM2di4lA==", + "version": "7.8.5", + "resolved": "https://registry.npmjs.org/semver/-/semver-7.8.5.tgz", + "integrity": "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA==", "dev": true, "license": "ISC", "bin": { @@ -4688,14 +4676,14 @@ } }, "node_modules/tinyglobby": { - "version": "0.2.15", - "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.15.tgz", - "integrity": "sha512-j2Zq4NyQYG5XMST4cbs02Ak8iJUdxRM0XI5QyxXuZOzKOINmWurp3smXu3y5wDcJrptwpSjgXHzIQxR0omXljQ==", + "version": "0.2.17", + "resolved": "https://registry.npmjs.org/tinyglobby/-/tinyglobby-0.2.17.tgz", + "integrity": "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g==", "dev": true, "license": "MIT", "dependencies": { "fdir": "^6.5.0", - "picomatch": "^4.0.3" + "picomatch": "^4.0.4" }, "engines": { "node": ">=12.0.0" @@ -4767,16 +4755,16 @@ } }, "node_modules/typescript-eslint": { - "version": "8.57.2", - "resolved": "https://registry.npmjs.org/typescript-eslint/-/typescript-eslint-8.57.2.tgz", - "integrity": "sha512-VEPQ0iPgWO/sBaZOU1xo4nuNdODVOajPnTIbog2GKYr31nIlZ0fWPoCQgGfF3ETyBl1vn63F/p50Um9Z4J8O8A==", + "version": "8.66.0", + "resolved": "https://registry.npmjs.org/typescript-eslint/-/typescript-eslint-8.66.0.tgz", + "integrity": "sha512-QlEbBPz/RuJ1XUHj29nm3t0F/O/cSlEnntozqPOYHnnTGAXFamnMBu5i9Vn6vhUPHGAjR+Vl+5J8vPN/BMUrJw==", "dev": true, "license": "MIT", "dependencies": { - "@typescript-eslint/eslint-plugin": "8.57.2", - "@typescript-eslint/parser": "8.57.2", - "@typescript-eslint/typescript-estree": "8.57.2", - "@typescript-eslint/utils": "8.57.2" + "@typescript-eslint/eslint-plugin": "8.66.0", + "@typescript-eslint/parser": "8.66.0", + "@typescript-eslint/typescript-estree": "8.66.0", + "@typescript-eslint/utils": "8.66.0" }, "engines": { "node": "^18.18.0 || ^20.9.0 || >=21.1.0" @@ -4787,7 +4775,7 @@ }, "peerDependencies": { "eslint": "^8.57.0 || ^9.0.0 || ^10.0.0", - "typescript": ">=4.8.4 <6.0.0" + "typescript": ">=4.8.4 <6.1.0" } }, "node_modules/ufo": { @@ -4937,7 +4925,6 @@ "integrity": "sha512-o5a9xKjbtuhY6Bi5S3+HvbRERmouabWbyUcpXXUA1u+GNUKoROi9byOJ8M0nHbHYHkYICiMlqxkg1KkYmm25Sw==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "esbuild": "^0.21.3", "postcss": "^8.4.43", @@ -4998,7 +4985,6 @@ "integrity": "sha512-+2ym1/+0VVrbhNyRoFFesVvBvHAVMZMK0rw60E3X/5349M1GuVdKeazuksqopEdvkKwKGs21Q729jX81/bkBJg==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@docsearch/css": "3.8.2", "@docsearch/js": "3.8.2", @@ -5110,7 +5096,6 @@ "integrity": "sha512-iV/sU9SzOlmA/0tygSmjkEN6Jbs3nPoIPFhCMLD2STrjgOU8DX7ZtzMhg4ahVwf5Rp9KoFzcXeB1ZrVbLBp5/Q==", "dev": true, "license": "MIT", - "peer": true, "dependencies": { "@vue/compiler-dom": "3.5.31", "@vue/compiler-sfc": "3.5.31", diff --git a/docs/package.json b/docs/package.json index f74991a4c..df1bd66d0 100644 --- a/docs/package.json +++ b/docs/package.json @@ -20,7 +20,7 @@ "eslint": "^10", "mermaid": "^11.12.3", "prettier": "^3.4.2", - "typescript-eslint": "8.57", + "typescript-eslint": "8.66", "vitepress": "^1.5.0", "vitepress-plugin-mermaid": "^2.0.17" } diff --git a/docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md b/docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md new file mode 100644 index 000000000..bec425f82 --- /dev/null +++ b/docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md @@ -0,0 +1,864 @@ +# The Cacheable Root: Latency Diagnosis and Stage 0 Design + +_Filename retains its original `esi-` prefix; the commit history and every +cross-reference point at it. The subject moved, the path did not._ + +**Issue:** IABTechLab/trusted-server#1009 · **Date:** 2026-08-08 · + +> **HISTORICAL RECORD — NOT THE CURRENT IMPLEMENTATION.** This document preserves the +> measurement and feasibility investigation. Every executable ESI tag, parser, and +> subrequest described below belongs to a rejected spike; do not use those sections to +> infer current runtime behavior. The final branch retains `assembly_mode = "esi"` only as +> the operator spelling for Fastly C2 plus exact byte-seam assembly. See +> [the merge-hardening design](./2026-08-12-1009-esi-merge-hardening-design.md). + +**Revised:** 2026-08-10 +**Baseline:** citations verified at `cfb98f4`; unchanged as of `b0ce56c3`. + +> ## ⚠️ Correction, 2026-08-10 — this document's original ESI verdict was wrong +> +> The first revision concluded that ESI was **structurally blocked**: that it +> presupposed a TS-owned template cache which did not exist, and that such a cache was +> in turn blocked on purge capability the platform did not offer. **Both claims are +> false**, and an external review was right to reject them. +> +> Verified against the pinned `fastly` 0.12.1: +> +> - **The cache boundary is native.** `fastly::cache::core` provides +> `insert(key, max_age).execute() -> StreamingBody` for arbitrary bytes, `lookup()` / +> `found()` to read them back, and `Transaction` with `must_insert()` for request +> collapsing. The two-stage design needs no separate KV or template service. +> - **Purge exists in-process.** `fastly::http::purge::purge_surrogate_key` purges from +> inside Compute; the management-API token scope cited in the original is irrelevant to +> it. Note which cache, though: `InsertBuilder::surrogate_keys([...])` is the **Core +> Cache** API and keys the transformed-template cache (C2). It does **not** key the HTTP +> read-through cache (C1) that Stage 0 turns on — purging that needs origin-supplied +> keys or the HTTP cache's own surrogate-key surface. +> - **The original pipeline ordering was backwards.** It said "order esi → lol*html, +> never the reverse." `lol_html` \_emits* the ESI include tags, so ESI must run after it. +> Correct order is in [§6.6](#66-the-esi-pipeline-corrected). +> +> The error was inspecting what this repository does and reporting it as what the +> platform permits — the same mistake this document criticises #1009 for making in the +> other direction. +> +> **ESI is therefore feasible and unvalidated, not rejected.** Validating it is +> [a separate plan](./2026-08-10-1009-esi-validation-spike.md). What survives +> here is the latency re-diagnosis and the Stage 0 optimisation, which are worth doing +> and are **not** an answer to #1009. + +## Document map — read this first + +#1009 is answered across three documents, not one. This is the only place that says +which owns what. + +| Document | Owns | +| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| **This spec** | Why the TTFB regression happens, what Stage 0 is and why, and the corrected ESI feasibility verdict | +| [Stage 0 plan](../plans/2026-08-08-1009-measurement-and-stage-0.md) | Implementing the measurement and the cache-bypass flag. **Does not close #1009.** | +| [ESI validation spike](./2026-08-10-1009-esi-validation-spike.md) | **Where #1009 is actually decided.** Four arms, safety gates, decision rule. | +| [Findings](../plans/2026-08-08-1009-measurement-findings.md) | Recorded results. Currently: Step A only, at `PROVISIONAL PASS`. | + +**If you want the ESI answer**, it is [§2](#2-why--the-three-findings) for the verdict, +[§6.6](#66-the-esi-pipeline-corrected) for the pipeline, and the spike plan for how it +gets validated. Everything else here is Stage 0 and the latency analysis behind it. + +**Decision requested:** approve the four items in §1. + +> **Orientation.** #1009 asks whether Edge Side Includes (ESI) can cache page fragments +> so that cacheable publisher HTML is separated from per-user ad state, recovering a +> TTFB regression that Trusted Server (TS) adds to navigations on a Next.js App Router +> publisher running on Fastly Compute. ESI can do this; whether it should is not settled +> here. Separately, the regression has a cheaper cause than the issue assumes. +> +> **This document deliberately carries no performance measurements.** Every conclusion +> below is derived from code at the pinned baseline, so it can be checked by reading the +> repository rather than by trusting a benchmark. Where a quantity is needed and unknown, +> it is named as unknown and [§3](#3-monday-morning) says how to obtain it. +> +> Terms used throughout: **the hold** = TS holding the HTTP response open at `` +> until the server-side auction (SSAT) resolves. **React #418** = the React +> hydration-mismatch error raised when `adInit()` mutates ad-slot subtrees during +> hydration; it is why bid application is deferred to `window.load`. It is a React error +> number, **not** a repository issue — the tracker is +> [#938](https://github.com/IABTechLab/trusted-server/issues/938). **The SSAT price +> defect** = a live mispricing bug named in #1009 (prices reading 100× high) — cited +> from #1009 and prior investigation, not re-verified here. + +--- + +## 1. Decision requested + +| # | Decision | Owner needed | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | +| D1 | **ESI is feasible and unvalidated.** Validate it via [the spike plan](./2026-08-10-1009-esi-validation-spike.md), not by deferring it. | Eng + product | +| D2 | **Fund ~3 days of measurement** (§3). No dependencies. Can start immediately. | Eng | +| D3 | **Approve Stage 0** — an operator flag disabling the origin cache bypass, subject to a **`FINAL PASS`** in §3. A `Vary` check alone is a `PROVISIONAL PASS` and is not a release gate. Rollback needs a purge path, not only a config push. | Eng | +| D4 | **Stages 1–2 queue behind the SSAT price defect and #938.** Stages 3b–4 unscheduled. ESI is not in this queue — see §7. | Product | + +Rationale for D4 in [§8](#8-priority). Everything this document recommends _against_ +doing is in [§7](#7-deferred-work-specified-not-scheduled), at deliberately lower +detail than the work it recommends. + +--- + +## 2. Why — the three findings + +**ESI is buildable on the pinned SDK, and unvalidated.** `lol_html` emits executable ESI +include tags into a shared template; `fastly::cache::core` stores that template; the +`esi` crate assembles per request on the way out. Everything that requires is +already a dependency. The real open questions are empirical, not architectural: does it +beat a plain client fetch by enough to justify a Fastly-only rendering path, and can +per-user leakage be excluded under cold MISS, warm HIT, stale revalidation, and fragment +failure. [The spike plan](./2026-08-10-1009-esi-validation-spike.md) answers +those; [§6.6](#66-the-esi-pipeline-corrected) gives the pipeline. + +Two constraints stay true regardless. ESI is **Fastly-only at every API level**, so it +is a per-platform accelerator rather than the architecture, and its maintenance cost +belongs in the decision. And its Dynamic Content Assembly must be **explicitly disabled** +— bid payloads carry partner-controlled creative markup, so under `DcaMode::Esi` an SSP +could embed an ESI include targeting an arbitrary URL and make the edge fetch it. Details in +[Appendix E](#appendix-e--esi-notes-condensed). + +**The auction is already out of band; the hold is ~free.** It is dispatched _before_ +the origin fetch and does not block — dispatched at `publisher.rs:2751-2755`, sent at `:2870` — +with a 500 ms budget. The actual cost is `with_cache_bypass` +(`publisher.rs:2867`), +which forces every ad-eligible navigation to miss the Fastly readthrough cache. + +**The two fixes are multiplicative.** Removing the bypass alone lets the previously +hidden auction surface as the new bottleneck. Removing the hold alone changes nothing, +because the auction was never the bottleneck. **Shipping the hold removal without the +bypass removal will measure no improvement and will read as the effort having failed** — +the most likely way this work gets judged unfairly. + +**Ordering is established; magnitude is not.** The ordering above follows from code and +needs no measurement. The _size_ of the win does — and the one quantity it depends on, +the origin build time under `Pass`, has never been measured. #1009's timings do not +supply it: they compare cached fetches against each other, not against an origin build. +**Quote no figure to a publisher until §3 Step C runs.** Full reasoning in +[§6](#6-the-analysis). + +--- + +## 3. Monday morning + +Three checks, ordered cheapest-first. Each needs a named owner before starting. + +**Step A — origin `Vary` and cookie check (minutes for the first pass).** `curl` the +origin with and without `RSC`, `Next-Router-*`, and the experiment header; inspect `Vary`, +`Cache-Control`, and `Set-Cookie`. **This first pass yields a `PROVISIONAL PASS` only** — +it is not what gates the flip. A `FINAL PASS` additionally requires a real authenticated +session, Basic Auth through TS, the experiment variant, representative routes, and +cached-hit render attribution. Do the cheap pass first because it is the +cheapest thing that unblocks anything. + +**Step B — what consumes TS's own response headers (under a day).** Request a TS-served +path that already emits `public, s-maxage` +(`http_util.rs:294-311`) +twice and look for `x-cache`/`age` on TS's _own_ response. **Gates the Stage 3a/3b +split** — see [§7](#7-deferred-work-specified-not-scheduled). + +**Step C — measure the hold directly (1 day + a measurement window).** + +The hold's cost is literally the duration of one `.await`: `collect_stream_auction` at +`publisher.rs:793`, plus the +two EOF variants in `hold_finish_ready_segments` and `hold_finish_tail_segments`. Two +`Instant`s around it yield **`hold_wait_ms`** — the number this entire document is +arguing about, measured rather than modelled. + +Emit two timings per ad-eligible navigation: + +| Metric | Why | +| ----------------- | ---------------------------------------------------------------------- | +| `hold_wait_ms` | **The decision.** How long the response was actually held for bids. | +| `origin_fetch_ms` | Attribution — how much of the win Stage 0 can claim. Origin TTFB only. | + +`hold_wait_ms` replaces the proxy comparison an earlier draft proposed. Comparing `O` +against `A` was an indirect way of asking "does the hold block?"; this asks it directly, +costs less to build, and removes the modelling error corrected in +[§6.2](#62-what-the-hold-actually-costs). + +Deliberately not measured: auction collect duration is already instrumented +(`OrchestrationResult::total_time_ms`, `auction/orchestrator.rs:285`, flowing to +`auction_events_raw`) — read it, don't rebuild it. Rewrite duration decides nothing and +would mean touching two finalizers. + +- **Mechanism: a `log::info!` line behind a debug flag, not `Server-Timing`.** A response + header would in fact work for the origin-fetch figure — that value is known before + headers commit — but a server-side log needs no browser harness to collect it, `log` is + this project's instrumentation crate, and the auction path already measures itself with + `web_time::Instant`. Gate it behind config: one line per eligible navigation is real log + spend and the instrumentation is temporary. +- **Sample: enough navigations per arm to separate the medians with confidence**, across + both page types, and state the N alongside any result. #1009's sample was small enough + that its conclusion did not survive contact with the code; replacing it with another + underpowered sample would repeat the error. + +**Step C has two outcomes, both actionable:** + +| `hold_wait_ms` median | Meaning | Effect on staging | +| --------------------- | ----------------------- | ------------------------------------------------------------ | +| Near zero | The hold is free | Proceed as staged: Stage 0 primary, Stage 2 protects its win | +| Materially non-zero | The hold **is** costing | **Staging inverts** — Stage 2 primary, Stage 0 secondary | + +The work does not change; its order and justification do. **The staging in §7 is +conditional on this measurement**, and the second outcome is a live possibility rather +than a formality — §6.2's argument for the first is weaker than an earlier draft claimed. + +Stage 1's bids-fetch timeout still needs a measured client-side figure rather than an +invented constant, but Step C is server-side and does not supply it. Capture it from the +browser harness when Stage 1 is actually scheduled. + +--- + +## 4. Stage 0 — the only build item recommended now + +Stop bypassing the read-through cache on ad-eligible navigations +(`publisher.rs:2867`). + +**Ship it as an operator flag, not a deletion.** Add +`publisher.bypass_origin_cache`, defaulting to today's behaviour, in the same release as +the Step C instrumentation. Then turn it off with `ts config push`. + +The diff is slightly larger than deleting a line, and that is the point. The risk being +gated here is **cache poisoning** — serving one representation in response to a request +for another. For that class of failure, rollback speed dominates diff size: a config push +reverts the read path in seconds where a release does not — but a config push **evicts +nothing**, so full rollback is flip, then purge or roll a versioned key namespace, then +observe past the origin TTL. The flag also buys an A/B on a byte-identical +build, removing build difference as a confound in the very measurement this depends on, +and allows flipping for a tester-cookie population before all traffic. + +Retire the flag once the change has held: flip the default, then delete the setting and +its branch. A temporary flag left in place becomes permanent configuration surface. + +### What to watch after the flip + +Two regression signals, both checked before the win is: + +- **`unexpected_origin_304` abandonment rate.** That reason + (`publisher.rs:2894-2916`, + emitted via `emit_abandoned_auction` at `:2360`) exists precisely because the ad-stack + path refuses cached and conditional origin responses. Re-enabling the cache is what + could revive it. **Any non-zero rate is a rollback signal** — it means a 304 is reaching + TS that the conditional-header strip was supposed to make impossible. +- **Representation mixing.** Spot-check that HTML navigations still return HTML and RSC + fetches still return `text/x-component`. A mismatch means the `Vary` risk materialized + despite a PASS verdict. Roll back immediately; this is cache poisoning, not a + performance regression. + +**Why it is safe in principle.** The conditional-header strip runs 34 lines earlier +under the same gate (`publisher.rs:2832-2836`, +which also strips `Range`/`If-Range`), so the request already reaches the cache +unconditional and a HIT returns a full body. [The 304-prevention design](./2026-07-22-ssat-root-document-304-prevention-design.md) +added the bypass as belt-and-braces and listed the TTFB cost under its own Risks. The +strip alone satisfies its invariant. + +**But it carries a risk that design never considered — and this is the blocking +precondition.** RSC fetches are not navigations +(`is_navigation_request` +requires `Sec-Fetch-Dest: document`), so they never set the bypass and **already flow +through the readthrough cache**, while HTML navigations are `PASS`. Removing the bypass +puts both representations under one cache key. #1009 states the origin varies on +`rsc`, `next-router-*`, and a publisher-specific experiment header — if that variance is +not declared via `Vary`, the +cache can serve a flight payload to an HTML navigation. + +The classification is also not airtight: `is_navigation_request` falls back to the +`Accept` header when Fetch Metadata is absent, and its own comment warns _"this path is +weaker — `fetch()` can set Accept: text/html"_ +(`http_util.rs:84-88`). + +**A FAIL is not merely a Stage 0 blocker — it is a live production defect.** RSC fetches +already transit the read-through cache today, because they never set the bypass. If the +origin varies on `Next-Router-*` without declaring it, TS is cross-serving RSC variants +right now. On a FAIL, file that immediately and treat "ask the origin to declare `Vary`" +as urgent rather than as the cheaper of two options. + +**The `Vary` check is necessary but not sufficient.** Turning the read-through cache on +for HTML navigations exposes three things a representation check does not cover, and all +three are a larger class than the RSC split: + +- **Client `Cookie`.** TS forwards client cookies to origin unchanged — there is no + `COOKIE` strip on the publisher path. Any cookie-personalized HTML (logged-in state, + paywall meter, publisher-side A/B assignment) becomes cross-servable unless the origin + declares `Vary: Cookie` or marks those responses private. +- **Origin `Set-Cookie`.** If the origin emits `Set-Cookie` alongside a shared-cacheable + `Cache-Control`, the read-through cache can replay one visitor's cookie to the next. + TS's own privacy net downgrades **TS's** response — it runs after the cache has already + stored the origin's. +- **`Authorization`.** #1009 describes a basic-auth-gated deployment. Responses to + authorized requests entering a shared cache needs its own check. + +So Step A must capture `Cache-Control` and `Set-Cookie` too, and repeat each request with +and without a session cookie. Same minutes of work; closes the bigger hole. + +**Two effort branches, and Step A's `Vary` result decides which** — note this selects the +_shape_ of Stage 0, while the `FINAL PASS` conditions decide _whether it ships at all_: + +| Step A result | Stage 0 is… | Effort | +| ---------------------- | --------------------------------------------- | ------ | +| Origin declares `Vary` | the flag, its tests, then a config push | 1–2 d | +| Origin does **not** | a TS-side cache-key discriminator — a feature | 4–8 d | + +The discriminator is the safer design either way, because it keys on the headers that +actually distinguish the representations rather than on the navigation classification. + +**Two benefits beyond TTFB, worth stating to a publisher:** + +- **Origin load drops.** The 304-prevention design explicitly accepted _"increasing + origin load"_ as a cost. This reverses it. +- **`stale-if-error` becomes reachable.** Under `Pass` an origin outage is a hard + failure. This needs a decision rather than a default: stale HTML carries stale slot + markup, and whether that beats an error is a product call. + +--- + +## 5. The trap in the deferred work — read this before scheduling Stages 1–2 + +The hold is load-bearing for something other than latency. The invariant is: + +> `ad_bids_state` must be `Some(..)` when `lol_html` processes the `` end tag. + +The end-tag handler (`html_processor.rs:381-395`) +locks that mutex once and falls back to `build_empty_bids_script()` on `None`. + +**Removing the hold without relocating collection renders a normal page with +`tsjs.bids = {}` and no server-side ads** — no error, no non-2xx, no ERROR log. On +Axum, Cloudflare, and Spin the loss is fully silent: +`publisher.rs:2248` holds a +bare `Option` with no guard, so not even a drop warning fires. **The +SSPs are billed regardless.** + +This is why Stage 2 is gated on three companions and a production soak, and why slot +fill cannot be the canary — see [§7](#7-deferred-work-specified-not-scheduled). + +--- + +## 6. The analysis + +### 6.1 Corrections to #1009's premises + +| # | #1009 states | Verified against `cfb98f4` | +| --- | -------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | Two per-user injection seams | Partly. `tsjs.adSlots` **content** is per-URL — `build_slot_json` emits config- and path-derived fields only. But its **presence** is gated on `should_run_ad_stack` (consent, bot, prefetch, kill switch), so it is request-dependent and **must not live in a shared template**. See §6.7. | +| 2 | Identity off-inline is a prerequisite | Privacy net is cookie-gated and returning navs set no cookie (`ec/finalize.rs:86-94`). **First-visit only** — but note the corollary: because returning navigations set no cookie, that net never fires for them and is **not** a backstop against shared-caching a per-user response. | +| 3 | Stamp at `:2882-2888` | `:2945-2963`, `private, no-store`, also removing `ETag`/`Last-Modified`/four CDN headers. **Seven headers.** | +| 4 | Three cacheability killers | Two more: `bypass_cache` and the `304→502 guard`. **The bypass is the cost.** | +| 5 | Two `!Send` pipelines is the hard part | `?Send` already pervasive. **Not the obstacle.** | +| 6 | Goal: root as a shared Fastly HIT | Nothing caches TS's own response on Compute; the A/B's `x-cache` is the **backend readthrough** cache. **Reframes the goal.** | + +Rows on #1009's `esi` compatibility check (holds), its drifted line numbers, and its +two broken `#1`/`#3` cross-references are in [Appendix A](#appendix-a--full-1009-correction-table). + +**Credit where due.** #1009 names the hold as blocker 1 and states it correctly. What +changes here is its _causal weight_. Likewise, #1009's own observation that TS _"shifts +the auction cost from client-side to server-side rather than adding new work"_ is the +argument for client-fill, which the issue then declines in favour of ESI. + +### 6.2 What the hold actually costs + +**An earlier draft of this section claimed a stronger argument than the code supports. +It was wrong, and the correction matters.** + +The hold does not key off `lol_html` at all. `BodyCloseHoldBuffer::push` +(`publisher.rs:2190-2202`) +scans the **decoded origin input** for ` Dispatch precedes the origin fetch, so the hold costs `max(0, A − T)`, where `A` is the +> auction collect duration and `T` is origin TTFB plus body transfer up to the `` +> byte. Since `` sits at the end of a document, `T` is close to the full download. + +`A` is bounded by `auction_timeout_ms`, resolved as +`creative_opportunities.auction_timeout_ms` falling back to `auction.timeout_ms` +(`publisher.rs:2680-2684`) +— check the resolution order against your own config rather than trusting a number; the +shipped example sets different values at each level. + +**This is a claim requiring measurement, not a proof.** §3 Step C measures the hold's +cost directly rather than inferring it. + +A finding that does survive, and belongs with [the ceiling](#64-the-ceiling): because +`HtmlWithPostProcessing` withholds all output until the final chunk, the streaming-prefix +design at `publisher.rs:1343-1348` +— whose comment promises "the client receives the document up to `` while the +auction rides alongside transfer" — is **inert on a Next.js publisher**. Every +`step.ready` yields empty bytes. That comment is misleading on exactly the publisher +under discussion. + +### 6.3 The quantity nobody has measured + +Write the fetch time under `Pass` as `O`. Recovery depends on it, and it has never been +captured. #1009's timings cannot supply it: they compare a POP hit against a +shield-served fetch, both of which are _cached_ paths, whereas `CacheOverride::Pass` +bypasses TS's read-through cache and its shield. + +Note `Pass` bypasses **TS's** caches only. It has no authority over any CDN the publisher +runs in front of their own origin — and #1009's `x-cache: MISS, MISS` on the TS-on arm +hints one may exist. So `O` may not be origin build time at all. Since `O` is the single +quantity this model depends on, that ambiguity is worth resolving in Step C rather than +assuming. + +What follows from code alone, without any number: + +| Configuration | Long pole after the change | Recovery | +| ------------------- | -------------------------- | ------------------------------ | +| Hold removal only | origin (still `PASS`) | **none** | +| Bypass removal only | the auction budget | partial — the auction surfaces | +| **Both** | the rewrite | **the full available win** | + +That ordering is what the staging rests on, and it is measurement-independent. The +magnitude of each row is not, and §3 Step C supplies it. + +### 6.4 The ceiling + +#1009 targets "approach the TS-off warm numbers." **Unreachable, structurally.** Those +numbers are TS-off _streaming_ a POP HIT. TS buffers the whole document before emitting +a byte (16 MB cap), so its floor is `full origin body download + full rewrite` — above a +streamed hit by construction, whatever the timings turn out to be. Set the target from +Step C's measured rewrite cost rather than from the TS-off baseline. Going below the +floor requires true origin streaming (#849), out of scope. A non-Next.js publisher with +no post-processor takes the streaming path and would see a lower floor. + +### 6.5 Confidence + +**High on the structural claims.** §6.2's argument, the bypass forcing a cache miss, the +the silent-empty-bids failure mode, the geo and `Vary` blockers, +and the fill-canary blindness are all read directly out of the code at `cfb98f4`. Anyone +can check them without running anything. + +**None on magnitude.** `O` is unmeasured and the rewrite cost is unmeasured. This +document does not estimate them, and no figure in it should be quoted as one. + +Worth stating plainly: #1009 reached the opposite causal conclusion from a small sample. +That is a caution about small samples generally, not only about that one — which is why +§3 Step C specifies the measurement rather than this document supplying a substitute +for it. + +### 6.6 The ESI pipeline, corrected + +An earlier revision of this document said "order esi → lol*html, never the reverse." +That is backwards. `lol_html` is what \_emits* the ESI include tags; ESI cannot process +tags that do not exist yet. The correct order: + +``` +origin → lol_html transform → fastly::cache::core → finalize headers → stream esi assembly → client + (one unconditional marker (shared template, (EC cookie, geo, (per request, + at the body-close seam; surrogate-keyed, unconditional fetch the + the head seam is NOT a TS-chosen TTL) private/no-store) fragment) + hole — adSlots presence + is request-gated, §6.7) nothing may change + after this point +``` + +The push/pull mismatch that the earlier revision treated as a blocker is real but +irrelevant: `lol_html` pushes, `esi` pulls, and **the cache is the buffer between them**. +That is not an obstacle to the two-stage design — it _is_ the two-stage design, which is +what #1009 proposed in the first place. + +Mechanism, all present in the pinned `fastly` 0.12.1: + +| Need | API | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| Store the template | `cache::core::insert(key, max_age).execute() -> StreamingBody` | +| Read it back | `cache::core::lookup(key)` → `found()` | +| Avoid a thundering herd | `cache::core::Transaction` — `must_insert()` / `must_insert_or_update()` | +| Invalidate **C2 only** | `InsertBuilder::surrogate_keys([...])` (Core Cache) + `fastly::http::purge::purge_surrogate_key`. Does **not** key C1 — see the row below. | +| Invalidate C1 | Origin-supplied surrogate keys, or the HTTP cache's own surrogate-key surface. Not the Core Cache API. | + +Purge runs **inside Compute**. The management-API token scope cited under +[Stage 4](#7-deferred-work-specified-not-scheduled) governs a different surface and does +not gate this. + +**Three caches, kept distinct.** Conflating them is what produced the original error: + +1. **Origin read-through** — raw origin bytes. What Stage 0 turns back on. +2. **Shared transformed template** — post-`lol_html`, pre-ESI, no per-user data. The ESI + target, and new. +3. **Assembled-response delivery cache** — the final per-user output. **Must never + exist.** Nothing in this document or the spike proposes one. + +**Validation constraint.** Viceroy 0.17 cannot exercise the customized read-through hooks +end to end. Unit tests can cover the transform and the security properties; MISS / HIT / +stale / shielding behaviour must run against a real Fastly test service. + +--- + +### 6.7 What may and may not live in a shared template + +A correction to §6.1 row 1, and the constraint that governs any shared-template design. + +The original framing — "`adSlots` is per-URL, so there is one per-user hole, not two" — +is half right and dangerously so. `build_slot_json` really does emit only config- and +path-derived fields. But whether the script is emitted **at all** is gated on +`should_run_ad_stack` (`publisher.rs:2920-2927`), which is +`is_get && is_navigation && !is_prefetch && !is_bot && has_matched_slots && +consent_allows_auction && auction_enabled`. + +So the _content_ is per-URL and the _presence_ is per-request. A shared object filled by +the first request would freeze that request's consent decision, bot classification, +prefetch status, and kill-switch state for every later reader. A consent-denied fill +serves a no-ads template to consenting users; a consenting fill serves ad markup to +someone who refused. + +**The rule for anything cached and shared:** + +| May live in the template | Must live in the per-request fragment | +| --------------------------------------- | ---------------------------------------------- | +| tsjs bundle script tag (content-hashed) | `tsjs.adSlots` — presence is request-gated | +| URL rewrites (per-host, in the key) | `tsjs.bids` | +| | GPT diagnostics bootstrap (cookie/query-gated) | +| | Integration head-inserts (request-scoped) | + +The test that catches this class is **byte-identity of the template across requests +differing in consent, bot classification, and prefetch status** — not an absence-of- +per-user-values scan, which the broken design would have passed. + +This applies to any shared-template work, ESI or client-fill alike. The +[spike plan](./2026-08-10-1009-esi-validation-spike.md) implements it. + +--- + +## 7. Deferred work, specified not scheduled + +**The full sequence, in one place.** Stage 0 is specified in [§4](#4-stage-0--the-only-build-item-recommended-now) +rather than repeated here; everything below it is deferred. + +| Stage | What | Status | +| ----- | ----------------------------------------------- | ---------------------------------------------- | +| **0** | Operator flag disabling the origin cache bypass | Recommended now. Gated on a `FINAL PASS`. §4. | +| 1 | Bid delivery off the response body | Deferred behind the correctness defects | +| 2 | Delete the `` hold | Deferred; one-way, needs a Stage 1 soak | +| 3a | Browser caching (`private, max-age` + `ETag`) | Specified, low risk, unscheduled | +| 3b | Shared cacheability | Blocked on geo suppression, `Vary`, and Step B | +| 4 | Purge wiring | Prerequisite for any TS-owned cache | + +**ESI is not a stage here.** It was Stage 5 in an earlier revision, queued behind the +rest. It no longer queues: it is feasible on the pinned SDK and is decided by +[its own spike plan](./2026-08-10-1009-esi-validation-spike.md), which runs +independently of Stages 1–4. The shared template cache it needs is `fastly::cache::core` +([§6.6](#66-the-esi-pipeline-corrected)), not a new service. + +Lower detail below is deliberate. Full specifications are in the appendices. + +**Stage 1 — bid delivery off the response body.** The client fetches `/_ts/page-bids` at +navigation generation 0. Endpoint, same-origin gate, wire shape, and client consumer +already exist. Three decisions must be made before planning: the `slots: []` precedence +rule when head-open already injected a non-empty `ts.adSlots`; the new terminal-event +emission point; and whether the dispatch/collect split survives at all. Plumbing detail +in [Appendix B](#appendix-b--stage-1-plumbing-condensed). Estimated 8–13 d, low-to-medium +confidence, uncertainty concentrated client-side. + +Three companions are mandatory, not optional: **suppress the server bids script +entirely** (not an empty one), **fail loud** (the end-tag handler takes bids by value so +a missing auction is a compile error), and **relocate telemetry** (navigation +`Completed` rows are emitted only from the collect functions, and the `ts-debug` dump +rides the same string). Behaviour change to accept: under client-fill the auction runs +only if the browser executes the fetch, so bots and JS-disabled clients stop triggering +server-side auctions — revenue-relevant, sign unknown. + +**Stage 2 — delete the hold.** 5–8 d. **Rollback is one-way**: it deletes the hold, the +dispatch/collect split, and twelve tests, so the only revert is a release. Ships only +after Stage 1 has run flag-on in production for a window defined _before_ Stage 1 +starts, with TS-attributed renders flat and `auction_events_raw` navigation rows intact. +Secondary wins: removes the duplicated per-codec decoder/encoder wiring, six compression +imports, and the non-parser-context `` runs _all_ attempts and concatenates every non-failed output** — not + first-success-wins, so primary/fallback pairs render both. Least obvious behaviour in + the crate. +- Single include, not per-slot: the auction is one operation producing all slots' bids. + +--- + +## Appendix F — deferred open items (condensed) + +Implementation-level, for unscheduled work only. Decisions needing a human are in +[§9](#9-decisions-needed-from-this-review). + +Should `collect_non_html_auction` (`publisher.rs:2388`) go with the hold or stay? Is +`body_close_hold_loop_stream` (`:2109`, no production caller) safe to delete, or is the +buffered-adapter streaming cutover (#495) still live? Does hidden-tab rAF behaviour +interact badly with a bids timeout? What are Fastly's pending-request semantics when a +`DispatchedAuction` drops mid-flight? Does `stale-if-error` on a cached root serve +acceptable content given stale slot markup? And the googletag shim discards listeners +queued before it loads (#1009 Part 1) — not filed, should be. + +--- + +## Appendix G — code-grounded seams + +All pinned to `cfb98f4`. + +| Concern | Location | +| --------------------------------------------------------- | ------------------------------------------------------------------------------- | +| Eligibility decision | `publisher.rs:2651`, `:2660` | +| `is_navigation_request` | `http_util.rs:73-98` | +| Auction dispatch (pre-origin, non-blocking) | `publisher.rs:2698-2760` | +| Auction overlap intent | `auction/orchestrator.rs:950-952` | +| Auction timeout resolution | `publisher.rs:2680-2684` (creative_opportunities, else auction.timeout_ms) | +| Conditional/range header strip | `publisher.rs:2832-2836` | +| Origin cache bypass | `publisher.rs:2866-2868` | +| Origin 304 → 502 guard | `publisher.rs:2894-2916` | +| `adSlots` build (content per-URL, presence request-gated) | `publisher.rs:2920`, `:3558-3577`, `:3501-3525` | +| Uncacheable stamp | `publisher.rs:2945-2963` | +| `` hold — sync / async / Fastly lazy | `publisher.rs:2235` / `:2109` / `:1318-1390` | +| Hold buffer | `publisher.rs:2177-2218` | +| Auction collect (HTML / non-HTML) | `publisher.rs:2431` (emits `:2456`) / `:2388` (emits `:2410`) | +| Abandonment emitter | `publisher.rs:2360` | +| Bids script build | `publisher.rs:3438-3491` | +| `/_ts/page-bids` | `publisher.rs:3611`, handler `:3723`, auction `:3903` | +| Injection seams (head / body-close) | `html_processor.rs:310-363` / `:381-395` | +| Post-processor buffering | `html_processor.rs:62-94` | +| Next.js post-processor registration | `integrations/nextjs/mod.rs:107` | +| Max buffered body (16 MB) | `settings.rs:77-79` | +| EC cookie issuance policy | `ec/finalize.rs:86-107` | +| Cookie-privacy net | `response_privacy.rs:20-61` | +| Geo response headers | `adapter-fastly/src/middleware.rs:194-200` | +| Cacheable-header precedent | `http_util.rs:294-311` | +| Management token lacks purge (wrong surface) | `adapter-fastly/src/management_api.rs:12` | +| In-process purge / surrogate keys | `fastly` 0.12.1 `cache::core`, `http::purge::purge_surrogate_key` | +| Client initial-ad gate | `js/lib/src/integrations/gpt/index.ts:536-555` | +| `adInit` bid application | `js/lib/src/integrations/gpt/index.ts:566`, `:652`, `:657-661` | +| Client SPA auction hook | `js/lib/src/integrations/gpt/index.ts:806`, `:859`, `:892-949` | +| Prior design that introduced the killers | `docs/superpowers/specs/2026-07-22-ssat-root-document-304-prevention-design.md` | diff --git a/docs/superpowers/archive/2026-08-10-1009-esi-validation-spike.md b/docs/superpowers/archive/2026-08-10-1009-esi-validation-spike.md new file mode 100644 index 000000000..1d83665ca --- /dev/null +++ b/docs/superpowers/archive/2026-08-10-1009-esi-validation-spike.md @@ -0,0 +1,967 @@ +# #1009 ESI Validation Spike + +> **HISTORICAL SPIKE — DO NOT IMPLEMENT.** This document records the investigation, +> including executable ESI tags, parser/subrequests, and a client-fill arm that were all +> removed. Every unchecked item below is historical, not remaining work. The accepted +> implementation keeps the public `esi` spelling but uses Fastly C2 plus exact byte-seam +> assembly. See +> [the merge-hardening design](../specs/2026-08-12-1009-esi-merge-hardening-design.md) and +> [implementation plan](./2026-08-12-1009-esi-merge-hardening.md). + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development +> (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps +> use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Decide #1009 on evidence. Build a shared-template pipeline behind a flag, run +ESI and client-fill against it, and produce a decision record that either adopts ESI, +adopts client-fill, or rejects both — with the Fastly-only maintenance cost priced in. + +**Architecture:** +`origin → lol_html transform → fastly::cache::core → finalize headers → stream assembly`. + +Headers finalize **before** assembly, not after — streaming responses on this adapter +commit headers first and then pipe chunks, so nothing can be set once assembly starts. + +The transform emits **one unconditional marker at the body-close seam**. Not two: the +head seam is not a template hole, because `tsjs.adSlots` presence is request-gated +(Task 3 Step 2). The cached object is a shared template with no per-user bytes and no +request-dependent decisions. Assembly is either the `esi` crate (edge) or a client fetch +(browser), selected per request by the arm allocator so both are measured on one build. + +**Tech Stack:** Rust 2024, `wasm32-wasip1`, `fastly` 0.12.1 (`cache::core`, `http::purge`), +`esi` 0.7, `lol_html`, a real Fastly test service for cache behaviour. + +**Spec:** `docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md` — +read the 2026-08-10 correction at the top and +[§6.6](./2026-08-08-esi-cacheable-root-validation-design.md#66-the-esi-pipeline-corrected) +before writing any code. + +**Control:** [the Stage 0 plan](./2026-08-08-1009-measurement-and-stage-0.md). Its +instrumentation and its bypass flag are prerequisites — this plan compares against them +and does not duplicate them. + +--- + +## Why this plan exists + +An earlier revision of the spec concluded ESI was structurally impossible. It was wrong: +`fastly::cache::core` provides the cache boundary natively, and purge runs inside Compute. +That correction reopens #1009 as an empirical question, and this plan is how it gets +answered. + +**What is genuinely uncertain**, and what each arm is for: + +1. Does a shared template plus per-request assembly beat today's inline path enough to + matter? +2. Does **edge** assembly (ESI) beat **client** assembly (a fetch) by enough to justify a + Fastly-only rendering path that must be maintained alongside the portable one? +3. Can per-user leakage be excluded across cold MISS, warm HIT, stale revalidation, + transform failure, and fragment failure? + +Question 3 is a gate, not a metric. A win on 1 and 2 with a failure on 3 is a rejection. + +## Three caches, never conflated + +The original error came from treating these as one thing. Every task below names which it +means. + +| # | Cache | Contents | Status | +| --- | --------------------------------- | ----------------------------- | ----------------------------------- | +| C1 | Origin read-through | raw origin bytes | Exists. Stage 0 turns it back on. | +| C2 | Shared transformed template | post-`lol_html`, pre-assembly | **New.** What this plan builds. | +| C3 | Assembled-response delivery cache | final per-user output | **Must never exist.** Not proposed. | + +If a task appears to require C3, stop — that is the leakage failure mode, not a design +option. + +## Arms + +Five, but only four are treatable as equivalent. + +| Arm | Root | Bids | Notes | +| ------- | ----------------------- | ---------------- | ---------------------------------------------------------- | +| **A0** | inline, C1 bypassed | inline `` | Today. The baseline. | +| **A1** | inline, C1 on | inline `` | Stage 0. Isolates the bypass from the template change. | +| **A2** | shared template from C2 | client fetch | Portable. Works on all four adapters. | +| **A3** | shared template from C2 | ESI at the edge | Fastly-only. The thing #1009 proposed. | +| **REF** | origin direct, TS off | publisher's own | **Reference, not an arm.** Different work, not comparable. | + +A0→A1 measures the bypass. A1→A2 measures the template split. A2→A3 measures edge versus +client assembly — **that difference is the entire case for ESI**, and it is the number +this plan exists to produce. + +**Do not compare A2 and A3 on root TTFB.** They serve the same C2 template, so their root +timings should be near-identical by construction; a null result there proves nothing. +ESI's claimed advantage is that bids arrive without a client round-trip, so measure: +**bids-ready time**, **`adInit` fire time**, and **first TS-attributed creative paint**. +Root TTFB stays as a guard that the template path did not regress, not as the comparison. + +REF is included because #1009 anchors on it, and excluded from pass/fail because TS-off +does no auction and no injection. Comparing against it measures the feature's existence, +not its implementation. + +--- + +## Task order and dependencies + +``` +Task 1 (esi compiles) ── DONE, PASS ──┐ + ├──> Task 3 (C2 cache) ─┬──> Task 4 (A2 client-fill) +Stage 0 plan (flag + instrumentation) ┘ ├──> Task 5 (A3 ESI) + └──> Task 6 (safety gates) + │ + Task 2 (real service) ─────────────────────────────┴──> Task 7 (decision) +``` + +**Task 2 is not a blocker on Tasks 3–6.** Everything those tasks need is exercisable under +Viceroy 0.17 — verified, see Task 2. The real service is required only for the +measurements Task 7 decides on, so provision it once there is something worth measuring. + +Task 6 runs against every arm, not once at the end. + +--- + +## Task 1: Confirm `esi` 0.7 builds on this toolchain + +Cheapest possible falsification. Do this before anything else. + +**Files:** `crates/trusted-server-adapter-fastly/Cargo.toml` + +- [ ] **Step 1: Add the dependency** + +```bash +cargo add esi@0.7 --package trusted-server-adapter-fastly +``` + +It belongs in the **Fastly adapter**, never in `trusted-server-core` — the crate is +hard-bound to `fastly::{Request, Response, Backend}` and core must stay portable. + +- [ ] **Step 2: Check it compiles for the real target** + +```bash +cargo check-fastly +``` + +Expected: clean. The crate declares edition 2021 with no `rust-version`, and pulls recent +`rand` and `nom`, so this is a genuine question on Rust 1.95.0 / `wasm32-wasip1`. + +- [ ] **Step 3: Check no shared dependency was forced to move** + +```bash +git diff --stat Cargo.lock +cargo check --manifest-path crates/trusted-server-integration-tests/Cargo.toml --tests \ + --target "$(rustc -vV | sed -n 's/^host: //p')" +``` + +**Correction, verified 2026-08-10:** an earlier revision of this step warned about a +desync between the root `Cargo.lock` and `crates/trusted-server-integration-tests/Cargo.lock`. +**That second lockfile does not exist** — the crate is a workspace member (root +`Cargo.toml:10`) and shares the root lockfile. The hazard cannot arise in that form. + +What does matter is whether adding `esi` forces an **existing** shared dependency to a new +version, since `regex`, `bytes`, and `log` are used across the workspace. Adding a new +major that coexists is harmless; moving an existing one is not. If one moves, fix with a +targeted `cargo update -p --precise ` — **never a full update**. + +**Already run and recorded** in [the findings](./2026-08-08-1009-measurement-findings.md): +no existing shared dependency moved. + +- [ ] **Step 4: Record and commit, or stop** + +**Task 1 is complete — verdict PASS, recorded 2026-08-10.** `esi` 0.7.1 compiles clean on +Rust 1.95.0 / `wasm32-wasip1`, all six clippy targets pass, and no existing shared +dependency moved. See [the findings](./2026-08-08-1009-measurement-findings.md). + +Had Step 2 failed, this plan would have stopped here with #1009 answered "not on this +toolchain." It did not. + +```bash +git add crates/trusted-server-adapter-fastly/Cargo.toml Cargo.lock +git commit -m "Add the esi crate to the Fastly adapter for the #1009 validation spike" +``` + +--- + +## Task 2: Local validation first, real service only for what needs it + +**Verified 2026-08-10 under Viceroy 0.17: the entire Core Cache surface this spike uses +works locally.** A probe exercised `cache::core::insert`, `lookup`, `finish`, `to_stream`, +and — the shape Task 3 Step 4 actually specifies — `Transaction::lookup`, +`must_insert_or_update`, `insert(...).surrogate_keys(...).execute_and_stream_back()`, and +hit-after-insert semantics. All passed. Recorded in +[the findings](./2026-08-08-1009-measurement-findings.md). + +That reorders this plan. An earlier revision made provisioning a Fastly service Task 2 and +a blocker on everything after it. It is not a blocker: **almost all of the correctness and +safety work is local**, and only the numbers and the cache topology need real +infrastructure. + +| Work | Where | +| ------------------------------------------------------------ | ------------ | +| C2 insert / lookup / transaction logic (Task 3) | **Local** | +| The `lol_html` transform and template byte-identity (Task 3) | **Local** | +| ESI assembly — the crate is pure Rust over `BufRead`/`Write` | **Local** | +| DCA off, dispatcher allowlist, injection refusal (Task 5) | **Local** | +| Fragment-failure degradation (Task 5) | **Local** | +| Header-finalization ordering, no-C3 assertions (Task 6) | **Local** | +| Cross-user leakage / request-neutrality gates (Task 6) | **Local** | +| Shielding behaviour | Real service | +| POP-level cache tiering (`x-cache`, `hit-state`, `age`) | Real service | +| Request collapsing under genuine concurrency | Real service | +| Stale revalidation timing at the edge | Real service | +| **Every performance number in Task 7's decision rule** | Real service | + +**So: build and prove correctness locally through Tasks 3, 5, and 6 before provisioning +anything.** If the design is wrong or leaks, that surfaces locally for free, and the +service is only needed once there is something worth measuring. + +Two caveats on the local scope. Viceroy is a single instance, so a passing `Transaction` +test proves the API works, **not** that collapsing behaves correctly under load. And local +timings are meaningless for the decision — do not let a fast local run substitute for +Task 7 evidence. + +### When the real service is needed + +- [ ] **Step 1: Provision it — after local correctness passes, not before** + +Separate from production. Confirm and record: whether the publisher backend is +**shielded**, and whether any Delivery service fronts the Compute service. Both change +what the numbers mean. + +```bash +fastly service list +fastly backend list --service-id --version latest +``` + +The shielding answer also settles an open question from the Stage 0 findings: #1009's +off-TS win came from a shield HIT, so whether the test service has one determines whether +its numbers transfer to production at all. + +- [ ] **Step 2: Extend the harness for lineage, not just correlation** + +The existing tester-cookie A/B has no way to join server timings to browser timings. A +root-only request ID is not enough either: under A3 the auction happens in a **fragment +subrequest**, so a root ID never reaches the auction telemetry. + +Propagate a **lineage ID plus the experiment arm** through the whole chain: + +``` +root request → C2 lookup → fragment subrequest → auction telemetry → browser render event +``` + +Generated at TS entry, forwarded into the fragment request, attached to the +`auction_events_raw` row, echoed as `x-ts-request-id`, and exposed to the browser harness +so render events carry it. Every timing log line includes both fields. + +Without this the experiment cannot join hold time, origin time, auction telemetry, browser +TTFB, and render outcome for the same pageview. **That is the difference between an +experiment and a pile of numbers.** + +- [ ] **Step 3: Capture C1 and C2 status separately** + +`x-cache`, `hit-state`, and `age` describe the **HTTP read-through cache (C1)**. They say +nothing about the **transformed-template cache (C2)**, which is a `cache::core` object +with no HTTP semantics. Recording only the former and calling it "cache status" would +attribute C2 hits and misses to the wrong tier. + +Emit both: the C1 headers as-is, plus an explicit `x-ts-c2` field carrying HIT / MISS / +STALE / BYPASS from the transaction outcome. Record the serving POP alongside. A median +that mixes cold-MISS and warm-HIT requests is meaningless, and arms cannot be compared +unless the mix is known — per tier. + +- [ ] **Step 4: Build a request-scoped arm allocator** + +`AssemblyMode` as specified in Task 3 is a **global** setting, but the sample plan below +requires randomized, non-sequential allocation. A global flip gives sequential blocks +instead, which confounds arm with time of day, cache warmth, and traffic mix. + +Allocate per request: hash the lineage ID into buckets, or key off the tester cookie. +The global setting stays as the kill switch and as the way to force a single arm; the +allocator is what the experiment actually uses. Record the assigned arm on every log line +and every telemetry row. + +- [ ] **Step 5: Define the sample plan before collecting anything** + +Write all of this into the findings document **before** the first measurement, and treat +it as fixed: + +| Element | What to state | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| Allocation | Requests per arm per route, and how arms are assigned | +| Randomization | Randomized or blocked by route and cache state — not sequential runs | +| Pilot variance | A small pilot to estimate variance, before sizing the real run | +| MDE and power | The smallest difference worth detecting, and the N that detects it | +| CI method | Which interval, computed how | +| Warmup and carryover | How cold MISS is forced, how warm HIT is confirmed, and how one arm's cache state is prevented from contaminating the next | + +Rationale: this whole effort exists because #1009 drew a causal conclusion from N=4 that +did not survive contact with the code. Repeating that with more arms and no power +calculation would be worse, not better — it would look rigorous while being equally +unfalsifiable. + +--- + +## Task 3: Build C2 — the shared transformed-template cache + +The core of the spike. Behind a flag, default off. + +**Files:** + +- `crates/trusted-server-core/src/publisher.rs` — emit **one** unconditional marker at the body-close seam (see Step 2; the head seam is not a template hole) +- `crates/trusted-server-core/src/settings.rs` — the mode flag +- `crates/trusted-server-adapter-fastly/src/` — the `cache::core` read/write + +- [ ] **Step 1: Add the assembly-mode setting** + +```rust +/// How per-user ad state reaches the page. +/// +/// `Inline` is today's behaviour: bids injected before ``, root uncacheable. +/// `ClientFill` and `Esi` both serve a shared template from the transformed-template +/// cache and fill the holes afterwards. Spike-only — remove with the spike. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AssemblyMode { + #[default] + Inline, + ClientFill, + Esi, +} +``` + +Default `Inline` so the flag is a no-op until set. Note the hazards the Stage 0 plan +already documents: `Settings` carries `#[serde(deny_unknown_fields)]`, `ts config push` is +typed, and `Publisher` has a hand-written `Default` plus eight exhaustive test literals +and a live doctest. + +- [ ] **Step 2: Make the template strictly request-neutral** + +**The obvious design is wrong and would leak.** An earlier draft kept `tsjs.adSlots` in +the shared template on the grounds that it is per-URL. Its _content_ is per-URL; its +_presence_ is not. It is gated on `should_run_ad_stack` (`publisher.rs:2920-2927`), which +is `is_get && is_navigation && !is_prefetch && !is_bot && has_matched_slots && +consent_allows_auction && auction_enabled`. + +So the first request to fill C2 would freeze **its own** consent decision, bot +classification, prefetch status, and kill-switch state into an object every later visitor +reads. A consent-denied first fill serves a no-ads template to consenting users; a +consenting first fill serves ad markup to a user who refused. + +**Rule: the template contains an unconditional inert placeholder and nothing else.** + +| Element | Where it lives | +| ------------------------- | -------------------------------------------------- | +| tsjs bundle script tag | Template — content-hashed, genuinely per-URL | +| URL rewrites | Template — per-host, in the cache key | +| `tsjs.adSlots` | **Fragment** — its presence is request-dependent | +| `tsjs.bids` | **Fragment** | +| GPT diagnostics bootstrap | **Fragment** — gated on a per-request cookie/query | + +Emit **one** unconditional marker at the body-close seam, identical on every request that +reaches the transform. Under `Esi` it is an executable ESI include tag; under +`ClientFill` it is nothing at all, with the client fetching unprompted. + +- [ ] **Step 3: Bypass C2 for anything that must not be shared** + +`cache::core` is not an HTTP cache — it will happily store whatever you hand it. Nothing +rejects private or authenticated responses for you. Refuse to insert when **any** holds: + +- The origin response carries `Set-Cookie`. +- The origin response is `private`, `no-store`, or `no-cache`. +- The request carried `Authorization`. +- The response is not 200 with an HTML content type. +- DataDome's request filter replaced the document. + +Audit every request-dependent rewrite before declaring the template neutral — the +integration head-inserts and the GPT-diagnostics bootstrap are both request-scoped and +must not reach C2. + +**Assert it, do not assume it.** A unit test over the transform output must fail on any +of: a bid value, an EC ID, a consent string, a geo value, a diagnostics bootstrap, or a +`Set-Cookie`. Then a second test must assert the template is **byte-identical** for two +requests differing in consent, bot classification, and prefetch status. That second test +is the one that catches this class of bug; the first would have passed on the broken +design. + +- [ ] **Step 4: Write and read C2 — with the real API** + +The builder is move-based and the insert and read handles are different objects. Naïve +code does not compile: + +```rust +// WRONG — surrogate_keys consumes the builder and returns it; this discards the +// return value and then uses a moved binding. And execute() gives a WRITE stream, +// so there is nothing to read back from it. +let mut insert = cache::core::insert(key, ttl); +insert.surrogate_keys(["ts-template"]); +let body = insert.execute()?; +``` + +Correct shape, using a transaction so a cold cache under load transforms once: + +```rust +use fastly::cache::core::{Transaction, CacheKey}; + +let tx = Transaction::lookup(CacheKey::from(key_bytes)).execute()?; + +// Order matters: a STALE entry sets BOTH found() and must_insert_or_update(). +// Testing found() first would serve the stale bytes and silently never fulfil the +// update obligation, leaving every concurrent waiter blocked until timeout. +let template: Body = if tx.must_insert_or_update() { + // Fetch and prepare BEFORE consuming `tx`. After `insert()` the transaction is + // gone and `cancel_insert_or_update()` is unreachable, so anything that can fail + // and does not need the writer belongs here. + let origin = match fetch_and_prepare_origin() { + Ok(origin) => origin, + Err(e) => { + tx.cancel_insert_or_update()?; // releases the obligation to a waiter + return fallback_uncached(e); + } + }; + + // `Transaction::insert(self)` consumes `tx` from this line on. + let (mut writer, found) = tx + .insert(template_ttl) + .surrogate_keys(["ts-template", &url_surrogate_key]) // chained, not discarded + .user_metadata(metadata_envelope) + .execute_and_stream_back()?; + + match stream_lol_html_output(origin, &mut writer) { + Ok(()) => { + writer.finish()?; // REQUIRED, and consumes `writer` + found.to_stream()? // fallible; there is no `to_body()` + } + Err(e) => { + // Also consumes `writer`, marking an unsuccessful end so no partial + // template is served. (A `StreamingBody` dropped without `finish()` is + // aborted anyway, but say it explicitly.) + writer.abandon()?; + return fallback_uncached(e); + } + } +} else if let Some(found) = tx.found() { + found.to_stream()? // C2 HIT — skip origin fetch and transform +} else { + unreachable!("a transaction is either obliged to insert or has found an item") +}; +``` + +Two ownership rules this shape exists to respect, both of which an earlier draft broke: +`Transaction::insert(self)` **consumes** the transaction, so a helper taking `&tx` cannot +call it and `cancel_insert_or_update` is unreachable afterwards; and `finish`/`abandon` +each consume the writer, so neither can be referenced from an arm that did not bind it. + +**Decide the stale policy explicitly.** `Found::is_stale()` and `is_usable()` exist, and +`stale_while_revalidate` can be set at insert. Serving stale while revalidating is a real +option — but it is a state machine, and `cache::core` implements none of it for you. The +spike should start by treating stale as a miss and only add stale-serve if the numbers +justify it. + +**`cache::core` carries no HTTP semantics.** Status, headers, content encoding, and +revalidation are all yours. Serialize what you need into `user_metadata` — at minimum the +content encoding, the transform schema version, and the origin `Vary` values the key was +built from — and decide explicitly whether the stored template is compressed. + +**Cache key must include**, beyond the origin's declared `Vary` (`rsc`, +`next-router-state-tree`, `next-router-prefetch`, `next-router-segment-prefetch`, +`Accept-Encoding` — measured, see the Stage 0 findings): + +- The full URL, explicitly. Do not rely on an ambient request key. +- **The assembly mode.** A2 and A3 emit different template bytes and would otherwise + poison each other's entries. +- **A template schema version**, bumped whenever the transform changes, so a deploy does + not read yesterday's shape. +- Request host and scheme, the enabled-integration set, and the tsjs content hash. + +Per-user signals must never appear in the key. If a signal cannot be excluded from the +template, it does not belong in C2 at all. + +### Design decided 2026-08-10: `cache::core`. Do not revisit read-through. + +An earlier revision left this open between `cache::core` and read-through caching with +`after_send` + `set_body_transform`. Investigated and verified against the pinned SDK and +Viceroy 0.17 source. **Read-through is not viable here** — not on preference, on three +hard blockers: + +1. **Viceroy stubs the entire HTTP Cache ABI**, and the SDK converts that into a _send + error_ rather than a fallback. `is_request_cacheable` returns + `Err(NotAvailable("HTTP Cache API primitives"))` + (`viceroy-lib-0.17.0/src/wiggle_abi/http_cache.rs:108-114`; 26 such stubs in that + file), which makes `must_use_host_caching()` true, which with a send hook set returns + `Err(SendErrorCause::HttpCacheApiUnsupported)` + (`fastly-0.12.1/src/http/request.rs:626-632`). **Setting `after_send` makes every + publisher origin fetch fail** under `fastly compute serve`, `cargo test-fastly`, and + the parity suite. The whole local loop dies. +2. **`with_cache_bypass` makes the hook silently dead.** `get_caching_mode` checks + `cache_override.is_pass()` **first** (`request.rs:612-615`) and returns host caching, so + `after_send` is never invoked and no error is raised. On exactly the requests in scope, + today, the hook would do nothing quietly. +3. **The closure bounds are incompatible with this codebase.** `with_after_send` requires + `Fn + Send + Sync + 'static` (`request.rs:545-550`). Everything the rewriter needs is + `!Send` by construction — `edgezero_core::body::Body` wraps a `LocalBoxStream` + deliberately, which is why the platform layer is `#[async_trait(?Send)]` throughout. + And `set_body_transform` is synchronous, so it could never await the auction collect. + +Read-through's appeal was real — `CandidateResponse::apply_and_stream_back` is +`execute_and_stream_back` with HTTP semantics attached, and TTL/SWR/vary/surrogate keys +derived from origin headers for free. It is simply unreachable from here. + +**Also settled: core cannot reach it at all.** `PlatformHttpRequest` +(`platform/http.rs:16-37`) is a plain data struct with no callback slot, and carrying one +would name `fastly::http::CandidateResponse` in portable core, breaking the other three +adapters. + +### Follow the existing null-object pattern + +`cache::core` fits the shape the repo already uses four times for a Fastly-only capability +behind a portable trait: `UnavailableHttpClient` (`platform/http.rs:216-243`), +`UnavailableKvStore` (`platform/kv.rs:14-17`), and the `RuntimeServices.kv_store` +field/accessor/builder (`platform/types.rs:170,222,269,330`). Add +`PlatformTemplateCache` the same way, and follow +`crates/trusted-server-adapter-fastly/src/ec_kv.rs` — 140 lines, the repo's only real +edge-storage read/write — rather than inventing a shape. + +**Return `EdgeBody`, not `Vec`.** `EdgeBody::Stream` exists, +`fastly_body_to_edge_stream` (`adapter-fastly/src/platform.rs:503`) already converts, and +`PublisherResponse::Buffered` tolerates a live stream (`publisher.rs:1019-1022`). + +### Exact insertion point + +**Immediately before `let mut platform_request = PlatformHttpRequest::new(...)`** — the +last line before `req` is consumed, and a few lines before the origin send. Everything +needed is in scope there: `settings`, `services`, the final URI and Host, `backend_name`, +`request_path`, `matched_slots`, `should_run_ad_stack`, `request_had_authorization`, +`request_host`, `request_scheme`. + +**One required move:** `assembly_mode` is currently computed _after_ the send, for the +logging call site. It depends only on `settings`, so hoist it above the insertion point. + +**Tee-ing is not needed.** With any post-processor registered — and the Next.js +integration always registers one — `HtmlWithPostProcessing` emits nothing until the final +chunk and then returns the whole transformed document as one contiguous buffer +(`html_processor.rs:92-97,148`). Two `write_all` calls on the same slice; no tee +abstraction, no extra copy. Still use `execute_and_stream_back`, but for transaction +correctness and request collapsing rather than for memory. On a hit the processor is never +built at all. + +- [ ] **Step 4b: close the risks the design investigation surfaced** + +Four, all specific to this codebase rather than to `cache::core` in general. + +**`Vary` is in the key list but nothing consumes it.** `c2_bypass_reason` checks +`Set-Cookie`, `Cache-Control`, `Authorization`, status and content type — **not `Vary`**. +Viceroy supports `WriteOptions.vary_rule`, so the mechanism exists; the gate has to use +it. Until then the key is missing a signal the origin explicitly declares, and Step A's +verdict is a `PROVISIONAL PASS`, not a release gate. + +**Resolved — `VarySpec`, commit `b688d667`.** Building the key exposed a problem this +plan states but does not solve: the key must cover everything the origin varies on, but +**a lookup happens before the fetch**, so on a cold key the origin's `Vary` is not yet +known. Three ways out — configure the list; two-phase lookup against a URL-keyed record +holding the last-seen `Vary`; or store the list alongside and re-key on mismatch. The +latter two are correct and double the lookups on every request. + +Configured is taken, **as a spike-grade choice rather than a production one**: Step A +already measured the origin's actual `Vary`, and a 60s TTL bounds drift to a minute +rather than indefinitely. + +The drift is guarded rather than merely accepted. `VarySpec::uncovered_by` runs _after_ +the origin responds, when its `Vary` is finally known, and names which headers the +configured spec missed. A template built under a key that did not cover something the +origin varies on **must not be stored** — a request differing only in that header would +read it. Naming the specific headers makes a stale config identifiable instead of +producing a generic refusal. + +Two decisions worth their tests. An absent header and a present-but-empty one key the +same, because the origin sees no difference between them. And `Vary: *` is not reported +as a named gap — it means uncacheable, which the eligibility gate handles, and reporting +it would produce a nonsense instruction to configure a header called `*`. + +Still open: wiring `uncovered_by` into `c2_bypass_reason` as a bypass reason, which +happens with the store call site. + +**Store bytes plus a metadata envelope; rebuild every header on a hit.** The publisher +path forces `private, no-store` and strips `ETag`/`Last-Modified`/CDN headers _after_ the +send. Replaying stored origin headers would fight that. Store only the transformed body +and a small `user_metadata` envelope — content encoding, content type, schema version, +tsjs hash — and construct every response header from scratch on a hit. Then no origin +header is ever replayed and the `Set-Cookie` privacy net is trivially safe. +`get_user_metadata` is implemented in Viceroy. + +**Content-Encoding belongs in the key.** The streaming pipeline pairs input encoding to +the same output encoding, so the transformed bytes inherit whatever the origin negotiated +from the client's `Accept-Encoding` — still gzip, deflate, br or identity after +`restrict_accept_encoding` narrows it. Either key on the negotiated encoding or normalize +to identity in the cache and re-encode on read. Getting this wrong serves brotli bytes to +a client that asked for gzip. + +**Host and scheme belong in the key.** The post-processed output is host-dependent by +construction: `request_host` and `request_scheme` reach `IntegrationHtmlContext`. + +- [ ] **Step 4c: file the wasted-dispatch follow-up** + +The auction is dispatched _before_ the insertion point. Under `Esi` and `ClientFill` the +root injects nothing, so that dispatch is already pure waste on this branch — and on a C2 +hit it is waste that must be cleaned up via `emit_abandoned_auction` or it leaks +telemetry. + +Keeping the lookup at the insertion point above is right for the spike: minimal diff, and +lookup latency overlaps the in-flight auction. Moving it earlier would eliminate the +wasted dispatch but serialize the lookup ahead of dispatch. **File it; do not fix it +here.** Suppressing root-level dispatch under the shared modes is Task 4's job, where it +also has to be reconciled with the exactly-one-auction gate. + +- [ ] **Step 5: Unit tests, then the target suite** + +```bash +cargo test -p trusted-server-core --target aarch64-apple-darwin assembly_mode +cargo test-fastly && cargo test-axum && cargo test-cloudflare && cargo test-spin +cargo fmt --all -- --check && cargo clippy-fastly +``` + +`ClientFill` must work on all four adapters. `Esi` is Fastly-only and must not break the +others' compilation. + +- [x] **Step 6: the call site — DONE.** `2db10639` (store), `2a2e6c6a` (lookup). + +The cache now engages end to end: a second request for the same URL is served without +touching the origin, and is byte-identical to what was stored. Verified by mutation — +disabling the lookup fails the hit test, so the hit is the cache answering rather than +the fixture answering twice. + +**Wiring the lookup corrected the key.** It carried the content encoding the _origin_ +chose, which does not exist at lookup time. That meant storing under `br` and looking up +under `gzip, br` — a cache that never hits. The field is now the `Accept-Encoding` sent +to the origin. Sound because negotiation is a function of what the origin was offered, +so identical offers yield identical choices; the chosen encoding stays in the metadata +and is what the served response declares. + +That made every key field request-derived, so **the key is built before the fetch** and +the response gate only authorizes storing it. A key that needed the response could only +ever authorize a store, never satisfy a read. + +**The lookup re-checks the request-derived disqualifications, and only those.** The +store gate is response-derived and cannot re-run, but need not: anything in the cache +passed it on the way in. What must re-run are properties of the _reader_ rather than of +the bytes — an authenticated request must not be served a shared template even when that +template is perfectly cacheable. + +**Shared modes take the buffered finalizer.** Storing needs every transformed byte and +streaming does not collect them. The branch keys on the store authorization rather than +on the assembly mode, so `Inline` never reaches it and the spike cannot regress the +shipped path by construction. A C2 _miss_ therefore buffers — the right trade, since a +miss is already paying an origin fetch and a full transform, and what the spike measures +is the hit, where there is no origin fetch to stream from at all. + +Every response header on a hit is constructed, never replayed, so no origin header can +reach a second visitor through the cache. + +The publisher tests use an in-memory cache double, so they prove the wiring rather than +the backing. The join they leave untested is the one `app.rs` makes: the publisher +reaches the cache as a `dyn PlatformTemplateCache` behind `RuntimeServices`, never as +the concrete type the Fastly tests exercise. That join is now executed under Viceroy +against the real Core Cache rather than only type-checked. + +**What this does not establish.** `ClientFill` and `Esi` still render a template with a +hole and nothing filling it. Task 4 and Task 5 remain the blockers on anything +deployable — a cache that works is necessary, not sufficient. + +--- + +## Task 4: Arm A2 — client-fill + +Mostly already specified. See +[the spec's Appendix B](./2026-08-08-esi-cacheable-root-validation-design.md#appendix-b--stage-1-plumbing-condensed) +for the client plumbing, the two-condition join gate, and the server contract; and +[§5](./2026-08-08-esi-cacheable-root-validation-design.md#5-the-trap-in-the-deferred-work--read-this-before-scheduling-stages-12) +for the silent-empty-bids trap, which applies in full. + +- [ ] **Step 1: Hoist the closure-trapped client state** — `pageBidsEndpoint`, + `requestPageBids`, and the `inflight`/`currentPath`/`lastAppliedPath` state, per + Appendix B. Do **not** route the initial load through `onNavigate`. +- [ ] **Step 2: Make `installScheduleInitialAdInit` a hydration-ready AND bids-settled + join**, with a bounded timeout that fires `adInit` untargeted rather than stranding + the slot. Derive the timeout from measured fetch latency, not a constant. +- [ ] **Step 3: Suppress the navigation-path dispatch** so exactly one auction runs per + pageview. Add a new `AuctionSource` for initial loads **plus the mechanism that + delivers it** — a header behind the same-origin gate, not a query parameter. +- [ ] **Step 4: Relocate terminal telemetry.** Navigation `Completed` is emitted only from + the collect functions; the `ts-debug` dump rides the same string. Both move. +- [ ] **Step 5: Verify exactly one auction per pageview** in `auction_events_raw`. Two is + a doubling of SSP spend and an immediate fail. + +--- + +## Task 5: Arm A3 — ESI at the edge + +- [x] **Step 0: the mechanism works — DONE.** `9539061e`, hardened in `0597f54e`. + +Verified under Viceroy with the real `esi` 0.7 crate rather than argued from docs: a +template carrying the `` seam's own ESI include tag comes back with the fragment +spliced in its place and no unresolved tag left. + +**The async/sync obstacle is dissolved, not worked around.** `esi`'s fragment dispatcher +is synchronous and this codebase's fragment producer is `async`; calling one from the +other means a nested executor, which panics. +`PendingFragmentContent::CompletedRequest` lets the dispatcher hand back an +already-built response, so the caller resolves the fragment in the normal async flow and +the dispatcher performs **no I/O at all** — no subrequest, no backend, no self-call, +nothing for Viceroy to stub. That also removes the need for a self-referencing backend +this plan would otherwise have required. + +**Step 2's instruction was right, and reading the crate showed why.** +`CacheConfig::is_includes_cacheable` defaults to **`true`**. A fragment carries one +visitor's bids, so the default caches per-user data and serves it to the next visitor — +silently, on a hit. `includes_force_ttl` is worse where set: it caches everything, +ignoring `private`, `no-store` and `Set-Cookie` alike. Both now stated explicitly, along +with `default_dca`/`inherit_parent_dca` (fragment bytes are data, never re-parsed as +ESI), `max_include_depth = 1`, and rendered caching / `edge_control` off because the +publisher path owns those headers. + +Nine tests. Four assert the configuration; the rest assert behaviour, including that a +fragment containing its own nested ESI include is spliced as text rather than dispatched, so +auction data cannot drive fragment requests. + +**What remains is the call site**, below. Emitting the include and resolving it are both +proven; connecting them is not done. + +- [ ] **Step 1: Wire `process_stream`, not the wrappers** + +`process_response` and `process_response_streaming` consume `self` _and_ send the response +themselves, which takes ownership away from the finalize / `ec_finalize` / apply-effects +ordering. `process_stream(&mut self, src: impl BufRead, out: &mut impl Write, …)` keeps it. + +Source is the C2 body. Sink is the client response body. + +**The ordering an earlier draft described is impossible.** It said EC cookie, geo, and the +privacy net run _after_ assembly. They cannot: streaming responses on this adapter +**commit headers first and then pipe chunks** +(`adapter-fastly/src/main.rs`, `send_edgezero_response`). Once ESI starts writing, no +header can change. + +The correct invariant: + +> **Finalize every header before a single body byte is written** — EC `Set-Cookie`, geo +> suppression, and an unconditional `Cache-Control: private, no-store` — **then** stream +> the assembly with no further header mutation. + +That means `private, no-store` is set unconditionally up front rather than derived from +what the assembly turns out to contain. Deriving it after the fact is not available, and +assuming it was is how a per-user response ends up shared-cacheable. + +- [ ] **Step 2: Disable DCA explicitly and allowlist the dispatcher** + +```rust +let config = esi::Configuration::default() + .with_escaped(false) + .with_default_dca(esi::DcaMode::None) // call the setter; do not rely on the default + .with_inherit_parent_dca(false); +``` + +Comments are not configuration. An earlier draft said DCA "stays at its default" — on a +pre-1.0 crate whose default could move in a patch release, and where this setting fails +**open**, that is not good enough. Call the setters. + +Also disable **fragment caching** explicitly, or mark the include `no-store="on"`. A +cached auction fragment is a per-user object in a shared cache — the C3 failure mode by +another route. + +The dispatcher must be **exact-path allowlisted**: a fragment URL that is not the bids +endpoint is refused, not fetched. The built-in dispatcher builds a dynamic backend per URL +host and panics on a hostless URL — never use it. + +Rationale in the spec's §2: bid payloads carry partner-controlled creative markup, so a +recursive parse would let an SSP make the edge fetch an arbitrary URL. **Add a unit test +that feeds a partner-controlled ESI include targeting `http://attacker.example/` through +a creative payload and asserts no fetch is attempted.** + +- [ ] **Step 3: The fragment must be a script, not the JSON endpoint** + +**`/_ts/page-bids` cannot be the ESI target.** It returns +`serde_json::json!({"slots":…, "bids":…})` (`publisher.rs:3987`), and ESI splices fragment +bytes in literally — the page would contain raw JSON where an executable script belongs. +Nothing would call `scheduleInitialAdInit`. + +Add a **dedicated fragment endpoint** returning the executable script — the same shape +`build_bids_script` produces today, plus the `adSlots` assignment that moved out of the +template in Task 3 Step 2. Either that, or use the `esi` crate's fragment-response +processor to wrap the JSON; the dedicated endpoint is simpler and easier to assert on. + +Three more things the naïve marker gets wrong: + +- **The same-origin gate will reject it.** `page_bids_request_allowed` + (`publisher.rs:3644`) requires `Sec-Fetch-Site: same-origin` or the `X-TSJS-Page-Bids` + header. An internal ESI subrequest carries neither. Give the fragment endpoint an + internal contract and a fixed backend rather than weakening that gate — it exists to + stop third parties burning SSP quota. +- **Parent context does not propagate.** EC identity, consent state, client IP, geo, User + Agent, and the correlation ID all live on the parent request. Forward an **explicitly + approved allowlist** of them into the fragment request. Forwarding everything is how a + fragment ends up more privileged than the parent. +- **Root dispatch must be suppressed.** The navigation path already dispatches an + auction. If A3 does not suppress it, every pageview runs two — doubling SSP and APS + spend. This applies to **A2 and A3 alike**. + +- [ ] **Step 4: Validate the whole URL, not the path** + +An exact-path allowlist alone permits `https://attacker.example/_ts/page-bids`. Validate +**scheme, authority, method, path, and query** — or better, ignore the marker's URL +entirely and dispatch to a fixed internal backend, treating the ESI include as a signal +rather than an address. + +Add a test that feeds an ESI include targeting +`https://attacker.example/_ts/page-bids` through a creative payload and asserts no +outbound fetch is attempted. + +- [ ] **Step 5: Deterministic synthetic fragment first** + +Before wiring the real auction, point the include at a fixed-content endpoint. This +separates "does the pipeline assemble correctly" from "does the auction behave," and the +two fail very differently. Only once assembly is proven does the fragment become the real +one. + +- [ ] **Step 6: Handle the flush hazard** + +`esi` flushes its output writer after each parse batch. Fastly's `StreamingBody` is a +`BufWriter`, so anything between esi and it must propagate `flush()` or nothing leaves the +Wasm heap. + +- [ ] **Step 7: Fragment failure must degrade, not break** + +Assert that a fragment timeout or non-2xx yields a page with empty bids rather than a 5xx +or a truncated document. Note the crate's non-obvious semantics: `alt` is attempted before +`onerror="continue"`, and `` runs **all** attempts and concatenates every +non-failed output — it is not first-success-wins. + +--- + +## Task 6: Safety gates — run against every arm + +Not a phase. Every one of these is a hard fail, independent of any performance result. + +- [x] **Zero cross-user leakage.** DONE — `76df2469`. Two synthetic users differing in EC + identity, consent jurisdiction and geo store a byte-identical template, each against + a fresh cache so the first cannot answer for the second. Forbidden-substring checks + are the second layer, since byte-identity also holds if both leak the same thing. + Mutation-verified: leaking `adSlots` through the head seam fails it. +- [x] **Cold MISS, warm HIT, stale revalidation** DONE — `76df2469`, and end to end under + `viceroy serve` (below). Stale reads as a miss; serving stale would mean serving a + template built by an older transform or bundle. + + The first stale test passed for the wrong reason and had to be rewritten: a zero TTL + produces an *absent* entry, not a stale one, so `is_stale()` was never reached — + confirmed by reverting the check and watching it stay green. Only a + `stale_while_revalidate` window makes an entry present-and-stale. + +- [x] **Transform failure** DONE — `76df2469`. A partial template in C2 is the worst + outcome available: a truncated document served to every later visitor, indefinitely, + with no error after the first request. Mutation-verified by storing before the cap + check. +- [ ] **Request collapsing** works: concurrent cold requests transform once. +- [x] **DCA disabled** DONE — `0597f54e`. Config asserted _and_ behaviour: a fragment + carrying its own nested ESI include is spliced as text rather than dispatched. + +- [ ] **Request collapsing** — not tested, and not testable here. Viceroy is + single-threaded, so the concurrent cold-request case cannot be produced. The racing + _writer_ path is covered (`a_second_put_on_a_fresh_entry_is_a_no_op`), which is the + correctness half; the collapsing half needs real concurrency. +- [ ] **Exactly one auction per pageview**, from `auction_events_raw`. +- [ ] **Cookie and privacy finalization ran BEFORE assembly**, not after — EC + `Set-Cookie` on first visit, geo suppression, and an unconditional + `Cache-Control: private, no-store`. Headers commit before the body streams on this + adapter, so "finalize after assembly" is not available; asserting it that way is how + a per-user response ends up shared-cacheable. ESI's streaming mode dropping + `$add_header` is a consequence of the same constraint, not a separate hazard. +- [ ] **Slot and bid attribution unchanged.** Same slots matched, same bids applied, same + renders attributed. Use TS-attributed renders — the SSAT line item, non-empty + `ts.bids`, `hb_adid` presence — **never slot fill**, which is blind to empty bids + because `adInit` defines slots regardless. +- [x] **No C3 — assert positively, not by absence.** DONE — `0adb578e`, and this gate's + wording caught a live bug. A C2 hit returns before the point where the publisher path + stamps `private, no-store`, so it served HTML with **no `Cache-Control` at all** — + heuristically cacheable, and therefore a shared cache of an assembled per-user + response. Checking for the _absence_ of `public`/`s-maxage`/`Surrogate-Control` would + have reported it as safe, because there was nothing present to forbid. Covered for + returning visitors specifically, where the cookie-privacy net never fires. + + Original wording, retained because it is what made the difference: Forbidding `public`, `s-maxage`, and + `Surrogate-Control` is **not sufficient**: a bare `Cache-Control: max-age=60` passes + that check and is still shared-cacheable, and that is exactly what the measured + origin sends. Require instead that every assembled response carries + `Cache-Control: private, no-store` and that `Expires`, `ETag`, `Last-Modified`, and + all four CDN cache directives are stripped. Test it for **returning** users + specifically — they set no EC cookie, so the cookie privacy net never fires and is + not a backstop here. + +--- + +## Task 7: The decision record + +**Files:** `docs/superpowers/plans/2026-08-10-1009-esi-decision-record.md` + +- [ ] **Step 1: Record every arm** with N, confidence interval, cache-tier mix, route mix, + and POP. Any arm missing those is not reportable. + +- [ ] **Step 2: Apply the decision rule, stated here before the data exists** + +**Adopt ESI only if all three hold:** + +1. Every Task 6 gate passes on A3. +2. A3 beats A2 on **bids-ready time, `adInit` fire time, and first TS-attributed creative + paint** — by a margin the reviewers ratify **before** collection, not chosen after + seeing the numbers. **Not root TTFB:** A2 and A3 serve the same C2 template, so their + root timings are near-identical by construction and a difference there would be noise. + Root TTFB is a non-regression guard only. +3. Render outcomes on A3 are non-inferior to A0. + +**Otherwise adopt A2 (client-fill)** if its gates pass and it beats A1. It is portable +across all four adapters and carries no Fastly-only maintenance burden. + +**Otherwise keep A1** — Stage 0 alone — and record #1009 as answered in the negative with +evidence. + +The margin in (2) exists because A3's cost is not its diff. It is a second rendering +architecture, Fastly-only, on a pre-1.0 crate, in the critical render path. A small win +does not pay for that. + +- [ ] **Step 3: Record what would change the answer**, so this does not get re-litigated + from scratch. At minimum: React #418 / [#938](https://github.com/IABTechLab/trusted-server/issues/938) + being fixed such that `adInit` can run synchronously, which is what would make edge + assembly's round-trip saving actually worth something. + +- [ ] **Step 4: Clean up.** Remove the spike flag or promote it to a real setting; purge + C2 (`purge_surrogate_key` on `ts-template`); remove the synthetic fragment endpoint; + and either land or delete the `esi` dependency. **A spike flag left in place becomes + permanent configuration surface.** + +--- + +## Reproducibility metadata + +Record with every result, or it cannot be re-run or trusted: commit SHA; `esi` and +`fastly` crate versions; Fastly service and version IDs; whether the backend is shielded; +`template_ttl`; the origin's `Cache-Control` and `Vary` at collection time; assembly mode; +routes; N per arm; and the cache-tier mix. + +## Out of scope + +- **Stages 1–2 of the spec** as production work. This spike may build parts of the + client-fill path to measure it; shipping it is a separate decision behind the + correctness defects. +- **Full RSC/flight partitioning.** `rsc_flight.rs` has no static/dynamic split. +- **Publisher-authored ESI.** Breaks the no-origin-changes promise. +- **A C3 delivery cache.** Not a deferred item — a thing that must not exist. + +## Definition of done + +- [ ] Task 1 verdict recorded: `esi` 0.7 builds on Rust 1.95.0 / `wasm32-wasip1`, or it + does not and the spike stopped. +- [ ] All four arms measured on one build, with correlation IDs joining server and browser + timings, and cache tier recorded per request. +- [ ] Every Task 6 gate has an explicit pass/fail per arm. +- [ ] Decision record exists, applies the pre-ratified rule, and names what would change + the answer. +- [ ] Cleanup complete: flag resolved, C2 purged, synthetic endpoint removed, dependency + landed or dropped. +- [ ] All CI gates pass: `cargo fmt --all -- --check`; the six clippy targets; the four + adapter test suites; the parity suite; JS build, test, and format; docs format. diff --git a/docs/superpowers/plans/2026-06-26-server-side-ad-template-cli.md b/docs/superpowers/plans/2026-06-26-server-side-ad-template-cli.md new file mode 100644 index 000000000..df781e518 --- /dev/null +++ b/docs/superpowers/plans/2026-06-26-server-side-ad-template-cli.md @@ -0,0 +1,2142 @@ +# Server-Side Ad Template CLI Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the unified `ts` CLI support for server-side ad-template static diagnostics and browser-backed verification described in `docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md`. + +**Architecture:** Keep the CLI host-only and thin: Clap parsing stays in `run.rs` / command adapter modules, shared app-config loading moves to `app_config.rs`, pure ad-template logic lives under `ad_templates/`, and Chrome/Chromium collection lives under `audit/`. Runtime gate rules are extracted into a small pure helper in `trusted-server-core` so the CLI does not duplicate server behavior. + +**Tech Stack:** Rust 2024 workspace, host-target `trusted-server-cli`, `clap`, EdgeZero typed app-config loader, `serde`/`serde_json` for stable JSON, `chromiumoxide` for browser-backed audit collection, local HTML fixture tests, and existing `trusted-server-core::creative_opportunities` matching. + +--- + +## Current State + +- Branch: `feature/ts-cli-ad-templates`. +- Static ad-template commands already exist in `crates/trusted-server-cli/src/config_ad_templates.rs`. +- The current branch does not contain #800 audit files. Port useful #800 pieces into the current #799 code shape; do not resurrect stale `args.rs` or `config_command.rs`. +- The spec was updated after review and is the source of truth: + `docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md`. +- Keep `.env` and operator-owned `trusted-server.toml` out of commits. + +## File Map + +### New files + +| File | Responsibility | +| -------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | +| `crates/trusted-server-cli/src/app_config.rs` | Shared effective app-config loader and shared `AppConfigArgs`. | +| `crates/trusted-server-cli/src/ad_templates/mod.rs` | Re-export focused ad-template CLI modules. | +| `crates/trusted-server-cli/src/ad_templates/expected.rs` | Path/URL normalization and expected-slot projection from runtime slot matching. | +| `crates/trusted-server-cli/src/ad_templates/compare.rs` | Pure DOM/GPT/APS evidence comparison, statuses, warnings, runtime gate output, strict failure decisions. | +| `crates/trusted-server-cli/src/ad_templates/output.rs` | Human and JSON rendering for static diagnostics and browser verification. | +| `crates/trusted-server-cli/src/audit/mod.rs` | Audit namespace entry point. | +| `crates/trusted-server-cli/src/audit/page.rs` | Generic page audit command ported from #800. | +| `crates/trusted-server-cli/src/audit/collector.rs` | Browser collector trait plus collected page/evidence structs. | +| `crates/trusted-server-cli/src/audit/browser.rs` | Chromiumoxide-backed browser collector, init scripts, optional scroll, page-level collection errors. | +| `crates/trusted-server-cli/src/audit/ad_templates.rs` | `ts audit ad-templates verify` orchestration. | +| `crates/trusted-server-cli/src/audit/ad_template_collector.js` | Read-only init script for GPT/APS/DOM evidence collection, included via `include_str!`. | + +### Modified files + +| File | Change summary | +| ---------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | +| `Cargo.toml` | Add workspace dependencies missing from this branch: `chromiumoxide`, `serde`, and `serde_json` if not present. | +| `crates/trusted-server-cli/Cargo.toml` | Add host-only CLI dependencies for browser audit and JSON output. | +| `crates/trusted-server-cli/src/lib.rs` | Register new `app_config`, `ad_templates`, and `audit` modules under `cfg(not(target_arch = "wasm32"))`. | +| `crates/trusted-server-cli/src/run.rs` | Add `Audit` command namespace, parser tests, and dispatch. | +| `crates/trusted-server-cli/src/config_ad_templates.rs` | Shrink to Clap adapter using shared loader/expected/output modules. | +| `crates/trusted-server-core/src/creative_opportunities.rs` | Add pure runtime gate helper types/functions shared by runtime and CLI. | +| `crates/trusted-server-core/src/publisher.rs` | Route existing server-side ad-stack gate through the shared helper without changing behavior. | + +## Implementation Rules + +- Use TDD for each task: write a failing test first, run it, implement the minimal code, re-run, then commit. +- Commit after each task using repo style: sentence case, imperative, no semantic prefix. +- Keep `trusted-server-cli` host-only. Do not introduce `tokio`, `chromiumoxide`, or filesystem/browser dependencies into core runtime or wasm adapter crates. +- Do not write real publisher domains or secrets in tests. Use `example.com`, `publisher.example`, and fictional IDs only. +- Prefer pure module tests over browser tests. Browser-backed fixture tests should use local HTML only and no GPT/APS network. + +## Task 0: Baseline And Branch Hygiene + +**Files:** + +- Verify: `docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md` +- Verify: `docs/superpowers/plans/2026-06-26-server-side-ad-template-cli.md` + +- [ ] **Step 1: Confirm branch and working tree** + + Run: + + ```bash + git status --short --branch + git log --oneline --decorate -5 + ``` + + Expected: on `feature/ts-cli-ad-templates`; no unrelated modified files besides the approved spec/plan docs. + +- [ ] **Step 2: Run docs format check before code work** + + Run: + + ```bash + cd docs && npm run format + ``` + + Expected: `All matched files use Prettier code style!` + +- [ ] **Step 3: Commit reviewed spec and plan** + + Run: + + ```bash + git add docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md docs/superpowers/plans/2026-06-26-server-side-ad-template-cli.md + git commit -m "Add server-side ad-template CLI implementation plan" + ``` + + Expected: docs-only commit. If the spec commit already exists separately, commit only the plan. + +## Task 1: Share Runtime Ad-Stack Gate Logic + +**Files:** + +- Modify: `crates/trusted-server-core/src/creative_opportunities.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` + +- [ ] **Step 1: Write failing core tests for the shared gate helper** + + Add tests near the existing `creative_opportunities` tests: + + ```rust + #[test] + fn ad_stack_gate_passes_for_eligible_navigation() { + let result = evaluate_ad_stack_gate(AdStackGateInput { + method_get: true, + navigation: true, + prefetch: false, + bot: false, + matched_slots: true, + consent_allows_auction: Some(true), + auction_enabled: true, + }); + + assert_eq!(result.expected, RuntimeAdStackExpected::Yes); + assert!(result.blocking_gates().is_empty()); + } + + #[test] + fn ad_stack_gate_blocks_known_kill_switch() { + let result = evaluate_ad_stack_gate(AdStackGateInput { + method_get: true, + navigation: true, + prefetch: false, + bot: false, + matched_slots: true, + consent_allows_auction: Some(true), + auction_enabled: false, + }); + + assert_eq!(result.expected, RuntimeAdStackExpected::No); + assert!(result.blocking_gates().contains(&AdStackGateName::AuctionEnabled)); + } + + #[test] + fn ad_stack_gate_is_unknown_when_consent_is_unknown() { + let result = evaluate_ad_stack_gate(AdStackGateInput { + method_get: true, + navigation: true, + prefetch: false, + bot: false, + matched_slots: true, + consent_allows_auction: None, + auction_enabled: true, + }); + + assert_eq!(result.expected, RuntimeAdStackExpected::Unknown); + } + + // Locks the spec §5.2 mirror invariant: with Some(consent) supplied for every + // input combination, `expected == Yes` must equal the legacy all-AND boolean. + #[test] + fn ad_stack_gate_with_known_consent_matches_legacy_boolean() { + for bits in 0u8..64 { + let input = AdStackGateInput { + method_get: bits & 1 != 0, + navigation: bits & 2 != 0, + prefetch: bits & 4 != 0, + bot: bits & 8 != 0, + matched_slots: bits & 16 != 0, + consent_allows_auction: Some(bits & 32 != 0), + auction_enabled: bits & 1 == 0, + }; + // Legacy semantics: all positive gates true, both negative gates false. + let legacy = input.method_get + && input.navigation + && !input.prefetch + && !input.bot + && input.matched_slots + && input.consent_allows_auction == Some(true) + && input.auction_enabled; + let got = evaluate_ad_stack_gate(input).expected == RuntimeAdStackExpected::Yes; + assert_eq!(got, legacy, "gate mismatch for bits={bits}"); + } + } + ``` + +- [ ] **Step 2: Run the focused test and verify it fails** + + Run: + + ```bash + # NOTE: trusted-server-core links the `fastly` crate and CANNOT build for the host + # triple — run core tests on the DEFAULT target (wasm32-wasip1 + viceroy runner), + # i.e. no `--target`. Only the host-only `trusted-server-cli` uses `--target `. + cargo test -p trusted-server-core creative_opportunities::tests::ad_stack_gate + ``` + + Expected: compile failure because `AdStackGateInput` / `evaluate_ad_stack_gate` do not exist. + +- [ ] **Step 3: Implement pure gate types and helper** + + Add public, serde-free types to `creative_opportunities.rs`: + + ```rust + #[derive(Debug, Clone, Copy, Eq, PartialEq)] + pub enum RuntimeAdStackExpected { + Yes, + No, + Unknown, + } + + #[derive(Debug, Clone, Copy, Eq, PartialEq)] + pub enum AdStackGateName { + MethodGet, + Navigation, + NotPrefetch, + NotBot, + MatchedSlots, + ConsentAllowsAuction, + AuctionEnabled, + } + + #[derive(Debug, Clone, Copy)] + pub struct AdStackGateInput { + pub method_get: bool, + pub navigation: bool, + pub prefetch: bool, + pub bot: bool, + pub matched_slots: bool, + pub consent_allows_auction: Option, + pub auction_enabled: bool, + } + + #[derive(Debug, Clone, Eq, PartialEq)] + pub struct AdStackGateResult { + pub expected: RuntimeAdStackExpected, + blocking_gates: Vec, + } + + impl AdStackGateResult { + pub fn blocking_gates(&self) -> &[AdStackGateName] { + &self.blocking_gates + } + } + ``` + + Implement `evaluate_ad_stack_gate(input)` so any known blocking boolean gate returns `No`, all known pass plus `Some(true)` consent returns `Yes`, and all known pass plus `None` consent returns `Unknown`. + + Mind the gate polarity, mirroring `should_run_server_side_ad_stack`: `method_get`, + `navigation`, `matched_slots`, and `auction_enabled` block when **false**, while + `prefetch` and `bot` block when **true** (their gate names `NotPrefetch` / `NotBot` + pass when the input bool is false). `consent_allows_auction` is the only tri-state + input: `Some(false)` blocks (No), `Some(true)` passes, `None` yields Unknown only + when no other gate already blocks. + +- [ ] **Step 4: Route `publisher.rs` through the helper** + + Replace the body of `should_run_server_side_ad_stack` with a call to `evaluate_ad_stack_gate`, preserving the existing function signature for low-risk runtime compatibility: + + ```rust + crate::creative_opportunities::evaluate_ad_stack_gate( + crate::creative_opportunities::AdStackGateInput { + method_get: is_get, + navigation: is_navigation, + prefetch: is_prefetch, + bot: is_bot, + matched_slots: has_matched_slots, + consent_allows_auction: Some(consent_allows_auction), + auction_enabled, + }, + ) + .expected + == crate::creative_opportunities::RuntimeAdStackExpected::Yes + ``` + +- [ ] **Step 5: Run focused tests** + + Run: + + ```bash + # Core tests run on the default wasm target via viceroy (no --target). + cargo test -p trusted-server-core publisher::tests + cargo test -p trusted-server-core creative_opportunities + ``` + + Expected: all focused tests pass (including the existing `should_run_server_side_ad_stack` truth-table tests in `publisher::tests`). + +- [ ] **Step 6: Commit** + + ```bash + git add crates/trusted-server-core/src/creative_opportunities.rs crates/trusted-server-core/src/publisher.rs + git commit -m "Share server-side ad stack gate evaluation" + ``` + +## Task 2: Extract Shared CLI App Config Loader + +**Files:** + +- Create: `crates/trusted-server-cli/src/app_config.rs` +- Modify: `crates/trusted-server-cli/src/lib.rs` +- Modify: `crates/trusted-server-cli/src/config_ad_templates.rs` + +- [ ] **Step 1: Write failing loader tests** + + Move the existing temp-project helpers from `config_ad_templates.rs` tests into `app_config.rs` tests and add: + + ```rust + #[test] + fn explicit_missing_app_config_does_not_fall_back() { + let temp = TempDir::new().expect("should create temp dir"); + let manifest_path = temp.path().join("edgezero.toml"); + fs::write(&manifest_path, "[app]\nname = \"trusted-server\"\n") + .expect("should write manifest"); + let missing_path = temp.path().join("missing.toml"); + + let args = AppConfigArgs { + app_config: Some(missing_path.clone()), + manifest: manifest_path, + no_env: true, + }; + + let err = load_settings(&args).expect_err("should reject missing explicit config"); + assert!( + err.contains(missing_path.to_string_lossy().as_ref()), + "error should mention the explicit missing path" + ); + } + ``` + +- [ ] **Step 2: Run focused test and verify it fails** + + Run: + + ```bash + cargo test -p trusted-server-cli app_config --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: compile failure because `app_config` module is not registered. + +- [ ] **Step 3: Implement `app_config.rs`** + + Move these items out of `config_ad_templates.rs`: + - `AppConfigArgs` + - `LoadedSettings` + - `load_settings` + - `resolve_app_config_path` + + Make the API explicit: + + ```rust + #[derive(Clone, Debug, Args)] + pub struct AppConfigArgs { + #[arg(long)] + pub app_config: Option, + #[arg(long, default_value = "edgezero.toml")] + pub manifest: PathBuf, + #[arg(long)] + pub no_env: bool, + } + + pub struct LoadedSettings { + pub app_config_path: PathBuf, + pub settings: Settings, + } + + pub fn load_settings(args: &AppConfigArgs) -> Result { + let manifest_loader = ManifestLoader::from_path(&args.manifest) + .map_err(|err| format!("failed to load {}: {err}", args.manifest.display()))?; + let app_name = manifest_loader.manifest().app.name.clone().ok_or_else(|| { + format!( + "{} has no [app].name; cannot resolve trusted-server.toml", + args.manifest.display() + ) + })?; + let app_config_path = + resolve_app_config_path(args.app_config.as_deref(), &args.manifest, &app_name); + + let mut opts = AppConfigLoadOptions::default(); + opts.env_overlay = !args.no_env; + let app_config = app_config::deserialize_app_config_with_options::( + &app_config_path, + &app_name, + &opts, + ) + .map_err(|err| format!("failed to load {}: {err}", app_config_path.display()))?; + + Ok(LoadedSettings { + app_config_path, + settings: app_config.into_settings(), + }) + } + + fn resolve_app_config_path( + explicit: Option<&Path>, + manifest_path: &Path, + app_name: &str, + ) -> PathBuf { + if let Some(path) = explicit { + return path.to_path_buf(); + } + let file_name = format!("{app_name}.toml"); + if let Some(parent) = manifest_path + .parent() + .filter(|parent| !parent.as_os_str().is_empty()) + { + parent.join(file_name) + } else { + PathBuf::from(file_name) + } + } + ``` + + Include the same top-level imports currently used by these helpers: + `std::path::{Path, PathBuf}`, `clap::Args`, + `edgezero_core::app_config::{self, AppConfigLoadOptions}`, + `edgezero_core::manifest::ManifestLoader`, + `trusted_server_core::config::TrustedServerAppConfig`, and + `trusted_server_core::settings::Settings`. + +- [ ] **Step 4: Register module and update imports** + + In `lib.rs`, add: + + ```rust + #[cfg(not(target_arch = "wasm32"))] + mod app_config; + ``` + + In `config_ad_templates.rs`, import: + + ```rust + use crate::app_config::{load_settings, AppConfigArgs}; + ``` + +- [ ] **Step 5: Run focused CLI tests** + + Run: + + ```bash + cargo test -p trusted-server-cli config_ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli app_config --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: existing static command behavior remains unchanged. + +- [ ] **Step 6: Commit** + + ```bash + git add crates/trusted-server-cli/src/app_config.rs crates/trusted-server-cli/src/config_ad_templates.rs crates/trusted-server-cli/src/lib.rs + git commit -m "Extract shared CLI app config loader" + ``` + +## Task 3: Add Expected-Slot Model + +**Files:** + +- Create: `crates/trusted-server-cli/src/ad_templates/mod.rs` +- Create: `crates/trusted-server-cli/src/ad_templates/expected.rs` +- Modify: `crates/trusted-server-cli/Cargo.toml` +- Modify: `crates/trusted-server-cli/src/lib.rs` +- Modify: `crates/trusted-server-cli/src/config_ad_templates.rs` + +- [ ] **Step 1: Write failing expected-slot tests** + + Add a test-only dependency to `crates/trusted-server-cli/Cargo.toml` so tests can + deserialize core slot config instead of constructing `CreativeOpportunitySlot` with + its `pub(crate)` `compiled_patterns` cache: + + ```toml + [target.'cfg(not(target_arch = "wasm32"))'.dev-dependencies] + toml = { workspace = true } + ``` + + In `expected.rs`, add tests for path normalization, full URL normalization, config-order preservation, resolved div ID, resolved GAM unit path, provider names, and matching page patterns: + + ```rust + fn creative_config_with_slots(patterns: &[&str]) -> CreativeOpportunitiesConfig { + let page_patterns = patterns + .iter() + .map(|pattern| format!("\"{pattern}\"")) + .collect::>() + .join(", "); + let toml = format!( + r#" + gam_network_id = "123" + auction_timeout_ms = 500 + price_granularity = "dense" + + [[slot]] + id = "atf" + gam_unit_path = "/123/news/atf" + div_id = "ad-atf-" + page_patterns = [{page_patterns}] + formats = [{{ width = 300, height = 250 }}] + floor_price = 0.50 + targeting = {{ zone = "atf" }} + + [slot.providers.prebid] + bidders = {{}} + "# + ); + let mut config = toml::from_str::(&toml) + .expect("should deserialize creative opportunities config"); + config.compile_slots(); + config + } + + #[test] + fn expected_slots_use_runtime_matcher_and_config_order() { + let config = creative_config_with_slots(["/news/*", "/"].as_slice()); + let expected = expected_slots_for_path("/news/story", &config) + .expect("should build expected slots"); + + assert_eq!(expected.path, "/news/story"); + assert_eq!(expected.slots.iter().map(|slot| slot.id.as_str()).collect::>(), ["atf"]); + assert_eq!(expected.slots[0].div_id, "ad-atf-"); + assert_eq!(expected.slots[0].gam_unit_path, "/123/news/atf"); + assert_eq!(expected.slots[0].providers, ["prebid"]); + } + + #[test] + fn normalize_path_or_url_strips_query_and_fragment() { + assert_eq!(normalize_path_or_url("https://www.example.com/news/story?x=1#top").expect("should normalize"), "/news/story"); + assert_eq!(normalize_path_or_url("news/story?x=1").expect("should normalize"), "/news/story"); + } + ``` + +- [ ] **Step 2: Run focused test and verify it fails** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_templates::expected --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: compile failure because module/types do not exist. + +- [ ] **Step 3: Implement expected-slot structs** + + Define pure structs that own strings and are stable for output: + + ```rust + #[derive(Debug, Clone, PartialEq)] + pub struct ExpectedSlots { + pub path: String, + pub slots: Vec, + } + + #[derive(Debug, Clone, PartialEq)] + pub struct ExpectedSlot { + pub id: String, + pub div_id: String, + pub gam_unit_path: String, + pub formats: Vec, + pub providers: Vec, + pub page_patterns: Vec, + } + + #[derive(Debug, Clone, PartialEq)] + pub struct ExpectedFormat { + pub width: u32, + pub height: u32, + // Mirrors `MediaType` rendered as a stable string (`"banner"`, `"video"`, `"native"`). + pub media_type: String, + } + ``` + + `div_id` and `gam_unit_path` are resolved (non-optional) strings. The core + `CreativeOpportunitySlot` stores `div_id` / `gam_unit_path` as `Option` + and the GAM unit path is composed with the configured GAM network ID; mirror the + existing `format_slot` resolution in `config_ad_templates.rs` so the CLI does not + invent a second resolution rule. Use + `trusted_server_core::creative_opportunities::match_slots`. Do not compile globs in CLI. + +- [ ] **Step 4: Register `ad_templates` and update static commands** + + In `lib.rs`, add: + + ```rust + #[cfg(not(target_arch = "wasm32"))] + mod ad_templates; + ``` + + Rewire `config_ad_templates.rs` onto the shared module, and remove the now-duplicated + local code so there is no name collision or dead `normalize_path_or_url`: + - delete the private `fn normalize_path_or_url` (currently `config_ad_templates.rs:448`) + and add `use crate::ad_templates::expected::{expected_slots_for_path, normalize_path_or_url};`; + - the existing `config_ad_templates::tests::normalizes_path_or_url_like_runtime_request_path` + test (currently `:661`) calls the local fn via `super::*` — either delete it (Task 3 + Step 1 already adds normalization tests in `expected.rs`) or repoint it at + `crate::ad_templates::expected::normalize_path_or_url`. Pick one so the test crate + still compiles at this commit. + +- [ ] **Step 5: Run focused tests** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_templates::expected --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli config_ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: all pass. + +- [ ] **Step 6: Commit** + + ```bash + git add crates/trusted-server-cli/Cargo.toml crates/trusted-server-cli/src/ad_templates/mod.rs crates/trusted-server-cli/src/ad_templates/expected.rs crates/trusted-server-cli/src/config_ad_templates.rs crates/trusted-server-cli/src/lib.rs + git commit -m "Add shared ad-template expected slot model" + ``` + +## Task 4: Add Stable Output And JSON Types + +**Files:** + +- Create: `crates/trusted-server-cli/src/ad_templates/output.rs` +- Modify: `crates/trusted-server-cli/Cargo.toml` +- Modify: `crates/trusted-server-cli/src/ad_templates/mod.rs` + +- [ ] **Step 1: Add CLI JSON dependencies** + + The CLI crate has **no plain `[dependencies]` table** — every runtime dep lives + under `[target.'cfg(not(target_arch = "wasm32"))'.dependencies]` (the workspace + default build target is `wasm32-wasip1` per `.cargo/config.toml`). Add the new deps + to that existing table; do **not** create a `[dependencies]` table, or they compile + for wasm and leak host-only crates into the wasm build: + + ```toml + [target.'cfg(not(target_arch = "wasm32"))'.dependencies] + # ... existing clap/url/etc ... + serde = { workspace = true } + serde_json = { workspace = true } + ``` + + Add workspace dependency `chromiumoxide = "0.9.1"` in `Cargo.toml` only in Task 7 when browser code is introduced. + +- [ ] **Step 2: Write failing JSON output tests** + + In `output.rs`, add tests that construct an in-memory verification result and assert exact JSON values: + + ```rust + #[test] + fn verification_json_contains_gate_state_and_extra_evidence() { + let result = VerificationReport::example_confirmed_with_extra_evidence(); + let value = serde_json::to_value(&result).expect("should serialize"); + + assert_eq!(value["ok"], true); + assert_eq!(value["pages"][0]["requested_path"], "/news/story"); + assert_eq!(value["pages"][0]["runtime_ad_stack_expected"], "unknown"); + assert_eq!(value["pages"][0]["extra_evidence"][0]["kind"], "gpt"); + assert_eq!(value["pages"][0]["warnings"][0]["code"], "redirected"); + } + + // Pins the spec §8 navigation_failed shape: error present, runtime/gates/ + // matched_slot_count keys ABSENT (skipped), final_url/path null. + #[test] + fn page_error_json_matches_navigation_failed_shape() { + let result = VerificationReport::example_navigation_failed(); + let value = serde_json::to_value(&result).expect("should serialize"); + let page = &value["pages"][0]; + + assert_eq!(page["error"]["code"], "navigation_failed"); + assert!(page["final_url"].is_null(), "final_url should be null"); + assert!(page["path"].is_null(), "path should be null"); + assert!(page.get("runtime_ad_stack_expected").is_none(), "runtime field absent on error page"); + assert!(page.get("gates").is_none(), "gates absent on error page"); + assert!(page.get("matched_slot_count").is_none(), "matched_slot_count absent on error page"); + assert_eq!(value["ok"], false); + } + ``` + +- [ ] **Step 3: Run focused test and verify it fails** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_templates::output --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: compile failure because output model does not exist. + +- [ ] **Step 4: Implement serializable output types** + + Model the **entire** `--json` wire tree from spec §8 (this is the single source of + truth for field names and ordering). Use owned `String` / `Vec` fields and + `#[serde(rename_all = "snake_case")]` so output is stable. Leaf enums: + + ```rust + #[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] + #[serde(rename_all = "snake_case")] + pub enum SlotStatus { Confirmed, Partial, Missing } + + #[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] + #[serde(rename_all = "snake_case")] + pub enum RuntimeAdStackExpectedJson { Yes, No, Unknown } + + impl From for RuntimeAdStackExpectedJson { + fn from(value: trusted_server_core::creative_opportunities::RuntimeAdStackExpected) -> Self { + use trusted_server_core::creative_opportunities::RuntimeAdStackExpected as Core; + match value { + Core::Yes => Self::Yes, + Core::No => Self::No, + Core::Unknown => Self::Unknown, + } + } + } + + #[derive(Debug, Clone, Copy, Eq, PartialEq, Serialize)] + #[serde(rename_all = "snake_case")] + pub enum GateState { Pass, Fail, Unknown } + ``` + + Top-level tree (field names and nesting must match §8 exactly): + + ```rust + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct VerificationReport { + pub ok: bool, + pub strict: bool, + pub pages: Vec, + pub warnings: Vec, + } + + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct PageJson { + pub url: String, + pub final_url: Option, + pub requested_path: String, + pub path: Option, + // Field ORDER matters: serde serializes in declaration order. Spec §8 places + // `error` immediately after `path` on the navigation_failed shape, so it must + // be declared here (not last). On normal pages `error` is None and skipped, so + // the runtime/gates/slots run in §8 order; on error pages the runtime/gates/ + // matched_slot_count are None and skipped, leaving url..path, error, slots, + // extra_evidence, warnings — exactly the §8 error shape. + #[serde(skip_serializing_if = "Option::is_none")] + pub error: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub runtime_ad_stack_expected: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub gates: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub matched_slot_count: Option, + pub slots: Vec, + pub extra_evidence: Vec, + pub warnings: Vec, + } + + // One field per gate name from spec §5.2 / §8, each a GateState. + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct Gates { + pub method_get: GateState, + pub navigation: GateState, + pub not_prefetch: GateState, + pub not_bot: GateState, + pub matched_slots: GateState, + pub auction_enabled: GateState, + pub consent_allows_auction: GateState, + } + + // Serialize for output JSON; Deserialize because the browser collector payload + // (Task 8) carries warning objects decoded into `BrowserAdEvidence.warnings`. + #[derive(Debug, Clone, Eq, PartialEq, Serialize, serde::Deserialize)] + pub struct Warning { + pub code: String, + pub message: String, + } + ``` + + Define the remaining nested JSON structs **explicitly** — do not serialize the + compare-module types directly. The compare types (`SlotResult`, `SlotEvidence`, + `GptSlotEvidence`, `ExtraEvidence`) carry a `phase` field and are not `Serialize`; + spec §8's `evidence.gpt` has **no** `phase` key and `configured` excludes `id` + and `page_patterns`. Mismatched reuse would emit extra keys. Wire structs: + + ```rust + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct SlotJson { + pub id: String, + pub status: SlotStatus, + pub phase: EvidencePhaseJson, + pub configured: ConfiguredJson, + pub evidence: SlotEvidenceJson, + pub warnings: Vec, + } + + // §8 `configured`: div_id, gam_unit_path, formats, providers — NO id/page_patterns. + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct ConfiguredJson { + pub div_id: String, + pub gam_unit_path: String, + pub formats: Vec, + pub providers: Vec, + } + + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct FormatJson { + pub width: u32, + pub height: u32, + pub media_type: String, + } + + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct SlotEvidenceJson { + pub dom_id: Option, + pub gpt: Option, + } + + // §8 `evidence.gpt`: gam_unit_path, div_id, sizes — NO phase. + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct GptEvidenceJson { + pub gam_unit_path: String, + pub div_id: String, + pub sizes: Vec<[u32; 2]>, + } + + #[derive(Debug, Clone, Copy, PartialEq, Serialize)] + #[serde(rename_all = "snake_case")] + pub enum EvidencePhaseJson { InitialLoad, Scroll } + + #[derive(Debug, Clone, PartialEq, Serialize)] + pub struct ExtraEvidenceJson { + pub kind: String, + pub phase: EvidencePhaseJson, + pub dom_id: Option, + pub gam_unit_path: Option, + pub sizes: Vec<[u32; 2]>, + pub reason: String, + } + ``` + + Note `sizes` serialize as `[[300,250]]` (arrays of two ints), matching §8 — use + `[u32; 2]` here even though the compare module uses `(u32, u32)` tuples; the + Task 9 assembly maps tuple → `[w, h]`. The conversion from the compare + `SlotResult`/`SlotEvidence`/`ExtraEvidence` to these JSON types (dropping `phase` + from `gpt`, dropping `id`/`page_patterns` from `configured`) lives in Task 9 Step 7. + `Warning` is the single warning type for the whole CLI; defined here and re-exported + from `ad_templates::mod` so `compare.rs` reuses it (plain data, not JSON logic). + Keep `example_confirmed_with_extra_evidence()` and similar fixtures behind + `#[cfg(test)]`. + +- [ ] **Step 5: Add verification human-render helpers** + + Add only the **browser-verification** page summary writers here (used by + `audit::ad_templates` in Task 9), writing to `&mut dyn Write`; no `println!` / + `eprintln!`. Do **not** add static match/check/explain writers in this task — + those are the existing `write_match_result`/`format_slot`/etc. functions that + Task 6 Step 3 **moves** out of `config_ad_templates.rs`. Keeping the static + relocation solely in Task 6 avoids two competing copies of the same helpers in + `output.rs`. The verification writers added here may be unused until Task 9 (a + warn-level `dead_code` lint that does not fail `cargo test`); add + `#[allow(dead_code)]` if clippy is run between Task 4 and Task 9. + +- [ ] **Step 6: Run focused tests** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_templates::output --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli config_ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: all pass. + +- [ ] **Step 7: Commit** + + ```bash + git add Cargo.toml crates/trusted-server-cli/Cargo.toml crates/trusted-server-cli/src/ad_templates/mod.rs crates/trusted-server-cli/src/ad_templates/output.rs + git commit -m "Add ad-template CLI output models" + ``` + +## Task 5: Add Pure Evidence Comparison + +**Files:** + +- Create: `crates/trusted-server-cli/src/ad_templates/compare.rs` +- Modify: `crates/trusted-server-cli/src/ad_templates/mod.rs` + +- [ ] **Step 1: Write failing comparison tests** + + Cover every spec status and warning case without launching Chrome. Define small + test constructors so tests do not couple to the full `BrowserAdEvidence` field + list (`page_bids` and `warnings` default empty, evidence items default to + `EvidencePhase::InitialLoad`): + + ```rust + fn dom(id: &str) -> DomEvidence { + DomEvidence { dom_id: id.to_string(), phase: EvidencePhase::InitialLoad } + } + + fn gpt_slot(gam_unit_path: &str, div_id: &str, sizes: &[(u32, u32)]) -> GptSlotEvidence { + GptSlotEvidence { + gam_unit_path: gam_unit_path.to_string(), + div_id: div_id.to_string(), + sizes: sizes.to_vec(), + phase: EvidencePhase::InitialLoad, + } + } + + fn aps(slot_id: &str, sizes: &[(u32, u32)]) -> ApsFetchBidsEvidence { + ApsFetchBidsEvidence { slot_id: slot_id.to_string(), sizes: sizes.to_vec(), phase: EvidencePhase::InitialLoad } + } + + // Non-banner format helper for the unsupported-format test. + fn expected_slot_video(id: &str, div_id: &str, gam_unit_path: &str) -> ExpectedSlot { + ExpectedSlot { + id: id.to_string(), + div_id: div_id.to_string(), + gam_unit_path: gam_unit_path.to_string(), + formats: vec![ExpectedFormat { width: 0, height: 0, media_type: "video".to_string() }], + providers: Vec::new(), + page_patterns: Vec::new(), + } + } + + fn evidence(doms: Vec, gpts: Vec, aps: Vec) -> BrowserAdEvidence { + BrowserAdEvidence { + dom_ids: doms, + gpt_slots: gpts, + aps_calls: aps, + page_bids: Vec::new(), + warnings: Vec::new(), + } + } + + #[test] + fn gpt_path_div_and_size_overlap_confirms_slot() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &["aps"]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed, "GPT path+div+size overlap should confirm"); + assert!(result.slots[0].warnings.is_empty(), "confirmed slot should carry no warnings"); + } + + #[test] + fn dom_only_is_partial() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence(vec![dom("ad-atf-0")], Vec::new(), Vec::new()); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Partial, "DOM-only evidence should be partial"); + assert!( + result.slots[0].warnings.iter().any(|w| w.code == "dom_without_gpt"), + "DOM-only slot should warn dom_without_gpt" + ); + } + + #[test] + fn no_dom_or_gpt_is_missing() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence(Vec::new(), Vec::new(), Vec::new()); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Missing, "no DOM/GPT evidence should be missing"); + } + + #[test] + fn prefix_dom_resolution_ignores_container_suffix() { + let expected = expected_slot("header", "ad-header-0-", "/123/homepage/header", &[(728, 90)], &[]); + // First candidate ends with `-container` and must be skipped; the framework-suffixed ID resolves. + let evidence = evidence( + vec![dom("ad-header-0--container"), dom("ad-header-0-_R_abc123")], + Vec::new(), + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].evidence.dom_id.as_deref(), Some("ad-header-0-_R_abc123"), "prefix match should skip -container"); + assert_eq!(result.slots[0].status, SlotStatus::Partial, "DOM-only prefix match is partial without GPT"); + } + + #[test] + fn unmatched_gpt_slot_becomes_extra_evidence() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![ + gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)]), + gpt_slot("/123/publisher/right-rail", "ad-right-rail-0", &[(300, 250)]), + ], + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed, "matched slot still confirms"); + assert_eq!(result.extra_evidence.len(), 1, "unmatched GPT slot becomes extra evidence"); + assert_eq!(result.extra_evidence[0].kind, "gpt"); + assert!(!result.strict_failed(), "extra evidence alone must not fail strict"); + } + + #[test] + fn auction_disabled_skips_strict_missing_failure() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence(Vec::new(), Vec::new(), Vec::new()); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::auction_disabled()); + + assert_eq!(result.runtime_ad_stack_expected, RuntimeAdStackExpected::No, "auction disabled should set No"); + assert_eq!(result.slots[0].status, SlotStatus::Missing, "static status is still reported"); + assert!(!result.strict_failed(), "missing slot must not fail strict when ad stack expected is No"); + } + + // §5.4: GPT path+div match but no numeric size overlap -> partial + warning. + #[test] + fn gpt_incompatible_sizes_is_partial() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(728, 90)])], + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Partial, "no size overlap should be partial"); + assert!(result.slots[0].warnings.iter().any(|w| w.code == "incompatible_sizes")); + } + + // §5.4/§5.6: matched slot with only non-banner formats -> partial + unsupported_format. + #[test] + fn non_banner_only_slot_is_partial() { + let expected = expected_slot_video("video", "ad-video-", "/123/news/video"); + let evidence = evidence( + vec![dom("ad-video-0")], + vec![gpt_slot("/123/news/video", "ad-video-0", &[(640, 480)])], + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Partial, "non-banner-only should be partial"); + assert!(result.slots[0].warnings.iter().any(|w| w.code == "unsupported_format")); + } + + // §5.4: GPT element ID may be `${resolved_dom_id}-container` and still confirm. + #[test] + fn gpt_container_element_id_confirms() { + let expected = expected_slot("atf", "ad-atf-0", "/123/news/atf", &[(300, 250)], &[]); + let evidence = evidence( + vec![dom("ad-atf-0"), dom("ad-atf-0-container")], + vec![gpt_slot("/123/news/atf", "ad-atf-0-container", &[(300, 250)])], + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed, "container element id is a valid GPT div match"); + } + + // §5.4: out-of-page GPT slot is partial (so it fails strict) plus a warning. + #[test] + fn out_of_page_gpt_slot_warns_and_is_partial() { + let expected = expected_slot("interstitial", "ad-oop-", "/123/news/oop", &[(300, 250)], &[]); + // gpt_slot with empty sizes models an out-of-page slot (no numeric sizes). + let evidence = evidence(vec![dom("ad-oop-0")], vec![gpt_slot("/123/news/oop", "ad-oop-0", &[])], Vec::new()); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Partial, "a sizeless slot against banner formats is partial"); + assert!(result.slots[0].warnings.iter().any(|w| w.code == "out_of_page_slot")); + } + + // §5.5: matching APS fetchBids -> no provider warning. + #[test] + fn aps_match_adds_no_warning() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &["aps"]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + vec![aps("atf", &[(300, 250)])], + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed); + assert!(!result.slots[0].warnings.iter().any(|w| w.code.starts_with("aps_")), "matching APS should not warn"); + } + + // §5.5: configured aps provider but no APS evidence -> provider warning, still confirmed, strict not failed. + #[test] + fn aps_missing_warns_but_keeps_confirmed() { + let expected = expected_slot("atf", "ad-atf-", "/123/news/atf", &[(300, 250)], &["aps"]); + let evidence = evidence( + vec![dom("ad-atf-0")], + vec![gpt_slot("/123/news/atf", "ad-atf-0", &[(300, 250)])], + Vec::new(), + ); + + let result = compare_page_evidence(&[expected], &evidence, RuntimeGateSummary::unknown_allowed()); + + assert_eq!(result.slots[0].status, SlotStatus::Confirmed, "missing APS does not flip status"); + assert!(result.slots[0].warnings.iter().any(|w| w.code == "aps_evidence_missing")); + assert!(!result.strict_failed(), "provider warning alone must not fail strict"); + } + ``` + + Add a `#[cfg(test)]` constructor in `compare.rs` tests that builds a real + `ExpectedSlot` (the Task 3 type) so comparison tests stay readable: + + ```rust + fn expected_slot(id: &str, div_id: &str, gam_unit_path: &str, sizes: &[(u32, u32)], providers: &[&str]) -> ExpectedSlot { + ExpectedSlot { + id: id.to_string(), + div_id: div_id.to_string(), + gam_unit_path: gam_unit_path.to_string(), + formats: sizes.iter().map(|&(width, height)| ExpectedFormat { width, height, media_type: "banner".to_string() }).collect(), + providers: providers.iter().map(|p| p.to_string()).collect(), + page_patterns: Vec::new(), + } + } + ``` + +- [ ] **Step 2: Run focused test and verify it fails** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_templates::compare --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: compile failure because comparison module does not exist. + +- [ ] **Step 3: Implement browser evidence structs** + + Define the minimum collector-independent input shape plus the comparison result + shape the tests assert against: + + All browser-evidence input structs derive `Debug, Clone` and `serde::Deserialize` + (Task 8 decodes them from the collector's `window.__tsAdTemplateEvidence` JSON); + `EvidencePhase` deserializes from `"initial_load"` / `"scroll"`. The comparison- + result structs derive `Debug` (so the Step 1 `assert_eq!`/`matches!` assertions + compile) and `Clone`. Sizes are `(u32, u32)` tuples internally; deserialize them + from JSON `[w, h]` arrays. + + ```rust + #[derive(Debug, Clone, Copy, Eq, PartialEq, serde::Deserialize)] + #[serde(rename_all = "snake_case")] + pub enum EvidencePhase { + InitialLoad, + Scroll, + } + + #[derive(Debug, Clone, serde::Deserialize)] + pub struct DomEvidence { + pub dom_id: String, + pub phase: EvidencePhase, + } + + #[derive(Debug, Clone, serde::Deserialize)] + pub struct GptSlotEvidence { + pub gam_unit_path: String, + pub div_id: String, + pub sizes: Vec<(u32, u32)>, + pub phase: EvidencePhase, + } + + // APS `apstag.fetchBids` evidence (spec §5.5): configured slot ID + observed sizes. + #[derive(Debug, Clone, serde::Deserialize)] + pub struct ApsFetchBidsEvidence { + pub slot_id: String, + pub sizes: Vec<(u32, u32)>, + pub phase: EvidencePhase, + } + + // DEFERRED in this implementation: `/__ts/page-bids` SPA observation (spec §5.2 + // "when available"). The struct/field are forward scaffolding so the collector and + // JSON can grow it later; Task 8 does NOT populate it and Task 4 JSON does NOT + // surface it in Phase 1. Tracked as a deferred item in Risks. Keep the field so + // `BrowserAdEvidence` deserialization stays forward-compatible (default empty). + #[derive(Debug, Clone, serde::Deserialize)] + pub struct PageBidsEvidence { + pub slot_id: String, + pub phase: EvidencePhase, + } + + // `Warning` is the shared CLI warning type defined in Task 4 (`output.rs`) and + // re-exported from `ad_templates::mod`. It is plain data reused here (not JSON + // logic). Because the collector payload carries warnings, give `Warning` BOTH + // `Serialize` (Task 4 output) and `Deserialize` (Task 8 decode) derives. + use crate::ad_templates::output::Warning; + + #[derive(Debug, Clone, serde::Deserialize)] + pub struct BrowserAdEvidence { + pub dom_ids: Vec, + pub gpt_slots: Vec, + pub aps_calls: Vec, + #[serde(default)] + pub page_bids: Vec, + #[serde(default)] + pub warnings: Vec, + } + + // Comparison output. Uses the core `RuntimeAdStackExpected` enum from Task 1 so + // pure comparison logic does not depend on the output/JSON module. Task 4's + // `RuntimeAdStackExpectedJson` is produced only at serialization time. + #[derive(Debug, Clone)] + pub struct PageVerificationResult { + pub runtime_ad_stack_expected: trusted_server_core::creative_opportunities::RuntimeAdStackExpected, + pub slots: Vec, + pub extra_evidence: Vec, + } + + #[derive(Debug, Clone)] + pub struct SlotResult { + pub id: String, + pub status: SlotStatus, + pub phase: EvidencePhase, + pub evidence: SlotEvidence, + pub warnings: Vec, + } + + #[derive(Debug, Clone)] + pub struct SlotEvidence { + pub dom_id: Option, + pub gpt: Option, + } + + #[derive(Debug, Clone)] + pub struct ExtraEvidence { + pub kind: String, + pub phase: EvidencePhase, + pub dom_id: Option, + pub gam_unit_path: Option, + pub sizes: Vec<(u32, u32)>, + pub reason: String, + } + ``` + + `RuntimeGateSummary` is the third argument to `compare_page_evidence`; it wraps + the core gate result. Provide `RuntimeGateSummary::unknown_allowed()` (expected + `Unknown`) and `RuntimeGateSummary::auction_disabled()` (expected `No`) test + constructors so comparison tests do not rebuild gate inputs by hand. + +- [ ] **Step 4: Implement DOM/GPT/APS rules** + + Status rules: + - DOM exact ID first, then first prefix match, **excluding `-container`** wrappers + (slot-root resolution, spec §5.3). + - GPT confirms when: GAM unit path matches, the GPT slot element ID equals the + resolved DOM ID **or** an existing `${resolved_dom_id}-container` element + (spec §5.4 — note this is the GPT element-ID match, distinct from the §5.3 DOM + root resolution that skips `-container`), and at least one numeric banner size + overlaps. + - GPT path/div match with no numeric size overlap → `partial` (warn `incompatible_sizes`). + - Matched slot whose configured formats are **all non-banner** (video/native) → + `partial` (warn `unsupported_format`); banner is the only Phase-1 confirmable type. + - DOM-only (no GPT) → `partial` (warn `dom_without_gpt`). + - No DOM and no GPT → `missing`. + + Size-compatibility warnings (spec §5.4 — all are warnings, none flip a confirmed + slot to fail): emit a `Warning` for each of: + - `fluid_size_ignored` — non-numeric observed sizes like `"fluid"` ignored for matching; + - `extra_observed_size` — observed GPT sizes not in the configured set; + - `configured_size_not_observed` — configured sizes never observed (when ≥1 was); + - `out_of_page_slot` — out-of-page GPT slot with no sizes observed; the slot is + reported `partial`, which fails `--strict`. + + Provider + extra evidence: + - APS: configured `providers.aps.slot_id` with matching `fetchBids` → no warning; + missing/ambiguous APS evidence → provider warning only (`aps_evidence_missing` / + `aps_evidence_ambiguous`), never flips status or fails `--strict` in Phase 1. + - Unmatched live DOM/GPT/APS evidence → structured `extra_evidence` (never fails strict). + + Define each warning `code` as a stable string constant so output and tests share them. + +- [ ] **Step 5: Implement strict decision method** + + Add an inherent method on the result so tests can call `result.strict_failed()`: + + ```rust + impl PageVerificationResult { + pub fn strict_failed(&self) -> bool { + use trusted_server_core::creative_opportunities::RuntimeAdStackExpected; + if self.runtime_ad_stack_expected == RuntimeAdStackExpected::No { + return false; + } + self.slots + .iter() + .any(|slot| matches!(slot.status, SlotStatus::Missing | SlotStatus::Partial)) + } + } + ``` + + - false when `runtime_ad_stack_expected == No`; + - true for any `missing` or `partial` slot when expected is `Yes` or `Unknown`; + - false for provider warnings and extra evidence alone (they are not slot statuses). + +- [ ] **Step 6: Run focused tests** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_templates::compare --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: all comparison tests pass. + +- [ ] **Step 7: Commit** + + ```bash + git add crates/trusted-server-cli/src/ad_templates/mod.rs crates/trusted-server-cli/src/ad_templates/compare.rs + git commit -m "Add pure ad-template evidence comparison" + ``` + +## Task 6: Refactor Static Commands Onto Shared Modules + +**Files:** + +- Modify: `crates/trusted-server-cli/src/config_ad_templates.rs` +- Modify: `crates/trusted-server-cli/src/ad_templates/output.rs` +- Modify: `crates/trusted-server-cli/src/run.rs` + +- [ ] **Step 1: Add characterization tests before refactor** + + These guard behavior across the Step 3 move, so each must assert **exact output + substrings** (capture the command's `Vec`/`String` output and + `assert!(out.contains("..."))`), not just run without panicking — a bare + smoke test cannot catch a wording regression. Mirror the existing assertion style + at `config_ad_templates.rs:570-657`. Pin, with concrete expected strings: + - `lint` not configured → e.g. `"creative_opportunities: not configured"`; + - `lint` with slots + auction disabled → slot count line + `"auction: disabled"`; + - `match --details` → slot div ID, GAM unit path, formats, providers lines; + - `check --expect-no-slots` → success message; + - `check` failure with missing and unexpected slots → the exact failure lines; + - `explain` → each gate line (including `"auction providers configured"`) and the + EdgeZero legacy-fallback warning text. + + Run the existing tests first and copy the real emitted strings so the + characterization assertions match current behavior exactly before refactoring. + +- [ ] **Step 2: Run tests before refactor** + + Run: + + ```bash + cargo test -p trusted-server-cli config_ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: characterization tests pass against the current implementation. + +- [ ] **Step 3: Move formatting into `ad_templates::output`** + + Move these helpers out of `config_ad_templates.rs`: + - `write_match_result` + - `write_gate` + - `format_slot` + - `format_format` + - `format_providers` + - `join_set` + - `plural` + + Keep command functions small: parse args, load config, call expected/gate logic, render. + +- [ ] **Step 4: Reuse shared gate helper in `explain`** + + Build `AdStackGateInput` from explain flags and config: + + ```rust + let gate = evaluate_ad_stack_gate(AdStackGateInput { + method_get, + navigation: !args.non_navigation, + prefetch: args.prefetch, + bot: args.bot, + matched_slots: !expected.slots.is_empty(), + consent_allows_auction: Some(!args.consent_denied), + auction_enabled: loaded.settings.auction.enabled, + }); + ``` + + Render the seven shared gate names from `gate` rather than hand-rolled boolean chains. + + **Preserve the explain-only provider gate.** The current `run_explain` + (`config_ad_templates.rs:270`) renders an eighth gate, + `"auction providers configured"` (`!loaded.settings.auction.providers.is_empty()`, + line 299), and ANDs it into its local `runs_ad_stack` decision (line 302). The + shared `evaluate_ad_stack_gate` helper and runtime `should_run_server_side_ad_stack` + intentionally have no provider-configured gate. Do not fold this into + `AdStackGateInput`. Keep `"auction providers configured"` as an explain-only + supplementary `write_gate(...)` line rendered alongside the shared result, and + keep it in `explain`'s own `runs_ad_stack` decision: + + ```rust + let providers_configured = !loaded.settings.auction.providers.is_empty(); + render_shared_gates(out, &gate)?; + write_gate(out, "auction providers configured", providers_configured)?; + let runs_ad_stack = + gate.expected == RuntimeAdStackExpected::Yes && providers_configured; + ``` + + This keeps `explain` output and behavior identical to the current implementation + (verified by the Step 1 characterization test) while still sharing the seven core + runtime gates with `publisher.rs`. + +- [ ] **Step 5: Run focused tests** + + Run: + + ```bash + cargo test -p trusted-server-cli config_ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: no output regressions except intentional wording updates covered by tests. + +- [ ] **Step 6: Commit** + + ```bash + git add crates/trusted-server-cli/src/config_ad_templates.rs crates/trusted-server-cli/src/ad_templates/output.rs crates/trusted-server-cli/src/run.rs + git commit -m "Refactor static ad-template commands" + ``` + +## Task 7: Port Generic Audit Namespace And Browser Collector + +**Files:** + +- Modify: `Cargo.toml` +- Modify: `crates/trusted-server-cli/Cargo.toml` +- Create: `crates/trusted-server-cli/src/audit/mod.rs` +- Create: `crates/trusted-server-cli/src/audit/page.rs` +- Create: `crates/trusted-server-cli/src/audit/collector.rs` +- Create: `crates/trusted-server-cli/src/audit/browser.rs` +- Modify: `crates/trusted-server-cli/src/lib.rs` +- Modify: `crates/trusted-server-cli/src/run.rs` + +- [ ] **Step 1: Add browser dependencies** + + Add `chromiumoxide` to the root `[workspace.dependencies]` table (inert until a + crate references it via `{ workspace = true }`): + + ```toml + [workspace.dependencies] + # ... existing entries ... + chromiumoxide = "0.9.1" + ``` + + Add the host deps to the CLI crate under its existing + `[target.'cfg(not(target_arch = "wasm32"))'.dependencies]` table — NOT a plain + `[dependencies]` table (workspace default target is wasm32; an unconditional dep + compiles for wasm and breaks the build / leaks host-only crates): + + ```toml + [target.'cfg(not(target_arch = "wasm32"))'.dependencies] + # ... existing clap/url/serde/etc ... + chromiumoxide = { workspace = true } + futures = { workspace = true } + tempfile = { workspace = true } + tokio = { workspace = true } + which = { workspace = true } + ``` + + Verify the root workspace already provides `futures`, `tempfile`, `tokio`, `which` + (it does on this branch); only `chromiumoxide` is a new workspace entry. + +- [ ] **Step 2: Write failing audit parser tests** + + In `run.rs` tests: + + ```rust + #[test] + fn audit_legacy_url_parses_as_page_alias() { + let args = parse(&["ts", "audit", "https://www.example.com/"]); + assert!(matches!(args.command, Command::Audit(_))); + } + + #[test] + fn audit_page_subcommand_parses() { + let args = parse(&["ts", "audit", "page", "https://www.example.com/"]); + assert!(matches!(args.command, Command::Audit(_))); + } + + #[test] + fn audit_ad_templates_verify_parses() { + let args = parse(&["ts", "audit", "ad-templates", "verify", "https://www.example.com/"]); + assert!(matches!(args.command, Command::Audit(_))); + } + + #[test] + fn audit_ad_templates_is_not_legacy_url() { + assert!(Args::try_parse_from(["ts", "audit", "ad-templates"]).is_err()); + } + ``` + +- [ ] **Step 3: Run parser tests and verify failure** + + Run: + + ```bash + cargo test -p trusted-server-cli audit_ --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: compile failure because `Audit` command does not exist. + +- [ ] **Step 4: Implement audit Clap namespace in current `run.rs` shape** + + Do not add stale #800 `args.rs`. Add a `Command::Audit(AuditArgs)` variant to the + existing `Command` enum, plus the full audit arg surface. The parser tests in + Step 2 exercise `ad-templates verify`, so the **entire** command surface (including + the verify args) must be defined here for those tests to compile. Task 9 implements + the verify _behavior_ only — it does not redefine these arg types. + + **Visibility:** `audit::run_audit` lives in `audit/mod.rs` and must name these + types in its signature and match their variants, so every audit arg type and its + fields are `pub(crate)` (not private). `PageAuditArgs` (from `audit/page.rs`, Task 7 + Step 5) and `AuditAdTemplatesVerifyArgs` are likewise `pub(crate)`/`pub`. `run.rs` + imports `AuditArgs` for the `Command::Audit(AuditArgs)` variant; everything else is + read by `audit/mod.rs`. (This mirrors `config_ad_templates::AdTemplatesCommand`, + which is `pub` and consumed by `run.rs`.) + + ```rust + // value parser shared by legacy_url and verify urls; rejects non-HTTP(S) schemes. + pub(crate) fn parse_http_url(raw: &str) -> Result { + let url = url::Url::parse(raw).map_err(|error| format!("invalid URL `{raw}`: {error}"))?; + match url.scheme() { + "http" | "https" => Ok(url), + other => Err(format!("unsupported URL scheme `{other}` (expected http or https)")), + } + } + + #[derive(Debug, clap::Args)] + pub(crate) struct AuditArgs { + #[command(subcommand)] + pub(crate) command: Option, + #[arg(value_parser = parse_http_url, hide = true)] + pub(crate) legacy_url: Option, + } + + #[derive(Debug, Subcommand)] + pub(crate) enum AuditSubcommand { + Page(PageAuditArgs), + #[command(name = "ad-templates", subcommand)] + AdTemplates(AuditAdTemplatesCommand), + } + + #[derive(Debug, Subcommand)] + pub(crate) enum AuditAdTemplatesCommand { + Verify(AuditAdTemplatesVerifyArgs), + } + + // Defined here (not Task 9) so parser tests compile. Task 9 fills in the handler. + #[derive(Debug, clap::Args)] + pub(crate) struct AuditAdTemplatesVerifyArgs { + #[command(flatten)] + pub config: AppConfigArgs, + #[arg(required = true, value_parser = parse_http_url)] + pub urls: Vec, + #[arg(long)] + pub strict: bool, + #[arg(long)] + pub json: bool, + #[arg(long)] + pub scroll: bool, + } + ``` + + Dispatch `Command::Audit(args)` to a single `audit::run_audit(args: AuditArgs)` + entry point (in `audit/mod.rs`) that normalizes the namespace: `legacy_url` (if + present) and `Page` both route to the generic page audit; `AdTemplates(Verify(..))` + routes to the verifier (a stub returning `Ok(())` until Task 9). If Clap cannot make + the optional-subcommand-plus-hidden-positional contract unambiguous, implement a + small `AuditArgs::normalize()` that rejects `legacy_url` values that are not HTTP(S). + Decide arg-type home consistently: keep them in `run.rs` as `pub(crate)` (as shown) + and import into `audit/mod.rs`, or move them next to `run_audit` in `audit/mod.rs` + and import `AuditArgs` into `run.rs` — either works, but do not split them. + +- [ ] **Step 5: Port minimal generic page audit** + + Port useful #800 concepts into `audit/page.rs`, but keep output read-only by default for now: + - parse/validate URL; + - call `AuditCollector::collect_page`; + - print summary with final URL, title, script/resource counts, warnings; + - no draft config generation in this PR unless #800 rebase keeps it explicitly. + +- [ ] **Step 6: Implement collector trait and browser collector base** + + `audit/collector.rs` — define the trait plus its concrete request/response types so + Task 9's `FakeCollector` and the verify orchestration have a contract to assert on: + + ```rust + pub trait AuditCollector { + fn collect_page(&self, request: BrowserCollectRequest) -> Result; + } + + pub struct BrowserCollectRequest { + pub url: url::Url, + // Pre-navigation init scripts (evaluate-on-new-document). Empty for plain page audit; + // Task 8 passes the ad-template collector script here. + pub init_scripts: Vec, + pub scroll: bool, + } + + pub struct CollectedPage { + pub final_url: url::Url, + pub title: String, + // Generic page-audit signals (counts only; no page HTML/cookies/storage). + pub script_count: usize, + pub resource_count: usize, + pub warnings: Vec, + // Present only when an ad-template init script was injected (Task 8); None for + // plain `ts audit page`. This is how `BrowserAdEvidence` rides on a CollectedPage. + pub ad_evidence: Option, + } + ``` + + `BrowserCollectRequest` carries `init_scripts` + `scroll` so ad-template verification + enables evidence hooks without changing the trait later. + + `audit/browser.rs` should port #800's: + - `which` browser lookup; + - isolated `TempDir` profile; + - current-thread Tokio runtime; + - `Browser::launch`; + - `page.goto`; + - `wait_for_navigation_response`; + - settle loop. + +- [ ] **Step 7: Run compile-focused CLI tests** + + Run: + + ```bash + cargo test -p trusted-server-cli audit_ --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: parser and non-browser unit tests pass. No test should require installed Chrome yet. + +- [ ] **Step 8: Commit** + + ```bash + git add Cargo.toml crates/trusted-server-cli/Cargo.toml crates/trusted-server-cli/src/audit crates/trusted-server-cli/src/lib.rs crates/trusted-server-cli/src/run.rs + git commit -m "Add audit namespace and browser collector base" + ``` + +## Task 8: Add Browser Ad-Template Evidence Collector + +**Files:** + +- Create: `crates/trusted-server-cli/src/audit/ad_template_collector.js` +- Modify: `crates/trusted-server-cli/src/audit/browser.rs` +- Modify: `crates/trusted-server-cli/src/audit/collector.rs` +- Modify: `crates/trusted-server-cli/src/ad_templates/compare.rs` + +- [ ] **Step 1: Write JS collector contract fixture tests** + + Add Rust unit tests that inspect generated init-script text and decode a mocked `window.__tsAdTemplateEvidence` JSON payload. These should not launch Chrome. + + Prefer **behavioral** assertions over brittle substring matching: where possible, + assert by decoding a mocked `window.__tsAdTemplateEvidence` payload into + `BrowserAdEvidence` and checking fields. For the few structural checks that must + inspect the script text, pin **exact** marker substrings (no "or equivalent", so + the pass condition is deterministic) — choose the markers to match the strings the + implementation will actually emit: + - `build_ad_template_init_script` output contains the literal `__TS_CONFIG` injection; + - contains the chosen googletag-hook marker (pick ONE and pin it, e.g. + `Object.defineProperty(window, "googletag"`); + - contains the `cmd.push` wrap marker; + - contains the `defineSlot` record marker; + - contains the `apstag.fetchBids` wrap marker; + - embeds only the configured div prefixes / provider IDs passed via `__TS_CONFIG` + (assert a non-configured prefix is absent). + +- [ ] **Step 2: Run tests and verify failure** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_template_collector --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: failure because collector script/builder does not exist. + +- [ ] **Step 3: Implement init script builder** + + In Rust, build script as: + + ```rust + pub fn build_ad_template_init_script(config: &AdTemplateCollectorConfig) -> Result { + let config_json = serde_json::to_string(config) + .map_err(|error| format!("failed to serialize ad-template collector config: {error}"))?; + Ok(format!(";(() => {{ const __TS_CONFIG = {config_json};\n{}\n}})();", include_str!("ad_template_collector.js"))) + } + ``` + + Keep the JS file generic; pass configured prefixes and APS slot IDs through `__TS_CONFIG`. + +- [ ] **Step 4: Implement read-only JS evidence collection** + + In `ad_template_collector.js`, write to `window.__tsAdTemplateEvidence`: + - `dom_ids`: matched IDs from configured prefixes, excluding `-container`; + - `gpt_slots`: record `defineSlot` calls observed **both** directly **and** when + dispatched from the `googletag.cmd` queue (wrap `cmd.push` so queued callbacks are + instrumented without changing their order — spec §7), **plus** a post-settle + `googletag.pubads().getSlots()` scrape. For each scraped slot capture + `getAdUnitPath()`, `getSlotElementId()`, and `getSizes()` so `getSlots()`-only + slots still carry numeric `sizes` for the §5.4 overlap rule. Normalize sizes from + both `defineSlot` input and `getSizes()` output: `[300,250]` → one `(300,250)`; + `[[300,250],[728,90]]` → two pairs; non-numeric (`"fluid"`) dropped from numeric + sizes and surfaced as a `fluid_size_ignored` warning; + - `aps_calls`: `fetchBids` payloads (configured slot IDs + sizes); + - `warnings`: collector warnings only ({code, message}), no page HTML/cookies/storage. + + Always call original page functions with unchanged arguments, and never override + `navigator.webdriver` (spec §7). + +- [ ] **Step 5: Add browser collector extraction** + + After settle and after optional scroll, evaluate: + + ```javascript + ;() => window.__tsAdTemplateEvidence || null + ``` + + Decode into `BrowserAdEvidence`. If decode fails, return a page warning rather than failing navigation. + +- [ ] **Step 6: Add deterministic scroll** + + In `audit/browser.rs`, implement `scroll` by evaluating: + + ```javascript + ;async () => { + const height = Math.max( + document.body.scrollHeight, + document.documentElement.scrollHeight + ) + for (const y of [ + Math.floor(height * 0.33), + Math.floor(height * 0.66), + height, + ]) { + window.scrollTo(0, y) + await new Promise((resolve) => setTimeout(resolve, 250)) + } + window.scrollTo(0, 0) + } + ``` + + Then wait for the same settle quiet period and collect evidence with `phase = "scroll"` where the JS script marks new observations. + +- [ ] **Step 7: Run focused tests** + + Run: + + ```bash + cargo test -p trusted-server-cli ad_template_collector --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli audit::browser --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: unit tests pass without launching Chrome. + +- [ ] **Step 8: Commit** + + ```bash + git add crates/trusted-server-cli/src/audit/ad_template_collector.js crates/trusted-server-cli/src/audit/browser.rs crates/trusted-server-cli/src/audit/collector.rs crates/trusted-server-cli/src/ad_templates/compare.rs + git commit -m "Collect browser ad-template evidence" + ``` + +## Task 9: Implement `ts audit ad-templates verify` + +**Files:** + +- Create: `crates/trusted-server-cli/src/audit/ad_templates.rs` +- Modify: `crates/trusted-server-cli/src/audit/mod.rs` +- Modify: `crates/trusted-server-cli/src/run.rs` +- Modify: `crates/trusted-server-cli/src/ad_templates/output.rs` + +- [ ] **Step 1: Write failing orchestration tests with a fake collector** + + Build a fake collector implementing `AuditCollector` and test: + - one confirmed page exits success in default mode; + - strict missing slot returns error; + - `[auction].enabled = false` returns runtime skipped and does not strict-fail missing evidence; + - one page navigation error plus one success sets JSON `ok = false`; + - invalid `ftp://` URL fails before fake collector is called; + - redirect uses final path for expected slots and emits redirect warning. + + Define the test scaffolding explicitly (no dangling helpers): + + ```rust + // Maps each requested URL to a canned outcome so orchestration is tested without Chrome. + struct FakeCollector { + pages: std::collections::HashMap>, + } + + impl FakeCollector { + // Success page: requested -> final_url, carrying the given ad evidence. + fn page(requested: &str, final_url: &str, evidence: BrowserAdEvidence) -> Self { + let mut pages = std::collections::HashMap::new(); + pages.insert( + requested.to_string(), + Ok(CollectedPage { + final_url: url::Url::parse(final_url).expect("valid final url"), + title: String::new(), + script_count: 0, + resource_count: 0, + warnings: Vec::new(), + ad_evidence: Some(evidence), + }), + ); + Self { pages } + } + // Helper to add a failing page for multi-URL tests. + fn with_error(mut self, requested: &str, message: &str) -> Self { + self.pages.insert(requested.to_string(), Err(message.to_string())); + self + } + } + + impl AuditCollector for FakeCollector { + fn collect_page(&self, request: BrowserCollectRequest) -> Result { + self.pages + .get(request.url.as_str()) + .cloned() + .unwrap_or_else(|| Err(format!("no fake page for {}", request.url))) + } + } + + impl BrowserAdEvidence { + // #[cfg(test)] fixture: one confirmed news slot (atf / ad-atf-0 / /123/news/atf, 300x250). + fn confirmed_news_slot() -> Self { + BrowserAdEvidence { + dom_ids: vec![DomEvidence { dom_id: "ad-atf-0".into(), phase: EvidencePhase::InitialLoad }], + gpt_slots: vec![GptSlotEvidence { + gam_unit_path: "/123/news/atf".into(), + div_id: "ad-atf-0".into(), + sizes: vec![(300, 250)], + phase: EvidencePhase::InitialLoad, + }], + aps_calls: Vec::new(), + page_bids: Vec::new(), + warnings: Vec::new(), + } + } + } + + // Runs the verify orchestration with `--json` over `urls` and returns parsed JSON. + // Loads a #[cfg(test)] effective config whose `/news/*` slot is the atf slot above. + fn run_verify_json(collector: &dyn AuditCollector, urls: impl IntoIterator) -> serde_json::Value { /* impl in test module */ } + + #[test] + fn verify_uses_final_url_for_matching_after_redirect() { + let collector = FakeCollector::page( + "https://www.example.com/", + "https://www.example.com/news/story", + BrowserAdEvidence::confirmed_news_slot(), + ); + let json = run_verify_json(&collector, ["https://www.example.com/"]); + + assert_eq!(json["pages"][0]["path"], "/news/story"); + // Warning order is unspecified; assert presence, not index 0. + let warnings = json["pages"][0]["warnings"].as_array().expect("warnings array"); + assert!( + warnings.iter().any(|w| w["code"] == "redirected"), + "redirect should emit a `redirected` warning" + ); + } + ``` + + `run_verify_json` calls the same `run_verify` entry point used in production but + with the fake collector injected and output captured; define it in the test module + so all six listed cases share it. + +- [ ] **Step 2: Run focused tests and verify failure** + + Run: + + ```bash + cargo test -p trusted-server-cli audit::ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: compile failure because verifier module does not exist. + +- [ ] **Step 3: Wire the verifier handler** + + `AuditAdTemplatesVerifyArgs` already exists from Task 7, Step 4. Replace the Task 7 + stub so `audit::run_audit` routes `AdTemplates(Verify(args))` into a new + `audit::ad_templates::run_verify(args)`. Do not redefine the arg struct. + +- [ ] **Step 4: Implement verification orchestration** + + For each URL: + 1. Collect browser page with ad-template init script and optional scroll. + 2. Parse final URL and normalize final path. + 3. Build expected slots for final path. + 4. Build gate summary using shared core gate helper with `consent_allows_auction = None`. + 5. Add redirect warning (`code = "redirected"`) if requested path differs from final path. + 6. Compare evidence (`compare_page_evidence`) to get a `PageVerificationResult`. + 7. **Assemble the wire `PageJson`** (Task 4 type) from the pieces the comparison + result does not carry: `url` / `final_url` / `requested_path` / `path`, + `gates` (map the gate summary's per-gate states to `GateState`), + `matched_slot_count`, `runtime_ad_stack_expected` (via the `From` impl on + `RuntimeAdStackExpectedJson`), then the `slots` / `extra_evidence` / `warnings` + from the comparison result. `PageVerificationResult` is intentionally URL- and + gate-agnostic; this step is where per-page request context is joined in. + 8. Preserve page-level errors as a `PageJson` with `error: Some(..)` and continue + remaining URLs. + +- [ ] **Step 5: Implement exit behavior** + - Default auditor-assist mode: return `Ok(())` for missing/partial evidence when no page-level collection errors occur. + - `--strict`: return `Err(String)` when any non-skipped page has missing/partial slot. + - Multi-URL page errors: JSON `ok=false`; command returns `Err(String)` after writing JSON/human output. + - Invalid schemes: fail before browser launch and before any output. + +- [ ] **Step 6: Run focused tests** + + Run: + + ```bash + cargo test -p trusted-server-cli audit::ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + cargo test -p trusted-server-cli ad_templates --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: verifier and pure comparison tests pass. + +- [ ] **Step 7: Commit** + + ```bash + git add crates/trusted-server-cli/src/audit/ad_templates.rs crates/trusted-server-cli/src/audit/mod.rs crates/trusted-server-cli/src/run.rs crates/trusted-server-cli/src/ad_templates/output.rs + git commit -m "Verify ad-template slots from browser evidence" + ``` + +## Task 10: Add Local Browser Fixture Tests + +**Files:** + +- Modify: `crates/trusted-server-cli/src/audit/browser.rs` +- Modify: `crates/trusted-server-cli/src/audit/ad_templates.rs` + +- [ ] **Step 1: Add test-only local HTTP fixture helper** + + In `audit::browser` tests, create a `TcpListener` serving static HTML from strings. Keep it test-only and host-target only. + + Fixture pages: + - direct `googletag.defineSlot`; + - `googletag.cmd.push`; + - late `window.googletag = { cmd: [] }`; + - late `window.apstag`; + - lazy slot created after scroll; + - redirect from `/` to `/news/story`; + - navigation returning 500. + +- [ ] **Step 2: Gate tests when Chrome is unavailable** + + Add helper: + + ```rust + fn chrome_available() -> bool { + ["chrome", "chromium", "google-chrome", "google-chrome-stable"] + .iter() + .any(|name| which::which(name).is_ok()) + } + ``` + + Each browser fixture test should early-return when unavailable. Do not use + `println!` / `eprintln!`; keep the skip reason in the helper name or a skipped + assertion message so clippy stays clean. This keeps CI portable unless Chrome is + installed. + +- [ ] **Step 3: Write fixture tests** + + Tests should assert the collector sees evidence, not real ad network behavior: + - direct GPT evidence confirms; + - command-queue GPT evidence confirms; + - APS `fetchBids` evidence removes APS provider warning; + - lazy slot appears only when `--scroll` is set; + - redirect result uses final path; + - failed page produces page-level error while other pages continue. + +- [ ] **Step 4: Run fixture tests locally** + + Run: + + ```bash + cargo test -p trusted-server-cli browser_fixture --target $(rustc -vV | sed -n 's/^host: //p') -- --nocapture + ``` + + Expected: pass when Chrome/Chromium exists; otherwise tests skip with explicit message. + +- [ ] **Step 5: Commit** + + ```bash + git add crates/trusted-server-cli/src/audit/browser.rs crates/trusted-server-cli/src/audit/ad_templates.rs + git commit -m "Add browser fixtures for ad-template verification" + ``` + +## Task 11: Update Documentation And Help Snapshots + +**Files:** + +- Modify: `docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md` if implementation decisions differ. +- Modify: `trusted-server.example.toml` only if command examples need harmless fictional config comments. +- Modify: `CLAUDE.md` only if verification commands or CLI command surface need to be documented. + +- [ ] **Step 1: Run CLI help manually** + + Run: + + ```bash + cargo run -p trusted-server-cli --target $(rustc -vV | sed -n 's/^host: //p') -- audit --help + cargo run -p trusted-server-cli --target $(rustc -vV | sed -n 's/^host: //p') -- audit ad-templates verify --help + cargo run -p trusted-server-cli --target $(rustc -vV | sed -n 's/^host: //p') -- config ad-templates --help + ``` + + Expected: nested audit commands are discoverable; hidden legacy `ts audit ` does not dominate help text. + +- [ ] **Step 2: Update docs if help text or behavior differs from spec** + + Keep examples using `https://www.example.com/` only. Do not mention real publisher sites. + +- [ ] **Step 3: Run docs format** + + Run: + + ```bash + cd docs && npm run format + ``` + + Expected: Prettier passes. + +- [ ] **Step 4: Commit** + + ```bash + git add docs trusted-server.example.toml CLAUDE.md + git commit -m "Document ad-template CLI verification" + ``` + + If no docs changed, skip the commit. + +## Task 12: Final Verification + +**Files:** + +- Verify all touched files. + +- [ ] **Step 1: Rust format** + + Run: + + ```bash + cargo fmt --all -- --check + ``` + + Expected: pass. + +- [ ] **Step 2: Host CLI tests** + + Run: + + ```bash + cargo test -p trusted-server-cli --target $(rustc -vV | sed -n 's/^host: //p') + ``` + + Expected: pass. Browser fixture tests either pass or explicitly skip when Chrome/Chromium is unavailable. + +- [ ] **Step 3: Workspace tests** + + Run: + + ```bash + cargo test --workspace + ``` + + Expected: pass. + +- [ ] **Step 4: Clippy** + + Run: + + ```bash + cargo clippy --workspace --all-targets --all-features -- -D warnings + ``` + + Expected: pass. + +- [ ] **Step 5: Wasm isolation proof** + + `trusted-server-adapter-fastly` does **not** depend on `trusted-server-cli`, so the + adapter build never compiles the CLI crate and cannot detect a CLI-crate dep leak. + The real proof is building the **CLI crate itself** for the wasm target (its modules + are `#[cfg(not(target_arch = "wasm32"))]`, so a wasm build must succeed with the + host-only deps compiled out). Note the workspace default target is already + `wasm32-wasip1`, so Steps 3–4 (`cargo test/clippy --workspace`) also build the CLI + crate for wasm — but make the isolation check explicit: + + ```bash + # Real CLI isolation proof: CLI crate must build for wasm with host deps excluded. + cargo build --package trusted-server-cli --target wasm32-wasip1 + # Adapter still built to confirm the production artifact is unaffected. + cargo build --package trusted-server-adapter-fastly --release --target wasm32-wasip1 + ``` + + Expected: both pass. If `chromiumoxide`/`tokio`/etc. leaked into a non-target-cfg + dependency table, the first command fails — that is the guard. + +- [ ] **Step 6: Docs format** + + Run: + + ```bash + cd docs && npm run format + ``` + + Expected: pass. + +- [ ] **Step 7: Inspect final diff** + + Run: + + ```bash + git status --short + git diff --stat origin/server-side-ad-templates-impl...HEAD + git log --oneline origin/server-side-ad-templates-impl..HEAD + ``` + + Expected: only intended CLI/core/doc files changed; no `.env`, operator `trusted-server.toml`, or generated browser artifacts included. + +## Risks And Watch Points + +- `chromiumoxide` must remain a host-only `trusted-server-cli` dependency. Any wasm build failure here means the dependency leaked. +- `ts audit ` compatibility must not swallow `ts audit ad-templates` as a URL. +- Runtime gate extraction (Task 1) only touches `should_run_server_side_ad_stack` + (the navigation gate). `/__ts/page-bids` is **intentionally NOT routed** through + `evaluate_ad_stack_gate` — its gate semantics differ (bot/prefetch skip the auction + but keep slots; no `is_navigation`/`is_get` gate). Its parity is preserved by + leaving it untouched, not by sharing the helper. Do not reroute page-bids. Keep + existing publisher and page-bids tests passing. +- The browser collector must not capture page HTML, cookies, storage, request bodies, or arbitrary DOM. Only collect configured-prefix DOM IDs and ad-related evidence. Never override `navigator.webdriver`. +- `runtime_ad_stack_expected = "unknown"` is normal for live consent state; do not over-model consent unless the collector can prove it. +- Browser fixture tests must not depend on real GPT/APS network calls. +- **Deferred in this implementation:** `/__ts/page-bids` SPA observation (spec §5.2 + "when available"). `PageBidsEvidence` exists as forward scaffolding but is not + collected (Task 8), surfaced in JSON (Task 4), or tested. Revisit if SPA route + verification is prioritized. +- Keep generation (`ts audit ad-templates generate`) out of this PR. diff --git a/docs/superpowers/plans/2026-07-06-cache-control-header-implementation-plan.md b/docs/superpowers/plans/2026-07-06-cache-control-header-implementation-plan.md new file mode 100644 index 000000000..8546ebf6e --- /dev/null +++ b/docs/superpowers/plans/2026-07-06-cache-control-header-implementation-plan.md @@ -0,0 +1,456 @@ +# Cache-Control Header Strategy Implementation Plan + +**Date:** 2026-07-06 +**Status:** Initial cache-header slice implemented in the current branch +**Spec:** `docs/superpowers/specs/2026-07-06-cache-control-header-design.md` + +## Scope + +Implement the **initial cache-header slice** from the current spec. The latest +spec resolves the initial-slice open questions and defers the larger dynamic +caching, template caching, streaming, and compression-offload work. + +Initial slice goals: + +1. Make TS-owned, hash-versioned TSJS responses cache correctly. +2. Make neutralized publisher Prebid compatibility responses safe to cache. +3. Add a structured, runtime-portable cache-policy model. +4. Add a configurable static/rehosted asset cache-rule engine so framework + assumptions are operator-controlled, not hard-coded. +5. Keep arbitrary publisher-origin assets origin-controlled unless an enabled + rule proves they are immutable-safe. + +Deferred follow-up features are listed separately below and should not be folded +into the initial cache-header PRs. + +## Decisions locked for the initial slice + +- SSAT-assembled HTML remains `Cache-Control: private, max-age=0` and strips + runtime edge-cache headers (`Surrogate-Control`, `Fastly-Surrogate-Control`, + `CDN-Cache-Control`, and `Cloudflare-CDN-Cache-Control`) whenever the ad stack + can inject per-user slot/bid state. +- TSJS keeps the current `/static/tsjs=...js?v=` canonical URL shape. + Matching hash/version requests receive immutable cache headers; missing or + mismatched hash/version requests keep short TTLs rather than redirecting. +- Runtime cache-key configuration must preserve the `v` query parameter for + `/static/tsjs=`. Fastly and Cloudflare include query strings in default cache + keys, but project-specific query normalization must not drop `v`. +- Framework-specific immutable paths, including Next.js `/_next/static/*`, must + be represented as configurable cache-rule presets. Do not add adapter- or + proxy-level hard-coded framework path checks. +- Operators decide which framework presets and publisher allowlists are enabled. + Arbitrary publisher CSS/JS/images remain origin-controlled unless an enabled + cache rule proves they are immutable-safe. +- TS-owned Prebid delivery is covered by deferred TSJS module URLs. Publisher + Prebid script URLs neutralized by TS are compatibility shims at stable URLs and + must use `no-store` or a very short TTL, not a year-long immutable policy. +- Fastly rehosted assets are TS-owned copies once TS rewrites/hosts them. A + matching rehost rule is authoritative over third-party origin cache defaults. Use + immutable only for TS-fingerprinted rehosted URLs, and preserve any later + TS/operator `private` or `no-store` decision as the final veto. +- Fastly and Cloudflare are the MVP runtime targets. This slice emits their + runtime-specific directives; actual Fastly storage integration and cache-key + verification are tracked in [#908](https://github.com/IABTechLab/trusted-server/issues/908). + Akamai mapping is deferred until Akamai is on the roadmap. +- Dynamic HTML/RSC/API caching, dynamic `Vary`/cache-key normalization, + origin-template caching, transformed-template caching, true publisher-origin + streaming, parser-context bid splice, EdgeZero streaming parity, and SSAT HTML + compression offload are deferred follow-up features. +- All personalized/cookie-bearing response hardening in `response_privacy.rs` and + adapter middleware stays in place and runs after any new policy application. + +## Original baseline before this implementation + +| Area | Current file(s) | Baseline | +| ---------------------------- | --------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | +| TSJS URL injection | `crates/trusted-server-core/src/tsjs.rs` | Injects `/static/tsjs=...js?v=`; current branch is moving hash work out of the hot path. | +| TSJS serving | `publisher.rs`, `http_util.rs` | Historically served through `serve_static_with_etag` with 5-minute browser/edge TTLs. | +| Cache policy primitives | `cache_policy.rs` | Current branch adds typed policy rendering; still needs final alignment with no-store and config rules. | +| Neutralized Prebid shim | `crates/trusted-server-core/src/integrations/prebid.rs` | `handle_script_handler` currently returns an empty JS shim with `public, max-age=31536000`; this must be changed. | +| Cache privacy | `publisher.rs`, `response_privacy.rs`, adapter middleware | Ad-stack HTML and cookie-bearing responses are downgraded to private/shared-uncacheable. | +| Rehosted asset cache policy | `proxy.rs` | Policy is effectively origin-controlled or `no-store, private`; no normalized immutable/SWR policy. | +| Dynamic HTML/RSC/API caching | none | Deferred. No initial-slice `Vary` rewriting or Next router-header special casing. | +| Origin-template cache | none | Deferred. No cache API/override/template key/surrogate-key implementation exists. | + +## Definition of done for the initial slice + +- TSJS hash-version-matching requests emit one-year immutable browser cache and + one-year edge cache headers. +- TSJS missing/mismatched hash requests keep short TTL behavior. +- TSJS injected hash generation no longer concatenates and hashes the full + bundle on every page view. +- Neutralized publisher Prebid shim responses use `no-store` or a very short TTL. +- Cache policy is represented as structured data and can emit Fastly + `Surrogate-Control`, generic `CDN-Cache-Control`, Cloudflare-specific + `Cloudflare-CDN-Cache-Control`, and `s-maxage` fallback headers. +- Cache policy can represent `no-store`/uncacheable responses as well as public + and private TTL policies. +- TS config expresses static/rehosted cache policy through structured rules with + match criteria, policy fields, and `enabled` flags. +- Built-in framework presets, including Next.js `/_next/static/*`, are + implemented through the shared rule engine and can be disabled/overridden. +- Arbitrary publisher-origin assets remain origin-controlled unless matched by an + enabled preset or publisher allowlist. +- Fastly TS-owned rehosted assets have explicit normalized policies instead of + blindly passing through third-party defaults. +- MVP adapters emit the correct edge-cache header from shared policy: Fastly + `Surrogate-Control`, Cloudflare `CDN-Cache-Control` / + `Cloudflare-CDN-Cache-Control`, or portable `s-maxage` fallback. Header + emission is complete; runtime storage and cache-key verification remain in + #908. +- Deferred features are documented as deferred and are not accidentally + implemented as hard-coded Next.js/dynamic-cache behavior. +- Tests and target-matched checks pass for touched crates/adapters. + +## Proposed PR sequence + +### PR 1 — Structured cache policy primitives + +Status: implemented in the current branch. + +#### Code changes + +- Keep/add a core module such as `crates/trusted-server-core/src/cache_policy.rs`. +- Define structured policy types: + - `CacheVisibility::{Public, Private}` + - `CachePolicy { visibility, browser_ttl, edge_ttl, stale_while_revalidate, +stale_if_error, immutable }` + - a `no-store` / uncacheable representation, either as a policy mode or a + dedicated helper, so neutralized shims and error responses do not need + ad-hoc strings; + - `EdgeCacheHeader::{SurrogateControl, CdnCacheControl, +CloudflareCdnCacheControl, SMaxageFallback, None}`. +- Add helpers that render policy into headers: + - browser `Cache-Control` + - Fastly `Surrogate-Control` + - generic `CDN-Cache-Control` + - Cloudflare-specific `Cloudflare-CDN-Cache-Control` + - portable `s-maxage` fallback. +- Keep helpers side-effect-limited: they should only mutate cache headers they + own and should not bypass `response_privacy` hardening. When applying private + or no-store policies, remove any existing edge-cache headers owned by the + helper so stale `Surrogate-Control`/CDN cache headers cannot survive. +- Add default policy constructors/constants for: + - immutable static; + - short TSJS fallback; + - neutralized Prebid shim (`no-store` or very short TTL); + - uncacheable private. + +#### Tests + +- Unit-test exact header rendering for immutable, short edge/browser split, + private, no-store, SWR/SIE, generic CDN, Cloudflare-specific CDN, and fallback + `s-maxage` policies. +- Test that `immutable` is omitted when browser TTL is absent or zero. +- Test that edge-header output is disabled for private/no-store responses, and + that applying private/no-store removes any pre-existing edge-cache header the + helper owns. + +### PR 2 — TSJS immutable hash-version serving + +Status: implemented in the current branch with runtime-specific edge-header +selection. + +#### Code changes + +- Extend `crates/trusted-server-js/build.rs` generated metadata with per-module + SHA-256 hashes. +- Update `trusted-server-js/src/bundle.rs`: + - `single_module_hash(id)` returns generated hash instead of hashing content; + - `concatenated_hash(ids)` hashes incrementally without concatenating a full + `String`, or caches the result per normalized module-id set; + - `concatenate_modules(ids)` can remain for serving the response body. +- Update `handle_tsjs_dynamic` in `publisher.rs`: + - parse `?v=` from the request URI; + - compare it with the canonical hash for the requested bundle; + - if it matches, apply immutable static policy plus `Vary: Accept-Encoding`, + ETag, and `X-Compress-Hint: on`; + - if missing/mismatched, keep short TTL policy plus ETag and + `X-Compress-Hint: on`. +- Keep the current canonical path shape (`/static/tsjs=...js?v=`). +- Document/verify that runtime cache-key configuration preserves the `v` query + parameter for `/static/tsjs=`. + +#### Tests + +- `tsjs_script_src` and deferred script tests still produce `?v=`. +- Matching `?v=` returns: + - `Cache-Control: public, max-age=31536000, immutable` + - runtime edge header via policy helper; + - `Vary: Accept-Encoding`; + - ETag. +- Missing/mismatched `?v=` returns short TTL and no `immutable`. +- Deferred disabled module still 404s. +- Hash helpers do not allocate the concatenated body just to hash it. + +### PR 3 — Neutralized publisher Prebid shim cache safety + +Fix the stable publisher Prebid compatibility route separately from TS-owned +Prebid delivery. + +#### Code changes + +- Update `PrebidIntegration::handle_script_handler` in + `crates/trusted-server-core/src/integrations/prebid.rs`. +- Replace the current year-long `public, max-age=31536000` response with either: + - `Cache-Control: no-store`, preferred for compatibility when integration + enablement/config can change; or + - a very short TTL if no-store is too conservative. +- Ensure no `Surrogate-Control`/CDN edge header is emitted for the neutralized + stable URL. +- Keep TS-owned Prebid bundle delivery on the deferred TSJS module path, where + matching `?v=` remains immutable. + +#### Tests + +- Neutralized Prebid script handler returns the empty compatibility script with + `no-store` or the chosen short TTL. +- Neutralized Prebid shim does not emit immutable or year-long cache headers. +- TSJS deferred Prebid still receives immutable headers when `?v=` matches. + +### PR 4 — Configurable static asset cache-rule engine + +Introduce operator-configurable static asset rules before applying immutable +upgrades to publisher-origin assets. + +#### Code changes + +- Add cache-rule settings rather than hard-coded path checks. Suggested shape: + - `CacheAssetRule { id, enabled, matcher, policy }` + - `CacheAssetMatcher::{PathPrefix, Glob, Regex, Extension, Preset}` + - `CacheAssetPreset::NextJsStatic` expands to `/_next/static/*` when enabled. +- Add cache settings under `Settings` (and `trusted-server.example.toml`) with + `#[serde(deny_unknown_fields)]` validation consistent with the rest of the + config model. +- Add a shared rule evaluator with deterministic precedence. Prefer an ordered + rule list where the first enabled match wins; reject duplicate rule IDs and + invalid matcher combinations during settings validation. +- Ship framework presets as data/config defaults or documented examples, not as + special cases in proxy/adapters. +- The Next.js preset may be present in example config, but operators must be able + to disable/override it. Do not silently apply it through a hard-coded branch. +- Support publisher-defined allowlist rules for other frameworks or + publisher-specific fingerprinted paths. +- Apply immutable policy only when an enabled rule/preset says the URL is + content-addressed, or for TS-owned validated hash URLs such as TSJS. + +#### Tests + +- With the Next.js preset enabled, `/_next/static/*` gets immutable policy. +- With the Next.js preset disabled, the same `/_next/static/*` remains + origin-controlled. +- Publisher-defined allowlist rule can mark a non-Next fingerprinted path + immutable. +- Non-matching publisher asset remains origin-controlled. +- Rule precedence is deterministic. +- Invalid regex/glob/config fails validation clearly. + +### PR 5 — MVP runtime edge-header mapping and docs + +Make the shared policy output explicit per runtime before wiring the rule engine +into more routes. This prevents new code from copying the current core +Fastly-specific `Surrogate-Control` behavior. This phase covers directive +rendering only; runtime storage is tracked in #908. + +#### Code changes + +- Stop requiring core helpers such as `handle_tsjs_dynamic` or + `serve_static_with_etag` to hard-code Fastly's `Surrogate-Control`. +- Choose one adapter boundary pattern and use it consistently: + - pass the runtime `EdgeCacheHeader`/policy emitter into core handlers; or + - return cache-policy metadata in response extensions and let adapters render + runtime-specific headers after route handling. +- Fastly adapter emits `Surrogate-Control` for edge TTLs. +- Cloudflare adapter emits `CDN-Cache-Control` or + `Cloudflare-CDN-Cache-Control`, depending on the chosen adapter convention. +- Portable/local fallback can use `s-maxage` inside `Cache-Control` when no + runtime-specific edge header is available. +- Akamai mapping remains absent/deferred; do not add untested Akamai behavior. +- Update `trusted-server.example.toml` and docs with disabled framework preset + examples and operator-owned allowlist examples. + +#### Tests + +- Fastly TSJS/static policy application emits `Surrogate-Control`. +- Cloudflare TSJS/static policy application emits the selected Cloudflare CDN + cache header and does not emit Fastly-only `Surrogate-Control`. +- Fallback policy emits `s-maxage` only for public/shared-cacheable responses. +- Private/no-store responses remove or avoid all edge-cache headers. + +### PR 6 — Apply static/rehosted policies to proxy responses + +Wire the rule engine into the routes that emit publisher-origin or rehosted +assets, using the runtime edge-header mapping from PR 5. + +#### Code changes + +- Extend `AssetProxyCachePolicy` in `proxy.rs` beyond + `OriginControlled`/`NoStorePrivate`, for example: + - `OriginControlled` + - `NoStorePrivate` + - `Normalized(CachePolicy)` from a matched enabled rule. +- Apply normalized policy at the asset handler, then reapply its runtime edge + directive after route finalization only when the finalized response is still + cacheable. Final `private` or `no-store` directives veto reapplication and + remove edge-cache headers. +- Preserve existing no-store/private handling for errors, signed failures, or + responses that set cookies/security headers. A matched TS-owned rehost rule + intentionally replaces the third-party origin's cache defaults before this + final privacy veto. +- Ensure operator `response_headers` cannot weaken protected private/no-store + decisions. +- For TS-owned rehosted copies: + - use immutable only for fingerprinted TS-owned URLs; + - use conservative edge/browser TTLs for stable rehosted URLs; + - keep dynamic/personalized endpoints uncached. + +#### Tests + +- Rehosted/fingerprinted route matched by an enabled rule gets immutable policy. +- Stable rehosted route gets the configured conservative policy, not a borrowed + third-party `no-store` unless configured. +- Rehosted error responses keep `no-store, private`. +- `Set-Cookie` response remains private/no-store and loses surrogate headers. +- Operator response headers cannot re-enable shared caching for protected + responses. + +## Initial config sketch + +Exact names can change during implementation, but keep the shape structured and +operator-controlled. + +```toml +[[cache.asset_rules]] +id = "nextjs-static" +enabled = false # operators may enable for Next.js publishers +preset = "nextjs-static" +visibility = "public" +browser_ttl_seconds = 31536000 +edge_ttl_seconds = 31536000 +immutable = true + +[[cache.asset_rules]] +id = "publisher-fingerprinted-assets-example" +enabled = false +path_globs = [ + "/assets/**/*.js", + "/assets/**/*.css", + "/assets/**/*.png", + "/assets/**/*.jpg", + "/assets/**/*.webp", + "/assets/**/*.avif", +] +fingerprint_style = "vite-base64-url" +visibility = "public" +browser_ttl_seconds = 31536000 +edge_ttl_seconds = 31536000 +immutable = true + +[cache.tsjs.versioned] +visibility = "public" +browser_ttl_seconds = 31536000 +edge_ttl_seconds = 31536000 +immutable = true + +[cache.tsjs.fallback] +visibility = "public" +browser_ttl_seconds = 300 +edge_ttl_seconds = 300 +stale_while_revalidate_seconds = 60 +stale_if_error_seconds = 86400 + +[cache.prebid_neutralized] +mode = "no-store" +``` + +Defaults should preserve current behavior unless a rule is explicitly enabled or +unless the response is TS-owned and hash-validated, such as TSJS. + +## Deferred follow-up backlog + +These remain valuable, but are intentionally outside the initial cache-header +slice. + +| Follow-up | Why deferred | Notes | +| --------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| True publisher-origin streaming | Requires platform/body boundary changes and adapter streaming semantics | Includes avoiding full `take_body_bytes()` materialization on Fastly and documenting/implementing non-Fastly streaming parity. | +| Parser-context bid splice | Requires HTML pipeline redesign | Replace raw ` **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Finish the SSAT debug-comment configuration with three explicit sensitivity modes while making default response-level metadata fail closed by both key and value schema. + +**Architecture:** Keep the existing configuration types in `settings.rs` and rendering in `publisher.rs`. Add an `Upstream` enum branch and a separate provider-diagnostic key set. In `Redacted`, reconstruct a small metadata object from validated values instead of copying arbitrary JSON; in `Upstream`, add only six named provider diagnostics; in `Full`, retain the existing raw-metadata behavior. + +**Tech Stack:** Rust, serde/serde_json, TOML, existing `trusted-server-core` tests and target-specific Cargo aliases. + +**Spec:** `docs/superpowers/specs/2026-07-20-ssat-debug-comment-config-design.md` + +--- + +## File Structure + +| File | Responsibility | +| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `crates/trusted-server-core/src/settings.rs` | Public config schema, safe selector keys, upstream diagnostic keys, normalization, config tests | +| `crates/trusted-server-core/src/publisher.rs` | Schema validation, safe message generation, three rendering branches, section/size protections, renderer tests | +| `trusted-server.example.toml` | Operator-facing mode and privacy documentation | +| `docs/superpowers/specs/2026-07-20-ssat-debug-comment-config-design.md` | Approved security contract; no behavioral edits during implementation unless a discovered contradiction is brought back for approval | + +No new production file or dependency is needed. The existing large modules already own these responsibilities, so splitting them during this security fix would add unrelated churn. + +--- + +### Task 1: Lock down the configuration boundary + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs:1890-2020` +- Test: `crates/trusted-server-core/src/settings.rs:2700-2770` + +- [ ] **Step 1: Write failing settings tests** + +Update `auction_debug_comment_options_default_matches_serde_defaults` to require exactly the safe selector keys: + +```rust +assert_eq!( + opts.metadata_keys, + vec![ + "error_type".to_string(), + "http_status".to_string(), + "message".to_string(), + ], + "should default to only schema-validated response metadata" +); +``` + +Add direct enum deserialization coverage: + +```rust +#[test] +fn auction_debug_comment_options_deserializes_upstream_verbosity() { + let options: AuctionDebugCommentOptions = + toml::from_str("verbosity = \"upstream\"") + .expect("should deserialize upstream verbosity"); + assert_eq!(options.verbosity, AuctionDebugCommentVerbosity::Upstream); +} +``` + +Change the normalization fixture to safe and unsafe-looking names so it proves normalization only, not authorization: + +```rust +let mut opts = AuctionDebugCommentOptions { + metadata_keys: vec![ + " http_status ".to_string(), + "".to_string(), + "debug".to_string(), + ], + ..AuctionDebugCommentOptions::default() +}; +opts.normalize(); +assert_eq!( + opts.metadata_keys, + vec!["http_status".to_string(), "debug".to_string()] +); +``` + +This deliberately leaves `debug` in normalized configuration; authorization must happen at render time. + +- [ ] **Step 2: Run the focused tests and confirm RED** + +Run: + +```bash +cargo test -p trusted-server-core --lib auction_debug_comment_options -- --nocapture +cargo test -p trusted-server-core --lib bad_verbosity_string_fails_config_load -- --nocapture +``` + +Expected: both commands fail to compile because Rust builds the complete lib test binary before applying the name filter, and the newly added test references the nonexistent `AuctionDebugCommentVerbosity::Upstream`. This is the intended RED state; the existing invalid-verbosity assertion is re-confirmed after the enum compiles in Step 4. + +- [ ] **Step 3: Implement the minimal settings change** + +Replace the current safe const with: + +```rust +pub(crate) const AUCTION_DEBUG_METADATA_ALLOWLIST: &[&str] = + &["error_type", "http_status", "message"]; + +pub(crate) const AUCTION_DEBUG_UPSTREAM_METADATA_KEYS: &[&str] = &[ + "errors", + "warnings", + "responsetimemillis", + "bidstatus", + "upstream_message", + "upstream_message_truncated", +]; +``` + +Add the enum variant between `Redacted` and `Full`: + +```rust +#[derive(Debug, Clone, Copy, Default, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AuctionDebugCommentVerbosity { + #[default] + Redacted, + Upstream, + Full, +} +``` + +Update the public docs on `metadata_keys` and `verbosity` to state: + +- `metadata_keys` selects only the three safe fields and cannot unlock upstream keys. +- `Upstream` adds six untyped provider diagnostic values and remains creative-truncated. +- `Full` copies all response metadata and does not truncate creatives. +- `Upstream` and `Full` must not be enabled in production. + +- [ ] **Step 4: Run focused tests and confirm GREEN** + +Run both commands from Step 2. + +Expected: all matching settings tests pass, including invalid-verbosity rejection. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/settings.rs +git commit -m "Harden SSAT debug comment configuration modes" +``` + +--- + +### Task 2: Make redacted rendering schema-safe + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs:60-70` +- Modify: `crates/trusted-server-core/src/publisher.rs:1870-2035` +- Test: `crates/trusted-server-core/src/publisher.rs:4250-4565` + +- [ ] **Step 1: Add a reusable test renderer with explicit metadata** + +Near the existing `dump_comment_for_creative_with_options`, add a helper that builds one `AuctionResponse::error("prebid", 12)`, attaches supplied metadata with the existing `with_metadata` builder, calls `prepend_auction_debug_comment`, and returns the rendered state. Do not add a production convenience API only for tests. + +- [ ] **Step 2: Write the failing default and subset tests** + +Replace `default_options_reproduce_current_behavior` with `default_options_apply_safe_response_metadata_schema`. Its fixture must contain: + +- valid `error_type = "http_status"` +- valid `http_status = 422` +- malicious raw `message` +- all six upstream diagnostic keys containing unique fictional identity-shaped markers +- `debug.resolvedrequest.user.id` + +Assert that output contains `error_type`, `http_status`, and the fixed message `Provider returned HTTP 422`; assert that none of the raw message, upstream markers, or `debug` subtree appear. + +Add `configured_metadata_subset_only_includes_selected_safe_keys` with `metadata_keys = ["http_status", "errors", "debug"]`. Assert only validated `http_status` survives; `errors` and `debug` cannot be unlocked by selector configuration. + +- [ ] **Step 3: Write the failing adversarial schema test** + +Add `redacted_mode_rejects_wrong_types_and_unknown_error_classifications`. Render separate cases containing: + +```rust +json!({"error_type": {"identity": "example-user-123"}}) +json!({"error_type": "provider_supplied_unknown", "message": "example-user-123"}) +json!({"http_status": "200 example-user-123"}) +json!({"http_status": 99}) +json!({"http_status": 600}) +json!({"message": {"identity": "example-user-123"}}) +``` + +Assert no identity marker or attacker-provided message appears. Also assert valid integer boundaries `100` and `599` survive, while non-integral JSON numbers do not. + +- [ ] **Step 4: Write the failing safe-message mapping test** + +Add a table-driven test for the fixed mappings: + +```rust +let cases = [ + ("parse_response", None, "Provider response could not be parsed"), + ("launch_failed", None, "Provider launch failed"), + ("transport", None, "Provider request failed"), + ("timeout", None, "Provider request timed out"), + ("http_status", Some(418), "Provider returned HTTP 418"), + ("http_status", None, "Provider returned an HTTP error"), +]; +``` + +For every case, include a malicious raw `metadata["message"]` and prove the renderer ignores it. + +- [ ] **Step 5: Run the focused publisher tests and confirm RED** + +Run: + +```bash +cargo test -p trusted-server-core --lib publisher::tests:: -- --nocapture +``` + +Expected: new privacy tests fail because current redacted rendering clones values by matching key names only. + +- [ ] **Step 6: Implement schema validation and safe message generation** + +Import both key constants. Add private helpers close to `redact_response_for_dump`: + +```rust +fn validated_error_type( + metadata: &std::collections::HashMap, +) -> Option<&str> { + let value = metadata.get("error_type")?.as_str()?; + matches!( + value, + "parse_response" | "launch_failed" | "transport" | "timeout" | "http_status" + ) + .then_some(value) +} + +fn validated_http_status( + metadata: &std::collections::HashMap, +) -> Option { + metadata + .get("http_status")? + .as_u64() + .filter(|status| (100..=599).contains(status)) +} + +fn safe_error_message(error_type: &str, http_status: Option) -> Option { + match error_type { + "parse_response" => Some("Provider response could not be parsed".to_string()), + "launch_failed" => Some("Provider launch failed".to_string()), + "transport" => Some("Provider request failed".to_string()), + "timeout" => Some("Provider request timed out".to_string()), + "http_status" => Some(http_status.map_or_else( + || "Provider returned an HTTP error".to_string(), + |status| format!("Provider returned HTTP {status}"), + )), + _ => None, + } +} +``` + +Implement `redacted_metadata_for_dump` so it reconstructs values only when each field is configured and valid. It must never clone `metadata["message"]`: + +```rust +fn redacted_metadata_for_dump( + metadata: &std::collections::HashMap, + options: &AuctionDebugCommentOptions, +) -> serde_json::Map { + let selected = |key: &str| { + AUCTION_DEBUG_METADATA_ALLOWLIST.contains(&key) + && options.metadata_keys.iter().any(|candidate| candidate == key) + }; + let error_type = validated_error_type(metadata); + let http_status = validated_http_status(metadata); + let mut safe = serde_json::Map::new(); + if selected("error_type") && let Some(value) = error_type { + safe.insert("error_type".to_string(), serde_json::json!(value)); + } + if selected("http_status") && let Some(value) = http_status { + safe.insert("http_status".to_string(), serde_json::json!(value)); + } + if selected("message") + && let Some(value) = error_type.and_then(|kind| safe_error_message(kind, http_status)) + { + safe.insert("message".to_string(), serde_json::json!(value)); + } + safe +} +``` + +- [ ] **Step 7: Add the `Upstream` and `Full` metadata branches** + +Build redacted metadata first. For `Upstream`, copy only keys in `AUCTION_DEBUG_UPSTREAM_METADATA_KEYS`; for `Full`, clone the entire map: + +```rust +let metadata = match options.verbosity { + AuctionDebugCommentVerbosity::Redacted => { + redacted_metadata_for_dump(&response.metadata, options) + } + AuctionDebugCommentVerbosity::Upstream => { + let mut metadata = redacted_metadata_for_dump(&response.metadata, options); + for key in AUCTION_DEBUG_UPSTREAM_METADATA_KEYS { + if let Some(value) = response.metadata.get(*key) { + metadata.insert((*key).to_string(), value.clone()); + } + } + metadata + } + AuctionDebugCommentVerbosity::Full => response + .metadata + .iter() + .map(|(key, value)| (key.clone(), value.clone())) + .collect(), +}; +``` + +Change creative handling so truncation applies to every mode except `Full`: + +```rust +if options.verbosity != AuctionDebugCommentVerbosity::Full + && let Some(creative) = &bid.creative +{ + value["creative"] = serde_json::Value::String(truncate_with_marker( + creative, + MAX_BID_CREATIVE_DUMP_BYTES, + )); +} +``` + +- [ ] **Step 8: Write and run upstream/full safety tests** + +Add: + +- `upstream_mode_includes_provider_diagnostics_but_not_debug_subtree` +- `verbosity_upstream_still_truncates_creative` +- `metadata_keys_empty_yields_empty_safe_metadata_in_redacted` + +The upstream fixture must include all six diagnostics plus a `debug` subtree. Assert all six diagnostics appear, `debug` does not, the configured safe subset still works, and the creative carries the existing truncation marker. Retain existing tests proving `Full` includes `debug`, skips per-creative truncation, and still respects the 256 KiB total cap. + +Extend `auction_debug_comment_neutralises_every_comment_terminator_vector` so its existing five attack vectors run once with `Redacted` and once with `Full`. Do not replace this table with the narrower two-vector full-mode test: every vector must prove that verbosity cannot bypass the unconditional HTML-comment safety boundary. + +Run: + +```bash +cargo test -p trusted-server-core --lib publisher::tests:: -- --nocapture +``` + +Expected: all publisher tests pass. + +- [ ] **Step 9: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs +git commit -m "Enforce SSAT debug response metadata schemas" +``` + +--- + +### Task 3: Correct operator-facing configuration documentation + +**Files:** + +- Modify: `trusted-server.example.toml:160-190` +- Verify against: `docs/superpowers/specs/2026-07-20-ssat-debug-comment-config-design.md` + +- [ ] **Step 1: Update the example configuration** + +Use exactly the safe default selector: + +```toml +[debug] +# NEVER enable in production. Injects an auction dump before . +# "redacted" validates response metadata but still includes bid-level fields +# and creative previews; it is not a fully anonymized dump. +auction_html_comment = false + +[debug.auction_html_comment_options] +include_provider_responses = true +include_mediator_response = true +include_bids = true +metadata_keys = ["error_type", "http_status", "message"] +# "redacted" (default), "upstream", or "full". +# "upstream" exposes six untyped provider diagnostic values that may contain +# request or identity data. "full" additionally exposes all response metadata +# and untruncated creatives. Never use either sensitive mode in production. +verbosity = "redacted" +``` + +Preserve nearby unrelated comments and settings from `main`. + +- [ ] **Step 2: Verify example parsing** + +Run: + +```bash +cargo test-cloudflare +cargo test-spin +``` + +Expected: both adapter suites pass, including paths that load `trusted-server.example.toml`. + +- [ ] **Step 3: Commit** + +```bash +git add trusted-server.example.toml +git commit -m "Document SSAT debug sensitivity modes" +``` + +--- + +### Task 4: Full verification and review + +**Files:** none unless verification exposes a scoped defect. + +- [ ] **Step 1: Run formatting checks** + +```bash +cargo fmt --all -- --check +cd docs && npm run format +``` + +Expected: no formatting differences. + +- [ ] **Step 2: Run native and adapter tests** + +```bash +cargo test -p trusted-server-core --lib +cargo test-axum +cargo test-cloudflare +cargo test-spin +cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity +``` + +Expected: all pass. + +- [ ] **Step 3: Run Fastly tests** + +```bash +cargo test-fastly +``` + +Expected: pass. If Viceroy fails before executing tests because macOS native certificate/keychain access is unavailable, record that exact environment blocker and separately run `cargo check-fastly`; do not describe Fastly tests as passing. + +- [ ] **Step 4: Run every target-specific lint alias** + +```bash +cargo clippy-fastly +cargo clippy-axum +cargo clippy-cloudflare +cargo clippy-cloudflare-wasm +cargo clippy-spin-native +cargo clippy-spin-wasm +``` + +Expected: no warnings or errors. + +- [ ] **Step 5: Run documentation and JavaScript gates required by repository CI** + +Use the repository-pinned Node version (`.tool-versions`, currently Node 24.12.0) for JavaScript commands: + +```bash +cd crates/trusted-server-js/lib && npm run format +cd crates/trusted-server-js/lib && npx vitest run +cd crates/trusted-server-js/lib && node build-all.mjs +``` + +Expected: all pass. If the current shell uses another Node major, invoke the pinned Node binary explicitly rather than changing lockfiles. + +- [ ] **Step 6: Inspect the final diff and request independent code review** + +```bash +git diff origin/main...HEAD --check +git status --short +``` + +Verify: + +- no conflict markers +- no unrelated changes +- safe selector contains only `error_type`, `http_status`, `message` +- every safe value is validated or generated +- upstream includes exactly six named diagnostic keys +- full remains the only mode that copies arbitrary response metadata +- upstream and redacted both truncate creatives +- byte cap and comment neutralization remain unconditional + +Then invoke `superpowers:requesting-code-review`. Fix any Critical or Important findings and rerun the affected gates before claiming completion. + +--- + +## Implementation Constraints + +- Follow test-driven development: observe each new security test fail before changing its production path. +- Do not alter bid-level `metadata`, `nurl`, `burl`, or creative contents beyond the existing preview truncation; issue #925 owns that separate boundary. +- Do not add provider instrumentation or change Prebid response parsing. +- Do not make either size cap configurable. +- Do not silently downgrade invalid verbosity strings; serde must reject them. +- Do not push the branch until the user has received the final verification and review results. diff --git a/docs/superpowers/plans/2026-08-06-server-side-ad-templates-cache-control.md b/docs/superpowers/plans/2026-08-06-server-side-ad-templates-cache-control.md new file mode 100644 index 000000000..b950d0a95 --- /dev/null +++ b/docs/superpowers/plans/2026-08-06-server-side-ad-templates-cache-control.md @@ -0,0 +1,248 @@ +# Dedicated Server-Side Ad Templates Switch and Cache Policy Plan + +> **For agentic workers:** Implement this plan task-by-task, keeping the dedicated +> template switch separate from the global auction configuration. + +**Goal:** Add an explicit on/off switch for server-side ad templates, while +retaining the browser-facing cache policy from issue #1007: + +- Server-side ad templates active: `Cache-Control: private, no-store`. +- Structurally inactive server-side ad templates: successful request-eligible GET + document HTML uses exactly `Cache-Control: max-age=60`, except origin + `private`/`no-store` policies remain unchanged. +- Non-200, non-GET, non-document, bot, prefetch, and consent-denied responses + keep the origin browser cache policy. +- CDN-specific cache headers must not change when templates are inactive. + +**Issue context:** The current cache-policy change uses the runtime +`should_run_ad_stack` gate. That gate is also affected by `[auction].enabled`, +which is not the right configuration boundary for publisher templates. A +browser can call `POST /auction`, and that endpoint is a separate server-run +auction API. The new switch must disable publisher HTML/page-bids template +delivery without disabling that API. + +## Configuration decision + +Add this field to the existing `[creative_opportunities]` section: + +```toml +[creative_opportunities] +enabled = true +``` + +Use `enabled = false` to turn off server-side ad templates while retaining the +slot definitions and keeping direct `POST /auction` behavior available. + +### Compatibility rules + +- The field defaults to `true` when omitted, preserving existing behavior for + deployments that already have `[creative_opportunities]` configured. +- The section remains optional. An absent section continues to mean that the + feature is unavailable. +- Serialize the default `true` value as omitted, matching the existing + rollback-compatibility pattern for newer creative-opportunity fields. An + explicit `false` must remain serialized so the setting is not silently lost. +- `auction.enabled` remains a separate auction/orchestrator setting. Do not use + it as the dedicated template switch and do not thread the new template flag + into `POST /auction`. + +## Current cache behavior to retain + +The existing HTML policy block in `publisher.rs` must remain structurally +consistent with the current issue #952 behavior: + +1. For an eligible request that runs the server-side ad stack and receives HTML: + - Set `Cache-Control: private, no-store`. + - Remove `ETag` and `Last-Modified`. + - Remove `Surrogate-Control`, `Fastly-Surrogate-Control`, `CDN-Cache-Control`, + and `Cloudflare-CDN-Cache-Control`. +2. For a request-eligible `200 OK` GET document HTML response where the + server-side ad stack is structurally inactive, including an explicit template + disable: + - Set exactly `Cache-Control: max-age=60`, replacing origin browser policies + as specified by issue #1007 unless the origin sends `private` or `no-store`. + - Leave validators and all CDN-specific cache headers untouched. +3. For non-200, non-GET, non-document, bot, prefetch, and consent-denied + responses, preserve the origin browser cache policy. +4. Apply request-scoped privacy finalization after this policy so GPT diagnostics + and cookie-bearing responses can still require `private, no-store`; a + cookie-bearing response therefore ends as `private, max-age=0` when it did + not already carry a stricter policy. + +## File map + +### Configuration and compatibility + +- `crates/trusted-server-core/src/creative_opportunities.rs` + - Add `CreativeOpportunitiesConfig::enabled` with a default-true serde + implementation and documentation. + - Add a small accessor if it improves readability, but keep the source of + truth in this config type. + - Update config constructors and serialization tests. +- `crates/trusted-server-core/src/settings.rs` + - Keep `creative_opportunities` parsing and runtime preparation compatible with + the new field. + - Make `creative_opportunity_slots()` return an empty slice when the section + is absent or explicitly disabled, so all adapters receive one consistent + runtime view. + - Add TOML and environment-override coverage for `enabled = false`. +- `crates/trusted-server-core/src/config.rs` + - Extend legacy-schema tests to prove default `enabled = true` is omitted from + serialized blobs and remains readable by older binaries. + - Prove an explicit `enabled = false` is serialized, making rollback failure + loud rather than silently re-enabling templates. +- `trusted-server.example.toml` + - Document `creative_opportunities.enabled` and show how to turn templates off + without deleting slot definitions. +- `docs/guide/configuration.md` + - Add the field to the creative-opportunities reference and document the + environment override: + `TRUSTED_SERVER__CREATIVE_OPPORTUNITIES__ENABLED=false`. + - Clarify that this switch controls publisher HTML/page-bids template + delivery, not direct `POST /auction` callers. +- `CHANGELOG.md` + - Add an entry describing the dedicated template switch and cache behavior. + +### Publisher execution and cache policy + +- `crates/trusted-server-core/src/publisher.rs` + - Include the dedicated flag in the initial publisher eligibility decision. + - Do not match, dispatch, or inject server-side ad templates when the flag is + false, even if slots are configured and `[auction].enabled` is true. + - Apply the issue #1007 inactive-HTML cache policy in this state. + - Update skip-reason diagnostics/telemetry so `ad_templates_disabled` is + distinguishable from `auction_disabled`, consent denial, bots, prefetch, and + no matching slots. + - Update `handle_page_bids` so an explicit template disable returns the normal + empty JSON shape (`slots: []`, `bids: {}`) rather than slot definitions. Keep + the current `404` behavior for an absent `[creative_opportunities]` section. + - Extend the existing SSAT cache-policy and eligibility tests. +- `crates/trusted-server-core/src/auction/endpoints.rs` + - Do not gate `POST /auction` on the new template flag. + - Add a regression test or test fixture proving that disabling + `creative_opportunities.enabled` does not suppress a direct auction request + when providers are configured. + - Separately document/verify the existing behavior of `[auction].enabled` for + this endpoint; do not conflate that global setting with the new template + switch. + +### Adapter propagation and browser behavior + +The adapters already pass `Settings::creative_opportunity_slots()` into the +publisher/page-bids handlers. Update and verify these call sites so the central +empty-slice behavior is honored; avoid adding four divergent config checks: + +- `crates/trusted-server-adapter-fastly/src/app.rs` +- `crates/trusted-server-adapter-axum/src/app.rs` +- `crates/trusted-server-adapter-cloudflare/src/app.rs` +- `crates/trusted-server-adapter-spin/src/app.rs` + +No route-level flag is needed if the core `Settings` accessor and handlers are +correct. Add adapter route assertions only where existing fixtures make them +useful. + +The browser runtime already defaults `window.tsjs.adSlots` and +`window.tsjs.bids` to empty values when the edge does not inject templates. If +terminology is updated, adjust these comments/tests without changing runtime +semantics: + +- `crates/trusted-server-js/lib/src/core/index.ts` +- `crates/trusted-server-js/lib/src/integrations/gpt/index.ts` +- Relevant page-bids tests under `crates/trusted-server-js/lib/test/integrations/gpt/` + +## Implementation tasks + +### Task 1: Add and serialize the dedicated setting + +- [ ] Add `enabled: bool` to `CreativeOpportunitiesConfig` with default `true`. +- [ ] Use `skip_serializing_if` so the default value does not appear in stored + config blobs; explicit `false` must serialize. +- [ ] Update all Rust struct literals in `creative_opportunities.rs` and + `publisher.rs` tests. +- [ ] Add parsing, default, false-value, and environment-override tests. +- [ ] Update the legacy compatibility tests in `config.rs`. + +### Task 2: Thread the setting through publisher eligibility + +- [ ] Update `should_run_server_side_ad_stack` to accept the dedicated template + flag as an explicit gate, with a descriptive parameter/doc comment. +- [ ] Ensure initial publisher slot matching and `Settings::creative_opportunity_slots` + do not expose slots when templates are disabled. +- [ ] Preserve the existing `[auction].enabled` and consent gates as separate + conditions. +- [ ] Add an `ad_templates_disabled` diagnostic/telemetry skip reason where the + current branch records a skipped auction. + +### Task 3: Apply the cache policy to the dedicated-off state + +- [ ] Keep the active-SSAT `private, no-store` behavior and validator/CDN header + removal unchanged. +- [ ] Keep the inactive-HTML `max-age=60` behavior from issue #1007. +- [ ] Verify that structurally inactive request-eligible responses replace the + browser-facing `Cache-Control` for `200 OK` GET document HTML, including + origin `no-cache` and zero-age policies, while preserving `private` and + `no-store`. +- [ ] Preserve `ETag`, `Last-Modified`, and every CDN-specific header. +- [ ] Verify that non-200, non-GET, non-document, bot, prefetch, and + consent-denied responses retain the origin browser cache policy. +- [ ] Verify that request-scoped GPT diagnostics and cookie privacy override this + policy. + +### Task 4: Gate SPA page-bids/template delivery + +- [ ] Include `co_config.enabled` in the `ad_stack_enabled` decision in + `handle_page_bids`. +- [ ] Return empty slots and bids for an explicit disable while retaining the + endpoint and its existing response privacy headers. +- [ ] Keep the absent-section `404` behavior unchanged. +- [ ] Add tests for enabled, disabled, absent, consent-denied, bot, and prefetch + cases as appropriate; preserve existing tests for `[auction].enabled=false`. + +### Task 5: Protect direct `POST /auction` from accidental coupling + +- [ ] Add a focused endpoint test with `creative_opportunities.enabled=false` + and a recording provider. +- [ ] Assert that the provider still sees the direct auction request and that + the response remains a normal OpenRTB response. +- [ ] If the test reveals that `[auction].enabled=false` also needs a separate + product decision for `/auction`, record that as a follow-up rather than + changing it as part of the template-switch work. + +### Task 6: Update docs, examples, comments, and adapter coverage + +- [ ] Update the example config, configuration guide, and changelog. +- [ ] Update stale comments that call `[auction].enabled` the universal template + kill switch. +- [ ] Verify all four adapter call sites use the centralized disabled-slot view. +- [ ] Run JS tests if comments or tests are touched; no JS behavior change is + expected. + +## Test plan + +Use target-matched commands; do not run bare workspace tests because the +workspace contains multiple runtime targets. + +- [ ] `cargo test-axum -p trusted-server-core publisher` +- [ ] `cargo test-fastly` +- [ ] `cargo test-axum` +- [ ] `cargo test-cloudflare` +- [ ] `cargo test-spin` +- [ ] `cargo fmt --all -- --check` +- [ ] `cargo clippy-fastly` +- [ ] `cargo clippy-axum` +- [ ] `cargo clippy-cloudflare` +- [ ] `cargo clippy-cloudflare-wasm` +- [ ] `cargo clippy-spin-native` +- [ ] `cargo clippy-spin-wasm` +- [ ] `cd crates/trusted-server-js/lib && npx vitest run` if JS tests/comments change +- [ ] `cd docs && npm run format` if documentation formatting is required + +## Non-goals + +- Do not change CDN-specific cache policy for inactive templates. +- Do not change adapter response privacy or cookie handling. +- Do not use `auction.rewrite_creatives` as the template switch; it controls + creative URL rewriting, not whether the server-side template stack runs. +- Do not gate or disable direct `POST /auction` as part of this feature. +- Do not remove slot definitions when the switch is off; the point of the switch + is to provide a reversible runtime control. diff --git a/docs/superpowers/plans/2026-08-08-1009-measurement-and-stage-0.md b/docs/superpowers/plans/2026-08-08-1009-measurement-and-stage-0.md new file mode 100644 index 000000000..98c6c221b --- /dev/null +++ b/docs/superpowers/plans/2026-08-08-1009-measurement-and-stage-0.md @@ -0,0 +1,1045 @@ +# #1009 Measurement and Stage 0 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Turn off the redundant origin cache bypass that the spec identifies as the +actual TTFB cost, behind an operator flag, and establish the measurement baseline that +later work is compared against. + +> **This plan does not close #1009.** It contains no ESI arm and no client-fill arm, so +> completing it cannot answer whether ESI separates cacheable content from per-user +> state. It is a **supporting optimisation and the experimental control** for +> [the ESI validation spike](../archive/2026-08-10-1009-esi-validation-spike.md), which is where +> #1009 is actually decided. Scoped and framed this way after external review on +> 2026-08-10. + +**Architecture:** Two investigation tasks that produce recorded findings and no code; one +code task that adds a config-gated timing log and makes the cache bypass operator- +controlled; and one config change that flips it, gated on the first investigation. +Nothing here touches the auction, the `` hold, or bid delivery — those are +Stages 1–2 in the spec and are explicitly out of scope. + +**Tech Stack:** Rust 2024 edition, `wasm32-wasip1`, Fastly Compute, `web_time::Instant` +for wasm-safe timing, `log` for instrumentation, Viceroy for adapter tests. + +**Spec:** `docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md` +(§3 Steps A/B/C and §4 Stage 0). Read §4 and §5 before starting Task 5. + +**Before pushing, run both documentation gates:** + +```bash +cd docs && npm run format && npm run build && cd .. +``` + +`npm run build` is not optional — `format` passes on documents with dead links, and that +shipped a broken docs build on this branch once already. + +**Two prettier gotchas, both hit while writing this plan.** CI gate 7 +(`cd docs && npm run format`) runs `prettier --check` over all of `docs/`, so both bite. + +1. **Not idempotent on embedded markdown fences.** The first `--write` reformats the + outer document and the embedded ` ```markdown ` block only settles on a second pass. + If `--check` still warns immediately after a `--write`, run `--write` again before + concluding anything is wrong. +2. **It mangles bare `snake_case` identifiers inside fences**, reading the underscores as + emphasis and rewriting `origin_fetch_ms` to `origin*fetch_ms`. **Always wrap + identifiers in backticks**, including inside fenced blocks and table cells. + +--- + +## Background an implementer needs + +Trusted Server proxies a publisher's origin, rewrites the HTML at the edge to inject ad +slot definitions and a JS bundle, and runs a server-side ad auction. For requests that +are eligible for that ad stack, `publisher.rs` currently does three things to the origin +request and response that together make the page uncacheable: + +1. strips conditional and range headers so the origin must return a full body, +2. sets a **cache bypass** so the Fastly read-through cache is skipped entirely, and +3. strips every cacheability header from the response. + +The spec establishes that (2) is redundant given (1) — by the time the request reaches +the cache it is already unconditional, so a cache HIT returns a full body anyway — and +that (2) is the dominant cost. This plan makes (2) operator-controlled and then turns it +off, after first confirming that is safe. + +**Why it might not be safe:** RSC (React Server Component) requests and ordinary HTML +navigations share the same URL and are distinguished only by request headers. RSC +requests are not classified as navigations, so they already flow through the cache while +HTML navigations bypass it. Removing the bypass puts both under one cache key. If the +origin does not declare `Vary` for those headers, the cache could serve one +representation in response to a request for the other. Task 1 checks this. + +**Terms:** _POP_ = Fastly edge point of presence. _shield_ = a designated POP that +backs other POPs. _read-through cache_ = Fastly's cache on the backend request path. +_bypass / `Pass`_ = skip that cache. + +--- + +## File structure + +| File | Responsibility in this plan | +| ---------------------------------------------------------------- | ---------------------------------------------------------------------------------- | +| `docs/superpowers/plans/2026-08-08-1009-measurement-findings.md` | **Create.** Recorded output of Tasks 1–2. Gates Task 5. | +| `crates/trusted-server-core/src/publisher.rs` | **Modify.** Timing log and the bypass flag (Task 3); tests (Task 5). | +| `crates/trusted-server-core/src/settings.rs` | **Modify.** `publisher.bypass_origin_cache` and `debug.publisher_timing` (Task 3). | +| `trusted-server.example.toml` | **Modify.** Document the new key (Task 5). | + +No new modules. No adapter changes: the `bypass_cache` platform capability and its +per-adapter mappings stay in place and keep their tests — the publisher-path call site +becomes operator-controlled rather than unconditional. + +## Task order and dependencies + +Only one edge is real. Do not serialize the rest. + +``` +Task 1 (origin Vary check) ──────┬──> Task 2 (appends to the findings file Task 1 creates) + │ + ├──> Task 5 (flip the flag) +Task 3 (instrumentation + flag) ─┘ +``` + +**Task 1 is externally blocked.** It needs the publisher origin hostname, which lives in +the operator's gitignored `trusted-server.toml`. Arrange access before starting, or the +plan stalls on its first step. + +Task 3 is independent and can start immediately. Task 2 only needs Task 1 far enough to +have created the findings document. Task 5 needs Task 1's verdict **and** Task 3's config +flag to exist. + +--- + +## Task 1: Step A — origin `Vary` check + +**Files:** + +- Create: `docs/superpowers/plans/2026-08-08-1009-measurement-findings.md` + +This task is an investigation. It writes no code and gates Task 5. + +- [ ] **Step 1: Get the origin URL** + +The publisher origin is operator config, not in the repo. Read it from the deployed +service config or ask the operator. Do **not** hardcode it into any committed file — the +findings document records the _result_, not the hostname. + +```bash +# The key is `publisher.origin_url` in the operator's trusted-server.toml +# (gitignored). Confirm the value before proceeding. +``` + +- [ ] **Step 2: Request the HTML representation and capture `Vary`** + +```bash +curl -sS -D - -o /dev/null "https:///" \ + -H 'Sec-Fetch-Dest: document' \ + -H 'Accept: text/html' +``` + +Expected: response headers. Record whether a `Vary` header is present and its value. + +- [ ] **Step 3: Request the RSC representation at the same URL** + +```bash +curl -sS -D - -o /dev/null "https:///" \ + -H 'RSC: 1' \ + -H 'Accept: text/x-component' +``` + +Expected: a different `Content-Type` (`text/x-component`) than Step 2, proving the two +representations share a URL. Record `Vary` again. + +- [ ] **Step 4: Probe the `Next-Router-*` headers** + +Do not skip this. The PASS criterion below names these headers, and an implementer who +tests only HTML and `RSC` can record a PASS that is wrong — which routes to Task 5a, the +one outcome this plan calls dangerous. + +```bash +for H in 'Next-Router-Prefetch: 1' 'Next-Router-State-Tree: %5B%22%22%5D'; do + echo "--- $H" + curl -sS -D - -o /dev/null "https:///" \ + -H 'RSC: 1' -H "$H" \ + | grep -iE '^(vary|content-type|content-length|cache-control|set-cookie):' +done +``` + +Compare `Content-Type` and `Content-Length` against the plain `RSC: 1` request from +Step 3. If either differs, the origin varies on that header and `Vary` must name it. + +Capture `Cache-Control` and `Set-Cookie` on every request in this task, not just this +one — see Step 5. + +- [ ] **Step 5: Probe cookie personalization — the bigger hole** + +The representation check above covers RSC-vs-HTML. It does **not** cover the larger +class: TS forwards client cookies to origin unchanged, so any cookie-personalized HTML +(logged-in state, paywall meter, publisher-side A/B assignment) becomes cross-servable +once the cache is on. + +**Do not compare body hashes.** Verified on the live origin: this page regenerates +~170 ad-slot container IDs as fresh 32-hex UUIDs on every request, so three requests give +three different hashes with byte-identical lengths, cookie or not. A hash comparison +reports a false FAIL every time. + +Normalize per-request identifiers, establish the no-cookie baseline drift first, then ask +whether the cookie arm differs by _more_ than that baseline: + +```bash +ORIGIN="https://"; HOSTH="Host: " +norm() { sed -E 's/[0-9a-f]{32}/UUID/g' "$1"; } + +for n in a b; do + curl -sS "$ORIGIN/" -H "$HOSTH" \ + -H 'Sec-Fetch-Dest: document' -H 'Accept: text/html' > "nc_$n.html" +done +curl -sS "$ORIGIN/" -H "$HOSTH" \ + -H 'Sec-Fetch-Dest: document' -H 'Accept: text/html' \ + -H 'Cookie: ' > ck.html + +echo "baseline drift: $(diff <(norm nc_a.html) <(norm nc_b.html) | grep -c '^[<>]')" +echo "with cookie: $(diff <(norm nc_a.html) <(norm ck.html) | grep -c '^[<>]')" +diff <(norm nc_a.html) <(norm ck.html) | head -20 +``` + +Send the `Host` override — the origin is a shared vhost and will not return the right +document without it. Read it from `publisher.origin_host_header_override`. + +**Step A has been run once and returned a PROVISIONAL PASS**, which is **not** sufficient +to flip the flag. See [the findings](./2026-08-08-1009-measurement-findings.md) for the +five untested conditions. Complete them and record a `FINAL PASS` before Task 5. + +Three failure shapes, any of which blocks Stage 0 independently of the `Vary` verdict: + +- Bodies differ by cookie **and** `Vary` does not name `Cookie` → cross-serving of + personalized HTML. +- Origin emits `Set-Cookie` alongside a shared-cacheable `Cache-Control` → the cache can + replay one visitor's cookie to the next. TS's privacy net does not help; it downgrades + **TS's** response, after the cache has already stored the origin's. +- The deployment is `Authorization`-gated (as #1009 describes) and authorized responses + are cacheable → same problem, different header. + +- [ ] **Step 6: Request with the experiment header, if the operator uses one** + +Repeat Step 2 with the publisher's experiment header set to two different values. +Record whether the bodies differ and whether `Vary` names that header. + +- [ ] **Step 7: Record the finding** + +Create `docs/superpowers/plans/2026-08-08-1009-measurement-findings.md`: + +```markdown +# #1009 measurement findings + +## Step A — origin `Vary` declaration + +**Date:** · **Checked by:** + +| Representation | `Content-Type` returned | `Content-Length` | `Vary` present? | `Vary` value | +| --------------------- | ----------------------- | ---------------- | --------------- | ------------ | +| HTML navigation | | | | | +| RSC | | | | | +| RSC + `Next-Router-*` | | | | | +| Experiment variant | | | | | + +**Cookie / auth exposure:** bodies differ by cookie? `Vary: Cookie` present? origin +`Set-Cookie` on a shared-cacheable response? `Authorization`-gated responses cacheable? + +**Verdict:** FINAL PASS / PROVISIONAL PASS / FAIL + +`FINAL PASS` = `Vary` names every request header the origin varies on (`RSC`, any +`Next-Router-*` or experiment header whose value changed the body, **and `Cookie` if +bodies differ by cookie**), no `Set-Cookie` rides a shared-cacheable response, **and** all +five conditions in Task 5's gate are recorded — a real authenticated session cookie, Basic +Auth through TS, the experiment variant, representative routes, and cached-hit +slot/render attribution. + +`PROVISIONAL PASS` = the `Vary` and cookie checks hold, but one or more of those five is +untested. **Not a release gate.** A first pass lands here. + +`FAIL` = any `Vary` or `Set-Cookie` criterion is unmet. + +**Consequence:** `FINAL PASS` → Task 5a (flip the flag). `PROVISIONAL PASS` → close the +gaps before Task 5 starts. `FAIL` → Task 5b (cache-key discriminator). See spec §4. + +**A FAIL is also a live production defect, not only a Stage 0 blocker.** RSC fetches are +not navigations, so they never set the bypass and **already transit the read-through +cache today**. If the origin varies undeclared on `Next-Router-*`, TS is cross-serving RSC +variants in production right now. File it immediately rather than deferring with Task 5b. +``` + +- [ ] **Step 8: Commit** + +CI gate 7 runs `prettier --check` across all of `docs/`, so format the findings file +before staging it — a filled-in markdown table will not be prettier-clean by hand. + +```bash +cd docs && npx prettier --write superpowers/plans/2026-08-08-1009-measurement-findings.md && cd .. +git add docs/superpowers/plans/2026-08-08-1009-measurement-findings.md +git commit -m "Record origin Vary findings for #1009 Stage 0 gate" +``` + +--- + +## Task 2: Step B — what consumes TS's own response headers + +**Files:** + +- Modify: `docs/superpowers/plans/2026-08-08-1009-measurement-findings.md` — **created by + Task 1 Step 6.** If Task 1 has not reached that step, create the file with just its + `# #1009 measurement findings` heading rather than blocking. + +Investigation. Determines whether the spec's Stage 3b has a consumer. Does not gate +Task 5, but it appends to Task 1's findings document — do not run the two concurrently +against that file. + +- [ ] **Step 1: Pick a path that already emits shared-cache headers** + +`serve_static_with_etag` emits `public, max-age=300, s-maxage=300` plus +`Surrogate-Control` — see `crates/trusted-server-core/src/http_util.rs:294-311`. It backs +the `/static/tsjs=` bundle route (`publisher.rs:303`, `:322`). Use that URL against +the deployed service. + +- [ ] **Step 2: Request it twice and inspect for cache markers** + +```bash +URL="https:///static/tsjs=" +curl -sS -D - -o /dev/null "$URL" | grep -iE 'x-cache|age:|x-served-by|hit-state' +sleep 2 +curl -sS -D - -o /dev/null "$URL" | grep -iE 'x-cache|age:|x-served-by|hit-state' +``` + +Expected on the second request: if a cache sits in front of the Compute service, an `age` +greater than zero or an `x-cache` containing `HIT`. + +**The probe above is weak evidence** — absence of `age` is equally consistent with "no +cache" and "cold cache". **The topology check below is the actual answer; run it first and +skip the probe if it is conclusive.** + +```bash +fastly service list +fastly service-version list --service-id +# Look for a Delivery service fronting the Compute service, and for shielding +# configured on the service rather than only on the origin backend. +``` + +A Compute service with no Delivery service in front and no fronting shield does not have +its own output cached — that is the configuration the spec assumes, and this step exists +to confirm or refute it rather than to leave it assumed. + +**While you have the service open, answer a second question that matters more than this +task does:** is the _publisher backend_ shielded on the TS service? + +```bash +fastly backend list --service-id --version active +# Look for a shield on the publisher origin backend. +``` + +#1009's entire off-TS advantage came from a **shield** HIT, not a POP HIT. Whether +Stage 0 recovers a shield HIT or only a single-POP HIT changes the size of the win +materially, and nothing else in this plan establishes it. + +- [ ] **Step 3: Record the finding** + +Append to the findings document: + +```markdown +## Step B — consumers of TS's own response headers + +**Verdict:** SHARED CACHE PRESENT / NO SHARED CACHE + +**Evidence:** + +**Consequence:** NO SHARED CACHE → spec Stage 3b is inert until a topology change; +deprioritize it and ship only Stage 3a (browser caching). SHARED CACHE PRESENT → +Stage 3b gains a consumer AND the per-user `x-geo-*` header leak in spec §7 becomes an +active privacy exposure rather than a theoretical one. Escalate immediately in that case. +``` + +- [ ] **Step 4: Commit** + +```bash +cd docs && npx prettier --write superpowers/plans/2026-08-08-1009-measurement-findings.md && cd .. +git add docs/superpowers/plans/2026-08-08-1009-measurement-findings.md +git commit -m "Record response-header cache consumer findings for #1009" +``` + +--- + +## Task 3: Step C — origin fetch timing, and the bypass flag + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-core/src/settings.rs` — `publisher.bypass_origin_cache`, + `default_bypass_origin_cache`, `debug.publisher_timing`, the `Publisher` `Default` impl, + eight test literals, and the `origin_host` doctest +- Modify: `crates/trusted-server-core/src/test_support.rs` (log-capture helper) +- Test: inside `mod ssat_cache_policy_tests` at `crates/trusted-server-core/src/publisher.rs:4541` + +**Use `web_time::Instant`, not `std::time::Instant`** — the workspace targets +`wasm32-wasip1` and `web_time` is the wasm-safe clock already used at +`crates/trusted-server-core/src/auction/orchestrator.rs:7`. + +### Two timings, and why these two + +Measure **`hold_wait_ms`** and **`origin_fetch_ms`**. Not the rewrite. + +`hold_wait_ms` is the decision. The hold's cost is literally the duration of one +`.await` — `collect_stream_auction` at `publisher.rs:793`, plus the two EOF variants in +`hold_finish_ready_segments` (`:869`) and `hold_finish_tail_segments` (`:896`). Two +`Instant`s around those calls answer "does the hold block?" directly, instead of +inferring it by comparing origin fetch against auction duration. + +`origin_fetch_ms` is attribution — how much of any win Stage 0 can claim. + +`rewrite_ms` decides nothing. Step C's verdict compares origin fetch against auction +collect, and the ceiling argument in spec §6.4 is structural — it needs no number. +Measuring the rewrite would mean instrumenting two finalizers +(`buffer_publisher_response_async` at `publisher.rs:1114`, and the +`async_stream::try_stream!` block at `publisher.rs:1286`), working around moves out of +`params` inside that block, and finding a correlation key that does not exist — +`OwnedProcessResponseParams` (`publisher.rs:1065-1087`) has no `request_path`, and adding +one means touching all 26 construction sites. + +None of that buys a decision. Skip it. If a rewrite figure is later wanted to set a +target, add it as a separate follow-on once the verdict is known. + +**Why a log line and not `Server-Timing`:** for `origin_fetch_ms` alone a response header +would in fact work — the value is known before headers commit. A log line is still +preferred because it is server-side (no dependence on a browser harness to collect it), +`log` is this project's instrumentation crate per `CLAUDE.md`, and the auction path +already measures itself the same way. The spec previously claimed `Server-Timing` cannot +work at all; that overbroad claim has already been corrected there. + +### Log volume — gate it + +The line sits after the origin send, so it fires for every publisher request that reaches +origin — tagged `ad_stack=false` for ineligible ones, not only for eligible navigations. +That is more useful for comparison and more log spend, and the instrumentation is +temporary either way. Gate it behind the existing debug surface rather than +emitting unconditionally: add a `#[serde(default)] pub publisher_timing: bool` to +`DebugConfig` (`crates/trusted-server-core/src/settings.rs:1872`), following +`ja4_endpoint_enabled` and `auction_html_comment` alongside it. Default `false`; enable +via `ts config push` for the measurement window, then disable. + +This also means the Step 1 test must set that flag in its settings fixture. + +The split is also what makes the Step 1 test achievable — `run_with_slots` +(`publisher.rs:4769`) invokes only `handle_publisher_request` and never drives either +finalizer, so a test asserting on a combined line could never pass. + +**What `origin_fetch_ms` actually measures.** `publisher.rs:2863-2865` sets +`.with_stream_response()` when the adapter supports it, so on Fastly `send()` returns at +response _headers_, not after the body downloads. `origin_fetch_ms` is therefore **origin +TTFB**, not full download time. Name it that way in the findings document. It is still +the correct before/after signal for Stage 0 — the bypass affects whether the request hits +a cache at all — but when comparing against auction `total_time_ms` in Step 9, compare +like with like and say which quantity each column holds. + +- [ ] **Step 1: Write the failing test** + +**Placement matters.** Add the test **inside `mod ssat_cache_policy_tests`** +(`publisher.rs:4541`), not the outer `mod tests` (`:4035`). Every helper it uses is +private to that nested module: `settings_with_enabled_auction_and_creative_opportunities` +(`:4684`), `article_slot` (`:4721`), `conditional_navigation_request` (`:4740`), +`queue_cacheable_html_response` (`:4752`), `run_with_slots` (`:4769`). Placed in the outer +module it will not resolve — and because two _other_ `article_slot` functions exist +(`:9593`, `:10276`) returning a different type, the failure surfaces as a confusing type +error rather than a missing-name error. + +**First, add the log-capture helper.** `crates/trusted-server-core/src/test_support.rs` +has none. Note its shape: the whole file is `#[cfg(test)] pub mod tests { … }`, so the +path is `crate::test_support::tests::capture_logs`, not `crate::test_support::capture_logs` +— see existing consumers at `auth.rs:103` and `config_payload.rs:48`. + +Two constraints the helper must respect or the test fails for unrelated reasons: + +- `log::set_boxed_logger` succeeds **once per process**. Install via a `OnceLock`/`Once` + and have `capture_logs()` return a guard that clears and then reads a shared buffer. +- Call `log::set_max_level(log::LevelFilter::Info)` or higher, or `log::info!` is filtered + out before it reaches the logger. +- **Do not have the guard hold the buffer's own `Mutex`.** The test body runs code that + calls `log::info!` on the same thread, and the logger must lock that same mutex to + append — `std::sync::Mutex` is not reentrant, so this **hangs** rather than failing. + Use two locks: a separate process-wide serialization mutex held by the guard, and the + buffer's own mutex taken and released per line by the logger. +- The buffer is process-global and every other concurrently-running `trusted-server-core` + test logs into it, so a `got: {captured}` diagnostic will be large. Assert with + `contains`, not equality. +- `log::set_max_level` is global for the test binary. Setting it to `Info` is fine, but it + affects every test in the process. + +```rust +#[tokio::test] +async fn eligible_navigation_logs_origin_fetch_duration() { + // Arrange + let logs = crate::test_support::tests::capture_logs(); + let mut settings = settings_with_enabled_auction_and_creative_opportunities(); + // The log line is gated; without this the assertions below can never pass. + settings.debug.publisher_timing = true; + let stub = Arc::new(StubHttpClient::new()); + queue_cacheable_html_response(&stub); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let slots = [article_slot()]; + + // Act + let _ = run_with_slots(&settings, &services, &slots, conditional_navigation_request()).await; + + // Assert + let captured = logs.contents(); + assert!( + captured.contains("publisher_timing"), + "eligible navigation should emit a publisher_timing log line, got: {captured}" + ); + assert!( + captured.contains("origin_fetch_ms="), + "publisher_timing should record origin_fetch_ms, got: {captured}" + ); +} +``` + +This test deliberately asserts only on the `publisher_timing` line. `run_with_slots` never +drives a finalizer, so `publisher_rewrite` is out of its reach — cover that separately if +at all, rather than contorting this test. + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cargo test -p trusted-server-core --target aarch64-apple-darwin \ + eligible_navigation_logs_origin_fetch_duration -- --nocapture +``` + +Expected: FAIL — no `publisher_timing` in the captured logs. (Substitute your host +triple; core tests run natively for fast iteration. The Viceroy run comes in Step 6.) + +- [ ] **Step 3: Time the origin fetch** + +In `publisher.rs`, at the top with the other imports, add: + +```rust +use web_time::Instant; +``` + +Then wrap the origin send. The current code is at `publisher.rs:2870`: + +```rust +let mut response = match services.http_client().send(platform_request).await { +``` + +Change it to: + +```rust +let origin_fetch_start = Instant::now(); +let mut response = match services.http_client().send(platform_request).await { +``` + +and immediately after the `match` completes (after the existing `};` that closes it, +before the existing `log::debug!("Publisher origin response received: ...")` at `:2888`): + +```rust +let origin_fetch_ms = u64::try_from(origin_fetch_start.elapsed().as_millis()).unwrap_or(u64::MAX); +``` + +**Make the bypass config-driven in the same change.** This is what lets Stage 0 ship as a +config flip rather than a second deploy — see Task 5. Replace the block at +`publisher.rs:2866-2868`: + +```rust +// Single source of truth for the request and the log line below. Operator- +// controlled so the read-through cache can be re-enabled without a release; +// see docs/superpowers/archive/2026-08-08-esi-cacheable-root-validation-design.md §4. +let cache_bypass = should_run_ad_stack && settings.publisher.bypass_origin_cache; +if cache_bypass { + platform_request = platform_request.with_cache_bypass(); +} +``` + +Add the setting to `Publisher` in `crates/trusted-server-core/src/settings.rs:29`, +**defaulting to today's behaviour** so this change is a no-op until deliberately flipped: + +```rust +/// Bypass the platform read-through cache on ad-eligible publisher navigations. +/// +/// `true` preserves the historical behaviour introduced by the SSAT 304-prevention +/// design. `false` lets those navigations use the read-through cache; the +/// conditional-header strip already guarantees a complete body on a cache HIT. +/// Temporary operator control for the Stage 0 rollout — remove once settled. +#[serde(default = "default_bypass_origin_cache")] +pub bypass_origin_cache: bool, +``` + +```rust +fn default_bypass_origin_cache() -> bool { + true +} +``` + +**Adding this field breaks nine sites. Update them in the same commit or Step 2 fails to +compile before it can produce the intended RED failure:** + +- The hand-written `Default` impl at `settings.rs:81-97`. +- Eight exhaustive test literals. The line numbers below anchor each + `let publisher = Publisher {` **opening**, not a field — add the new field inside each + brace: `settings.rs:3553`, `:3564`, `:3575`, `:3586`, `:3597`, `:3608`, `:3621`, + `:3635`. `clippy-fastly` runs `--all-targets`, so these gate lint too. +- The rustdoc example for `origin_host`, whose literal opens at `settings.rs:130`. + **This is a live doctest** and the host-triple test command below does not skip + doctests. + +While there, mirror the existing default-agreement test +`publisher_default_max_buffered_body_bytes_matches_config_default` (`settings.rs:3648`) — +it exists to catch a hand-written `Default` diverging from a serde default, which is +exactly the shape this field re-introduces. One assertion. + +Then emit the line, immediately after computing `origin_fetch_ms`, gated on the debug +flag from the section above: + +```rust +if settings.debug.publisher_timing { + log::info!( + "publisher_timing origin_fetch_ms={origin_fetch_ms} \ + cache_bypass={cache_bypass} ad_stack={should_run_ad_stack}" + ); +} +``` + +- [ ] **Step 4: Instrument `hold_wait_ms` — the decision metric** + +This is the number the whole effort turns on, and it needs **one edit in one function**. + +`collect_stream_auction` (`publisher.rs:2431`) is the only function that awaits the +auction collect, and all three call sites reach it: + +| Call site | Path | +| ------------------- | ------------------------------------------------------------ | +| `publisher.rs:793` | `hold_collect_close_tail` — Fastly lazy stream | +| `publisher.rs:2257` | `body_close_hold_loop`, EOF arm — Axum, Cloudflare, Spin | +| `publisher.rs:2311` | `body_close_hold_loop`, mid-stream arm — same three adapters | + +Instrument the callee, not the callers. It already destructures `settings` out of +`AuctionCollectDeps` (`:2436`), so the debug flag is in scope with no new plumbing, and +one edit covers every adapter. + +Wrap the `collect_dispatched_auction` await at `:2447-2449`: + +```rust + let hold_wait_start = Instant::now(); + let result = orchestrator + .collect_dispatched_auction(dispatched, services, &collect_ctx) + .await; + if settings.debug.publisher_timing { + let hold_wait_ms = + u64::try_from(hold_wait_start.elapsed().as_millis()).unwrap_or(u64::MAX); + log::info!("publisher_hold hold_wait_ms={hold_wait_ms}"); + } +``` + +`settings` here is `&&Settings` from the destructure — deref as needed; the compiler will +say so. + +**Do not instrument `hold_finish_ready_segments` (`:869`) or `hold_finish_tail_segments` +(`:896`).** Neither awaits the collect. The first returns `close_found` for its caller to +act on; the second delegates to `hold_collect_close_tail` at `:909`. Instrumenting them +would double-count. + +**Do not instrument the auction itself.** `OrchestrationResult::total_time_ms` +(`orchestrator.rs:285`, struct at `:1449`, per-provider at `:365`) already flows to +`auction_events_raw`. `hold_wait_ms` measures something different and more useful: how +long the _response_ waited, which is near zero when the auction finished during transfer +even though `total_time_ms` is large. + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cargo test -p trusted-server-core --target aarch64-apple-darwin \ + eligible_navigation_logs_origin_fetch_duration -- --nocapture +``` + +Expected: PASS. + +- [ ] **Step 6: Run the full publisher test module under the real target** + +A format-changing edit to this file can break tests far from the one you added, and the +Viceroy runner aborts on the first panic — so run the whole suite, not a filtered subset. + +```bash +cargo test-fastly +``` + +Expected: PASS. `app::tests` DNS `Error` lines in the output are pre-existing noise. + +- [ ] **Step 7: Verify format and lint** + +```bash +cargo fmt --all -- --check +cargo clippy-fastly +``` + +Expected: both clean. + +- [ ] **Step 8: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs \ + crates/trusted-server-core/src/settings.rs \ + crates/trusted-server-core/src/test_support.rs +git commit -m "Add an operator switch for the origin cache bypass and log origin fetch time" +``` + +Staging without `settings.rs` leaves a tree that does not compile. + +- [ ] **Step 9: Deploy and collect** + +Deploy first. Then enable the log — it is gated and off by default: + +```bash +# In the operator's trusted-server.toml, under [debug]: +# publisher_timing = true +ts config push +``` + +**Deploy before pushing, not after.** `Settings`, `Publisher`, and `DebugConfig` all carry +`#[serde(deny_unknown_fields)]`, and `ts config push` validates against the typed schema +(`crates/trusted-server-cli` → `run_config_push_typed::`). So the +`ts` binary must be rebuilt from this commit (`cargo install-cli`), and pushing the new +keys before the new WASM is live would break config load on the deployed build. +`trusted-server.example.toml:121-125` records this same hazard for +`auction.rewrite_creatives`. + +Then capture the **bypass-on baseline only**. Do not try to collect an off arm here — +turning the bypass off _is_ Task 5, which is gated on Task 1's verdict and forbidden on a +FAIL. The off arm is collected in Task 5 Step 8. + +Capture enough navigations to separate the medians with confidence, across both a homepage and an article path, with the bypass +both on and off. Record the N alongside the result. + +Append to the findings document: + +```markdown +## Step C — server-side latency breakdown + +**N per arm:** · **Paths:** · **Date:** + +| Arm | `origin_fetch_ms` = origin TTFB (median) | auction `total_time_ms` (median) | `rewrite_ms` (median) | +| ---------- | ---------------------------------------- | -------------------------------- | --------------------- | +| bypass on | | | | +| bypass off | | | | + +Read the asymmetry carefully. `origin_fetch_ms` is origin **TTFB** — the send returns at +response headers because `.with_stream_response()` is set — whereas `total_time_ms` is +the auction's full duration. The comparison below is still the right one, but it is not +comparing two like quantities. + +**Verdict:** HOLD IS FREE / HOLD IS COSTING + +Read it off `hold_wait_ms` directly — no model, no comparison against auction duration. + +HOLD IS FREE = `hold_wait_ms` median near zero. The auction finishes during body +transfer. Proceed as staged in spec §7: Stage 0 primary, Stage 2 protects its win. + +HOLD IS COSTING = `hold_wait_ms` median materially non-zero. **Staging inverts** — +Stage 2 becomes primary and Stage 0 secondary. The work does not change, only its order. +Spec §6.2 argues for the first outcome but explicitly does not prove it, so treat the +second as a live possibility. +``` + +- [ ] **Step 10: Commit the findings** + +```bash +cd docs && npx prettier --write superpowers/plans/2026-08-08-1009-measurement-findings.md && cd .. +git add docs/superpowers/plans/2026-08-08-1009-measurement-findings.md +git commit -m "Record server-side latency breakdown for #1009" +``` + +--- + +> **Task 4 (spec correction) was completed while this plan was being written.** §3's +> mechanism bullet, §4's operator-flag framing, and the `unexpected_origin_304` watch are +> all already in the spec. Nothing to do; the task is removed rather than left as a +> no-op an implementer would stall on. + +--- + +## Task 5: Stage 0 — turn the origin cache bypass off + +**Gate:** do not flip the flag until Task 1 has recorded a **`FINAL PASS`**. There are +three verdicts, not two. + +- **`FINAL PASS`** → Task 5a (config flip). +- **`PROVISIONAL PASS`** → **stop.** Not a release gate. This is the current state. It + means the representation split is declared correctly under the conditions tested, and + that those conditions were too narrow to flip production on. +- **`FAIL`** → Task 5b. Do **not** flip; it can serve an RSC payload to an HTML + navigation. + +**`FINAL PASS` requires all five, each recorded in the findings document:** + +| Condition | Why the provisional run is insufficient | +| -------------------------------------------------------- | ---------------------------------------------------- | +| A real authenticated or state-bearing session cookie | `sessionid=abc123` is synthetic and proves nothing | +| Basic Auth exercised **through TS**, not just the origin | #1009 describes a gated deployment | +| The experiment variant named in #1009 | Absent from the origin's `Vary`; unexplained | +| Representative routes — article, section, search | Only the homepage was probed | +| Cached-hit slot and render attribution | The randomized div IDs are an unverified interaction | + +Any one of these unrecorded means the verdict stays `PROVISIONAL PASS` and Task 5 does +not start. + +Because Task 3 made the bypass config-driven, Stage 0 ships as a **config change on an +already-deployed build** — no second release, and the read path reverts with another +config push rather than a revert. That matters here: the failure mode this gates on is +cache poisoning, where minutes of exposure are worse than a slow rollout. + +**But a config push is not a full rollback.** It stops HTML navigations reading from +cache; it evicts nothing already stored. See Step 4's rollback sequence — flip, then purge +or roll a versioned namespace, then observe past the origin TTL. Until a raw origin cache purge path +exists, the tail is "wait out the origin TTL," and that must be an accepted, recorded +risk before the flip. + +### Task 5a: flip the flag (Task 1 verdict = `FINAL PASS`) + +**Files:** + +- Modify: the operator's `trusted-server.toml` (gitignored) +- Modify: `crates/trusted-server-core/src/publisher.rs` — the test, and later the default +- Modify: `trusted-server.example.toml` — document the key + +- [ ] **Step 1: Add a test covering the flag in both positions** + +The existing test at `publisher.rs:4824` +(`eligible_navigation_bypasses_cache_and_returns_non_storable_html`) asserts `vec![true]` +and must **keep passing** while the default is `true` — it now documents the default +rather than the only behaviour. Leave it, and add a sibling next to it: + +```rust +#[tokio::test] +async fn eligible_navigation_uses_read_through_cache_when_bypass_disabled() { + // Arrange + let mut settings = settings_with_enabled_auction_and_creative_opportunities(); + settings.publisher.bypass_origin_cache = false; + let stub = Arc::new(StubHttpClient::new()); + queue_cacheable_html_response(&stub); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let slots = [article_slot()]; + + // Act + let response = + run_with_slots(&settings, &services, &slots, conditional_navigation_request()).await; + let response_head = response_head(response); + + // Assert + assert_eq!( + stub.recorded_cache_bypass_flags(), + vec![false], + "disabling bypass_origin_cache should let the navigation use the read-through \ + cache; the conditional-header strip already guarantees a full body on a HIT" + ); + assert_eq!( + recorded_header( + stub.recorded_request_headers().first().expect("should record request"), + header::IF_NONE_MATCH.as_str() + ), + None, + "conditional headers must still be stripped with the bypass disabled" + ); + assert!( + response_head + .headers + .get(header::CACHE_CONTROL) + .and_then(|v| v.to_str().ok()) + .is_some_and(|v| v.contains("no-store")), + "the synthesized document must stay non-storable regardless of the bypass flag" + ); +} +``` + +Those last two assertions are the point of the test: the flag must change **only** the +cache mode, leaving the conditional-header strip and the response non-storability intact. + +**Leave `publisher.rs:4941` and `:5160` unchanged** — they already assert `vec![false]` +for non-eligible requests and must keep doing so. `Range`/`If-Range` stripping is covered +by `eligible_range_navigation_fetches_complete_html` (`publisher.rs:4883`), unaffected. + +- [ ] **Step 2: Run both tests** + +```bash +cargo test -p trusted-server-core --target aarch64-apple-darwin \ + eligible_navigation -- --nocapture +``` + +Expected: both the existing default-behaviour test and the new flag-disabled test PASS. +If Task 3's `bypass_origin_cache` field is not yet in place, the new test will not +compile — land Task 3 first. + +- [ ] **Step 3: Document both keys in the example config** + +Add to `trusted-server.example.toml` under `[debug]` (line 149, alongside +`ja4_endpoint_enabled` and `auction_html_comment`): + +```toml +# Emit a `publisher_timing` log line per publisher origin fetch. Temporary +# instrumentation for the #1009 latency measurement; leave false in production. +publisher_timing = false +``` + +And under `[publisher]`: + +```toml +# Bypass the platform read-through cache on ad-eligible navigations. +# `true` is the historical default. Set `false` to let those navigations use the +# read-through cache — only after confirming the origin declares `Vary` for every +# header it varies on (see the Stage 0 precondition). +bypass_origin_cache = true +``` + +- [ ] **Step 4: Flip it in the operator config and push** + +```bash +# In the operator's trusted-server.toml, under [publisher]: +# bypass_origin_cache = false +ts config push +``` + +Note from prior operational experience in this repo: the environment-variable overlay is +scalar-only **and** only overrides keys that already exist in the TOML. Adding the key to +the operator's file is required; setting only an env var will be silently dropped. + +**Rollback is a config push plus an eviction — not a config push alone.** Pushing `true` +again stops HTML navigations reading from cache, but evicts nothing: objects already +cached, including those RSC and other request classes keep reading, persist until they +expire. The origin's `max-age=60` bounds that, but does not remove it. + +Full rollback: + +1. Push `bypass_origin_cache = true`. +2. Purge — **and note this is the raw origin cache, not the shared transformed-template cache.** + `InsertBuilder::surrogate_keys` belongs to the Core Cache API and applies to the + transformed-template cache the ESI spike builds. It has no effect on the HTTP read-through + cache that Stage 0 turns on. Purging the raw origin cache + requires either surrogate keys the **origin** supplies on its responses, or the HTTP + cache's own request/candidate surrogate-key surface. Confirm which is available before + relying on it. + + **Neither is wired today.** If the flip ships before one exists, the rollback story is + "wait out the origin TTL" — roughly a minute, per the Step A findings. That is + survivable, but it must be an accepted risk recorded before the flip rather than a + discovery during an incident. + +3. Observe past the origin TTL before declaring the incident closed. + +- [ ] **Step 5: Run the full suite across every adapter** + +```bash +cargo test-fastly && cargo test-axum && cargo test-cloudflare && cargo test-spin +``` + +Expected: all PASS. If `platform/test_support.rs:797` or `:888` fail, they are testing +the stub's own recording behaviour rather than publisher behaviour — read them before +changing anything. + +- [ ] **Step 6: Format and lint every target** + +```bash +cargo fmt --all -- --check +cargo clippy-fastly && cargo clippy-axum && cargo clippy-cloudflare \ + && cargo clippy-cloudflare-wasm && cargo clippy-spin-native && cargo clippy-spin-wasm +``` + +Expected: all clean. + +- [ ] **Step 7: Commit the code and config-template changes** + +```bash +git add crates/trusted-server-core/src/publisher.rs trusted-server.example.toml +git commit -m "Add an operator switch for the publisher origin cache bypass" +``` + +- [ ] **Step 8: Watch for the failure modes, not just the win** + +After the flip, check three things before declaring success. The first two are regression +signals, not confirmations. + +1. **`unexpected_origin_304` abandonment telemetry.** This reason + (`publisher.rs:2896`, emitted via `emit_abandoned_auction` at `:2360`) exists because + the ad-stack path refuses cached and conditional origin responses. Re-enabling the + cache is precisely what could revive it. **Any non-zero rate is a rollback signal** — + it means a 304 is reaching TS, which the conditional-header strip was supposed to make + impossible. Push `true` and investigate before continuing. +2. **Representation mixing.** Spot-check that HTML navigations still return HTML and RSC + fetches still return `text/x-component`. A mismatch is the Task 1 risk having + materialized despite a PASS verdict — roll back immediately, this is cache poisoning. +3. **`origin_fetch_ms` and `cache_bypass=false`** in the `publisher_timing` logs. This is + the win, and it is the _last_ thing to check, not the first. + +- [ ] **Step 9: Record and commit** + +Append the before/after medians and the three checks above to the findings document, +format it, and commit. + +- [ ] **Step 10: Retire the flag (follow-up, not now)** + +Once the flip has held for a sustained period, flip the default to `false` in +`default_bypass_origin_cache`, then remove the setting and the branch entirely. Track it; +a temporary flag left in place becomes permanent configuration surface. + +### Task 5b: cache-key discriminator (Task 1 verdict = FAIL) + +**Do not implement from this plan.** A FAIL means the origin serves multiple +representations at one URL without declaring `Vary`, so removing the bypass requires TS +to add its own cache-key discriminator — a feature, not a deletion, and materially larger +than Stage 0 as scoped here. + +Escalate with the Task 1 findings and write a separate plan. Two things that plan must +address, both from spec §4: + +1. The discriminator must key on the request headers that actually distinguish the + representations (`RSC`, `Next-Router-*`, the experiment header), **not** on the + navigation classification. `is_navigation_request` + (`crates/trusted-server-core/src/http_util.rs:73-98`) falls back to the `Accept` + header when Fetch Metadata is absent, and its own comment warns that `fetch()` can set + `Accept: text/html` — so a fetch-based request can be misclassified as a navigation. +2. Whether the origin should simply be asked to declare `Vary`, which is cheaper than + building the discriminator and fixes the problem for every consumer rather than only + for TS. + +--- + +## Out of scope + +Named so nobody widens this plan mid-flight. All are specified in the spec. + +- **Stages 1–2** — moving bid delivery off the response body and deleting the `` + hold. Spec §7 and §8 put these behind the correctness defects. Spec §5 explains why + starting them casually produces a silent revenue loss. +- **Stages 3a/3b** — response cacheability. 3b is additionally gated on Task 2. +- **Stages 4–5** — purge capability, TS-owned template cache, ESI. +- **Removing the `bypass_cache` platform capability.** Task 5a removes one call site only. + +--- + +## Definition of done + +- [ ] Findings document records verdicts for Steps A, B, and C, each with its date, its + N where applicable, and the consequence spelled out. +- [ ] `publisher_timing` and `publisher_hold` lines are emitted in production and + readable, and `hold_wait_ms` has a recorded median. +- [ ] Task 1 recorded a **`FINAL PASS`** — all five conditions in Task 5's gate closed, + not merely the provisional run. +- [ ] Either Task 5a is shipped, or Task 1 returned FAIL and both a production defect and + a follow-up plan for 5b exist. +- [ ] A purge path or versioned cache-key namespace exists **before** the flip, or the + "wait out the TTL" rollback is explicitly accepted and recorded as a risk. +- [ ] **The win is measured client-side, not from `origin_fetch_ms`.** That figure is + origin TTFB and excludes body download, rewrite, and post-processing — it is + attribution, not the outcome. #1009 already has a working tester-cookie browser A/B + measuring the TTFB the publisher actually complained about; use it for before/after. +- [ ] `unexpected_origin_304` rate is zero and representations are not mixed (Task 5a + Step 8) — both checked **before** the win is claimed. +- [ ] All CI gates pass: `cargo fmt --all -- --check`; the six clippy targets; the four + adapter test suites; the parity suite; JS build, test, and format; docs format. diff --git a/docs/superpowers/plans/2026-08-08-1009-measurement-findings.md b/docs/superpowers/plans/2026-08-08-1009-measurement-findings.md new file mode 100644 index 000000000..3582077da --- /dev/null +++ b/docs/superpowers/plans/2026-08-08-1009-measurement-findings.md @@ -0,0 +1,503 @@ +# #1009 measurement findings + +Recorded output of the checks in +[the plan](./2026-08-08-1009-measurement-and-stage-0.md). Results only — the origin +hostname is operator config and is deliberately not reproduced here. + +> **Final implementation note, 2026-08-12.** The opt-in `esi` mode now uses Fastly +> Core Cache plus an exact inert byte seam. The parser and client-fill experiments were +> removed. Real-origin numbers in this file remain evidence about the observed path, not +> a clean before/after benchmark. Current operational semantics are documented in +> [the configuration guide](../../guide/configuration.md). + +## Step A — origin `Vary` declaration and cookie exposure + +**Date:** 2026-08-08 · **Method:** direct `curl` against the publisher origin with the +configured `origin_host_header_override`, homepage path. + +### Representation split + +| Representation | `Content-Type` | `Cache-Control` | `Set-Cookie` | +| ------------------------------------ | ------------------ | --------------- | ------------ | +| HTML navigation | `text/html` | `max-age=60` | none | +| `RSC: 1` | `text/x-component` | `max-age=60` | none | +| `RSC: 1` + `Next-Router-Prefetch: 1` | `text/x-component` | `max-age=60` | none | + +`Vary`, identical on every response: + +``` +vary: rsc, next-router-state-tree, next-router-prefetch, next-router-segment-prefetch, Accept-Encoding +``` + +The origin declares **every** header that distinguishes the representations, including +`next-router-segment-prefetch`, which the plan's probe list did not think to check. The +HTML/RSC split at one URL is real and correctly declared. + +### Cookie personalization + +Hash comparison was useless here and the plan's probe as written would have produced a +false FAIL — see the method note below. After normalizing per-request identifiers: + +| Comparison | Differing lines | +| ------------------------------ | --------------- | +| no-cookie A vs no-cookie B | 2 | +| no-cookie A vs **with cookie** | 2 | + +Both diffs are the same single `generationTimestamp` field in the RSC payload. **The +cookie changes nothing.** Byte lengths were identical across all three responses +(1,432,944). + +Cookie sent: `ts-tester=true; sessionid=abc123; ts-ec=probe`. + +### Verdict: **PROVISIONAL PASS** — not sufficient to gate a production flip + +Downgraded 2026-08-10 after external review. Everything below held under the conditions +tested; the conditions tested are narrower than the gate requires. + +What passed: + +- `Vary` names every request header the origin varies on. ✅ +- Bodies did not differ by the cookie sent, so `Vary: Cookie` was not required **for + that cookie**. ✅ +- No `Set-Cookie` on a shared-cacheable response. ✅ +- Origin returns 200 without credentials at this layer. ✅ + +**What was not tested, and each of these can flip the verdict:** + +| Gap | Why it matters | +| ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `sessionid=abc123` is not a real session | A synthetic value proves nothing about a state-bearing publisher session. An authenticated or paywall-metered session is exactly the case that would personalize. | +| One route (homepage) only | Article, section, and search routes may personalize differently. | +| Experiment variant never exercised | #1009 says the origin varies on one. It is absent from `Vary` — see Residual uncertainty below. | +| Basic Auth through TS untested | #1009 describes a gated deployment. Only the origin was probed directly. | +| Cached-hit slot resolution untested | The randomized div IDs below are an unverified interaction, not a cleared one. | + +**Consequence:** Stage 0 still takes the operator-flag path rather than the cache-key +discriminator, and no live cross-serving defect is indicated. But this is **not** a +release gate. Close the table above before flipping the flag in production. + +## Two findings the checks were not looking for + +### 1. The origin already intends this page to be shared-cached + +`cache-control: max-age=60` with a correct `Vary` and no `Set-Cookie`. The origin has +been cacheable all along; Trusted Server opted out of it. That is the spec's §4 framing +confirmed from the other side, and it strengthens the case that the bypass was +belt-and-braces rather than load-bearing. + +It also bounds the win: a 60-second TTL means Stage 0 buys a cache hit only within that +window. Whether that translates into a meaningful hit rate depends on request volume per +URL, which is not measured here. + +### 2. Ad-slot div IDs are randomized per request — and this interacts with Stage 0 + +The only per-request variance in the document is ~170 lines of ad-slot container IDs, +each a fresh 32-hex UUID: + +``` +ad-in_content-f75fa7fba54a4fc2a2d787f51c1837dd-in_content-0 +ad-in_content-a968b27e3ee2424f8bb1c19560abf2b1-in_content-0 ← same slot, next request +``` + +Under the bypass, Trusted Server sees fresh IDs on every request. **Once the cache is on, +every visitor within a 60-second window receives the same IDs.** + +This is very likely fine — `tsjs.adSlots` is built from configured slot definitions, not +scraped from origin markup, and injection is a prefix match on the configured `div_id`. +But it is an untested interaction between Stage 0 and the slot-matching path, and it was +not in anyone's risk list. **This is a release gate, not a note.** Verify slot matching resolves against a cached +document before flipping the flag, and watch TS-attributed renders across the flip +rather than only `origin_fetch_ms`. + +## Method note — a defect in the plan's Step A probe + +The plan's cookie check compares `shasum` of the response bodies. On this origin that +test always fails, cookie or not, because of the randomized div IDs above. Three requests +produced three different hashes with byte-identical lengths. + +**Correct method:** normalize per-request identifiers before comparing, e.g. +`sed -E 's/[0-9a-f]{32}/UUID/g'`, and diff the normalized bodies rather than hashing +them. Establish the no-cookie baseline drift first, then compare the cookie arm against +that baseline — a cookie arm is only interesting if it differs by _more_ than the +baseline does. Fix the plan before anyone re-runs this. + +## Residual uncertainty + +#1009 states the origin varies on an experiment header as well as `rsc` and +`next-router-*`. **No experiment header appears in the origin's `Vary` list**, and the +RSC payload's `experiments` key did not differ across any of the requests made here. + +Three readings, unresolved: the issue was imprecise; experiments are assigned +client-side; or they key on a cookie value this probe did not supply. The `Vary` +declaration is authoritative for cache correctness and it is thorough enough to name four +Next-specific headers, so this is unlikely to be a cache-safety gap. Worth one question +to whoever wrote that line in #1009 rather than further probing. + +## Rollback caveat, added 2026-08-10 + +The plan described flipping the flag back as a seconds-long rollback. That is +incomplete. Re-enabling the bypass stops **HTML navigations** reading from cache; it +evicts nothing. Objects already cached — including those RSC and other request classes +continue to read — persist until they expire. + +Two mitigations, both real: + +- The origin's `max-age=60` bounds read-through exposure to roughly a minute. +- Purge exists in-process — `fastly::http::purge::purge_surrogate_key`. An earlier claim + that TS had no purge capability was wrong; it has no _wiring_, which is buildable. + +**But note which cache.** `InsertBuilder::surrogate_keys` belongs to the **Core Cache** +API and applies to the shared transformed-template cache the ESI spike would build. It has +**no effect on the raw origin cache** that Stage 0 turns on. Purging the raw origin cache needs +surrogate keys the _origin_ supplies on its responses, or the HTTP cache's own +request/candidate surrogate-key surface. Confirm which is available before relying on it — +an earlier revision of this document conflated the two. + +**The shared transformed-template cache's purge is locally testable; the raw origin cache's is not.** Verified 2026-08-10: Viceroy 0.17 +implements `purge_surrogate_key` against the same in-process cache it serves reads from +(`viceroy-lib-0.17.0/src/wiggle_abi/fastly_purge_impl.rs:10-32`), soft purge included. So +the purge-based rollback for the shared template cache the spike builds can be exercised end +to end without a Fastly service. That does nothing for Stage 0, whose exposure is the raw origin cache. + +Rollback is therefore: flip the flag, **then** purge the raw origin cache by whichever mechanism is actually +available (or roll a versioned key namespace), **then** observe past the origin TTL before +declaring the incident closed. With no raw origin cache purge path wired, the tail is the TTL itself — +roughly a minute here, and a recorded risk rather than a surprise. + +## ESI spike Task 1 — does `esi` 0.7 build on this toolchain? + +**Date:** 2026-08-10 · **Verdict: PASS.** The cheapest falsifier for the ESI question +clears. #1009 is not closed by a toolchain limit. + +| Check | Result | +| ------------------------------------------------------------------------------------------ | -------------------- | +| `cargo add esi@0.7 --package trusted-server-adapter-fastly` | resolved `esi 0.7.1` | +| `cargo check-fastly` (Rust 1.95.0 / `wasm32-wasip1`) | clean | +| `cargo fmt --all -- --check` | clean | +| All six clippy targets (fastly, axum, cloudflare, cloudflare-wasm, spin-native, spin-wasm) | clean | +| `cargo check --manifest-path crates/trusted-server-integration-tests/Cargo.toml --tests` | clean | + +**Nine new transitive dependencies:** `esi 0.7.1`, `nom 8.0.0`, `rand 0.10.2`, +`rand_core 0.10.1`, `chacha20 0.10.1`, `cpufeatures 0.3.0`, `atoi 2.0.0`, +`html-escape 0.2.15`, `md5 0.8.1`. + +**No existing shared dependency moved.** `regex` stays 1.12.4, `bytes` 1.12.0, `log` +0.4.33. `nom` and `rand` gain new majors that coexist with the existing 7.1.3 / 0.8.6 / +0.9.4 rather than replacing them — the best available outcome, since a forced bump on a +shared dep is what would have made this expensive. + +### A claim in the spike plan was wrong + +Task 1 Step 3 told the implementer to check for a desync between the root `Cargo.lock` and +`crates/trusted-server-integration-tests/Cargo.lock`. **That second lockfile does not +exist.** The integration-tests crate is a workspace member (root `Cargo.toml:10`) and +shares the root lockfile, so the desync hazard cannot arise in that form. The plan has +been corrected. The dual-lockfile constraint was real at some earlier point; it is not the +current layout. + +### Viceroy 0.17 supports the whole Core Cache surface this spike needs + +**Date:** 2026-08-10 · **Verdict: PASS.** Probed directly under +`cargo test -p trusted-server-adapter-fastly --target wasm32-wasip1`, then removed: + +| API | Result | +| ------------------------------------------------------------------------ | ------ | +| `cache::core::insert(key, ttl).execute()` → write → `finish()` | works | +| `cache::core::lookup(key).execute()` → `Found::to_stream()` | works | +| `Transaction::lookup(key).execute()` → `must_insert_or_update()` | works | +| `Transaction::insert(ttl).surrogate_keys([…]).execute_and_stream_back()` | works | +| Second transactional lookup reports a hit, no obligation | works | + +That is the entire API surface the spike's Task 3 Step 4 specifies, including the +transaction and stream-back shapes. + +**Consequence: provisioning a Fastly service is not a prerequisite.** An earlier revision +of the spike plan made it Task 2 and a blocker on everything downstream. Almost all of the +correctness and safety work — the shared template-cache logic, the transform, template byte-identity, +ESI assembly, DCA and dispatcher refusal, fragment-failure degradation, header ordering, +and the leakage gates — runs locally. The plan is re-sequenced accordingly. + +**What still needs a real service:** shielding behaviour, POP-level cache tiering, +request collapsing under genuine concurrency (Viceroy is a single instance, so a passing +`Transaction` test proves the API works and not that collapsing is correct under load), +stale revalidation timing, and **every performance number in the decision rule**. Local +timings are meaningless for the decision. + +### Not yet verified + +Compiling and a cache round-trip are not an implementation. Nothing yet exercises the +`lol_html` transform into the shared template cache, ESI assembly, or any runtime behaviour on the publisher path, +and the `esi` dependency is added but unused. + +## ESI spike Task 3 — implementation progress + +**Date:** 2026-08-10. All of it behaviour-neutral under the default +`AssemblyMode::Inline`; nothing here changes a shipped code path. + +| Step | State | +| ----------------------------------- | ------------------------------------------------------------------------------- | +| 1 — `AssemblyMode` setting | **Done.** `Option` on `CreativeOpportunitiesConfig`. | +| 2 — head-seam neutrality gate | **Done.** `template_ad_slots_script`, three byte-identity tests. | +| 2b — body-close decoupling | **Done.** `BodyCloseInjection`, `body_close_injection`. | +| 2c — emit the marker under `Esi` | **Not done.** Blocked on the fragment endpoint; see below. | +| 3 — template-cache eligibility gate | **Done.** `template_cache_bypass_reason`, eight tests. Logs only, no cache I/O. | +| 4 — template-cache read/write | **Not started.** Design choice open; see below. | + +### What is deliberately absent + +**No marker is emitted under `Esi`.** The marker must point at a fragment endpoint +returning an **executable script**. `/_ts/page-bids` returns JSON +(`publisher.rs`, `handle_page_bids`) and ESI splices fragment bytes verbatim, so aiming +at it would put raw JSON where a script belongs. That endpoint does not exist, and a +marker with nothing behind it is worse than no marker. A test pins the current answer so +it changes deliberately. + +**No cache read or write.** `template_cache_bypass_reason` has a real call site that logs its verdict, +which makes the decision observable during the spike without mutating anything. Task 3 +Step 4 is blocked on choosing between read-through-with-body-transform and explicit +`cache::core` — the plan names that as a decision to make before writing code, and it is +under investigation rather than assumed. + +### A defect this work introduced and then caught + +Gating the head seam on neutrality made `ad_slots_script` `None` under the shared modes. +The body-close element handler read exactly that value to decide whether to inject at all, +so shared modes silently stopped injecting anything at `` — a side effect of a +`` change. Safe, since emitting nothing cannot leak, but wrong in the way the spec +warns about: the gate has to be "did this response carry bids", not "does this page have +slots". + +Found by reading the handler while starting the next step, not by a failing test. Fixed by +replacing the inference with a named decision. The test that now guards it asserts +body-close is identical whether or not the head script is present — a decision that read +the head script would be _accidentally_ correct today, because that script is always +absent under shared modes, and wrong the moment that changes. + +Worth recording because it is the same shape as the bug the whole task exists to prevent: +something that looks correct and quietly does nothing. + +### Coverage and its limits + +Fourteen new tests. `fmt`, all six clippy targets, and all four adapter suites pass, with +1850 core tests under Viceroy. `clippy --all-targets` caught a benchmark construction site +that all four test suites missed — the suites are not the whole gate. + +**The neutrality guarantee is narrower than it looks.** The tests prove `tsjs.adSlots` is +neutral. They say nothing about the other things injected at the same seam — integration +`head_inserts`, the gpt-diagnostics bootstrap, the RSC placeholder rewriter — which the +spec flags as needing an audit and which that audit has not yet covered. Until it does, +treat request-neutrality as asserted for one element rather than established for the +template. + +## Code review of the Task 3 commits — three HIGH findings, all closed + +**Date:** 2026-08-11. An independent review of the four implementation commits found +three HIGH issues. The default `Inline` path was verified unchanged byte-for-byte, so +none was a live regression — but all three were invariants this branch exists to +establish and none was enforced or tested. + +### 1. The auction dispatched under shared modes with nothing to consume it + +`assembly_mode` was computed _after_ the dispatch decision, so flipping to `client_fill` +or `esi` would still have sent real SSP bid requests, held the response for the full +auction budget, and discarded the result — because both injection seams now return +nothing — with no error, no warning and no log. + +Exactly the silent-waste signature §5 of the design doc is about, reached by an +incomplete feature flag rather than by removing the hold. Fixed by hoisting +`assembly_mode` above the dispatch and gating on `root_auction_is_useful`. + +The test derives the invariant rather than asserting per-variant: a root auction is +useful exactly when a seam will consume its result. A new mode cannot make the dispatch +gate and the injection decisions disagree without failing it. + +### 2. The template-cache gate ignored the forwarded client `Cookie` + +TS forwards client cookies to origin unchanged — there is no `Cookie` strip on the +publisher path. So a response can be cookie-personalized while carrying no `Set-Cookie` +itself (session established earlier), no `Cache-Control` at all, status 200, HTML — and +every condition in the gate reported it cacheable. + +§4 of the design doc names this. The plan's own Task 3 Step 3 checklist missed it, so +the implementation matching the checklist exactly still had the hole. Now disqualifying +until an origin `Vary` covering `Cookie` is verified. + +### 3. Request-neutrality was asserted for one element, not the seam + +The head seam still injected integration `head_inserts` and the GPT-diagnostics +bootstrap unconditionally. + +Audited both. **`head_inserts` is clean** — all three implementations (datadome, didomi, +gpt) take the context parameter unused, so output depends on configuration, not the +request. **GPT diagnostics is not** — cookie- or query-activated, and documented as an +immutable request-scoped decision. + +It did not leak, but only by coincidence: `requires_private_no_store()` is a strict +superset of the conditions under which either script is emitted, and that stamp lands +before the template-cache gate reads response headers, so the gate refused. Two independent +conditions that happened to align, with nothing enforcing the relationship. + +Fixed on both sides — the processor receives no diagnostics decision under shared modes, +**and** a test enumerates every combination of the decision's three fields asserting that +anything which injects also requires the stamp. The gate is the guarantee; the invariant +test is the backstop if the gate is ever removed. + +### What this says about the tests that existed + +All three findings were in code the existing tests covered — and passed. The tests +exercised the pure decision functions with hand-built inputs and never the rendered +``/`` bytes. That is still true: **no test renders a full document through +`create_html_processor` and compares two requests byte-for-byte.** The plan's Task 3 +Step 2 requires exactly that, and it remains the most valuable missing test. + +### Reviewer's gate, adopted + +Do not proceed to Task 3 Step 4 (actual template-cache read/write) or expose `AssemblyMode` to any +test or staging traffic until the full-document byte-identity test exists. The three +fixes above close the known holes; that test is what would catch the next one. + +## Task 3 complete — the shared template cache engages end to end + +`2db10639` (store), `2a2e6c6a` (lookup), plus `b688d667`/`577eb85a` for the `Vary` +handling. A second request for the same URL is now served without touching the origin, +byte-identical to what was stored. + +**Three problems only appeared once the code had to run**, none of them visible in the +plan or in review: + +1. **The key needed the origin's `Vary`, but a lookup precedes the fetch.** Resolved with + an operator-stated list plus a post-response drift guard that refuses to store under a + key that missed something. Spike-grade: a two-phase lookup is the correct answer and + doubles the lookups. +2. **The key carried the encoding the _origin_ chose**, which also does not exist at + lookup time — storing under `br`, looking up under `gzip, br`, a cache that never hits. + Now keyed on what was sent to the origin. +3. **Storing needs every transformed byte; streaming does not collect them.** Shared modes + take the buffered finalizer, branching on the store authorization rather than the + assembly mode, so `Inline` cannot reach it. + +Each was a case where the design read as complete and the implementation had a hole in +it. That is the same pattern as the three review findings above, arriving one layer down. + +**Verified by mutation, not just by green tests.** Disabling the lookup fails the hit +test, so the hit is the cache answering rather than the fixture answering twice; dropping +the `Authorization` re-check fails the authenticated test; reading only the first `Vary` +header value, and disabling the drift guard, each fail their own tests. The reviewer's +gate above was satisfied first: the byte-identity tests it demanded exist and were +themselves mutation-checked. + +**Still not deployable.** `ClientFill` and `Esi` render a template with a hole and +nothing filling it — Task 4 and Task 5. A cache that works is necessary, not sufficient. + +## Local end-to-end run — the Esi arm renders + +`viceroy serve` against a stub origin, config pushed into a scratchpad `fastly.toml` so +nothing tracked was modified. Served document: + +```html +

Stub article

+
+

Body copy.

+ +``` + +No executable ESI tag. One origin fetch for two requests. `private, no-store` on the hit. +Cached template 353 bytes against 467 served, so the cache holds the pre-assembly +template. All three fragment formats behave: script, JSON, and `400` on a typo. `Inline` +unaffected — two fetches for two requests, no template-cache activity, no markers. + +### The bug only a running server could find + +With the auction **enabled**, the template cache never engaged: two origin fetches, marker unresolved. + +TS stamps its own `private, no-store` when `should_run_ad_stack` is true. The template-cache gate ran +after that stamp, read it as the origin's declaration, concluded `OriginNotShareable`, and +refused — **on every page that serves ads**, which is every page that matters. + +The more important half is why no test caught it. The fixture left the auction disabled +and passed `slots: &[]`, so `should_run_ad_stack` was false in every test, the stamp never +fired, and the ordering was unobservable. Every template-cache assertion had been made against the one +configuration where the template cache's hardest condition does not apply. + +Demonstrated both ways: with the old fixture, reintroducing the bug passes all seven +tests; with the corrected fixture it fails six. + +### Pattern across this branch + +Five bugs now share one shape — compiled, passed every existing test, and were wrong: + +1. The head-seam gate silently disabled body-close injection (`d9e05973`). +2. The key held the encoding the origin _chose_, so the cache could never hit (`2a2e6c6a`). +3. A template-cache hit served with no `Cache-Control` at all (`0adb578e`). +4. A template-cache hit dropped its in-flight auction, billing SSPs for nothing (`b3ac59a6`). +5. The gate read TS's own header as the origin's (`4c557347`). + +Three were found by writing the test the plan asked for. One needed a running server. None +were found by review — including my own, twice over on the same gate. + +The stale-cache test is the same failure in miniature: it passed while never reaching +`is_stale()`, and only mutation testing exposed that. A test that passes for the wrong +reason is worse than no test, because it is counted as coverage. + +## Independent review — two blockers, and two reasons the cache would have measured nothing + +An independent reviewer read `main...HEAD` and, importantly, **demonstrated** findings by +running code rather than inferring them. Four things it found that review-by-reading had +not. + +**A POST was answered from a cached GET.** `handle_publisher_request` is the `*`-method +fallback route, so a publisher path that renders on GET and accepts a form or webhook on +POST reaches it for both. The origin never saw the mutating request; the caller got `200` +and a page. Fixed at key construction, since the key governs lookup and store alike. + +**`Vary: Accept-Encoding` disqualified everything.** The key has a dedicated +`accept_encoding` field, so such an origin is already keyed correctly — but the coverage +check consulted only the operator-configured list and reported a gap. Every compressing +origin sends that header, so **the shared template cache would have stored nothing against any real origin**. +This is worse than a plain bug: the spike would have measured a hit rate near zero and +reported it as a result. + +**Cookies excluded essentially every repeat visitor.** Any cookie disqualified in both +directions, and TS sets its own identity cookie. The population that could ever see a warm +hit was roughly first-ever page views and cookie-less clients. The design notes called +this the "first-nav exception"; it is the common case, not the exception. Now opt-in via +`origin_is_cookie_independent`, with the `Vary: Cookie` drift guard overriding a wrong +assertion. + +**`ClientFill` had no end-to-end coverage.** The reviewer reintroduced a diagnostics leak +scoped to that mode and all 1889 tests passed. Investigation showed that specific mutation +is unreachable — `requires_private_no_store()` is a strict superset of the injection +condition and stamps before the gate reads headers — but only by a coincidence between two +independent conditions. Both the coverage gap and the coincidence are now pinned by tests. + +### What the review says about the review process + +The reviewer's confirmed findings all came from **running** something. Its clean bills — +no leak in the template itself, no `Inline` regression — came with positive evidence: +tracing that integration context types carry no per-reader field at all, and separately +proving the leakage test has teeth by breaking the store/assemble order and watching it +fail. + +Two of my own comments were wrong, and it caught both by checking rather than reading: +one claimed three call sites where there are two, the other described a dispatcher +mechanism that stopped existing when assembly moved to `CompletedRequest`. + +Verified afterwards against a running server with the origin advertising +`Vary: Accept-Encoding`: a cookie-bearing repeat visitor now costs one origin fetch across +two requests, and a POST still reaches the origin. + +## Step B — consumers of TS's own response headers + +Not yet run. + +## Step C — hold and origin fetch timings + +Not yet run. diff --git a/docs/superpowers/plans/2026-08-10-1009-esi-validation-spike.md b/docs/superpowers/plans/2026-08-10-1009-esi-validation-spike.md new file mode 100644 index 000000000..5b9a455e7 --- /dev/null +++ b/docs/superpowers/plans/2026-08-10-1009-esi-validation-spike.md @@ -0,0 +1,967 @@ +# #1009 ESI Validation Spike + +> **HISTORICAL SPIKE — DO NOT IMPLEMENT.** This document records the investigation, +> including executable ESI tags, parser/subrequests, and a client-fill arm that were all +> removed. Every unchecked item below is historical, not remaining work. The accepted +> implementation keeps the public `esi` spelling but uses Fastly C2 plus exact byte-seam +> assembly. See +> [the merge-hardening design](../specs/2026-08-12-1009-esi-merge-hardening-design.md) and +> [implementation plan](./2026-08-12-1009-esi-merge-hardening.md). + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development +> (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps +> use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Decide #1009 on evidence. Build a shared-template pipeline behind a flag, run +ESI and client-fill against it, and produce a decision record that either adopts ESI, +adopts client-fill, or rejects both — with the Fastly-only maintenance cost priced in. + +**Architecture:** +`origin → lol_html transform → fastly::cache::core → finalize headers → stream assembly`. + +Headers finalize **before** assembly, not after — streaming responses on this adapter +commit headers first and then pipe chunks, so nothing can be set once assembly starts. + +The transform emits **one unconditional marker at the body-close seam**. Not two: the +head seam is not a template hole, because `tsjs.adSlots` presence is request-gated +(Task 3 Step 2). The cached object is a shared template with no per-user bytes and no +request-dependent decisions. Assembly is either the `esi` crate (edge) or a client fetch +(browser), selected per request by the arm allocator so both are measured on one build. + +**Tech Stack:** Rust 2024, `wasm32-wasip1`, `fastly` 0.12.1 (`cache::core`, `http::purge`), +`esi` 0.7, `lol_html`, a real Fastly test service for cache behaviour. + +**Spec:** `docs/superpowers/specs/2026-08-08-esi-cacheable-root-validation-design.md` — +read the 2026-08-10 correction at the top and +[§6.6](../specs/2026-08-08-esi-cacheable-root-validation-design.md#66-the-esi-pipeline-corrected) +before writing any code. + +**Control:** [the Stage 0 plan](./2026-08-08-1009-measurement-and-stage-0.md). Its +instrumentation and its bypass flag are prerequisites — this plan compares against them +and does not duplicate them. + +--- + +## Why this plan exists + +An earlier revision of the spec concluded ESI was structurally impossible. It was wrong: +`fastly::cache::core` provides the cache boundary natively, and purge runs inside Compute. +That correction reopens #1009 as an empirical question, and this plan is how it gets +answered. + +**What is genuinely uncertain**, and what each arm is for: + +1. Does a shared template plus per-request assembly beat today's inline path enough to + matter? +2. Does **edge** assembly (ESI) beat **client** assembly (a fetch) by enough to justify a + Fastly-only rendering path that must be maintained alongside the portable one? +3. Can per-user leakage be excluded across cold MISS, warm HIT, stale revalidation, + transform failure, and fragment failure? + +Question 3 is a gate, not a metric. A win on 1 and 2 with a failure on 3 is a rejection. + +## Three caches, never conflated + +The original error came from treating these as one thing. Every task below names which it +means. + +| # | Cache | Contents | Status | +| --- | --------------------------------- | ----------------------------- | ----------------------------------- | +| C1 | Origin read-through | raw origin bytes | Exists. Stage 0 turns it back on. | +| C2 | Shared transformed template | post-`lol_html`, pre-assembly | **New.** What this plan builds. | +| C3 | Assembled-response delivery cache | final per-user output | **Must never exist.** Not proposed. | + +If a task appears to require C3, stop — that is the leakage failure mode, not a design +option. + +## Arms + +Five, but only four are treatable as equivalent. + +| Arm | Root | Bids | Notes | +| ------- | ----------------------- | ---------------- | ---------------------------------------------------------- | +| **A0** | inline, C1 bypassed | inline `` | Today. The baseline. | +| **A1** | inline, C1 on | inline `` | Stage 0. Isolates the bypass from the template change. | +| **A2** | shared template from C2 | client fetch | Portable. Works on all four adapters. | +| **A3** | shared template from C2 | ESI at the edge | Fastly-only. The thing #1009 proposed. | +| **REF** | origin direct, TS off | publisher's own | **Reference, not an arm.** Different work, not comparable. | + +A0→A1 measures the bypass. A1→A2 measures the template split. A2→A3 measures edge versus +client assembly — **that difference is the entire case for ESI**, and it is the number +this plan exists to produce. + +**Do not compare A2 and A3 on root TTFB.** They serve the same C2 template, so their root +timings should be near-identical by construction; a null result there proves nothing. +ESI's claimed advantage is that bids arrive without a client round-trip, so measure: +**bids-ready time**, **`adInit` fire time**, and **first TS-attributed creative paint**. +Root TTFB stays as a guard that the template path did not regress, not as the comparison. + +REF is included because #1009 anchors on it, and excluded from pass/fail because TS-off +does no auction and no injection. Comparing against it measures the feature's existence, +not its implementation. + +--- + +## Task order and dependencies + +``` +Task 1 (esi compiles) ── DONE, PASS ──┐ + ├──> Task 3 (C2 cache) ─┬──> Task 4 (A2 client-fill) +Stage 0 plan (flag + instrumentation) ┘ ├──> Task 5 (A3 ESI) + └──> Task 6 (safety gates) + │ + Task 2 (real service) ─────────────────────────────┴──> Task 7 (decision) +``` + +**Task 2 is not a blocker on Tasks 3–6.** Everything those tasks need is exercisable under +Viceroy 0.17 — verified, see Task 2. The real service is required only for the +measurements Task 7 decides on, so provision it once there is something worth measuring. + +Task 6 runs against every arm, not once at the end. + +--- + +## Task 1: Confirm `esi` 0.7 builds on this toolchain + +Cheapest possible falsification. Do this before anything else. + +**Files:** `crates/trusted-server-adapter-fastly/Cargo.toml` + +- [ ] **Step 1: Add the dependency** + +```bash +cargo add esi@0.7 --package trusted-server-adapter-fastly +``` + +It belongs in the **Fastly adapter**, never in `trusted-server-core` — the crate is +hard-bound to `fastly::{Request, Response, Backend}` and core must stay portable. + +- [ ] **Step 2: Check it compiles for the real target** + +```bash +cargo check-fastly +``` + +Expected: clean. The crate declares edition 2021 with no `rust-version`, and pulls recent +`rand` and `nom`, so this is a genuine question on Rust 1.95.0 / `wasm32-wasip1`. + +- [ ] **Step 3: Check no shared dependency was forced to move** + +```bash +git diff --stat Cargo.lock +cargo check --manifest-path crates/trusted-server-integration-tests/Cargo.toml --tests \ + --target "$(rustc -vV | sed -n 's/^host: //p')" +``` + +**Correction, verified 2026-08-10:** an earlier revision of this step warned about a +desync between the root `Cargo.lock` and `crates/trusted-server-integration-tests/Cargo.lock`. +**That second lockfile does not exist** — the crate is a workspace member (root +`Cargo.toml:10`) and shares the root lockfile. The hazard cannot arise in that form. + +What does matter is whether adding `esi` forces an **existing** shared dependency to a new +version, since `regex`, `bytes`, and `log` are used across the workspace. Adding a new +major that coexists is harmless; moving an existing one is not. If one moves, fix with a +targeted `cargo update -p --precise ` — **never a full update**. + +**Already run and recorded** in [the findings](./2026-08-08-1009-measurement-findings.md): +no existing shared dependency moved. + +- [ ] **Step 4: Record and commit, or stop** + +**Task 1 is complete — verdict PASS, recorded 2026-08-10.** `esi` 0.7.1 compiles clean on +Rust 1.95.0 / `wasm32-wasip1`, all six clippy targets pass, and no existing shared +dependency moved. See [the findings](./2026-08-08-1009-measurement-findings.md). + +Had Step 2 failed, this plan would have stopped here with #1009 answered "not on this +toolchain." It did not. + +```bash +git add crates/trusted-server-adapter-fastly/Cargo.toml Cargo.lock +git commit -m "Add the esi crate to the Fastly adapter for the #1009 validation spike" +``` + +--- + +## Task 2: Local validation first, real service only for what needs it + +**Verified 2026-08-10 under Viceroy 0.17: the entire Core Cache surface this spike uses +works locally.** A probe exercised `cache::core::insert`, `lookup`, `finish`, `to_stream`, +and — the shape Task 3 Step 4 actually specifies — `Transaction::lookup`, +`must_insert_or_update`, `insert(...).surrogate_keys(...).execute_and_stream_back()`, and +hit-after-insert semantics. All passed. Recorded in +[the findings](./2026-08-08-1009-measurement-findings.md). + +That reorders this plan. An earlier revision made provisioning a Fastly service Task 2 and +a blocker on everything after it. It is not a blocker: **almost all of the correctness and +safety work is local**, and only the numbers and the cache topology need real +infrastructure. + +| Work | Where | +| ------------------------------------------------------------ | ------------ | +| C2 insert / lookup / transaction logic (Task 3) | **Local** | +| The `lol_html` transform and template byte-identity (Task 3) | **Local** | +| ESI assembly — the crate is pure Rust over `BufRead`/`Write` | **Local** | +| DCA off, dispatcher allowlist, injection refusal (Task 5) | **Local** | +| Fragment-failure degradation (Task 5) | **Local** | +| Header-finalization ordering, no-C3 assertions (Task 6) | **Local** | +| Cross-user leakage / request-neutrality gates (Task 6) | **Local** | +| Shielding behaviour | Real service | +| POP-level cache tiering (`x-cache`, `hit-state`, `age`) | Real service | +| Request collapsing under genuine concurrency | Real service | +| Stale revalidation timing at the edge | Real service | +| **Every performance number in Task 7's decision rule** | Real service | + +**So: build and prove correctness locally through Tasks 3, 5, and 6 before provisioning +anything.** If the design is wrong or leaks, that surfaces locally for free, and the +service is only needed once there is something worth measuring. + +Two caveats on the local scope. Viceroy is a single instance, so a passing `Transaction` +test proves the API works, **not** that collapsing behaves correctly under load. And local +timings are meaningless for the decision — do not let a fast local run substitute for +Task 7 evidence. + +### When the real service is needed + +- [ ] **Step 1: Provision it — after local correctness passes, not before** + +Separate from production. Confirm and record: whether the publisher backend is +**shielded**, and whether any Delivery service fronts the Compute service. Both change +what the numbers mean. + +```bash +fastly service list +fastly backend list --service-id --version latest +``` + +The shielding answer also settles an open question from the Stage 0 findings: #1009's +off-TS win came from a shield HIT, so whether the test service has one determines whether +its numbers transfer to production at all. + +- [ ] **Step 2: Extend the harness for lineage, not just correlation** + +The existing tester-cookie A/B has no way to join server timings to browser timings. A +root-only request ID is not enough either: under A3 the auction happens in a **fragment +subrequest**, so a root ID never reaches the auction telemetry. + +Propagate a **lineage ID plus the experiment arm** through the whole chain: + +``` +root request → C2 lookup → fragment subrequest → auction telemetry → browser render event +``` + +Generated at TS entry, forwarded into the fragment request, attached to the +`auction_events_raw` row, echoed as `x-ts-request-id`, and exposed to the browser harness +so render events carry it. Every timing log line includes both fields. + +Without this the experiment cannot join hold time, origin time, auction telemetry, browser +TTFB, and render outcome for the same pageview. **That is the difference between an +experiment and a pile of numbers.** + +- [ ] **Step 3: Capture C1 and C2 status separately** + +`x-cache`, `hit-state`, and `age` describe the **HTTP read-through cache (C1)**. They say +nothing about the **transformed-template cache (C2)**, which is a `cache::core` object +with no HTTP semantics. Recording only the former and calling it "cache status" would +attribute C2 hits and misses to the wrong tier. + +Emit both: the C1 headers as-is, plus an explicit `x-ts-c2` field carrying HIT / MISS / +STALE / BYPASS from the transaction outcome. Record the serving POP alongside. A median +that mixes cold-MISS and warm-HIT requests is meaningless, and arms cannot be compared +unless the mix is known — per tier. + +- [ ] **Step 4: Build a request-scoped arm allocator** + +`AssemblyMode` as specified in Task 3 is a **global** setting, but the sample plan below +requires randomized, non-sequential allocation. A global flip gives sequential blocks +instead, which confounds arm with time of day, cache warmth, and traffic mix. + +Allocate per request: hash the lineage ID into buckets, or key off the tester cookie. +The global setting stays as the kill switch and as the way to force a single arm; the +allocator is what the experiment actually uses. Record the assigned arm on every log line +and every telemetry row. + +- [ ] **Step 5: Define the sample plan before collecting anything** + +Write all of this into the findings document **before** the first measurement, and treat +it as fixed: + +| Element | What to state | +| -------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| Allocation | Requests per arm per route, and how arms are assigned | +| Randomization | Randomized or blocked by route and cache state — not sequential runs | +| Pilot variance | A small pilot to estimate variance, before sizing the real run | +| MDE and power | The smallest difference worth detecting, and the N that detects it | +| CI method | Which interval, computed how | +| Warmup and carryover | How cold MISS is forced, how warm HIT is confirmed, and how one arm's cache state is prevented from contaminating the next | + +Rationale: this whole effort exists because #1009 drew a causal conclusion from N=4 that +did not survive contact with the code. Repeating that with more arms and no power +calculation would be worse, not better — it would look rigorous while being equally +unfalsifiable. + +--- + +## Task 3: Build C2 — the shared transformed-template cache + +The core of the spike. Behind a flag, default off. + +**Files:** + +- `crates/trusted-server-core/src/publisher.rs` — emit **one** unconditional marker at the body-close seam (see Step 2; the head seam is not a template hole) +- `crates/trusted-server-core/src/settings.rs` — the mode flag +- `crates/trusted-server-adapter-fastly/src/` — the `cache::core` read/write + +- [ ] **Step 1: Add the assembly-mode setting** + +```rust +/// How per-user ad state reaches the page. +/// +/// `Inline` is today's behaviour: bids injected before ``, root uncacheable. +/// `ClientFill` and `Esi` both serve a shared template from the transformed-template +/// cache and fill the holes afterwards. Spike-only — remove with the spike. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Deserialize, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AssemblyMode { + #[default] + Inline, + ClientFill, + Esi, +} +``` + +Default `Inline` so the flag is a no-op until set. Note the hazards the Stage 0 plan +already documents: `Settings` carries `#[serde(deny_unknown_fields)]`, `ts config push` is +typed, and `Publisher` has a hand-written `Default` plus eight exhaustive test literals +and a live doctest. + +- [ ] **Step 2: Make the template strictly request-neutral** + +**The obvious design is wrong and would leak.** An earlier draft kept `tsjs.adSlots` in +the shared template on the grounds that it is per-URL. Its _content_ is per-URL; its +_presence_ is not. It is gated on `should_run_ad_stack` (`publisher.rs:2920-2927`), which +is `is_get && is_navigation && !is_prefetch && !is_bot && has_matched_slots && +consent_allows_auction && auction_enabled`. + +So the first request to fill C2 would freeze **its own** consent decision, bot +classification, prefetch status, and kill-switch state into an object every later visitor +reads. A consent-denied first fill serves a no-ads template to consenting users; a +consenting first fill serves ad markup to a user who refused. + +**Rule: the template contains an unconditional inert placeholder and nothing else.** + +| Element | Where it lives | +| ------------------------- | -------------------------------------------------- | +| tsjs bundle script tag | Template — content-hashed, genuinely per-URL | +| URL rewrites | Template — per-host, in the cache key | +| `tsjs.adSlots` | **Fragment** — its presence is request-dependent | +| `tsjs.bids` | **Fragment** | +| GPT diagnostics bootstrap | **Fragment** — gated on a per-request cookie/query | + +Emit **one** unconditional marker at the body-close seam, identical on every request that +reaches the transform. Under `Esi` it is an executable ESI include tag; under +`ClientFill` it is nothing at all, with the client fetching unprompted. + +- [ ] **Step 3: Bypass C2 for anything that must not be shared** + +`cache::core` is not an HTTP cache — it will happily store whatever you hand it. Nothing +rejects private or authenticated responses for you. Refuse to insert when **any** holds: + +- The origin response carries `Set-Cookie`. +- The origin response is `private`, `no-store`, or `no-cache`. +- The request carried `Authorization`. +- The response is not 200 with an HTML content type. +- DataDome's request filter replaced the document. + +Audit every request-dependent rewrite before declaring the template neutral — the +integration head-inserts and the GPT-diagnostics bootstrap are both request-scoped and +must not reach C2. + +**Assert it, do not assume it.** A unit test over the transform output must fail on any +of: a bid value, an EC ID, a consent string, a geo value, a diagnostics bootstrap, or a +`Set-Cookie`. Then a second test must assert the template is **byte-identical** for two +requests differing in consent, bot classification, and prefetch status. That second test +is the one that catches this class of bug; the first would have passed on the broken +design. + +- [ ] **Step 4: Write and read C2 — with the real API** + +The builder is move-based and the insert and read handles are different objects. Naïve +code does not compile: + +```rust +// WRONG — surrogate_keys consumes the builder and returns it; this discards the +// return value and then uses a moved binding. And execute() gives a WRITE stream, +// so there is nothing to read back from it. +let mut insert = cache::core::insert(key, ttl); +insert.surrogate_keys(["ts-template"]); +let body = insert.execute()?; +``` + +Correct shape, using a transaction so a cold cache under load transforms once: + +```rust +use fastly::cache::core::{Transaction, CacheKey}; + +let tx = Transaction::lookup(CacheKey::from(key_bytes)).execute()?; + +// Order matters: a STALE entry sets BOTH found() and must_insert_or_update(). +// Testing found() first would serve the stale bytes and silently never fulfil the +// update obligation, leaving every concurrent waiter blocked until timeout. +let template: Body = if tx.must_insert_or_update() { + // Fetch and prepare BEFORE consuming `tx`. After `insert()` the transaction is + // gone and `cancel_insert_or_update()` is unreachable, so anything that can fail + // and does not need the writer belongs here. + let origin = match fetch_and_prepare_origin() { + Ok(origin) => origin, + Err(e) => { + tx.cancel_insert_or_update()?; // releases the obligation to a waiter + return fallback_uncached(e); + } + }; + + // `Transaction::insert(self)` consumes `tx` from this line on. + let (mut writer, found) = tx + .insert(template_ttl) + .surrogate_keys(["ts-template", &url_surrogate_key]) // chained, not discarded + .user_metadata(metadata_envelope) + .execute_and_stream_back()?; + + match stream_lol_html_output(origin, &mut writer) { + Ok(()) => { + writer.finish()?; // REQUIRED, and consumes `writer` + found.to_stream()? // fallible; there is no `to_body()` + } + Err(e) => { + // Also consumes `writer`, marking an unsuccessful end so no partial + // template is served. (A `StreamingBody` dropped without `finish()` is + // aborted anyway, but say it explicitly.) + writer.abandon()?; + return fallback_uncached(e); + } + } +} else if let Some(found) = tx.found() { + found.to_stream()? // C2 HIT — skip origin fetch and transform +} else { + unreachable!("a transaction is either obliged to insert or has found an item") +}; +``` + +Two ownership rules this shape exists to respect, both of which an earlier draft broke: +`Transaction::insert(self)` **consumes** the transaction, so a helper taking `&tx` cannot +call it and `cancel_insert_or_update` is unreachable afterwards; and `finish`/`abandon` +each consume the writer, so neither can be referenced from an arm that did not bind it. + +**Decide the stale policy explicitly.** `Found::is_stale()` and `is_usable()` exist, and +`stale_while_revalidate` can be set at insert. Serving stale while revalidating is a real +option — but it is a state machine, and `cache::core` implements none of it for you. The +spike should start by treating stale as a miss and only add stale-serve if the numbers +justify it. + +**`cache::core` carries no HTTP semantics.** Status, headers, content encoding, and +revalidation are all yours. Serialize what you need into `user_metadata` — at minimum the +content encoding, the transform schema version, and the origin `Vary` values the key was +built from — and decide explicitly whether the stored template is compressed. + +**Cache key must include**, beyond the origin's declared `Vary` (`rsc`, +`next-router-state-tree`, `next-router-prefetch`, `next-router-segment-prefetch`, +`Accept-Encoding` — measured, see the Stage 0 findings): + +- The full URL, explicitly. Do not rely on an ambient request key. +- **The assembly mode.** A2 and A3 emit different template bytes and would otherwise + poison each other's entries. +- **A template schema version**, bumped whenever the transform changes, so a deploy does + not read yesterday's shape. +- Request host and scheme, the enabled-integration set, and the tsjs content hash. + +Per-user signals must never appear in the key. If a signal cannot be excluded from the +template, it does not belong in C2 at all. + +### Design decided 2026-08-10: `cache::core`. Do not revisit read-through. + +An earlier revision left this open between `cache::core` and read-through caching with +`after_send` + `set_body_transform`. Investigated and verified against the pinned SDK and +Viceroy 0.17 source. **Read-through is not viable here** — not on preference, on three +hard blockers: + +1. **Viceroy stubs the entire HTTP Cache ABI**, and the SDK converts that into a _send + error_ rather than a fallback. `is_request_cacheable` returns + `Err(NotAvailable("HTTP Cache API primitives"))` + (`viceroy-lib-0.17.0/src/wiggle_abi/http_cache.rs:108-114`; 26 such stubs in that + file), which makes `must_use_host_caching()` true, which with a send hook set returns + `Err(SendErrorCause::HttpCacheApiUnsupported)` + (`fastly-0.12.1/src/http/request.rs:626-632`). **Setting `after_send` makes every + publisher origin fetch fail** under `fastly compute serve`, `cargo test-fastly`, and + the parity suite. The whole local loop dies. +2. **`with_cache_bypass` makes the hook silently dead.** `get_caching_mode` checks + `cache_override.is_pass()` **first** (`request.rs:612-615`) and returns host caching, so + `after_send` is never invoked and no error is raised. On exactly the requests in scope, + today, the hook would do nothing quietly. +3. **The closure bounds are incompatible with this codebase.** `with_after_send` requires + `Fn + Send + Sync + 'static` (`request.rs:545-550`). Everything the rewriter needs is + `!Send` by construction — `edgezero_core::body::Body` wraps a `LocalBoxStream` + deliberately, which is why the platform layer is `#[async_trait(?Send)]` throughout. + And `set_body_transform` is synchronous, so it could never await the auction collect. + +Read-through's appeal was real — `CandidateResponse::apply_and_stream_back` is +`execute_and_stream_back` with HTTP semantics attached, and TTL/SWR/vary/surrogate keys +derived from origin headers for free. It is simply unreachable from here. + +**Also settled: core cannot reach it at all.** `PlatformHttpRequest` +(`platform/http.rs:16-37`) is a plain data struct with no callback slot, and carrying one +would name `fastly::http::CandidateResponse` in portable core, breaking the other three +adapters. + +### Follow the existing null-object pattern + +`cache::core` fits the shape the repo already uses four times for a Fastly-only capability +behind a portable trait: `UnavailableHttpClient` (`platform/http.rs:216-243`), +`UnavailableKvStore` (`platform/kv.rs:14-17`), and the `RuntimeServices.kv_store` +field/accessor/builder (`platform/types.rs:170,222,269,330`). Add +`PlatformTemplateCache` the same way, and follow +`crates/trusted-server-adapter-fastly/src/ec_kv.rs` — 140 lines, the repo's only real +edge-storage read/write — rather than inventing a shape. + +**Return `EdgeBody`, not `Vec`.** `EdgeBody::Stream` exists, +`fastly_body_to_edge_stream` (`adapter-fastly/src/platform.rs:503`) already converts, and +`PublisherResponse::Buffered` tolerates a live stream (`publisher.rs:1019-1022`). + +### Exact insertion point + +**Immediately before `let mut platform_request = PlatformHttpRequest::new(...)`** — the +last line before `req` is consumed, and a few lines before the origin send. Everything +needed is in scope there: `settings`, `services`, the final URI and Host, `backend_name`, +`request_path`, `matched_slots`, `should_run_ad_stack`, `request_had_authorization`, +`request_host`, `request_scheme`. + +**One required move:** `assembly_mode` is currently computed _after_ the send, for the +logging call site. It depends only on `settings`, so hoist it above the insertion point. + +**Tee-ing is not needed.** With any post-processor registered — and the Next.js +integration always registers one — `HtmlWithPostProcessing` emits nothing until the final +chunk and then returns the whole transformed document as one contiguous buffer +(`html_processor.rs:92-97,148`). Two `write_all` calls on the same slice; no tee +abstraction, no extra copy. Still use `execute_and_stream_back`, but for transaction +correctness and request collapsing rather than for memory. On a hit the processor is never +built at all. + +- [ ] **Step 4b: close the risks the design investigation surfaced** + +Four, all specific to this codebase rather than to `cache::core` in general. + +**`Vary` is in the key list but nothing consumes it.** `c2_bypass_reason` checks +`Set-Cookie`, `Cache-Control`, `Authorization`, status and content type — **not `Vary`**. +Viceroy supports `WriteOptions.vary_rule`, so the mechanism exists; the gate has to use +it. Until then the key is missing a signal the origin explicitly declares, and Step A's +verdict is a `PROVISIONAL PASS`, not a release gate. + +**Resolved — `VarySpec`, commit `b688d667`.** Building the key exposed a problem this +plan states but does not solve: the key must cover everything the origin varies on, but +**a lookup happens before the fetch**, so on a cold key the origin's `Vary` is not yet +known. Three ways out — configure the list; two-phase lookup against a URL-keyed record +holding the last-seen `Vary`; or store the list alongside and re-key on mismatch. The +latter two are correct and double the lookups on every request. + +Configured is taken, **as a spike-grade choice rather than a production one**: Step A +already measured the origin's actual `Vary`, and a 60s TTL bounds drift to a minute +rather than indefinitely. + +The drift is guarded rather than merely accepted. `VarySpec::uncovered_by` runs _after_ +the origin responds, when its `Vary` is finally known, and names which headers the +configured spec missed. A template built under a key that did not cover something the +origin varies on **must not be stored** — a request differing only in that header would +read it. Naming the specific headers makes a stale config identifiable instead of +producing a generic refusal. + +Two decisions worth their tests. An absent header and a present-but-empty one key the +same, because the origin sees no difference between them. And `Vary: *` is not reported +as a named gap — it means uncacheable, which the eligibility gate handles, and reporting +it would produce a nonsense instruction to configure a header called `*`. + +Still open: wiring `uncovered_by` into `c2_bypass_reason` as a bypass reason, which +happens with the store call site. + +**Store bytes plus a metadata envelope; rebuild every header on a hit.** The publisher +path forces `private, no-store` and strips `ETag`/`Last-Modified`/CDN headers _after_ the +send. Replaying stored origin headers would fight that. Store only the transformed body +and a small `user_metadata` envelope — content encoding, content type, schema version, +tsjs hash — and construct every response header from scratch on a hit. Then no origin +header is ever replayed and the `Set-Cookie` privacy net is trivially safe. +`get_user_metadata` is implemented in Viceroy. + +**Content-Encoding belongs in the key.** The streaming pipeline pairs input encoding to +the same output encoding, so the transformed bytes inherit whatever the origin negotiated +from the client's `Accept-Encoding` — still gzip, deflate, br or identity after +`restrict_accept_encoding` narrows it. Either key on the negotiated encoding or normalize +to identity in the cache and re-encode on read. Getting this wrong serves brotli bytes to +a client that asked for gzip. + +**Host and scheme belong in the key.** The post-processed output is host-dependent by +construction: `request_host` and `request_scheme` reach `IntegrationHtmlContext`. + +- [ ] **Step 4c: file the wasted-dispatch follow-up** + +The auction is dispatched _before_ the insertion point. Under `Esi` and `ClientFill` the +root injects nothing, so that dispatch is already pure waste on this branch — and on a C2 +hit it is waste that must be cleaned up via `emit_abandoned_auction` or it leaks +telemetry. + +Keeping the lookup at the insertion point above is right for the spike: minimal diff, and +lookup latency overlaps the in-flight auction. Moving it earlier would eliminate the +wasted dispatch but serialize the lookup ahead of dispatch. **File it; do not fix it +here.** Suppressing root-level dispatch under the shared modes is Task 4's job, where it +also has to be reconciled with the exactly-one-auction gate. + +- [ ] **Step 5: Unit tests, then the target suite** + +```bash +cargo test -p trusted-server-core --target aarch64-apple-darwin assembly_mode +cargo test-fastly && cargo test-axum && cargo test-cloudflare && cargo test-spin +cargo fmt --all -- --check && cargo clippy-fastly +``` + +`ClientFill` must work on all four adapters. `Esi` is Fastly-only and must not break the +others' compilation. + +- [x] **Step 6: the call site — DONE.** `2db10639` (store), `2a2e6c6a` (lookup). + +The cache now engages end to end: a second request for the same URL is served without +touching the origin, and is byte-identical to what was stored. Verified by mutation — +disabling the lookup fails the hit test, so the hit is the cache answering rather than +the fixture answering twice. + +**Wiring the lookup corrected the key.** It carried the content encoding the _origin_ +chose, which does not exist at lookup time. That meant storing under `br` and looking up +under `gzip, br` — a cache that never hits. The field is now the `Accept-Encoding` sent +to the origin. Sound because negotiation is a function of what the origin was offered, +so identical offers yield identical choices; the chosen encoding stays in the metadata +and is what the served response declares. + +That made every key field request-derived, so **the key is built before the fetch** and +the response gate only authorizes storing it. A key that needed the response could only +ever authorize a store, never satisfy a read. + +**The lookup re-checks the request-derived disqualifications, and only those.** The +store gate is response-derived and cannot re-run, but need not: anything in the cache +passed it on the way in. What must re-run are properties of the _reader_ rather than of +the bytes — an authenticated request must not be served a shared template even when that +template is perfectly cacheable. + +**Shared modes take the buffered finalizer.** Storing needs every transformed byte and +streaming does not collect them. The branch keys on the store authorization rather than +on the assembly mode, so `Inline` never reaches it and the spike cannot regress the +shipped path by construction. A C2 _miss_ therefore buffers — the right trade, since a +miss is already paying an origin fetch and a full transform, and what the spike measures +is the hit, where there is no origin fetch to stream from at all. + +Every response header on a hit is constructed, never replayed, so no origin header can +reach a second visitor through the cache. + +The publisher tests use an in-memory cache double, so they prove the wiring rather than +the backing. The join they leave untested is the one `app.rs` makes: the publisher +reaches the cache as a `dyn PlatformTemplateCache` behind `RuntimeServices`, never as +the concrete type the Fastly tests exercise. That join is now executed under Viceroy +against the real Core Cache rather than only type-checked. + +**What this does not establish.** `ClientFill` and `Esi` still render a template with a +hole and nothing filling it. Task 4 and Task 5 remain the blockers on anything +deployable — a cache that works is necessary, not sufficient. + +--- + +## Task 4: Arm A2 — client-fill + +Mostly already specified. See +[the spec's Appendix B](../specs/2026-08-08-esi-cacheable-root-validation-design.md#appendix-b--stage-1-plumbing-condensed) +for the client plumbing, the two-condition join gate, and the server contract; and +[§5](../specs/2026-08-08-esi-cacheable-root-validation-design.md#5-the-trap-in-the-deferred-work--read-this-before-scheduling-stages-12) +for the silent-empty-bids trap, which applies in full. + +- [ ] **Step 1: Hoist the closure-trapped client state** — `pageBidsEndpoint`, + `requestPageBids`, and the `inflight`/`currentPath`/`lastAppliedPath` state, per + Appendix B. Do **not** route the initial load through `onNavigate`. +- [ ] **Step 2: Make `installScheduleInitialAdInit` a hydration-ready AND bids-settled + join**, with a bounded timeout that fires `adInit` untargeted rather than stranding + the slot. Derive the timeout from measured fetch latency, not a constant. +- [ ] **Step 3: Suppress the navigation-path dispatch** so exactly one auction runs per + pageview. Add a new `AuctionSource` for initial loads **plus the mechanism that + delivers it** — a header behind the same-origin gate, not a query parameter. +- [ ] **Step 4: Relocate terminal telemetry.** Navigation `Completed` is emitted only from + the collect functions; the `ts-debug` dump rides the same string. Both move. +- [ ] **Step 5: Verify exactly one auction per pageview** in `auction_events_raw`. Two is + a doubling of SSP spend and an immediate fail. + +--- + +## Task 5: Arm A3 — ESI at the edge + +- [x] **Step 0: the mechanism works — DONE.** `9539061e`, hardened in `0597f54e`. + +Verified under Viceroy with the real `esi` 0.7 crate rather than argued from docs: a +template carrying the `` seam's own ESI include tag comes back with the fragment +spliced in its place and no unresolved tag left. + +**The async/sync obstacle is dissolved, not worked around.** `esi`'s fragment dispatcher +is synchronous and this codebase's fragment producer is `async`; calling one from the +other means a nested executor, which panics. +`PendingFragmentContent::CompletedRequest` lets the dispatcher hand back an +already-built response, so the caller resolves the fragment in the normal async flow and +the dispatcher performs **no I/O at all** — no subrequest, no backend, no self-call, +nothing for Viceroy to stub. That also removes the need for a self-referencing backend +this plan would otherwise have required. + +**Step 2's instruction was right, and reading the crate showed why.** +`CacheConfig::is_includes_cacheable` defaults to **`true`**. A fragment carries one +visitor's bids, so the default caches per-user data and serves it to the next visitor — +silently, on a hit. `includes_force_ttl` is worse where set: it caches everything, +ignoring `private`, `no-store` and `Set-Cookie` alike. Both now stated explicitly, along +with `default_dca`/`inherit_parent_dca` (fragment bytes are data, never re-parsed as +ESI), `max_include_depth = 1`, and rendered caching / `edge_control` off because the +publisher path owns those headers. + +Nine tests. Four assert the configuration; the rest assert behaviour, including that a +fragment containing its own nested ESI include is spliced as text rather than dispatched, so +auction data cannot drive fragment requests. + +**What remains is the call site**, below. Emitting the include and resolving it are both +proven; connecting them is not done. + +- [ ] **Step 1: Wire `process_stream`, not the wrappers** + +`process_response` and `process_response_streaming` consume `self` _and_ send the response +themselves, which takes ownership away from the finalize / `ec_finalize` / apply-effects +ordering. `process_stream(&mut self, src: impl BufRead, out: &mut impl Write, …)` keeps it. + +Source is the C2 body. Sink is the client response body. + +**The ordering an earlier draft described is impossible.** It said EC cookie, geo, and the +privacy net run _after_ assembly. They cannot: streaming responses on this adapter +**commit headers first and then pipe chunks** +(`adapter-fastly/src/main.rs`, `send_edgezero_response`). Once ESI starts writing, no +header can change. + +The correct invariant: + +> **Finalize every header before a single body byte is written** — EC `Set-Cookie`, geo +> suppression, and an unconditional `Cache-Control: private, no-store` — **then** stream +> the assembly with no further header mutation. + +That means `private, no-store` is set unconditionally up front rather than derived from +what the assembly turns out to contain. Deriving it after the fact is not available, and +assuming it was is how a per-user response ends up shared-cacheable. + +- [ ] **Step 2: Disable DCA explicitly and allowlist the dispatcher** + +```rust +let config = esi::Configuration::default() + .with_escaped(false) + .with_default_dca(esi::DcaMode::None) // call the setter; do not rely on the default + .with_inherit_parent_dca(false); +``` + +Comments are not configuration. An earlier draft said DCA "stays at its default" — on a +pre-1.0 crate whose default could move in a patch release, and where this setting fails +**open**, that is not good enough. Call the setters. + +Also disable **fragment caching** explicitly, or mark the include `no-store="on"`. A +cached auction fragment is a per-user object in a shared cache — the C3 failure mode by +another route. + +The dispatcher must be **exact-path allowlisted**: a fragment URL that is not the bids +endpoint is refused, not fetched. The built-in dispatcher builds a dynamic backend per URL +host and panics on a hostless URL — never use it. + +Rationale in the spec's §2: bid payloads carry partner-controlled creative markup, so a +recursive parse would let an SSP make the edge fetch an arbitrary URL. **Add a unit test +that feeds a partner-controlled ESI include targeting `http://attacker.example/` through +a creative payload and asserts no fetch is attempted.** + +- [ ] **Step 3: The fragment must be a script, not the JSON endpoint** + +**`/_ts/page-bids` cannot be the ESI target.** It returns +`serde_json::json!({"slots":…, "bids":…})` (`publisher.rs:3987`), and ESI splices fragment +bytes in literally — the page would contain raw JSON where an executable script belongs. +Nothing would call `scheduleInitialAdInit`. + +Add a **dedicated fragment endpoint** returning the executable script — the same shape +`build_bids_script` produces today, plus the `adSlots` assignment that moved out of the +template in Task 3 Step 2. Either that, or use the `esi` crate's fragment-response +processor to wrap the JSON; the dedicated endpoint is simpler and easier to assert on. + +Three more things the naïve marker gets wrong: + +- **The same-origin gate will reject it.** `page_bids_request_allowed` + (`publisher.rs:3644`) requires `Sec-Fetch-Site: same-origin` or the `X-TSJS-Page-Bids` + header. An internal ESI subrequest carries neither. Give the fragment endpoint an + internal contract and a fixed backend rather than weakening that gate — it exists to + stop third parties burning SSP quota. +- **Parent context does not propagate.** EC identity, consent state, client IP, geo, User + Agent, and the correlation ID all live on the parent request. Forward an **explicitly + approved allowlist** of them into the fragment request. Forwarding everything is how a + fragment ends up more privileged than the parent. +- **Root dispatch must be suppressed.** The navigation path already dispatches an + auction. If A3 does not suppress it, every pageview runs two — doubling SSP and APS + spend. This applies to **A2 and A3 alike**. + +- [ ] **Step 4: Validate the whole URL, not the path** + +An exact-path allowlist alone permits `https://attacker.example/_ts/page-bids`. Validate +**scheme, authority, method, path, and query** — or better, ignore the marker's URL +entirely and dispatch to a fixed internal backend, treating the ESI include as a signal +rather than an address. + +Add a test that feeds an ESI include targeting +`https://attacker.example/_ts/page-bids` through a creative payload and asserts no +outbound fetch is attempted. + +- [ ] **Step 5: Deterministic synthetic fragment first** + +Before wiring the real auction, point the include at a fixed-content endpoint. This +separates "does the pipeline assemble correctly" from "does the auction behave," and the +two fail very differently. Only once assembly is proven does the fragment become the real +one. + +- [ ] **Step 6: Handle the flush hazard** + +`esi` flushes its output writer after each parse batch. Fastly's `StreamingBody` is a +`BufWriter`, so anything between esi and it must propagate `flush()` or nothing leaves the +Wasm heap. + +- [ ] **Step 7: Fragment failure must degrade, not break** + +Assert that a fragment timeout or non-2xx yields a page with empty bids rather than a 5xx +or a truncated document. Note the crate's non-obvious semantics: `alt` is attempted before +`onerror="continue"`, and `` runs **all** attempts and concatenates every +non-failed output — it is not first-success-wins. + +--- + +## Task 6: Safety gates — run against every arm + +Not a phase. Every one of these is a hard fail, independent of any performance result. + +- [x] **Zero cross-user leakage.** DONE — `76df2469`. Two synthetic users differing in EC + identity, consent jurisdiction and geo store a byte-identical template, each against + a fresh cache so the first cannot answer for the second. Forbidden-substring checks + are the second layer, since byte-identity also holds if both leak the same thing. + Mutation-verified: leaking `adSlots` through the head seam fails it. +- [x] **Cold MISS, warm HIT, stale revalidation** DONE — `76df2469`, and end to end under + `viceroy serve` (below). Stale reads as a miss; serving stale would mean serving a + template built by an older transform or bundle. + + The first stale test passed for the wrong reason and had to be rewritten: a zero TTL + produces an *absent* entry, not a stale one, so `is_stale()` was never reached — + confirmed by reverting the check and watching it stay green. Only a + `stale_while_revalidate` window makes an entry present-and-stale. + +- [x] **Transform failure** DONE — `76df2469`. A partial template in C2 is the worst + outcome available: a truncated document served to every later visitor, indefinitely, + with no error after the first request. Mutation-verified by storing before the cap + check. +- [ ] **Request collapsing** works: concurrent cold requests transform once. +- [x] **DCA disabled** DONE — `0597f54e`. Config asserted _and_ behaviour: a fragment + carrying its own nested ESI include is spliced as text rather than dispatched. + +- [ ] **Request collapsing** — not tested, and not testable here. Viceroy is + single-threaded, so the concurrent cold-request case cannot be produced. The racing + _writer_ path is covered (`a_second_put_on_a_fresh_entry_is_a_no_op`), which is the + correctness half; the collapsing half needs real concurrency. +- [ ] **Exactly one auction per pageview**, from `auction_events_raw`. +- [ ] **Cookie and privacy finalization ran BEFORE assembly**, not after — EC + `Set-Cookie` on first visit, geo suppression, and an unconditional + `Cache-Control: private, no-store`. Headers commit before the body streams on this + adapter, so "finalize after assembly" is not available; asserting it that way is how + a per-user response ends up shared-cacheable. ESI's streaming mode dropping + `$add_header` is a consequence of the same constraint, not a separate hazard. +- [ ] **Slot and bid attribution unchanged.** Same slots matched, same bids applied, same + renders attributed. Use TS-attributed renders — the SSAT line item, non-empty + `ts.bids`, `hb_adid` presence — **never slot fill**, which is blind to empty bids + because `adInit` defines slots regardless. +- [x] **No C3 — assert positively, not by absence.** DONE — `0adb578e`, and this gate's + wording caught a live bug. A C2 hit returns before the point where the publisher path + stamps `private, no-store`, so it served HTML with **no `Cache-Control` at all** — + heuristically cacheable, and therefore a shared cache of an assembled per-user + response. Checking for the _absence_ of `public`/`s-maxage`/`Surrogate-Control` would + have reported it as safe, because there was nothing present to forbid. Covered for + returning visitors specifically, where the cookie-privacy net never fires. + + Original wording, retained because it is what made the difference: Forbidding `public`, `s-maxage`, and + `Surrogate-Control` is **not sufficient**: a bare `Cache-Control: max-age=60` passes + that check and is still shared-cacheable, and that is exactly what the measured + origin sends. Require instead that every assembled response carries + `Cache-Control: private, no-store` and that `Expires`, `ETag`, `Last-Modified`, and + all four CDN cache directives are stripped. Test it for **returning** users + specifically — they set no EC cookie, so the cookie privacy net never fires and is + not a backstop here. + +--- + +## Task 7: The decision record + +**Files:** `docs/superpowers/plans/2026-08-10-1009-esi-decision-record.md` + +- [ ] **Step 1: Record every arm** with N, confidence interval, cache-tier mix, route mix, + and POP. Any arm missing those is not reportable. + +- [ ] **Step 2: Apply the decision rule, stated here before the data exists** + +**Adopt ESI only if all three hold:** + +1. Every Task 6 gate passes on A3. +2. A3 beats A2 on **bids-ready time, `adInit` fire time, and first TS-attributed creative + paint** — by a margin the reviewers ratify **before** collection, not chosen after + seeing the numbers. **Not root TTFB:** A2 and A3 serve the same C2 template, so their + root timings are near-identical by construction and a difference there would be noise. + Root TTFB is a non-regression guard only. +3. Render outcomes on A3 are non-inferior to A0. + +**Otherwise adopt A2 (client-fill)** if its gates pass and it beats A1. It is portable +across all four adapters and carries no Fastly-only maintenance burden. + +**Otherwise keep A1** — Stage 0 alone — and record #1009 as answered in the negative with +evidence. + +The margin in (2) exists because A3's cost is not its diff. It is a second rendering +architecture, Fastly-only, on a pre-1.0 crate, in the critical render path. A small win +does not pay for that. + +- [ ] **Step 3: Record what would change the answer**, so this does not get re-litigated + from scratch. At minimum: React #418 / [#938](https://github.com/IABTechLab/trusted-server/issues/938) + being fixed such that `adInit` can run synchronously, which is what would make edge + assembly's round-trip saving actually worth something. + +- [ ] **Step 4: Clean up.** Remove the spike flag or promote it to a real setting; purge + C2 (`purge_surrogate_key` on `ts-template`); remove the synthetic fragment endpoint; + and either land or delete the `esi` dependency. **A spike flag left in place becomes + permanent configuration surface.** + +--- + +## Reproducibility metadata + +Record with every result, or it cannot be re-run or trusted: commit SHA; `esi` and +`fastly` crate versions; Fastly service and version IDs; whether the backend is shielded; +`template_ttl`; the origin's `Cache-Control` and `Vary` at collection time; assembly mode; +routes; N per arm; and the cache-tier mix. + +## Out of scope + +- **Stages 1–2 of the spec** as production work. This spike may build parts of the + client-fill path to measure it; shipping it is a separate decision behind the + correctness defects. +- **Full RSC/flight partitioning.** `rsc_flight.rs` has no static/dynamic split. +- **Publisher-authored ESI.** Breaks the no-origin-changes promise. +- **A C3 delivery cache.** Not a deferred item — a thing that must not exist. + +## Definition of done + +- [ ] Task 1 verdict recorded: `esi` 0.7 builds on Rust 1.95.0 / `wasm32-wasip1`, or it + does not and the spike stopped. +- [ ] All four arms measured on one build, with correlation IDs joining server and browser + timings, and cache tier recorded per request. +- [ ] Every Task 6 gate has an explicit pass/fail per arm. +- [ ] Decision record exists, applies the pre-ratified rule, and names what would change + the answer. +- [ ] Cleanup complete: flag resolved, C2 purged, synthetic endpoint removed, dependency + landed or dropped. +- [ ] All CI gates pass: `cargo fmt --all -- --check`; the six clippy targets; the four + adapter test suites; the parity suite; JS build, test, and format; docs format. diff --git a/docs/superpowers/plans/2026-08-12-1009-esi-merge-hardening.md b/docs/superpowers/plans/2026-08-12-1009-esi-merge-hardening.md new file mode 100644 index 000000000..51a942765 --- /dev/null +++ b/docs/superpowers/plans/2026-08-12-1009-esi-merge-hardening.md @@ -0,0 +1,294 @@ +# #1009 ESI Merge and Hardening Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use `superpowers:executing-plans` to implement +> this plan task-by-task. This plan is intentionally executed inline because the operator +> explicitly prohibited subagents. + +**Goal:** Merge current `main` and make the opt-in ESI byte-seam/shared-template path correct, +private, cache-semantic, compressed, observable, and operationally reversible. + +**Architecture:** Fastly Core Cache holds identity-encoded reader-neutral templates behind a +transaction acquired before origin work. Every request assembles its own slots and structured bid +map at an exact inert seam, encodes the result for that client, and receives a final immutable +private/no-store policy. + +**Tech Stack:** Rust 1.95, Fastly Compute/Core Cache, `edgezero_core` HTTP types, `lol_html`, +TypeScript/Vitest, Viceroy, shell harness. + +> **Implementation status, 2026-08-12:** Tasks 1–12 are complete on the branch. The Viceroy +> harness passed in both modes after running outside the filesystem sandbox so it could read the +> macOS native-certificate keychain. + +--- + +### Task 1: Merge live main and preserve auction contracts + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-core/src/integrations/gpt_diagnostics.rs` +- Modify: `crates/trusted-server-js/lib/src/core/types.ts` +- Modify: `crates/trusted-server-js/lib/src/integrations/gpt/index.ts` +- Test: adjacent Rust and Vitest modules + +- [x] Merge `origin/main` with `git merge --no-ff origin/main`. +- [x] Resolve the `AdBidsState`/`write_bids_to_state` conflict by building one structured map with + `auction_id`, storing both map and script, and returning its delivered slot IDs. +- [x] Add/adjust tests proving ESI and inline retain `hb_auction_id`, APS renderer metadata, and + delivered-winner attribution. +- [x] Run the focused Rust and GPT tests. +- [x] Complete the merge commit. + +### Task 2: Remove mechanisms outside the approved ESI byte-seam design + +**Files:** + +- Modify: `crates/trusted-server-core/src/creative_opportunities.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Delete: `crates/trusted-server-core/src/platform/template_assembly.rs` +- Modify: `crates/trusted-server-core/src/platform/mod.rs` +- Modify: `crates/trusted-server-core/src/platform/types.rs` +- Delete: `crates/trusted-server-adapter-fastly/src/esi_assembly.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` +- Modify: `crates/trusted-server-adapter-fastly/Cargo.toml` +- Modify: `Cargo.lock` + +- [x] Update mode tests to specify only `inline` and `esi`; watch the old client-fill expectations + fail or stop compiling. +- [x] Remove `ClientFill`, executable fragment serialization, assembler traits/registration, and + the `esi` crate. +- [x] Update comments to call the production path byte-seam assembly. +- [x] Run focused configuration, publisher, and Fastly adapter tests. +- [x] Commit the scope cleanup. + +### Task 3: Canonicalize and bound the template key + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Modify: `crates/trusted-server-core/src/creative_opportunities.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` + +- [x] Write failing tests for absent versus empty `Vary`, repeated raw values, invalid configured + names, punctuation-colliding purge URLs, changed origin host override, and changed creative + configuration. +- [x] Replace string pairs with a typed canonical `Vary` value preserving presence and all bytes. +- [x] Hash a length-prefixed canonical key and hash the URL-specific surrogate key. +- [x] Include publisher origin identity and the complete template-shaping fingerprint. +- [x] Run focused key/configuration tests and commit. + +### Task 4: Enforce request and origin cache semantics + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-core/Cargo.toml` if HTTP-date parsing needs a direct dependency + +- [x] Write failing tests for response `max-age=0`, positive max age, repeated/malformed cache + directives, `Age` exhaustion, expired/malformed `Expires`, missing freshness, invalid `Vary`, + and request no-cache/no-store/range/conditional bypasses. +- [x] Add a typed cache eligibility result carrying the positive remaining TTL. +- [x] Parse relevant response directives fail-closed and cap, never extend, origin freshness. +- [x] Add request-side bypass classification before lookup. +- [x] Make unsupported/backend-failed cache lookups fall back to inline processing on non-Fastly + adapters rather than buffering a cacheless ESI path. +- [x] Run focused eligibility tests and commit. + +### Task 5: Move request collapse before the origin fetch + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Modify: `crates/trusted-server-core/src/platform/types.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/template_cache.rs` + +- [x] Verify the reservation is acquired before origin work and the Fastly transaction contract + blocks same-key waiters. Viceroy is single-threaded, so it cannot directly reproduce two + truly concurrent cold requests. +- [x] Introduce a lookup outcome with an opaque insert reservation and explicit cancellation. +- [x] Implement Fastly `Transaction::lookup` before origin work and consume/cancel its obligation + on every exit path. +- [x] Ensure invalid fresh entries become replaceable rather than causing repeated refetches. +- [x] Run focused Core Cache/Viceroy tests and commit. + +### Task 6: Make privacy and policy-header parity final + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-core/src/response_privacy.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` +- Modify: `crates/trusted-server-core/src/integrations/registry.rs` tests + +- [x] Write failing tests for repeated CSP/CSP-Report-Only, omitted COOP/COEP/CORP/HSTS/Link, + unknown cached header metadata, duplicate required metadata fields, and a late integration + changing `Cache-Control` to public. +- [x] Capture all ordered values, expand the safe allowlist, and decode metadata strictly. +- [x] Replay with `append`, then apply the assembled-response privacy policy last. +- [x] Preserve and reassert private/no-store after request-filter effects in Fastly's final send. +- [x] Run focused header/privacy tests and commit. + +### Task 7: Bypass shared templates for request-private diagnostics + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` + +- [x] Write a failing warm-cache test activated by diagnostics query and another by diagnostics + cookie. +- [x] Make `requires_private_no_store()` a lookup/store disqualifier. +- [x] Verify ordinary diagnostics-disabled requests still hit the template cache. +- [x] Run focused diagnostics/template-cache tests and commit. + +### Task 8: Re-encode assembled responses for the reader + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` + +- [x] Write failing cold/warm tests requiring gzip/br clients to receive a matching encoded body + and proving reader encoding no longer partitions the stored template. +- [x] Keep the origin offer within the reader's supported codings so a response-gate bypass remains + lossless, while decoding every stored template to identity. +- [x] Carry the selected response encoding separately from identity template metadata. +- [x] Encode buffered assembly after splicing and stream hit prefix/seam/suffix through one encoder. +- [x] Handle `identity;q=0` without serving an unacceptable representation. +- [x] Emit the correct `Vary: Accept-Encoding` response semantics after final encoding. +- [x] Run focused compression tests and commit. + +### Task 9: Make marker failures safe + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` + +- [x] Write failing tests for HTML with no explicit ``, a publisher-authored marker + collision, and a corrupt cached marker. +- [x] Record/validate a schema-bound seam location or use a collision-resistant marker contract. +- [x] Cancel storage and fall back safely when the optimization cannot produce one seam. +- [x] Run focused miss/hit assembly tests and commit. + +### Task 10: Add operational observability and harden the harness + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/template_cache.rs` +- Modify: `scripts/template-cache-local-test.sh` +- Modify: `.github/workflows/test.yml` + +- [x] Write failing tests for distinct backend-error versus not-found status and template-cache response-state + reporting. +- [x] Preserve backend errors and emit bounded template-cache status without exposing key material. +- [x] Change the harness to operate on a temporary manifest and fail on missing/non-numeric probe + output or empty response bodies. +- [x] Test both cold and warm integrity and execute the generated scheduler payload contract. +- [x] Add the ESI harness to CI where Viceroy prerequisites are available. +- [x] Run shell syntax/static checks and commit. + +### Task 11: Document configuration, semantics, and rollback + +**Files:** + +- Modify: `trusted-server.example.toml` +- Modify: `docs/guide/configuration.md` +- Modify: `docs/superpowers/archive/2026-08-10-1009-esi-validation-spike.md` +- Modify: `docs/superpowers/specs/2026-08-11-1009-streaming-assembly-architecture.md` +- Modify: relevant #1009 findings documents + +- [x] Document that `esi` means Fastly shared template cache plus byte-seam assembly, not parser execution or final + assembled-response caching. +- [x] Document `template_cache_vary`, cookie independence, freshness, metrics, purge, rollback + ordering, and limitations on non-Fastly adapters. +- [x] Close or supersede stale spike checkboxes and remove claims contradicted by the final code. +- [x] Run docs format/build and commit. + +### Task 12: Full verification + +**Files:** none expected beyond fixes discovered by verification + +- [x] Run `cargo fmt --all -- --check`. +- [x] Run all four adapter test aliases and the parity suite. +- [x] Run all six clippy aliases. +- [x] Build the Fastly release WASM. +- [x] Run JS tests, build, and format under pinned Node 24.12.0. +- [x] Run docs format/build. +- [x] Run `scripts/template-cache-local-test.sh esi` and `inline` if the environment exposes the required + local certificate store; otherwise report the exact environment blocker. +- [x] Run `git diff --check`, inspect the merge graph, and confirm the worktree contains only + intended changes. + +### Task 13: Interpret Fastly Surrogate-Control conservatively + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `docs/guide/configuration.md` +- Modify: `docs/superpowers/specs/2026-08-12-1009-esi-merge-hardening-design.md` + +- [x] Write a failing gate test using `Cache-Control: max-age=60` plus the observed publisher + `Surrogate-Control` policy (`max-age=1200`, `stale-while-revalidate=21600`, and + `stale-if-error=604800`). +- [x] Write failing tests proving the shorter standard/surrogate freshness wins, stale windows do + not extend fresh reuse, restrictive directives are refused, and unknown, duplicate, or + malformed directives fail closed. +- [x] Parse only Fastly's supported `max-age`, `stale-while-revalidate`, and `stale-if-error` + directives; continue refusing every other vendor CDN policy field. +- [x] Keep request `Cache-Control: max-age=0` as an intentional template-cache bypass so reload preserves its + revalidation semantics. +- [x] Run focused tests, `cargo test-fastly`, target-matched formatting/clippy, both local harness + modes, and verify the observed publisher policy progresses from `miss-stored` to `hit` in + the local Fastly runtime on an ordinary navigation. + +### Task 14: Allow browser reloads to reuse a fresh ESI template + +Task 14 supersedes Task 13's conservative request `max-age=0` bypass after end-to-end testing +proved that the template cache reuses only the neutral template and still creates a new private response and +auction. + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `docs/superpowers/specs/2026-08-12-1009-esi-merge-hardening-design.md` + +- [x] Write a failing end-to-end test proving `Cache-Control: max-age=0` reruns the auction but + does not refetch the reader-neutral publisher template. +- [x] Treat only a valid zero request max age as compatible with the template cache; continue bypassing positive + or malformed constraints and every explicit revalidation directive. +- [x] Verify the focused tests, formatting, and Fastly clippy, then commit independently. + +### Task 15: Make the ESI template-cache ceiling configurable + +Task 15 supersedes Task 13's shorter-of-standard-and-surrogate rule. The final behavior follows +Fastly edge precedence while retaining restrictive directives as hard refusals. + +**Files:** + +- Modify: `crates/trusted-server-core/src/creative_opportunities.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/template_cache.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` +- Modify: `trusted-server.example.toml` +- Modify: `docs/guide/configuration.md` +- Modify: `docs/superpowers/specs/2026-08-12-1009-esi-merge-hardening-design.md` + +- [x] Add failing configuration tests for the 60-second default, an explicit 1,200-second ceiling, + zero, values above one day, and omission from serialized rollback-compatible config. +- [x] Add failing freshness tests proving Fastly precedence, age deduction, and the configured + ceiling for the observed `Cache-Control: max-age=60` plus + `Surrogate-Control: max-age=1200` response. +- [x] Implement `template_cache_max_age_seconds` under `[creative_opportunities]` and thread its + resolved duration into template-cache eligibility. +- [x] Remove the Fastly adapter's second hard-coded 60-second cap; the already-authorized + per-entry max age becomes the sole insertion lifetime. +- [x] Update the example and operator guide, without editing the tracked deployment + `fastly.toml`. +- [x] Run focused red/green tests, full adapter tests and clippy gates, documentation checks, and + inspect the final diff with `fastly.toml` excluded. diff --git a/docs/superpowers/plans/2026-08-14-1009-esi-parser-assembly.md b/docs/superpowers/plans/2026-08-14-1009-esi-parser-assembly.md new file mode 100644 index 000000000..ddef73903 --- /dev/null +++ b/docs/superpowers/plans/2026-08-14-1009-esi-parser-assembly.md @@ -0,0 +1,104 @@ +# #1009 ESI Parser Assembly Implementation Plan + +> **Execution note:** Implemented inline in the current checkout, without a worktree or +> subagents, as requested. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Use the repaired ESI parser on authorized cold template-cache misses without changing the existing warm-hit streaming behavior. + +**Architecture:** The template cache retains the inert schema-v4 seam. Core delegates cold assembly through a platform trait; Fastly converts the seam to one synthetic ESI include and resolves it from the already-collected per-reader script. Parser failure falls back to core's validated byte split, while warm hits continue to stream by byte seam. + +**Tech Stack:** Rust 1.95, `wasm32-wasip1`, Fastly Compute/Viceroy, `stackpop/esi` pinned by Git revision, `error-stack`. + +--- + +### Task 1: Restore a platform assembly boundary + +**Files:** + +- Create: `crates/trusted-server-core/src/platform/template_assembly.rs` +- Modify: `crates/trusted-server-core/src/platform/mod.rs` +- Modify: `crates/trusted-server-core/src/platform/types.rs` +- Test: `crates/trusted-server-core/src/platform/template_assembly.rs` + +- [x] Add a failing object-safety/default-behavior test for `PlatformTemplateAssembler`. +- [x] Run the focused core test and confirm it fails because the boundary is absent. +- [x] Add the trait, error type, unavailable default, runtime service field, builder method, + accessor, and test support. +- [x] Run the focused tests and confirm they pass. + +### Task 2: Delegate only cold-miss assembly + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Test: `crates/trusted-server-core/src/publisher.rs` + +- [x] Add a recording assembler to the template-cache end-to-end tests. +- [x] Add a test asserting one platform call on a cold miss and no additional call on the + subsequent warm hit. +- [x] Add a test asserting platform failure returns a complete byte-seam response. +- [x] Add tests for `x-ts-assembly` values on parser, fallback, and warm paths. +- [x] Run each test first and confirm the expected failure. +- [x] Change `assemble_if_shared` to call the platform assembler after storage, fall back + to the validated byte split on error, and return the assembly method. +- [x] Set `x-ts-assembly` without changing `x-ts-template-cache` or privacy headers. +- [x] Re-run the template-cache end-to-end test module. + +### Task 3: Add the repaired Fastly ESI adapter + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/Cargo.toml` +- Modify: `Cargo.lock` +- Create: `crates/trusted-server-adapter-fastly/src/esi_assembly.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/app.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` +- Test: `crates/trusted-server-adapter-fastly/src/esi_assembly.rs` + +- [x] Add failing adapter tests for a large Next.js script followed by the seam, an + unexpected publisher ESI directive, an unexpected dispatcher URL, and verbatim + fragment content. +- [x] Run the focused Fastly test filter and confirm the missing module/implementation + fails. +- [x] Pin `https://github.com/stackpop/esi.git` at + `4c53feab4d22ad9a84641b4c46f3f63bc6d197e2`. +- [x] Implement the explicit no-cache/no-DCA ESI configuration and synthetic completed + fragment dispatcher. +- [x] Register `FastlyTemplateAssembler` in per-request runtime services. +- [x] Run the focused Fastly tests and confirm they pass. + +### Task 4: Preserve cache schema and documentation truth + +**Files:** + +- Modify: `docs/superpowers/specs/2026-08-11-1009-streaming-assembly-architecture.md` +- Modify: `docs/guide/configuration.md` +- Modify: `scripts/template-cache-local-test.sh` +- Test: `crates/trusted-server-core/src/platform/template_cache.rs` + +- [x] Add/adjust tests proving schema version 4 and the inert stored marker remain + unchanged. +- [x] Extend the local harness to require `esi-parser` on the miss and `byte-seam` on the + hit. +- [x] Update architecture and operator documentation to describe the hybrid path and + pinned fork accurately. +- [x] Run formatting and the harness's static checks. + +### Task 5: Full verification and signed commit + +**Files:** + +- Review every modified file. + +- [x] Run `cargo fmt --all -- --check`. +- [x] Run every target-matched Clippy alias from `CLAUDE.md`. +- [x] Run `cargo test-fastly`, `cargo test-axum`, `cargo test-cloudflare`, and + `cargo test-spin`. +- [x] Run the integration parity test. +- [x] Run JS tests/build/format and docs format. +- [x] Run the template-cache local harness when Viceroy and its certificate environment are + available; otherwise report that environmental gap explicitly. +- [x] Run `git diff --check`, inspect staged scope, and confirm no operator configuration + or secrets are staged. +- [x] Create one SSH-signed commit only after every required gate is green. +- [x] Verify the commit signature locally and report the exact commit ID and test counts. diff --git a/docs/superpowers/plans/2026-08-18-admin-diagnostics-review-fixes.md b/docs/superpowers/plans/2026-08-18-admin-diagnostics-review-fixes.md new file mode 100644 index 000000000..b13430a45 --- /dev/null +++ b/docs/superpowers/plans/2026-08-18-admin-diagnostics-review-fixes.md @@ -0,0 +1,576 @@ +# Admin Diagnostics Review Fixes Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Close every PR #928 review finding by making admin diagnostics fail closed, preventing publisher/KV side effects, preserving raw KV JSON, and documenting the API. + +**Architecture:** Core settings owns the canonical admin-template-to-auth-probe mapping and runtime admin namespace classification. Core EC admin code owns one shared fallback-denial response so each adapter only adds a small guard at its publisher fallback boundary. Fastly dispatches the read-only EIDs diagnostic before EC setup, while raw JSON display and typed interpretation remain separate inside the core handler. + +**Tech Stack:** Rust 2024, `http`, `serde_json`, `error-stack`, EdgeZero adapter routers, Fastly/Viceroy, Markdown documentation. + +**Design spec:** `docs/superpowers/specs/2026-08-18-admin-diagnostics-review-fixes-design.md` + +--- + +## File map + +- Modify `crates/trusted-server-core/src/settings.rs`: canonical admin route/auth + probes, admin namespace classification, startup validation, and tests. +- Modify `crates/trusted-server-core/src/auth.rs`: runtime fail-closed behavior + and regression tests. +- Modify `crates/trusted-server-core/src/ec/admin.rs`: shared diagnostic + fallback denial, lossless raw JSON display, and unit tests. +- Modify `crates/trusted-server-adapter-fastly/src/app.rs`: fallback guard, + early read-only EIDs dispatch, and adapter tests. +- Modify each portability adapter's `src/app.rs` and `tests/routes.rs`: fallback + guard and cross-adapter route regressions. +- Modify `docs/guide/api-reference.md`: operator-facing contract. + +No new crate, dependency, schema type, or test-only production seam is needed. + +### Task 1: Make admin authentication coverage parameter-aware and fail closed + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs:2194-2279` +- Modify: `crates/trusted-server-core/src/settings.rs:4876-4970` +- Modify: `crates/trusted-server-core/src/auth.rs:29-55` +- Test: `crates/trusted-server-core/src/auth.rs:79-315` + +- [ ] **Step 1: Add a failing literal-template startup regression** + +Build settings TOML whose first handler covers the four non-parameterized admin +routes and whose second handler covers only literal braces: + +```rust +[[handlers]] +path = "^/_ts/admin/(keys/rotate|keys/deactivate|ec|eids)$" +username = "admin" +password = "strong-test-password" + +[[handlers]] +path = "^/_ts/admin/ec/[{]id[}]$" +username = "admin" +password = "strong-test-password" +``` + +Assert `Settings::from_toml` fails and identifies `/_ts/admin/ec/{id}` as +uncovered. + +- [ ] **Step 2: Run the regression and verify RED** + +```bash +cargo test-fastly from_toml_rejects_literal_parameter_template_auth_coverage +``` + +Expected: FAIL because current validation accepts the literal template match. + +- [ ] **Step 3: Add and run a failing concrete-handler password regression** + +Add settings with a concrete-ID handler regex +`^/_ts/admin/ec/[a-f0-9]{64}[.][a-z0-9]{6}$` and placeholder password +`change-me-admin-password`. Assert finalization rejects it as an admin handler. + +```bash +cargo test-fastly from_toml_rejects_placeholder_password_for_concrete_admin_ec_handler +``` + +Expected: FAIL because password validation also uses the literal template. + +- [ ] **Step 4: Implement one canonical template-to-auth-probe mapping** + +Keep `Settings::ADMIN_ENDPOINTS` as canonical templates. Add a fixed fictional +valid EC probe and a helper used by both coverage and password validation: + +```rust +const ADMIN_EC_ID_AUTH_PROBE: &str = concat!( + "/_ts/admin/ec/", + "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + ".abc123", +); + +fn admin_auth_probe(path: &'static str) -> &'static str { + match path { + "/_ts/admin/ec/{id}" => ADMIN_EC_ID_AUTH_PROBE, + path => path, + } +} +``` + +Make `uncovered_admin_endpoints` report the template but match its probe. Make +`validate_admin_handler_passwords` use the same helper. Update stale comments. + +- [ ] **Step 5: Run both settings regressions and verify GREEN** + +```bash +cargo test-fastly literal_parameter_template_auth_coverage +cargo test-fastly placeholder_password_for_concrete_admin_ec_handler +``` + +Expected: PASS. + +- [ ] **Step 6: Add failing runtime fail-closed auth tests** + +In `auth.rs`, deserialize settings directly with `toml::from_str` to bypass +startup finalization. With the literal-template-only configuration, send a +concrete valid EC request and assert `enforce_basic_auth` returns a configuration +error rather than `Ok(None)`. Add `/_ts/administrator` as a boundary case that +must remain public when no handler matches. + +- [ ] **Step 7: Run runtime tests and verify RED** + +```bash +cargo test-fastly concrete_admin_path_without_matching_handler_fails_closed +``` + +Expected: FAIL because current auth returns `Ok(None)`. + +- [ ] **Step 8: Implement the runtime namespace invariant** + +Add: + +```rust +#[must_use] +pub fn is_admin_path(path: &str) -> bool { + path == "/_ts/admin" || path.starts_with("/_ts/admin/") +} +``` + +When `handler_for_path` returns `None`, make `enforce_basic_auth` return +`TrustedServerError::Configuration` for an admin path and retain `Ok(None)` for +all other paths. + +- [ ] **Step 9: Run core tests and target-matched suite** + +```bash +cargo test-fastly admin_path +cargo test-fastly uncovered_admin_endpoints +cargo test-fastly +``` + +Expected: PASS without warnings. + +- [ ] **Step 10: Commit Task 1** + +```bash +git add crates/trusted-server-core/src/settings.rs crates/trusted-server-core/src/auth.rs +git commit -m "Fail closed for concrete admin routes" +``` + +### Task 2: Add a shared local denial for diagnostic fallback requests + +**Files:** + +- Modify: `crates/trusted-server-core/src/ec/admin.rs:20-45` +- Modify: `crates/trusted-server-core/src/ec/admin.rs:432-464` +- Test: `crates/trusted-server-core/src/ec/admin.rs:464-925` + +- [ ] **Step 1: Add failing table-driven fallback tests** + +Test a new +`deny_admin_diagnostic_fallback(&Request) -> Option>` +helper. For bare EC, single-ID EC, and EIDs, every non-GET publisher fallback +method must return `405`, `Allow: GET`, and `Cache-Control: no-store`. GET and +non-GET requests to trailing/extra-segment EC/EIDs forms must return `404` and +`no-store`. An unrelated publisher path must return `None`. + +- [ ] **Step 2: Run focused tests and verify RED** + +```bash +cargo test-fastly admin_diagnostic_fallback +``` + +Expected: compilation FAIL because the helper does not exist. + +- [ ] **Step 3: Implement classification and response construction** + +Add a private path-shape classifier and documented public helper. Its core +response logic is: + +```rust +let mut response = if shape.is_valid_resource() && req.method() != Method::GET { + json_error(StatusCode::METHOD_NOT_ALLOWED, "method not allowed") +} else { + json_error(StatusCode::NOT_FOUND, "admin diagnostic route not found") +}; +if response.status() == StatusCode::METHOD_NOT_ALLOWED { + response + .headers_mut() + .insert(header::ALLOW, HeaderValue::from_static("GET")); +} +``` + +Reuse `json_error`/`json_response`, which already set JSON and `no-store`. +Classify only the EC/EIDs families. EC bare and exactly one non-empty ID segment +are valid resource shapes; EIDs exact is valid; suffix/trailing forms are +malformed. A valid GET that somehow reaches fallback returns local `404`. + +- [ ] **Step 4: Run focused and target-matched tests** + +```bash +cargo test-fastly admin_diagnostic_fallback +cargo test-fastly +``` + +Expected: PASS. + +- [ ] **Step 5: Commit Task 2** + +```bash +git add crates/trusted-server-core/src/ec/admin.rs +git commit -m "Deny admin diagnostics in publisher fallback" +``` + +### Task 3: Wire the denial guard into every adapter + +**Files:** + +- Modify/Test: `crates/trusted-server-adapter-fastly/src/app.rs` +- Modify: `crates/trusted-server-adapter-axum/src/app.rs` +- Test: `crates/trusted-server-adapter-axum/tests/routes.rs` +- Modify: `crates/trusted-server-adapter-cloudflare/src/app.rs` +- Test: `crates/trusted-server-adapter-cloudflare/tests/routes.rs` +- Modify: `crates/trusted-server-adapter-spin/src/app.rs` +- Test: `crates/trusted-server-adapter-spin/tests/routes.rs` + +- [ ] **Step 1: Add Fastly adapter regressions and verify RED** + +Using authenticated router requests, test the valid diagnostic shapes against +`POST`, `HEAD`, `OPTIONS`, `PUT`, `PATCH`, and `DELETE`, asserting `405`, +`Allow: GET`, and `no-store`. Test authenticated GET and POST requests for +trailing/extra-segment forms, asserting `404` and `no-store`. + +```bash +cargo test-fastly authenticated_admin_diagnostic_fallback +``` + +Expected: FAIL because requests enter publisher fallback. + +- [ ] **Step 2: Wire Fastly and verify GREEN** + +Import the helper and call it at the start of `dispatch_fallback`, before GPT +preparation, filters, integration routing, or publisher handling: + +```rust +if let Some(response) = deny_admin_diagnostic_fallback(&req) { + return response; +} +``` + +```bash +cargo test-fastly authenticated_admin_diagnostic_fallback +cargo test-fastly +``` + +Expected: PASS. + +- [ ] **Step 3: Add Axum regressions and verify RED** + +Add the same matrices in `tests/routes.rs` using `make_service()`. + +```bash +cargo test-axum authenticated_admin_diagnostic_fallback +``` + +Expected: FAIL. + +- [ ] **Step 4: Wire Axum and verify GREEN** + +Call the helper at the start of Axum's fallback `dispatch` before publisher +handling. + +```bash +cargo test-axum authenticated_admin_diagnostic_fallback +cargo test-axum +``` + +Expected: PASS. + +- [ ] **Step 5: Add Cloudflare regressions and verify RED** + +Use `request_builder()` plus `route(test_router(), req)`. + +```bash +cargo test-cloudflare authenticated_admin_diagnostic_fallback +``` + +Expected: FAIL. + +- [ ] **Step 6: Wire Cloudflare and verify GREEN** + +Call the helper before Cloudflare integration/publisher dispatch. + +```bash +cargo test-cloudflare authenticated_admin_diagnostic_fallback +cargo test-cloudflare +``` + +Expected: PASS. + +- [ ] **Step 7: Add Spin regressions and verify RED** + +Use the existing Spin router helpers with the same matrices. + +```bash +cargo test-spin authenticated_admin_diagnostic_fallback +``` + +Expected: FAIL. + +- [ ] **Step 8: Wire Spin and verify GREEN** + +Call the helper before Spin integration/publisher dispatch. + +```bash +cargo test-spin authenticated_admin_diagnostic_fallback +cargo test-spin +``` + +Expected: PASS. + +- [ ] **Step 9: Commit Task 3** + +```bash +git add crates/trusted-server-adapter-fastly/src/app.rs \ + crates/trusted-server-adapter-axum/src/app.rs \ + crates/trusted-server-adapter-axum/tests/routes.rs \ + crates/trusted-server-adapter-cloudflare/src/app.rs \ + crates/trusted-server-adapter-cloudflare/tests/routes.rs \ + crates/trusted-server-adapter-spin/src/app.rs \ + crates/trusted-server-adapter-spin/tests/routes.rs +git commit -m "Keep admin diagnostics out of publisher fallback" +``` + +### Task 4: Make Fastly EIDs diagnostics structurally read-only + +**Files:** + +- Modify/Test: `crates/trusted-server-adapter-fastly/src/app.rs:501-595` +- Reference: `crates/trusted-server-adapter-fastly/src/main.rs:184-247` +- Reference: `crates/trusted-server-core/src/ec/finalize.rs:77-106` + +- [ ] **Step 1: Add and run a failing finalization-state regression** + +Create an authenticated browser-shaped `GET /_ts/admin/eids` request with valid +EC, EID, and shared-ID cookies. Assert `200` and: + +```rust +assert!( + response.extensions().get::().is_none(), + "admin EIDs diagnostics should not attach EC finalization state" +); +``` + +```bash +cargo test-fastly admin_eids_diagnostic_skips_ec_finalization +``` + +Expected: FAIL because `execute_named` attaches `EcFinalizeState`. + +- [ ] **Step 2: Add the early EIDs dispatch** + +Before GPT preparation or EC setup, build the registry, call the handler, map +errors through `http_error`, and return without `attach_dispatch_extensions`: + +```rust +if matches!(handler, NamedRouteHandler::AdminEidsLookup) { + let result = PartnerRegistry::from_config(&state.settings.ec.partners) + .and_then(|registry| handle_admin_eids_lookup(®istry, &req)); + return Ok(result.unwrap_or_else(|error| http_error(&error))); +} +``` + +Make the normal route arm explicitly unreachable or remove it cleanly. Update +module lifecycle comments. + +- [ ] **Step 3: Run focused and Fastly suites** + +```bash +cargo test-fastly admin_eids_diagnostic_skips_ec_finalization +cargo test-fastly +``` + +Expected: PASS. + +- [ ] **Step 4: Commit Task 4** + +```bash +git add crates/trusted-server-adapter-fastly/src/app.rs +git commit -m "Keep admin EID diagnostics read only" +``` + +### Task 5: Preserve raw parseable KV entries and metadata + +**Files:** + +- Modify/Test: `crates/trusted-server-core/src/ec/admin.rs:47-278` +- Modify/Test: `crates/trusted-server-core/src/ec/admin.rs:464-925` + +- [ ] **Step 1: Add and run a failing lossless-display regression** + +Seed a valid raw entry with unknown top-level, consent, and partner fields; +legacy map-shaped `seen_domains`; valid auction partner data; and metadata with +an unknown field. Assert the final response preserves all raw values and shape, +keeps numeric timestamps, adds ISO companions, preserves metadata, and still +derives auction EIDs. + +```bash +cargo test-fastly parseable_legacy_entry_and_metadata_preserve_raw_json +``` + +Expected: FAIL because typed reserialization drops and normalizes data. + +- [ ] **Step 2: Add and run failing schema/collision tests** + +For valid JSON that cannot deserialize as `KvEntry`, assert raw `entry` remains +present with `entry_error` and no `raw_body`. For stored `created_iso` and +`consent.updated_iso`, assert neither is overwritten. + +```bash +cargo test-fastly valid_json_with_invalid_kv_schema_remains_visible +cargo test-fastly stored_iso_fields_are_not_overwritten +``` + +Expected: FAIL. + +- [ ] **Step 3: Separate raw display from typed interpretation** + +Parse `lookup.body` as `JsonValue` for `payload.entry`, then independently as +`KvEntry` for tombstone, validation, and auction. Invalid JSON sets +`entry_error` plus lossy `raw_body`; valid JSON with an invalid schema remains +visible and sets only `entry_error`. + +Replace `entry_json_with_iso_timestamps(&KvEntry)` with a helper accepting +`&mut JsonValue`. Read raw `created` and `consent.updated` as `u64`, and use +`entry(...).or_insert(...)` for ISO companions so stored collisions win. + +Parse metadata independently as `JsonValue` for display and `KvMetadata` only +for diagnostics. Typed metadata failure must not erase parseable raw metadata; +invalid JSON remains in `metadata_error`. + +- [ ] **Step 4: Run admin and Fastly suites** + +```bash +cargo test-fastly ec::admin::tests +cargo test-fastly +``` + +Expected: PASS, including existing corrupt-entry, validation, timestamp, and +auction tests. + +- [ ] **Step 5: Commit Task 5** + +```bash +git add crates/trusted-server-core/src/ec/admin.rs +git commit -m "Preserve raw admin EC diagnostic records" +``` + +### Task 6: Document the operator API contract + +**Files:** + +- Modify: `docs/guide/api-reference.md:1-20` +- Modify: `docs/guide/api-reference.md:497-590` +- Modify: `docs/guide/api-reference.md:746-772` + +- [ ] **Step 1: Add the Admin Diagnostic Endpoints section** + +Document Basic Auth and sensitive data; both EC lookup forms; response fields +and the raw/typed error matrix; `401`, `400`, `404`, `405`, and `501`; Fastly +support and portability `501`; EIDs cookie inputs, payload, always-`200` +post-auth semantics, and all-adapter support; JSON/no-store behavior; `Allow: +GET`; and the live-consent limitation. Use only fictional/example data. + +- [ ] **Step 2: Update navigation and protected endpoints** + +Add the section to the API category list and the three routes to Protected +Endpoints. + +- [ ] **Step 3: Format and inspect documentation** + +```bash +cd docs && npm run format +git diff --check +git diff -- docs/guide/api-reference.md +``` + +Expected: formatting passes, no whitespace errors, and the contract matches +implemented statuses and headers. + +- [ ] **Step 4: Commit Task 6** + +```bash +git add docs/guide/api-reference.md +git commit -m "Document admin EC and EID diagnostics" +``` + +### Task 7: Verify the complete review resolution + +**Files:** Verify all files changed by Tasks 1-6. + +- [ ] **Step 1: Run formatting** + +```bash +cargo fmt --all -- --check +cd docs && npm run format +``` + +Expected: PASS. + +- [ ] **Step 2: Run all adapter tests** + +```bash +cargo test-fastly +cargo test-axum +cargo test-cloudflare +cargo test-spin +``` + +Expected: PASS. + +- [ ] **Step 3: Run parity tests** + +```bash +cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity +``` + +Expected: PASS. + +- [ ] **Step 4: Run all target-matched lint gates** + +```bash +cargo clippy-fastly +cargo clippy-axum +cargo clippy-cloudflare +cargo clippy-cloudflare-wasm +cargo clippy-spin-native +cargo clippy-spin-wasm +``` + +Expected: PASS with `-D warnings`. + +- [ ] **Step 5: Inspect final branch state** + +```bash +git diff main...HEAD --check +git status --short +git log --oneline --decorate -10 +``` + +Expected: no uncommitted implementation changes and a focused commit sequence. + +- [ ] **Step 6: Request code review** + +Invoke `superpowers:requesting-code-review` with the approved spec and plan. +Address only verified findings and rerun affected tests after corrections. + +- [ ] **Step 7: Verify before completion** + +Invoke `superpowers:verification-before-completion`, confirm fresh output for +every claimed gate, and report environmental limitations rather than claiming +success. + +- [ ] **Step 8: Prepare review-thread resolution notes** + +Map each of the five findings to its implementing commit and test evidence. Do +not post or resolve GitHub threads without separate user authorization. diff --git a/docs/superpowers/plans/2026-08-18-contiguous-generated-slot-tables.md b/docs/superpowers/plans/2026-08-18-contiguous-generated-slot-tables.md new file mode 100644 index 000000000..0fb8977a0 --- /dev/null +++ b/docs/superpowers/plans/2026-08-18-contiguous-generated-slot-tables.md @@ -0,0 +1,46 @@ +# Contiguous Generated Slot Tables Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Keep generated creative-opportunity slot and provider tables contiguous with their parent section. + +**Architecture:** Normalize the document positions carried by generated `toml_edit` tables before inserting them into the target document. Anchor the whole generated subtree at the target creative section and rely on stable serialization order. + +**Tech Stack:** Rust, `toml_edit`, Cargo tests + +--- + +### Task 1: Reproduce the position collision + +**Files:** + +- Modify/Test: `crates/trusted-server-cli/src/commands/audit/generate/slot_toml.rs` + +- [ ] Add `splice_keeps_generated_slots_and_providers_contiguous` with a late creative section and unrelated tables at colliding positions. +- [ ] Assert no unrelated table header occurs between `[creative_opportunities]`, all generated slots, and their provider subtables. +- [ ] Add `splice_groups_a_new_creative_section_with_its_slots` for an input that has no creative section, proving the newly created parent and generated subtree share the final anchor. +- [ ] Run each focused test with `cargo test_cli_macos -- --exact` and confirm both ordering assertions fail. + +### Task 2: Normalize imported table positions + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/slot_toml.rs` + +- [ ] Add a small recursive helper using `Table::set_position`, `Table::iter_mut`, and `ArrayOfTables::iter_mut` to assign one anchor position to every table in the generated slot subtree. +- [ ] Use the existing creative table's position; for a newly created section, allocate one greater than the greatest parsed position and explicitly assign that anchor to both the new parent and its generated subtree. +- [ ] Run `cargo test_cli_macos commands::audit::generate::slot_toml::tests::splice_keeps_generated_slots_and_providers_contiguous -- --exact` and confirm it passes. +- [ ] Run `cargo test_cli_macos commands::audit::generate::slot_toml::tests::splice_groups_a_new_creative_section_with_its_slots -- --exact` and confirm it passes. +- [ ] Run `cargo test_cli_macos commands::audit::generate::slot_toml::tests` and confirm the complete module suite passes. + +### Task 3: Verify and deliver + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/slot_toml.rs` + +- [ ] Run `./scripts/test-cli.sh`. +- [ ] Run `cargo fmt --all -- --check`. +- [ ] Run `cargo clippy --package trusted-server-cli --target aarch64-apple-darwin --all-targets -- -D warnings`. +- [ ] Confirm `trusted-server.toml` and the user's existing `fastly.toml` change remain untouched. +- [ ] Commit the verified generator fix on the current feature branch. diff --git a/docs/superpowers/plans/2026-08-18-pr-823-review-resolution.md b/docs/superpowers/plans/2026-08-18-pr-823-review-resolution.md new file mode 100644 index 000000000..873168438 --- /dev/null +++ b/docs/superpowers/plans/2026-08-18-pr-823-review-resolution.md @@ -0,0 +1,723 @@ +# PR 823 Review Resolution Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Resolve every actionable finding in PR 823 review `4958563121`, verify the branch, publish it, and answer all 28 inline threads. + +**Architecture:** Correct the review findings at four existing seams: core runtime gate APIs, pure CLI projection/comparison, crawl generation and TOML persistence, and the shared browser session. Keep page-controlled work bounded, use one source of truth for runtime/browser behavior, and preserve operator-authored configuration outside the managed creative-opportunities fields (`slot`, `gam_network_id`, `section_root`, and `section_segment`). + +**Tech Stack:** Rust 2024, clap 4, toml_edit 0.23, chromiumoxide 0.9, Tokio current-thread runtime, serde/serde_json, embedded JavaScript collector, mdBook documentation, GitHub CLI. + +--- + +## File Map + +- `crates/trusted-server-core/src/creative_opportunities.rs`: allocation-free gate evaluation, gate diagnostics, pattern validation, consent semantics. +- `crates/trusted-server-core/src/publisher.rs`: named gate input at the runtime call site. +- `crates/trusted-server-cli/src/ad_templates/{expected,compare,output}.rs`: runtime-equivalent projection, typed formats, confirmability, safe output. +- `crates/trusted-server-cli/src/commands/config/ad_templates.rs`: static command validation, gate parity, lint, escaping. +- `crates/trusted-server-cli/src/commands/audit/{collector,browser,ad_templates,ad_template_collector.js}.rs`: shared browser options/session and verifier behavior. +- `crates/trusted-server-cli/src/commands/audit/generate/{browser_collector,evidence,gpt_slots,crawl_plan,page_patterns,unit_template,slot_toml,mod,validate}.rs`: crawl evidence, inference, persistence, and dry-run safety. +- `crates/trusted-server-cli/src/commands/audit/{mod,page}.rs`, `crates/trusted-server-cli/src/run.rs`, `crates/trusted-server-cli/src/main.rs`: clap contracts and exit outcomes. +- `docs/guide/cli.md`, `scripts/test-cli.sh`, `.github/workflows/test.yml`: operator contract and enforced browser CI. + +## Task 1: Make the runtime gate API allocation-free and reusable + +**Files:** + +- Modify: `crates/trusted-server-core/src/creative_opportunities.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` + +- [ ] **Step 1: Add failing core tests** + +Add tests that sweep all 64 boolean combinations with `consent_allows_auction: None`, assert the expected `No`/`Unknown` result, assert `blocking_gates()` derives diagnostics without an owned `Vec`, and exercise the specific page-pattern validation error. + +Use a borrowed/static iterator contract: + +```rust +pub fn blocking_gates(self) -> impl Iterator { + AdStackGateName::ALL + .into_iter() + .filter(move |gate| gate.blocks(self.input)) +} + +pub fn validate_page_pattern(pattern: &str) -> Result<(), String> { + compile_page_pattern(pattern).map(|_| ()) +} +``` + +- [ ] **Step 2: Run the narrow tests and confirm RED** + +Run: + +```bash +cargo test --package trusted-server-core --target "$(rustc -vV | awk '/host:/ {print $2}')" ad_stack_gate -- --nocapture +``` + +Expected: failure because the unknown-consent sweep and allocation-free diagnostic API are not implemented. + +- [ ] **Step 3: Implement the minimal core change** + +Store the original `AdStackGateInput` in `AdStackGateResult`, compute `expected` with boolean expressions rather than `Vec::push`, expose a zero-allocation iterator over a `const ALL`, make `compile_page_pattern` crate-private, and add `validate_page_pattern`. Document that `None` means unknown and differs from denied (`Some(false)`). Preserve the detailed glob error in `compile_patterns`. + +Delete `should_run_server_side_ad_stack`; construct `AdStackGateInput` with named fields in `publisher.rs`. Import the gate types at module scope. + +- [ ] **Step 4: Verify GREEN** + +Run the narrow command again, then: + +```bash +cargo test-fastly creative_opportunities +cargo test-axum creative_opportunities +``` + +Expected: all selected tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/creative_opportunities.rs crates/trusted-server-core/src/publisher.rs +git commit -m "Align ad stack gate diagnostics with runtime" +``` + +## Task 2: Align expected-slot projection and comparison with runtime behavior + +**Files:** + +- Modify: `crates/trusted-server-cli/src/ad_templates/expected.rs` +- Modify: `crates/trusted-server-cli/src/ad_templates/compare.rs` +- Modify: `crates/trusted-server-cli/src/ad_templates/output.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/ad_templates.rs` + +- [ ] **Step 1: Add failing projection and comparison tests** + +Cover: + +- an unrenderable dynamic slot is omitted from expected slots and does not make `matched_slots` pass; +- the diagnostic says the runtime omits the slot for that path; +- `MediaType` remains typed through comparison; +- video/native-only slots produce `Unconfirmable` and do not fail strict; +- a sizeless out-of-page slot against banner-configured formats is `Partial` and fails strict; +- an incompatible banner is still `Partial` and fails strict; +- a missing slot has `phase: None` and JSON omits `phase`; +- server-side APS configuration alone does not emit `aps_evidence_missing`; +- collector warnings are appended to page warnings; +- human output contains expectation, gates, matched count, extra evidence, and warnings; +- bidi override/isolate characters are escaped. + +The central type changes are: + +```rust +pub struct ExpectedFormat { + pub width: u32, + pub height: u32, + pub media_type: MediaType, +} + +pub enum SlotStatus { + Confirmed, + Partial, + Missing, + Unconfirmable, +} + +pub struct SlotResult { + pub phase: Option, + // existing fields +} +``` + +- [ ] **Step 2: Run the narrow tests and confirm RED** + +Run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" ad_templates::expected +cargo test --package trusted-server-cli --target "$HOST_TARGET" ad_templates::compare +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::ad_templates +``` + +Expected: new assertions fail on current projection/status/warning behavior. + +- [ ] **Step 3: Implement projection, comparison, and output changes** + +Filter `match_slots` with `render_gam_unit_path(...).map(...)` while building `ExpectedSlot`. Remove the unconditional client-side APS check. Compute confirmability before assigning status. Map typed media values to strings only in `to_slot_json`. Make JSON phase `Option` with `skip_serializing_if = "Option::is_none"`. Extend warnings with `evidence.warnings` after decode. + +Extend `is_terminal_control` with `0x202A..=0x202E` and `0x2066..=0x2069`. Apply `escape_terminal_text` to every human-facing page/config-derived field. + +- [ ] **Step 4: Verify GREEN** + +Run all three narrow commands again. + +Expected: all selected tests pass with no warnings. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-cli/src/ad_templates crates/trusted-server-cli/src/commands/audit/ad_templates.rs +git commit -m "Match ad template verification to runtime behavior" +``` + +## Task 3: Correct static CLI contracts and process exit semantics + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/config/ad_templates.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/mod.rs` +- Modify: `crates/trusted-server-cli/src/run.rs` +- Modify: `crates/trusted-server-cli/src/main.rs` +- Modify: `crates/trusted-server-cli/Cargo.toml` +- Modify: `Cargo.lock` + +- [ ] **Step 1: Add failing parser, normalization, lint, and outcome tests** + +Add tests proving: + +- bare and full-URL forms normalize spaces, dot segments, tabs, queries, and fragments identically; +- `/r?to=https://example.com` remains a bare path; +- `check` requires exactly one expectation mode and rejects `--allow-extra-slots --expect-no-slots` through clap; +- `--method` accepts a valid `http::Method` and uses exact GET semantics; +- `lint` reports each invalid configured pattern; +- `explain` uses `gate.expected` even when providers are empty and prints provider state separately; +- `--edgezero-enabled` is rejected because the unsupported model is removed; +- bare `ts audit` displays help rather than a drifting manual error; +- parser coverage includes lint, explain, generate, verify profiles/options, and the no-`--adapter` contract; +- an assertion outcome maps to exit 1 and a tool error maps to exit 2. + +Use an explicit process outcome: + +```rust +#[derive(Debug, Clone, Copy, Eq, PartialEq)] +pub enum RunOutcome { + Success, + AssertionFailed, +} + +impl RunOutcome { + pub const fn exit_code(self) -> i32 { + match self { + Self::Success => 0, + Self::AssertionFailed => 1, + } + } +} +``` + +Tool failures remain `Err(String)` and therefore exit 2. Assertion commands write their failure to stderr before returning `AssertionFailed`, avoiding `log::error!` filtering. + +- [ ] **Step 2: Run parser/static tests and confirm RED** + +Run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::config::ad_templates +cargo test --package trusted-server-cli --target "$HOST_TARGET" run::tests +``` + +Expected: current hand-rolled validation, normalization, and exit behavior fail the new tests. + +- [ ] **Step 3: Implement the CLI contract** + +Use a dummy HTTPS base with `Url::options().base_url(...)` for bare paths after anchored scheme detection on the pre-query slice. Add clap `ArgGroup`, `conflicts_with`, `arg_required_else_help`, typed `http::Method`, and browser settle validation. Add `http = { workspace = true }` to the CLI host dependencies. + +Return `RunOutcome` from dispatchable CI commands. Keep edgezero delegated errors as tool errors. Remove the unsupported EdgeZero flag/text and route gate output through `blocking_gates()`. + +- [ ] **Step 4: Verify GREEN** + +Run the two narrow commands again and confirm all tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add Cargo.lock crates/trusted-server-cli/Cargo.toml crates/trusted-server-cli/src/main.rs crates/trusted-server-cli/src/run.rs crates/trusted-server-cli/src/commands/audit/mod.rs crates/trusted-server-cli/src/commands/config/ad_templates.rs +git commit -m "Define ad template CLI assertion contracts" +``` + +## Task 4: Make the injected collector bounded and behavior-preserving + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/ad_template_collector.js` +- Modify: `crates/trusted-server-cli/src/commands/audit/collector.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/browser.rs` + +- [ ] **Step 1: Add failing JavaScript-contract and decoder tests** + +Add tests/fixtures for an out-of-`u32` size beside a valid slot, a truthy `googletag.cmd` without `push`, multiple `cmd.push` arguments, 512-character capture limits, and non-enumerable/closure-local wrapping. Replace the existing `contains("cmd.push")` assertion with assertions that the no-op wrapper is absent. + +The JavaScript bounds are: + +```javascript +const __TS_MAX_STRING = 512 +function __ts_text(value) { + return String(value).slice(0, __TS_MAX_STRING) +} + +if (width > 4294967295 || height > 4294967295) return null +``` + +The setter must always retain the publisher value: + +```javascript +set(value) { + try { + internal = wrap(value) + } catch (error) { + internal = value + __ts_push(__ts_ev.warnings, { + code: "wrap_failed", + message: __ts_text(error), + }) + } +} +``` + +- [ ] **Step 2: Run the narrow tests and confirm RED** + +Run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::collector +cargo test --package trusted-server-cli --target "$HOST_TARGET" collector_payload +``` + +Expected: current script permits oversized integers and retains the behavior-changing wrapper. + +- [ ] **Step 3: Implement minimal collector changes** + +Guard all page-derived strings through `__ts_text`, enforce numeric upper bounds, delete the `cmd.push` wrapper, use a closure-local `WeakSet` for wrapped objects, and install wrapped functions with non-enumerable `Object.defineProperty`. Soften the header claim to “observes without capturing page data.” + +Before serde decode, stringify the evidence inside the page and return a small +sentinel instead of the payload when the serialized string exceeds 1 MiB +(`MAX_EVIDENCE_PAYLOAD_BYTES = 1_048_576`). On the Rust side, the sentinel +produces an `ad_evidence_too_large` warning and `ad_evidence: None`; it does not +fail navigation or the whole collection. This bounds CDP transfer and Rust +decode/allocation while preserving a precise operator diagnostic. + +- [ ] **Step 4: Verify GREEN** + +Run the narrow commands again and confirm all tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-cli/src/commands/audit/ad_template_collector.js crates/trusted-server-cli/src/commands/audit/collector.rs crates/trusted-server-cli/src/commands/audit/browser.rs +git commit -m "Bound browser ad template evidence collection" +``` + +## Task 5: Unify browser launch, session reuse, and settling + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/collector.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/browser.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/ad_templates.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/page.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/mod.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/browser_collector.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/collector.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/mod.rs` + +- [ ] **Step 1: Add failing fake-collector and browser configuration tests** + +Cover one launch/session for multiple URLs, root included in the profile batch, page close on success/error, host-only `Path=/` cookies, explicit final-URL failure, same-host HTTP-to-HTTPS acceptance, host/downgrade/port refusal, new-headless 1280x800 defaults, headful/profile/proxy/consent parity, `$CHROME` parity, and generic/legacy default-on consent. + +Extend the trait with a default batch method so fakes remain simple: + +```rust +pub trait AuditCollector { + fn collect_page(&self, request: BrowserCollectRequest) -> Result; + + fn collect_pages( + &self, + requests: &[BrowserCollectRequest], + ) -> Vec> { + requests.iter().cloned().map(|request| self.collect_page(request)).collect() + } +} +``` + +The real browser implementation overrides `collect_pages` to create one runtime, +temporary profile, browser, handler, and sequentially closed pages. + +- [ ] **Step 2: Run narrow tests and confirm RED** + +Run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::browser +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::ad_templates +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::browser_collector +``` + +Expected: verifier launches per URL, browser defaults diverge, and tabs/cookies/final URL handling fail new assertions. + +- [ ] **Step 3: Implement shared browser configuration and batching** + +Move executable resolution and launch-option construction into `browser.rs` as crate-visible helpers used by both collectors. Flatten shared browser options into generate and verify, while keeping generation-only pacing/crawl flags local. Build cookies with explicit domain from `url.host_str()` and `path = Some("/".to_string())`; do not set `url` simultaneously. + +In each page collector, capture the inner result, always call bounded `page.close().await`, then return the captured result. Batch verify requests via `collect_pages`. Include the root in each profile's batch rather than collecting it in a throwaway session. Use `spawn_blocking` for scraper analysis before folding results. + +- [ ] **Step 4: Bound post-navigation work and correct settle semantics** + +Install `performance.setResourceTimingBufferSize(100000)` before navigation. Make `settle` return warnings and wrap every `evaluate`, URL/title read, scroll operation, and evidence read in a per-operation timeout. Accrue quiet only after `document.readyState` is `interactive` or `complete`; sleep `min(remaining_quiet, 250ms)` so short quiet values are honored. Treat `wait_for_navigation` timeout as a warning after successful `goto`. + +Propagate GPT/link/sitemap evaluation errors as notes, set `await_promise` for sitemap discovery, and warn when only the main frame is inspected while child frames exist. + +- [ ] **Step 5: Verify GREEN** + +Run all three narrow commands again. If Chrome is available, also run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::browser::tests:: -- --ignored --test-threads=1 +``` + +Expected: unit/fake tests pass; browser fixtures execute and pass when Chrome exists. + +- [ ] **Step 6: Commit** + +```bash +git add crates/trusted-server-cli/src/commands/audit +git commit -m "Share browser sessions across ad template audits" +``` + +## Task 6: Preserve crawl evidence and make inference conservative + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/evidence.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/gpt_slots.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/page_patterns.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/crawl_plan.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/unit_template.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/mod.rs` + +- [ ] **Step 1: Add failing inference tests** + +Add focused tests for: + +- `annonsü1`/`annonsü2` and `ünicode-ad-a`/`ünicode-ad-b` prefixes; +- desktop-empty/mobile-present and the inverse; +- two disjoint unrelated placements retained, two with a useful prefix or three fragments refused; +- same-page normalized UUID collisions retained with raw div IDs and all formats; +- 16+ digit numeric stable segments retained; +- comma-separated SRA `dids` ignored; +- locale `/en` pattern emitted as `/en` and every emitted glob matches its source path; +- glob metacharacters escaped with `glob::Pattern::escape`; +- percent-encoded noise/extension paths and `.html`/`.htm`/`.php` treatment; +- dropped-section notes capped at ten plus “and N more”; +- both ambiguous template rows result in explicit `Refuse`; +- real crawl evidence can infer `section_segment = 1`; +- refused slots do not appear in rendered output and their reasons appear in notes. + +- [ ] **Step 2: Run narrow tests and confirm RED** + +Run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::evidence +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::gpt_slots +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::page_patterns +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::crawl_plan +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::unit_template +``` + +Expected: each new regression reproduces its review finding. + +- [ ] **Step 3: Implement evidence-preserving discovery** + +Use the last matching `char_indices` byte boundary for shared prefixes. Remove an empty-page marker whenever a later profile yields slots. Require `(useful shared prefix || group size >= 3)` before classifying disjoint same-shape slots as fragments; emit an ambiguity diagnostic otherwise. + +Group normalized collisions within a page before deduplication. When a group has multiple raw div IDs, keep raw entries, make their generated IDs unique, and attach a collision note. Restrict ephemeral hex matching to tokens containing at least one `a..f`, or an explicit UUID shape; never treat all-digit identifiers as hashes. Reject gampad fallback when parsed `dids` contains a comma. + +- [ ] **Step 4: Implement conservative patterns/templates** + +Emit the observed short path for locale landing pages, escape literal prefixes, decode only for filtering while retaining encoded request paths for matching, and cap notes. Teach crawl planning to carry/infer the section depth used by page-pattern generation. + +Delete the tautological witness check and move its explanatory invariant into `analyse_slot` docs. Keep the existing conservative `Refuse` result for non-derivable slugs and unwitnessed roots. Filter all `Refuse` decisions before `RenderSlot` creation and push each reason into notes. + +- [ ] **Step 5: Verify GREEN** + +Run all five narrow commands again, then: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate +``` + +Expected: the generate module suite passes. + +- [ ] **Step 6: Commit** + +```bash +git add crates/trusted-server-cli/src/commands/audit/generate +git commit -m "Preserve ad template crawl evidence" +``` + +## Task 7: Make slot persistence and dry-run output safe + +**Files:** + +- Modify: `Cargo.toml` +- Modify: `Cargo.lock` +- Modify: `crates/trusted-server-cli/Cargo.toml` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/slot_toml.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/mod.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/validate.rs` + +- [ ] **Step 1: Add failing persistence tests** + +Cover: + +- trailing comments after the final slot; +- a multiline string line beginning `[foo]`; +- an array continuation beginning `[300, 250]`; +- non-contiguous slot tables; +- byte-identical unrelated sections/comments and CRLF preservation; +- end-to-end `--replace` through `run_update_slots`; +- dry-run source file byte identity; +- stdout contains only a zero-context unified diff of managed + creative-opportunities changes and does not contain `admin_password` or + unrelated config; +- notes/rollback warning go to stderr; +- a concurrent source edit between initial read and write is refused; +- rerun unions formats and reports broad-prefix collapse. + +Change `run_update_slots` to accept separate writers: + +```rust +pub(crate) fn run_update_slots( + request: &UpdateSlotsRequest<'_>, + collectors: &[(&str, &dyn AuditCollector)], + out: &mut dyn Write, + err: &mut dyn Write, +) -> CliResult<()>; +``` + +- [ ] **Step 2: Run persistence tests and confirm RED** + +Run: + +```bash +HOST_TARGET="$(rustc -vV | awk '/host:/ {print $2}')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" commands::audit::generate::slot_toml +cargo test --package trusted-server-cli --target "$HOST_TARGET" update_slots +``` + +Expected: current line scanner corrupts/preserves incorrectly and dry-run leaks the complete config. + +- [ ] **Step 3: Implement a TOML-aware managed edit** + +Parse the source as `DocumentMut` and update the complete managed field set: +`creative_opportunities.slot`, `gam_network_id`, `section_root`, and +`section_segment`. Insert the generated array-of-tables and upsert only scalar +values that generation actually inferred. A generated `None` preserves the +existing scalar on both merge and `--replace`; absence of fresh evidence is +never an instruction to delete operator configuration. Retain decorations on +all other items. Before returning, parse both documents and compare canonical +clones with all four managed fields removed; return an error if any other item +differs. Preserve CRLF after serialization. Add regression cases in Step 1 for +an unresolved network ID and literal-only rerun retaining existing +`gam_network_id`/section policy. + +Document `splice_creative_slots` at its definition and remove the orphaned comments. Replace the `let _ = network_id` presence check with `keys.network_id.is_none()` logic. + +- [ ] **Step 4: Implement secret-safe dry-run and stale-read protection** + +Add `similar` as a workspace/CLI dependency and render a zero-context unified +diff between the old and new managed creative-opportunities projection. The +projection contains only `gam_network_id`, `section_root`, `section_segment`, +and the slot array, so every generated scalar change is visible without +including unrelated operator keys: + +```rust +let diff = similar::TextDiff::from_lines(old_managed, new_managed); +writeln!(out, "{}", diff.unified_diff().context_radius(0).header("configured creative opportunities", "generated creative opportunities"))?; +``` + +Send all notes to `err`. Immediately before atomic rename, re-read the config and compare it with the original bytes; refuse on mismatch. Do not perform this check on dry-run because no write occurs. + +In `merge_render_slots`, union discovered formats into a matching existing slot and count how many discovered slots map to each existing prefix; report counts greater than one. + +- [ ] **Step 5: Verify GREEN** + +Run both narrow commands again and confirm all tests pass. + +- [ ] **Step 6: Commit** + +```bash +git add Cargo.toml Cargo.lock crates/trusted-server-cli/Cargo.toml crates/trusted-server-cli/src/commands/audit/generate +git commit -m "Preserve operator config during slot generation" +``` + +## Task 8: Complete documentation, test hygiene, and CI enforcement + +**Files:** + +- Modify: `docs/guide/cli.md` +- Modify: `scripts/test-cli.sh` +- Modify: `.github/workflows/test.yml` +- Modify: `crates/trusted-server-cli/src/lib.rs` +- Modify: touched Rust tests and comments under `crates/trusted-server-cli/src/` + +- [ ] **Step 1: Add/restore parser and CI guard tests** + +Restore the `audit` no-`--adapter` parser test. Add a script contract that sets `TS_AUDIT_BROWSER_TESTS=1`; browser fixture tests panic when that variable is set and Chrome cannot be resolved. Configure the workflow with a browser setup action or the runner's installed Chrome path and export `CHROME` before `scripts/test-cli.sh`. + +- [ ] **Step 2: Replace sensitive-looking fixtures and stale assertions** + +Replace sensitive or customer-shaped fixtures introduced by this PR with fictional network IDs, publisher names, URL shapes, and neutral div tokens. Update comments to describe shapes rather than customers. + +Correct all touched `expect` messages to start with `should`, remove redundant crate/file `dead_code` allowances and annotate only genuinely deferred fields, reorder `Audit`, simplify the Prebid query parser so keys—not substrings—are matched, and bind legacy URLs directly without an impossible `expect`. + +- [ ] **Step 3: Document the complete operator contract** + +In `docs/guide/cli.md`, document: + +- `config ad-templates lint|match|check|explain` and every flag; +- shared `--app-config`, `--manifest`, and `--no-env` behavior; +- `audit ad-templates generate|verify` browser/profile/proxy/consent/settle flags; +- dry-run stdout diff versus stderr notes; +- exit 0 success, exit 1 assertion drift, exit 2 tool/configuration error; +- refused slots are omitted with reasons; +- locale-prefixed inference and section depth; +- `Unconfirmable` strict behavior and optional evidence phase. + +Update the existing design/output examples where the wire contract changed. + +- [ ] **Step 4: Run format and focused checks** + +Run: + +```bash +cargo fmt --all -- --check +cd docs && npm run format +``` + +Expected: both commands exit 0. + +- [ ] **Step 5: Commit** + +```bash +git add .github/workflows/test.yml scripts/test-cli.sh docs crates/trusted-server-cli/src +git commit -m "Document and enforce ad template audit contracts" +``` + +## Task 9: Run full verification and repair regressions + +**Files:** + +- Modify only files implicated by a failing check. + +- [ ] **Step 1: Run format and CLI/browser tests** + +```bash +cargo fmt --all -- --check +./scripts/test-cli.sh +``` + +Expected: exit 0; browser fixture output shows tests executed rather than skipped. + +- [ ] **Step 2: Run repository target suites** + +```bash +cargo test-fastly +cargo test-axum +cargo test-cloudflare +cargo test-spin +``` + +Expected: all suites exit 0. + +- [ ] **Step 3: Run all target-matched clippy gates** + +```bash +cargo clippy-fastly +cargo clippy-axum +cargo clippy-cloudflare +cargo clippy-cloudflare-wasm +cargo clippy-spin-native +cargo clippy-spin-wasm +cargo clippy --manifest-path crates/trusted-server-cli/Cargo.toml --target "$(rustc -vV | sed -n 's/host: //p')" --all-targets -- -D warnings +``` + +Expected: all commands exit 0 with no warnings. + +- [ ] **Step 4: Run cross-adapter parity gates** + +```bash +cargo fmt --manifest-path crates/trusted-server-integration-tests/Cargo.toml -- --check +cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity +cargo clippy --manifest-path crates/trusted-server-integration-tests/Cargo.toml --all-targets -- -D warnings +``` + +Expected: formatting, parity tests, and integration-test clippy exit 0. + +- [ ] **Step 5: Run JavaScript and documentation checks** + +```bash +cd crates/trusted-server-js/lib && npx vitest run && npm run format && node build-all.mjs +cd ../../.. && cd docs && npm run format +``` + +Expected: tests/build/format exit 0. + +- [ ] **Step 6: Inspect the final diff against the review** + +Run: + +```bash +git diff --check origin/main...HEAD +git status --short +``` + +Walk the 28-thread traceability table and every summary category in the design spec. Confirm each has a code/doc/test resolution or an evidence-backed response. + +- [ ] **Step 7: Commit any verification-only corrections** + +If verification required changes, inspect `git diff --name-only`, stage each +listed path explicitly (never `git add .`), and commit them as `Resolve ad +template review regressions`. Record those exact paths in the execution log. +Skip this commit when verification required no changes. + +## Task 10: Publish and answer GitHub review threads + +**Files:** + +- No repository files unless publication reveals a conflict. + +- [ ] **Step 1: Push the verified branch** + +```bash +git push origin feature/ts-cli-ad-templates +``` + +Expected: push succeeds and PR 823 shows the verified head commit. + +- [ ] **Step 2: Correct the PR description** + +Change the legacy alias statement to say bare `ts audit ` aliases to `ts audit generate `. Preserve all unrelated PR-body content. + +- [ ] **Step 3: Reply to every inline thread** + +For each ID in the spec traceability table, post through: + +```bash +gh api repos/IABTechLab/trusted-server/pulls/823/comments//replies -f body='' +``` + +Each reply must name the concrete behavior changed and, where useful, the focused test. For question threads, state the chosen behavior: union formats and diagnose broad prefixes; default consent assumption on; keep conservative refusal and align docs; allow only same-host HTTP-to-HTTPS upgrades; remove the unsupported EdgeZero model. + +- [ ] **Step 4: Verify publication** + +Query PR 823's head SHA, review comments, checks, and unresolved threads. Confirm all 28 inline comments have one reply and no reply claims a fix absent from the pushed diff. + +- [ ] **Step 5: Report the result** + +Summarize commits, verification commands, any environment limitation, PR link, and thread reply count. Do not claim checks pass without fresh output from Task 9. diff --git a/docs/superpowers/plans/2026-08-18-pre-navigation-cookie-install.md b/docs/superpowers/plans/2026-08-18-pre-navigation-cookie-install.md new file mode 100644 index 000000000..c574c4104 --- /dev/null +++ b/docs/superpowers/plans/2026-08-18-pre-navigation-cookie-install.md @@ -0,0 +1,49 @@ +# Pre-navigation Cookie Installation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Allow domain/path-scoped operator cookies to be installed before the audit's first navigation. + +**Architecture:** Add one browser-level cookie installation helper beside `host_cookie`, and call it before creating each audit page. Preserve explicit host-only and root-path scope while avoiding `Page::set_cookie`'s `about:blank` validation. + +**Tech Stack:** Rust, chromiumoxide/CDP, Tokio, Cargo tests + +--- + +### Task 1: Reproduce the pre-navigation failure + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/browser.rs` + +- [ ] Add a Chrome-backed test that installs a host-only cookie before navigating away from `about:blank` and asserts it reaches the first document. +- [ ] Exercise the existing `BrowserCollector` end-to-end against a local HTTP fixture, supplying the cookie through `BrowserCollectRequest`, so the RED test compiles before the fix exists. +- [ ] Run `cargo test_cli_macos commands::audit::browser::tests::supplied_cookie_reaches_first_navigation -- --ignored --exact --nocapture` and confirm it fails with `Blank page can not have cookie`. +- [ ] Add a Chrome-backed error test against the wished-for `set_browser_cookies` API, using an invalid cookie name, and assert the error contains the name but not the secret value. +- [ ] Run that error test and confirm RED because `set_browser_cookies` does not exist yet. + +### Task 2: Install cookies at browser scope + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/browser.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/browser_collector.rs` + +- [ ] Add `set_browser_cookies(&Browser, &[(String, String)], &Url) -> Result<(), String>` beside `host_cookie`; install one cookie per browser call so failures retain name-only context without exposing values. +- [ ] Invoke it before page creation in both collectors and remove page-level cookie installation. +- [ ] Run `cargo test_cli_macos commands::audit::browser::tests::supplied_cookie_reaches_first_navigation -- --ignored --exact --nocapture` and confirm it passes. +- [ ] Run the focused error test and confirm it passes. + +### Task 3: Verify the change + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/browser.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/browser_collector.rs` + +- [ ] Run `cargo test_cli_macos commands::audit::browser::tests`. +- [ ] Run `cargo test_cli_macos commands::audit::generate::browser_collector::tests`. +- [ ] Run `./scripts/test-cli.sh` to exercise the portable host-target suite and ignored browser fixtures. +- [ ] Run `cargo fmt --all -- --check`. +- [ ] Run `cargo clippy --package trusted-server-cli --target aarch64-apple-darwin --all-targets -- -D warnings`. +- [ ] Inspect the diff to confirm no cookie values are logged and `fastly.toml` remains untouched. diff --git a/docs/superpowers/plans/2026-08-18-ssat-debug-comment-format.md b/docs/superpowers/plans/2026-08-18-ssat-debug-comment-format.md new file mode 100644 index 000000000..bbe214363 --- /dev/null +++ b/docs/superpowers/plans/2026-08-18-ssat-debug-comment-format.md @@ -0,0 +1,232 @@ +# SSAT Debug Comment Output Format Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a backward-compatible `compact`/`pretty` configuration option for the outer JSON in SSAT auction debug comments. + +**Architecture:** Extend `AuctionDebugCommentOptions` with a serde-backed format enum that defaults to compact. Keep the existing dump value and safety pipeline unchanged, selecting only `serde_json::to_string` versus `serde_json::to_string_pretty` before terminator neutralization and the 256 KiB cap. + +**Tech Stack:** Rust 2024, serde, serde_json, TOML configuration, existing `trusted-server-core` unit tests. + +--- + +## File Map + +- Modify `crates/trusted-server-core/src/settings.rs`: define the output-format enum, add the option and default, and test TOML behavior. +- Modify `crates/trusted-server-core/src/publisher.rs`: select compact or pretty JSON serialization and test rendering/safety invariants. +- Modify `trusted-server.example.toml`: document the new setting and accepted values. + +### Task 1: Add the configuration type + +**Files:** + +- Modify: `crates/trusted-server-core/src/settings.rs:1945-2030` +- Test: `crates/trusted-server-core/src/settings.rs:2710-2780` + +- [ ] **Step 1: Write failing settings tests** + +Extend `auction_debug_comment_options_default_matches_serde_defaults` with: + +```rust +assert_eq!( + opts.format, + AuctionDebugCommentFormat::Compact, + "should default to compact output" +); +``` + +Add focused tests: + +```rust +#[test] +fn auction_debug_comment_options_deserializes_pretty_format() { + let options: AuctionDebugCommentOptions = toml::from_str(r#"format = "pretty""#) + .expect("should deserialize pretty format"); + assert_eq!(options.format, AuctionDebugCommentFormat::Pretty); +} + +#[test] +fn auction_debug_comment_options_bad_format_fails_config_load() { + let result: Result = + toml::from_str(r#"format = "expanded""#); + assert!( + result.is_err(), + "unrecognized format must fail to deserialize, not silently fall back" + ); +} +``` + +- [ ] **Step 2: Run the tests and verify RED** + +Run: + +```bash +cargo test-fastly auction_debug_comment_options -- --nocapture +``` + +Expected: compilation fails because `AuctionDebugCommentFormat` and `format` do not exist. + +- [ ] **Step 3: Implement the minimal configuration surface** + +Add `format` after `verbosity` in `AuctionDebugCommentOptions`: + +```rust +/// JSON representation used for the outer auction dump. +#[serde(default)] +pub format: AuctionDebugCommentFormat, +``` + +Set `format: AuctionDebugCommentFormat::Compact` in the hand-written default and define: + +```rust +/// JSON representation used for the outer `ts-debug` auction dump. +#[derive(Debug, Clone, Copy, Default, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum AuctionDebugCommentFormat { + #[default] + Compact, + Pretty, +} +``` + +- [ ] **Step 4: Run the focused tests and verify GREEN** + +Run `cargo test-fastly auction_debug_comment_options -- --nocapture`. + +Expected: all matching settings tests pass. + +- [ ] **Step 5: Commit the configuration change** + +```bash +git add crates/trusted-server-core/src/settings.rs +git commit -m "Configure SSAT debug comment output format" +``` + +### Task 2: Render pretty outer JSON without transforming values + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs:2027-2130` +- Test: `crates/trusted-server-core/src/publisher.rs:4330-4945` + +- [ ] **Step 1: Add a test helper for extracting an uncapped dump** + +Extract the existing parsing logic into `dump_json_from_comment`, returning the dump substring and parsed `serde_json::Value`. It must split after `dump=` and before the comment's final newline/terminator so it works for both one-line and indented JSON. + +- [ ] **Step 2: Write failing rendering tests** + +Add tests proving: + +1. Default compact output still contains `dump={"provider_responses":` and no newline immediately after the opening object. +2. Pretty output contains `dump={\n "provider_responses":`. +3. Compact and pretty uncapped dumps deserialize to equal JSON values. +4. In Full mode, metadata containing `{"requestbody": "{\"id\":\"request-1\"}"}` retains `requestbody` as a JSON string in pretty output. + +Extend the existing comment-terminator test to iterate over both +`AuctionDebugCommentFormat::{Compact, Pretty}` for the tested verbosity modes. +Add a pretty/full case to the total-cap test and assert the existing +`(truncated` marker remains present. + +- [ ] **Step 3: Run the rendering tests and verify RED** + +Run: + +```bash +cargo test-fastly auction_debug_comment -- --nocapture +``` + +Expected: the pretty-layout assertion fails because rendering still always uses compact serialization. + +- [ ] **Step 4: Implement format-selected serialization** + +Import `AuctionDebugCommentFormat` alongside the existing options types. Replace the single serializer call with: + +```rust +let serialized = match options.format { + AuctionDebugCommentFormat::Compact => serde_json::to_string(&dump), + AuctionDebugCommentFormat::Pretty => serde_json::to_string_pretty(&dump), +}; +let dump = render_dump( + serialized.unwrap_or_else(|error| format!("")), +); +``` + +Do not alter the dump value, nested strings, neutralization, cap, or comment envelope. + +- [ ] **Step 5: Run the focused tests and verify GREEN** + +Run `cargo test-fastly auction_debug_comment -- --nocapture`. + +Expected: all matching rendering tests pass. + +- [ ] **Step 6: Commit the renderer change** + +```bash +git add crates/trusted-server-core/src/publisher.rs +git commit -m "Pretty print SSAT debug comment dumps" +``` + +### Task 3: Document and verify the completed feature + +**Files:** + +- Modify: `trusted-server.example.toml:180-190` + +- [ ] **Step 1: Document the setting** + +Add to `[debug.auction_html_comment_options]`: + +```toml +# "compact" (default) or "pretty". Pretty formats only the outer dump; +# JSON request/response bodies remain strings exactly as captured. +format = "compact" +``` + +- [ ] **Step 2: Run formatting** + +Run `cargo fmt --all` and then `cargo fmt --all -- --check`. + +Expected: both exit successfully. + +- [ ] **Step 3: Run required tests** + +Run each command separately: + +```bash +cargo test-fastly +cargo test-axum +cargo test-cloudflare +cargo test-spin +``` + +Expected: every command exits successfully with zero failed tests. + +- [ ] **Step 4: Run target-matched lint checks** + +Run each command separately: + +```bash +cargo clippy-fastly +cargo clippy-axum +cargo clippy-cloudflare +cargo clippy-cloudflare-wasm +cargo clippy-spin-native +cargo clippy-spin-wasm +``` + +Expected: every command exits successfully with warnings denied. + +- [ ] **Step 5: Inspect the final diff and configuration compatibility** + +Run `git diff --check` and inspect `git diff origin/main...HEAD` plus the remaining working-tree diff. Confirm compact is the serde/default value, pretty changes whitespace only, nested strings are preserved, and both safety protections remain unconditional. + +- [ ] **Step 6: Commit documentation and any formatting changes** + +```bash +git add crates/trusted-server-core/src/settings.rs crates/trusted-server-core/src/publisher.rs trusted-server.example.toml +git commit -m "Document SSAT debug comment formatting" +``` + +- [ ] **Step 7: Push the branch and update PR #943** + +Run `git push origin feat/ssat-debug-comment-config`, then verify PR state and checks with `gh pr view 943` and `gh pr checks 943`. diff --git a/docs/superpowers/plans/2026-08-19-ad-template-generation-progress.md b/docs/superpowers/plans/2026-08-19-ad-template-generation-progress.md new file mode 100644 index 000000000..ca4245314 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-ad-template-generation-progress.md @@ -0,0 +1,167 @@ +# Ad-template Generation Progress Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Show immediate, safe, profile-aware progress while `ts audit ad-templates generate` performs a long browser crawl. + +**Architecture:** Add typed progress events to the `AuditCollector` boundary so the browser can report work before buffered page results are returned. Render and flush those events from `run_update_slots` on stderr, using only URL paths. Preserve crawl/progress errors over teardown errors while always closing and waiting for Chrome. + +**Tech Stack:** Rust 2024, `std::io::Write`, existing `url`, `tokio`, `chromiumoxide`, and CLI test helpers; no new dependency. + +--- + +### Task 1: Define and render safe progress events + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/collector.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/mod.rs` + +- [ ] **Step 1: Write failing renderer and writer tests** + +Add tests in `generate/mod.rs` for events covering launch, `1/?`, `2/17`, and finalization. Assert that `https://user:pass@publisher.example/news?token=secret#fragment` renders only `/news`, terminal control bytes are escaped, stdout remains untouched, and a counting writer records an explicit `flush()`. Add writers that fail independently on `write()` and `flush()` and assert a CLI output error. + +- [ ] **Step 2: Run the focused tests and confirm RED** + +Run: + +```bash +cargo test --package trusted-server-cli --target aarch64-apple-darwin progress -- --nocapture +``` + +Expected: FAIL because the progress event and renderer do not exist. + +- [ ] **Step 3: Add the progress model and renderer** + +In `collector.rs`, define a small event enum and callback type: + +```rust +pub(crate) enum CollectionProgress<'a> { + Launching, + Loading { + current: usize, + total: Option, + url: &'a Url, + }, + Planning, + Finalizing, +} + +pub(crate) type ProgressSink<'a> = + &'a mut dyn FnMut(CollectionProgress<'_>) -> CliResult<()>; +``` + +Add concise doc comments to the enum, every variant, and the callback alias. The +callback documentation must state that returning an error stops new collection +work but does not bypass an already-launched browser's finalization/close/wait. + +In `generate/mod.rs`, add a `write_collection_progress` helper that accepts a profile label, formats only `url.path()` (or `/` when empty), sanitizes it with `escape_terminal_text`, writes one line to stderr, and immediately calls `flush()`. Render and test `Planning` between the root load and subsequent page loads. + +- [ ] **Step 4: Run the focused tests and confirm GREEN** + +Run the command from Step 2. Expected: all progress renderer/writer tests pass. + +### Task 2: Propagate progress through collectors with teardown-safe failures + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/collector.rs` +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/browser_collector.rs` +- Modify: `scripts/test-cli.sh` + +- [ ] **Step 1: Write failing collector tests** + +Test default `collect_pages` and `collect_site` count semantics, including an attempted page whose collection fails. The exact dynamic-site sequence is root `1/?`, planning, then follow-ups `2/total` through `total/total`; totals include the root and failed attempts advance the count. Add a Chrome-backed test whose progress callback fails during collection. It must return the progress error only after the browser teardown path completes. Extend the existing result-combination unit tests to cover first-error preservation across a collection/planning error, a later finalization-progress error, close error, and wait error, while proving finalization, close, and wait were all attempted. + +- [ ] **Step 2: Run the focused tests and confirm RED** + +Run: + +```bash +cargo test --package trusted-server-cli --target aarch64-apple-darwin commands::audit::generate::browser_collector::tests -- --nocapture +cargo test --package trusted-server-cli --target aarch64-apple-darwin commands::audit::generate::collector::tests -- --nocapture +``` + +Expected: FAIL because collectors do not accept or emit progress callbacks. + +- [ ] **Step 3: Add callbacks to the collector boundary** + +Extend `collect_pages` and `collect_site` with `ProgressSink`. Default collectors emit `Loading` before each page. The root of a dynamically planned site emits `current: 1, total: None`, followed by `Planning`; after planning, default `collect_site` iterates follow-ups itself with an explicit offset so they report `2/total` onward. Fixed batches emit totals including the root, and failed attempts still consume their position. + +Pass the callback into `with_browser`. Adapt `BrowserAuditCollector::collect_page` with an explicit no-op progress sink because single-page artifact generation has no command progress writer. Emit `Launching` before browser launch, `Loading` immediately before each navigation, `Planning` immediately before invoking the root planner, and `Finalizing` before close/wait. Track only the first crawl/progress error: on callback failure, stop scheduling pages, still attempt finalization, `browser.close()`, and `browser.wait()`, then return that first error ahead of teardown errors. + +Extend `scripts/test-cli.sh` with a second ignored-test filter for +`commands::audit::generate::browser_collector::tests::` so the new Chrome-backed +progress-failure test is actually executed under `TS_AUDIT_BROWSER_TESTS=1` and +single-threaded, alongside the existing three browser audit fixtures. + +- [ ] **Step 4: Run unit and Chrome-backed tests and confirm GREEN** + +Run the focused command, then: + +```bash +./scripts/test-cli.sh +``` + +Expected: collector unit tests and all four Chrome-backed tests pass. + +### Task 3: Wire profile-aware progress into generation + +**Files:** + +- Modify: `crates/trusted-server-cli/src/commands/audit/generate/mod.rs` + +- [ ] **Step 1: Write failing generation tests** + +Update `run_update_slots` tests to assert the stderr buffer contains progress for the first profile's `1/?` root, planning, later known totals, the second profile's `1/total` root, and finalization. Assert dry-run diff/success output on stdout contains no progress lines. Add an ordering test with a shared observable writer and fake collector: from inside `collect_site`, after invoking and flushing the progress callback but before returning, assert the progress bytes are already visible. + +- [ ] **Step 2: Run the focused tests and confirm RED** + +Run: + +```bash +cargo test --package trusted-server-cli --target aarch64-apple-darwin update_slots -- --nocapture +``` + +Expected: FAIL because `run_update_slots` does not provide progress callbacks. + +- [ ] **Step 3: Connect callbacks and profiles** + +Create a progress closure for the first profile and pass it to `collect_site`. Pass `err` through `crawl_sections`, create a closure for each later profile, and pass it to `collect_pages`. Keep notes and final summary behavior unchanged. + +- [ ] **Step 4: Run the focused tests and confirm GREEN** + +Run the command from Step 2. Expected: all generation tests pass and progress appears only in stderr. + +### Task 4: Verify and ship + +**Files:** + +- Verify all modified files plus the two design documents. + +- [ ] **Step 1: Format and lint** + +```bash +cargo fmt --all -- --check +cargo clippy --package trusted-server-cli --target aarch64-apple-darwin --all-targets --all-features -- -D warnings +git diff --check +cd docs && npm run format +``` + +Expected: all commands exit 0. + +- [ ] **Step 2: Run the complete local CLI suite** + +```bash +./scripts/test-cli.sh +``` + +Expected: unit, config, proxy, documentation, and Chrome-backed tests pass. + +- [ ] **Step 3: Review the scoped diff** + +Confirm no cookie values, real publisher data, or changes to the pre-existing `fastly.toml` modification are included. Request an independent code review and address concrete findings. + +- [ ] **Step 4: Commit and push** + +Stage only the progress implementation and its design/plan documents. Commit with `Show ad-template generation progress`, push `feature/ts-cli-ad-templates`, and confirm local HEAD matches the remote branch. diff --git a/docs/superpowers/plans/2026-08-19-pr-1013-review-remediation.md b/docs/superpowers/plans/2026-08-19-pr-1013-review-remediation.md new file mode 100644 index 000000000..fbcfd8c98 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-pr-1013-review-remediation.md @@ -0,0 +1,470 @@ +# PR #1013 Review Remediation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Resolve all technically sound actionable review feedback on PR #1013 while preserving ordinary proxy behavior and the ESI spike's fail-safe contracts. + +**Architecture:** Keep the existing publisher, template-cache, and adapter boundaries. Add narrow typed invariants at those boundaries: fallible Fastly body reads, an explicit response-privacy marker, shared publisher-ESI detection, and fallible metadata encoding. Behavioral changes are test-first; mechanical review cleanup follows once runtime contracts are green. + +**Tech Stack:** Rust 2024, `error-stack`, Fastly SDK/Viceroy, `edgezero_core` HTTP types, TypeScript, Vitest, Bash/GitHub Actions. + +**Spec:** `docs/superpowers/specs/2026-08-19-pr-1013-review-remediation-design.md` + +--- + +## File Map + +- `crates/trusted-server-adapter-fastly/src/template_cache.rs`: fallible cache-body reads, insert length metadata, shared purge key, metadata-error propagation. +- `crates/trusted-server-adapter-fastly/src/main.rs`: terminal response effects keyed by an explicit privacy marker. +- `crates/trusted-server-adapter-fastly/src/esi_assembly.rs`: consume the shared publisher-ESI detector and cover comment blocks. +- `crates/trusted-server-core/src/response_privacy.rs`: define and attach the typed terminal-private marker. +- `crates/trusted-server-core/src/platform/template_assembly.rs`: own the shared ESI-directive detector. +- `crates/trusted-server-core/src/platform/template_cache.rs`: shared purge constant, normalized keys, fallible metadata encoding, reservation and schema tests. +- `crates/trusted-server-core/src/platform/mod.rs`: public platform roster/export consistency. +- `crates/trusted-server-core/src/publisher.rs`: unconditional encoding restriction, collision bypass, unused-argument and page-bids cleanup, rustdoc/harness notes. +- `crates/trusted-server-core/src/creative_opportunities.rs` and `src/integrations/gpt_diagnostics.rs`: consolidate adjacent impl blocks and test conventions. +- `crates/trusted-server-js/lib/src/integrations/gpt/index.ts` and `crates/trusted-server-core/src/integrations/gpt_bootstrap.js`: one-shot initial scheduler contract. +- `crates/trusted-server-js/lib/test/integrations/gpt/*.test.ts`: executable scheduling contracts. +- `Cargo.toml`, `crates/trusted-server-adapter-fastly/Cargo.toml`, `.github/workflows/test.yml`, `scripts/template-cache-local-test.sh`, `trusted-server.example.toml`, and `docs/guide/configuration.md`: dependency, CI, harness, and operator-facing cleanup. +- `docs/superpowers/archive/` plus cross-references: archive the two superseded documents. + +### Task 1: Make Fastly cache I/O fail safely + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/template_cache.rs` +- Test: `crates/trusted-server-adapter-fastly/src/template_cache.rs` + +- [ ] **Step 1: Add a failing fallible-reader regression test** + +Extract the byte-reading decision behind a private helper generic over `std::io::Read`, then test it with a reader that returns bytes followed by `io::Error`. The assertion must expect `ReadFoundError::Invalid(TemplateCacheMiss::Truncated)` (or `Backend` if investigation shows the adapter consistently classifies transport failures that way) and must not panic. + +- [ ] **Step 2: Run the focused test and verify RED** + +Run: `cargo test -p trusted-server-adapter-fastly --target wasm32-wasip1 cache_body_read_error` + +Expected: FAIL because the current path uses `Body::into_bytes` and has no fallible helper/classification. + +- [ ] **Step 3: Replace the panicking SDK conversion** + +Import `std::io::Read as _`, call `read_to_end` on `found.to_stream()?`, map the error through `ReadFoundError`, and retain the post-read `metadata.body_len` check. Do not use `into_bytes`. + +- [ ] **Step 4: Add `.known_length(body.len() as u64)` to direct `put`** + +Place it beside `surrogate_keys` and `user_metadata`, matching the reservation insert builder. + +- [ ] **Step 5: Run the adapter cache suite and verify GREEN** + +Run: `cargo test -p trusted-server-adapter-fastly --target wasm32-wasip1 template_cache` + +Expected: all template-cache tests PASS, including the new read-error case. + +- [ ] **Step 6: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/template_cache.rs +git commit -m "Make Fastly template cache reads fallible" +``` + +### Task 2: Preserve injection when encoding negotiation fails + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Test: `crates/trusted-server-core/src/publisher.rs` + +- [ ] **Step 1: Add a failing end-to-end publisher test** + +In the ESI publisher tests, send a navigation with `Accept-Encoding: zstd, gzip;q=0, deflate;q=0, br;q=0, identity;q=0`. This fully refuses every representation TS can assemble, so `negotiate_reader_compression` must fail. Queue an origin response that would be undecodable if the header leaked, then assert the recorded origin request advertises `identity` and the returned HTML contains TSJS injection. + +- [ ] **Step 2: Verify RED** + +Run: `cargo test-fastly esi_unsupported_reader_encoding_still_injects_tsjs` + +Expected: FAIL because ESI mode currently skips `restrict_accept_encoding` when `reader_supports_assembly` is false. + +- [ ] **Step 3: Apply the minimal fix** + +Call `restrict_accept_encoding(&mut req)` unconditionally before the origin fetch. Keep reader assembly eligibility separate; it controls shared assembly, not whether the origin offer is processable. + +- [ ] **Step 4: Verify GREEN and the helper matrix** + +Run: `cargo test-fastly publisher_proxy` + +Run: `cargo test-fastly esi_unsupported_reader_encoding_still_injects_tsjs` + +Expected: all matching tests PASS. + +- [ ] **Step 5: Commit** + +```bash +git add crates/trusted-server-core/src/publisher.rs +git commit -m "Preserve injection for unsupported encodings" +``` + +### Task 3: Scope terminal privacy re-enforcement to TS-owned responses + +**Files:** + +- Modify: `crates/trusted-server-core/src/response_privacy.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs` +- Test: `crates/trusted-server-core/src/response_privacy.rs` +- Test: `crates/trusted-server-adapter-fastly/src/main.rs` + +- [ ] **Step 1: Add two failing terminal-policy tests** + +Extend the Fastly tests so an assembled response is explicitly marked and remains `private, no-store` after hostile late effects. Add a companion test whose unmarked origin response starts with `Cache-Control: private, max-age=600`, `ETag`, and `Last-Modified`; after terminal effects it must retain all three. + +- [ ] **Step 2: Verify RED** + +Run: `cargo test -p trusted-server-adapter-fastly --target wasm32-wasip1 terminal_response` + +Expected: the ordinary-origin preservation test FAILS because current code infers TS ownership from the header value. + +- [ ] **Step 3: Add a typed response extension** + +Define a documented marker such as `TerminalPrivateResponse` in `response_privacy.rs`. Make `enforce_synthesized_html_cache_privacy` and the cached-template stamping path insert it when TS creates per-reader output. Keep `enforce_private_no_store` as the pure header mutation used during terminal re-enforcement. + +- [ ] **Step 4: Consume the marker in Fastly terminal effects** + +Replace the `is_private_or_no_store` snapshot with `response.extensions().get::().is_some()`. Apply late effects first, then re-enforce only for marked responses. Leave the existing Set-Cookie privacy guard last. + +- [ ] **Step 5: Verify GREEN across core and Fastly** + +Run: `cargo test-fastly response_privacy` + +Run: `cargo test -p trusted-server-adapter-fastly --target wasm32-wasip1 terminal_response` + +Expected: marked response remains terminal-private; unmarked origin-private response is unchanged. + +- [ ] **Step 6: Commit** + +```bash +git add crates/trusted-server-core/src/response_privacy.rs crates/trusted-server-core/src/publisher.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Scope terminal privacy to synthesized responses" +``` + +### Task 4: Refuse publisher ESI and seam collisions without mutating bytes + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_assembly.rs` +- Modify: `crates/trusted-server-core/src/platform/mod.rs` +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/esi_assembly.rs` +- Test: `crates/trusted-server-core/src/publisher.rs` +- Test: `crates/trusted-server-adapter-fastly/src/esi_assembly.rs` + +- [ ] **Step 1: Rewrite the collision test to express the desired behavior** + +Change `an_origin_marker_collision_is_normalized_before_store` into a regression asserting the cold response preserves the publisher marker bytes, the template cache stores no entry, and a second request reaches origin again. Add a script-string collision fixture so the test proves no HTML-comment-only neutralizer is involved. + +- [ ] **Step 2: Add failing ESI-comment tests** + +Test ``, uppercase `` unchanged. +- Modify `crates/trusted-server-core/src/publisher.rs`: rename `HEADER_X_TS_C2_CACHE`, `C2ResponseState`, `C2BypassReason`, `C2CachePolicy`, `set_c2_response_state`, `request_bypasses_c2`, `c2_bypass_reason`, `c2_cache_ttl`, local variables, test modules, assertions, comments, and `c2_template_cache` log messages; emit only `x-ts-template-cache` and test that the old header is absent. +- Modify `crates/trusted-server-core/src/html_processor.rs`: rename the active `reserved-c2-seam` test fixture to a template-cache seam name and update its assertions/comments. This fixture is not the schema-history marker exception. +- Modify `crates/trusted-server-core/src/creative_opportunities.rs`, `crates/trusted-server-core/src/platform/types.rs`, and `crates/trusted-server-core/src/response_privacy.rs`: rewrite active cache comments and validation/test messages to “template cache” terminology. +- Modify `crates/trusted-server-adapter-fastly/src/template_cache.rs` and `crates/trusted-server-adapter-fastly/src/esi_assembly.rs`: rewrite active comments and the legacy-read log to `template_cache`; do not alter Fastly cache operations or ESI behavior. + +Harness, CI, and operator material: + +- Rename `scripts/c2-local-test.sh` to `scripts/template-cache-local-test.sh`; update its function name, header parser, log regexes, comments, and usage text. +- Modify `.github/workflows/test.yml` so the step name and both invocations call `scripts/template-cache-local-test.sh`. +- Modify `trusted-server.example.toml` and `docs/guide/configuration.md` so operator-facing prose, diagnostics, and harness commands say “template cache” and use `X-TS-Template-Cache`. +- Modify these non-archived active plans/specifications where the occurrences describe this cache: `docs/superpowers/plans/2026-08-08-1009-measurement-and-stage-0.md`, `docs/superpowers/plans/2026-08-08-1009-measurement-findings.md`, `docs/superpowers/plans/2026-08-12-1009-esi-merge-hardening.md`, `docs/superpowers/plans/2026-08-14-1009-esi-parser-assembly.md`, `docs/superpowers/plans/2026-08-19-pr-1013-review-remediation.md`, `docs/superpowers/specs/2026-08-11-1009-streaming-assembly-architecture.md`, `docs/superpowers/specs/2026-08-12-1009-esi-merge-hardening-design.md`, `docs/superpowers/specs/2026-08-14-1009-esi-parser-assembly-design.md`, and `docs/superpowers/specs/2026-08-19-pr-1013-review-remediation-design.md`. Preserve unrelated `c2` substrings such as checksums, cookie/EC identifiers, creative IDs, and third-party fixture content. +- Do not edit `docs/superpowers/archive/**`. Do not rewrite the exact schema-history marker. Leave `docs/superpowers/specs/2026-08-19-template-cache-terminology-design.md` as the migration record of the old/new names and compatibility effects; its before/after references are an explicit exception. + +The implementation worker should use `@superpowers:test-driven-development` for the focused contract changes, `@superpowers:subagent-driven-development` or `@superpowers:executing-plans` for this task sequence, and `@superpowers:verification-before-completion` before claiming completion. + +### Task 1: Establish failing key-namespace and public-header contracts + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` (test module near `rendered_key_is_fixed_size_and_contains_no_request_material`) +- Modify: `crates/trusted-server-core/src/publisher.rs` (the existing cold/warm end-to-end response-state test) + +- [ ] **Step 1: Run the current focused baseline.** + +Run: + +```bash +cargo test-fastly template_cache::tests::rendered_key_is_fixed_size_and_contains_no_request_material +cargo test-fastly publisher::c2_end_to_end_tests::a_second_request_is_served_from_the_cache_without_touching_the_origin +``` + +Expected: both commands PASS against the old `ts-c2-v4-...` key and `x-ts-c2-cache` header, establishing that the existing behavior is green before changing expected contracts. + +- [ ] **Step 2: Add the deterministic new namespace expectation first.** + +In `platform/template_cache.rs`, change the fixed-length expectation to use `ts-template-cache-v{TEMPLATE_SCHEMA_VERSION}-`, assert that the rendered key starts with that prefix, and add an exact fixture assertion for `key()`: + +```rust +assert_eq!( + key().to_cache_key(), + "ts-template-cache-v4-54431eb4ea82644d6378717a8c3f18302fafbf739e684598da79e392b16900a6" +); +``` + +This exact value proves both the visible prefix and the canonical hash-domain bytes changed; it must not be replaced by a length-only assertion. + +- [ ] **Step 3: Add the public-header expectation first.** + +In the cold/warm publisher test, read `x-ts-template-cache` using `HeaderName::from_static` and assert the old `x-ts-c2-cache` header is absent on both the cold and warm responses. Keep the existing `miss-stored` and `hit` values and origin/body assertions unchanged. Use the new raw header name in this test before renaming the production constant so the test compiles and fails at the observable contract. + +- [ ] **Step 4: Run the focused contracts and verify they fail for the intended old values.** + +Run: + +```bash +cargo test-fastly template_cache::tests::rendered_key_is_fixed_size_and_contains_no_request_material +cargo test-fastly publisher::c2_end_to_end_tests::a_second_request_is_served_from_the_cache_without_touching_the_origin +``` + +Expected: the key test FAILS with the old `ts-c2-v4-...` output (including the old digest), and the publisher test FAILS because the response still emits `x-ts-c2-cache` instead of `x-ts-template-cache`. No policy, body, or cache-state assertion should fail for another reason. + +- [ ] **Step 5: Commit the red contract tests.** + +Do not commit source implementation changes yet. Commit only the two focused test expectation changes: + +```bash +git add crates/trusted-server-core/src/platform/template_cache.rs crates/trusted-server-core/src/publisher.rs +git commit -m "Specify template cache namespace and header" +``` + +### Task 2: Migrate the opaque template-cache key namespace + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs` +- Test: `crates/trusted-server-core/src/platform/template_cache.rs` + +- [ ] **Step 1: Replace both namespace components and terminology in the implementation.** + +Change only the namespace inputs/labels and active prose: hash `b"ts-template-cache"` instead of `b"ts-c2"`, render `ts-template-cache-v{schema_version}-{digest}`, and describe the shared transformed template without the C1/C2/C3 numbered taxonomy. Keep `TEMPLATE_SCHEMA_VERSION` at `4`, keep the exact historical v3 marker, keep surrogate keys, and keep all key fields/order and hash algorithm unchanged. + +- [ ] **Step 2: Run the key-focused suite.** + +Run: + +```bash +cargo test-fastly template_cache +``` + +Expected: PASS, including the exact deterministic namespace assertion, fixed-length/key-format assertion, all-field-distinctness tests, delimiter-collision test, Vary case/order tests, metadata tests, and reservation tests. The old namespace must not be accepted as an alias or read path. + +- [ ] **Step 3: Commit the namespace boundary.** + +```bash +git add crates/trusted-server-core/src/platform/template_cache.rs +git commit -m "Move template cache keys to named namespace" +``` + +### Task 3: Rename publisher state, policy APIs, logs, and public diagnostics + +**Files:** + +- Modify: `crates/trusted-server-core/src/publisher.rs` +- Test: `crates/trusted-server-core/src/publisher.rs` + +- [ ] **Step 1: Rename the response-state API and all active internal identifiers.** + +Use `TemplateCacheResponseState`, `HEADER_X_TS_TEMPLATE_CACHE`, `set_template_cache_response_state`, `TemplateCacheBypassReason`, `TemplateCachePolicy`, `request_bypasses_template_cache`, `template_cache_bypass_reason`, and `template_cache_ttl`. Rename local `c2_*` variables and the `c2_store_authorization_tests`, `c2_end_to_end_tests`, and `c2_gate_tests` modules to `template_cache_*`. Replace every active `c2_template_cache` log prefix with `template_cache`; preserve the same bounded values (`hit`, `miss-stored`, `miss-store-error`, `miss-reserved`, `bypass-request`, `bypass-response`, `unsupported`, `invalid`, `backend-error`). Rewrite comments/assertion messages to “template cache” without changing logic. + +- [ ] **Step 2: Emit only the new public header.** + +Make the renamed setter insert `HeaderValue::from_static(state.as_str())` under `x-ts-template-cache`. Do not emit `x-ts-c2-cache` as an alias. Update all publisher tests that use the constant or literal to the renamed constant/new literal, while retaining the explicit old-header-absent assertion from Task 1. + +- [ ] **Step 3: Run the focused publisher suites.** + +Run: + +```bash +cargo test-fastly template_cache_store_authorization_tests +cargo test-fastly template_cache_end_to_end_tests +cargo test-fastly template_cache_gate_tests +``` + +Expected: PASS. Cold, warm, reserved, bypass, unsupported, invalid, and backend-error states retain their existing values; miss/hit assembly, origin counts, privacy headers, diagnostics bypass, policy gates, and body identity remain unchanged. The new header is present for each relevant state and the old header is absent. + +- [ ] **Step 4: Commit the publisher boundary.** + +```bash +git add crates/trusted-server-core/src/publisher.rs +git commit -m "Name template cache diagnostics" +``` + +### Task 4: Finish active Rust terminology and seam fixtures + +**Files:** + +- Modify: `crates/trusted-server-core/src/html_processor.rs` +- Modify: `crates/trusted-server-core/src/creative_opportunities.rs` +- Modify: `crates/trusted-server-core/src/platform/types.rs` +- Modify: `crates/trusted-server-core/src/response_privacy.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/template_cache.rs` +- Modify: `crates/trusted-server-adapter-fastly/src/esi_assembly.rs` + +- [ ] **Step 1: Rename the active seam fixture and supporting prose.** + +Change `reserved-c2-seam` to `reserved-template-cache-seam` in the HTML processor test and its source/collision assertions. Rewrite comments and rustdoc that call the shared transformed template “C2”; use “template cache” or “shared transformed template.” Do not touch the exact `ts-c2-v3` schema-history marker in `platform/template_cache.rs`. + +- [ ] **Step 2: Rename Fastly adapter log/comment terminology.** + +Change the legacy read warning to `template_cache legacy read failed` and update Fastly/ESI rustdoc. Do not change error classification, cache key construction (which already consumes the core key), transaction behavior, or assembly output. + +- [ ] **Step 3: Run the focused supporting suites.** + +Run: + +```bash +cargo test-fastly html_processor +cargo test-fastly template_cache +cargo test-fastly publisher +``` + +Expected: PASS; the renamed fixture still proves the transform-owned terminal seam, the key suite still proves the new namespace, and publisher behavior remains unchanged apart from terminology/header names. + +- [ ] **Step 4: Commit the remaining Rust terminology.** + +```bash +git add crates/trusted-server-core/src/html_processor.rs crates/trusted-server-core/src/creative_opportunities.rs crates/trusted-server-core/src/platform/types.rs crates/trusted-server-core/src/response_privacy.rs crates/trusted-server-adapter-fastly/src/template_cache.rs crates/trusted-server-adapter-fastly/src/esi_assembly.rs +git commit -m "Describe shared templates consistently" +``` + +### Task 5: Rename the local harness and its CI caller + +**Files:** + +- Rename: `scripts/c2-local-test.sh` → `scripts/template-cache-local-test.sh` +- Modify: `scripts/template-cache-local-test.sh` +- Modify: `.github/workflows/test.yml` + +- [ ] **Step 1: Rename the script without creating a compatibility shim.** + +Use `git mv scripts/c2-local-test.sh scripts/template-cache-local-test.sh`. Update usage comments, `c2_state` to `template_cache_state`, the `x-ts-c2-cache` parser to `x-ts-template-cache`, all `c2_template_cache` log patterns to `template_cache`, and prose describing the inert marker. Keep `esi` and `inline` argument behavior and all timing/body/origin-count assertions intact. + +- [ ] **Step 2: Update CI callers.** + +Rename the workflow step to “Run template cache ESI local harness” and update both CI commands to `BID_DELAY=3 ./scripts/template-cache-local-test.sh esi` and `BID_DELAY=3 ./scripts/template-cache-local-test.sh inline`. + +- [ ] **Step 3: Run shell and harness verification.** + +Run: + +```bash +bash -n scripts/template-cache-local-test.sh +BID_DELAY=3 ./scripts/template-cache-local-test.sh esi +BID_DELAY=3 ./scripts/template-cache-local-test.sh inline +``` + +Expected: syntax check PASS; both harness modes PASS, with cold `miss-stored`, warm `hit`, new `X-TS-Template-Cache` parsing, expected origin counts, seam/assembly integrity, and no old-header/log matches. If local Viceroy prerequisites are unavailable, record that environmental block explicitly and run the same commands in CI before completion; do not add a legacy script shim. + +- [ ] **Step 4: Commit the harness/CI boundary.** + +```bash +git status --short scripts .github/workflows/test.yml +git add -A -- scripts .github/workflows/test.yml +git commit -m "Rename template cache local harness" +``` + +### Task 6: Update active operator documentation, examples, plans, and specs + +**Files:** + +- Modify: `trusted-server.example.toml` +- Modify: `docs/guide/configuration.md` +- Modify: the nine active plans/specs listed in the file map + +- [ ] **Step 1: Update the operator example and guide.** + +Replace numbered-cache prose with “template cache,” update all diagnostic examples to `X-TS-Template-Cache`, and update harness commands to `scripts/template-cache-local-test.sh`. Preserve the configuration keys, safety caveats, bounded state values, rollback instructions, `ts-template` surrogate key, and all behavior descriptions. Explain raw origin caching/final assembly by those names where an old C1/C2 taxonomy was used. + +- [ ] **Step 2: Update active plans/specs mechanically but semantically.** + +Rename cache-related `C2`, `c2_*`, `x-ts-c2-cache`, and `scripts/c2-local-test.sh` references to the named terminology, including completed checklist text and historical findings. Rewrite sentences that distinguish cache layers in terms of raw origin bytes, shared template cache, and final assembled response. Leave unrelated IDs, hashes, cookie/EC identifiers, and third-party content unchanged. + +- [ ] **Step 3: Verify documentation formatting and active references.** + +Run: + +```bash +cd docs && npm run format +cd .. +rg -n -i --glob '!docs/superpowers/archive/**' --glob '!docs/superpowers/specs/2026-08-19-template-cache-terminology-design.md' --glob '!docs/superpowers/plans/2026-08-19-template-cache-terminology.md' 'X-TS-C2-Cache|x-ts-c2-cache|ts-c2|c2_template_cache|C2Response|C2Bypass|C2Cache|c2_bypass|c2_cache|c2-local-test|reserved-c2-seam|\bC2\b|\bc2\b' crates/trusted-server-core/src/platform/template_cache.rs crates/trusted-server-core/src/publisher.rs crates/trusted-server-core/src/html_processor.rs crates/trusted-server-core/src/creative_opportunities.rs crates/trusted-server-core/src/platform/types.rs crates/trusted-server-core/src/response_privacy.rs crates/trusted-server-adapter-fastly/src scripts trusted-server.example.toml docs/guide docs/superpowers/plans docs/superpowers/specs +``` + +Expected: exactly three results are retained: the exact +`` schema-history marker in +`platform/template_cache.rs`, plus the two negative `x-ts-c2-cache` compatibility-test +literals in `publisher.rs` that prove the old header is absent on cold and warm +responses. Inspect those three results rather than weakening the search. The excluded +migration design and implementation plan may retain their explicit old/new compatibility +references; archived documents and unrelated substrings are not migration failures. + +- [ ] **Step 4: Commit the documentation boundary.** + +```bash +git add trusted-server.example.toml docs/guide/configuration.md docs/superpowers/plans/2026-08-08-1009-measurement-and-stage-0.md docs/superpowers/plans/2026-08-08-1009-measurement-findings.md docs/superpowers/plans/2026-08-12-1009-esi-merge-hardening.md docs/superpowers/plans/2026-08-14-1009-esi-parser-assembly.md docs/superpowers/plans/2026-08-19-pr-1013-review-remediation.md docs/superpowers/specs/2026-08-11-1009-streaming-assembly-architecture.md docs/superpowers/specs/2026-08-12-1009-esi-merge-hardening-design.md docs/superpowers/specs/2026-08-14-1009-esi-parser-assembly-design.md docs/superpowers/specs/2026-08-19-pr-1013-review-remediation-design.md +git commit -m "Use template cache terminology in documentation" +``` + +### Task 7: Run full verification and review the migration diff + +**Files:** + +- Test/verify: all files changed by Tasks 1–6 + +- [ ] **Step 1: Run Rust formatting and target-matched tests.** + +Run: + +```bash +cargo fmt --all -- --check +cargo test-fastly +cargo test-axum +cargo test-cloudflare +cargo test-spin +cargo fmt --manifest-path crates/trusted-server-integration-tests/Cargo.toml -- --check +cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity +``` + +Expected: all commands PASS. The Fastly suite is the authoritative shared-template-cache test target; Axum and Cloudflare confirm the terminology/API changes do not break adapters that use the unavailable-cache fallback. + +- [ ] **Step 2: Run target-matched Clippy and JS tests/formatting.** + +Run: + +```bash +cargo clippy-fastly +cargo clippy-axum +cargo clippy-cloudflare +cargo clippy-cloudflare-wasm +cargo clippy-spin-native +cargo clippy-spin-wasm +cargo clippy --manifest-path crates/trusted-server-integration-tests/Cargo.toml --all-targets -- -D warnings +cd crates/trusted-server-js/lib && npx vitest run && npm run format && node build-all.mjs +cd ../../.. +``` + +Expected: all commands PASS with no new warnings, and JavaScript tests/formatting remain green; no JS behavior should have changed. + +- [ ] **Step 3: Re-run the renamed harness and documentation search.** + +Run: + +```bash +BID_DELAY=3 ./scripts/template-cache-local-test.sh esi +BID_DELAY=3 ./scripts/template-cache-local-test.sh inline +rg -n -i --glob '!docs/superpowers/archive/**' --glob '!docs/superpowers/specs/2026-08-19-template-cache-terminology-design.md' --glob '!docs/superpowers/plans/2026-08-19-template-cache-terminology.md' 'X-TS-C2-Cache|x-ts-c2-cache|ts-c2|c2_template_cache|C2Response|C2Bypass|C2Cache|c2_bypass|c2_cache|c2-local-test|reserved-c2-seam|\bC2\b|\bc2\b' crates/trusted-server-core/src/platform/template_cache.rs crates/trusted-server-core/src/publisher.rs crates/trusted-server-core/src/html_processor.rs crates/trusted-server-core/src/creative_opportunities.rs crates/trusted-server-core/src/platform/types.rs crates/trusted-server-core/src/response_privacy.rs crates/trusted-server-adapter-fastly/src scripts trusted-server.example.toml docs/guide docs/superpowers/plans docs/superpowers/specs +git diff --check +git status --short +``` + +Expected: both harness modes PASS; the active-cache search returns exactly the three +retained results (the v3 schema-history marker and the two negative `x-ts-c2-cache` +compatibility-test literals); `git diff --check` PASS; and `git status --short` is empty +(no stale `scripts/c2-local-test.sh`, generated artifacts, or unrelated edits). +Confirm the only other retained old spellings live in the excluded migration design, +implementation plan, archived records, and explicitly unrelated substrings. + +- [ ] **Step 4: Review the final diff before handoff.** + +Use `git diff HEAD~6..HEAD --stat` and `git diff HEAD~6..HEAD --` (adjust the commit range if additional logical commits were made) to confirm the changes are terminology-only: no schema-version bump, no dual header, no old-namespace read, no policy/TTL/eligibility change, no template-byte change, and no script shim. Follow `@superpowers:verification-before-completion` and report command evidence before claiming completion. diff --git a/docs/superpowers/plans/2026-08-20-pr-1013-round-3-review-remediation.md b/docs/superpowers/plans/2026-08-20-pr-1013-round-3-review-remediation.md new file mode 100644 index 000000000..ad8f36e05 --- /dev/null +++ b/docs/superpowers/plans/2026-08-20-pr-1013-round-3-review-remediation.md @@ -0,0 +1,448 @@ +# PR #1013 Round-3 Review Remediation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Resolve every actionable round-2 and round-3 PR #1013 review finding and return the branch with complete local verification and a rerun browser-integration check. + +**Architecture:** Preserve the existing publisher, Fastly terminal-hook, template-cache, and GPT scheduler boundaries. Complete the privacy invariant through a shared response marker helper, preserve GPT one-shot state on the shared `tsjs` object, explicitly discharge failed Fastly reservations, and make the remaining test/comment/doc changes locally without broad refactoring. + +**Tech Stack:** Rust 2024, `error-stack`, Fastly Core Cache/Viceroy, TypeScript, Vitest/jsdom, Cargo target aliases, GitHub Actions/CLI. + +--- + +## File Map + +- `crates/trusted-server-core/src/response_privacy.rs`: own generic terminal-private stamping and keep the synthesized-HTML wrapper. +- `crates/trusted-server-core/src/publisher.rs`: apply terminal-private marking to page-bids/invalid-304 responses; add page-bids and ESI coverage; repair cache terminology comments. +- `crates/trusted-server-adapter-fastly/src/main.rs`: prove late filter effects cannot weaken a page-bids response carrying the marker. +- `crates/trusted-server-adapter-fastly/src/template_cache.rs`: cancel invalid reservations, preserve error context, test released obligations, and correct partial-write documentation. +- `crates/trusted-server-core/src/platform/template_cache.rs`: complete CR/LF rejection coverage. +- `crates/trusted-server-core/src/html_processor.rs`: clarify CSP nonce safety net and collision assertion text. +- `crates/trusted-server-core/src/integrations/gpt_bootstrap.js`: use shared document-level latch state. +- `crates/trusted-server-js/lib/src/core/types.ts`: type and document the internal latch. +- `crates/trusted-server-js/lib/src/integrations/gpt/index.ts`: preserve the latch across fallback-to-bundle scheduler replacement. +- `crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts`: mirror slot semantics and cover bootstrap state. +- `crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts`: cover the real bootstrap-to-bundle handoff. +- `docs/guide/configuration.md`: qualify request-side `max-age` bypass wording. + +### Task 1: Complete terminal-private page-bids coverage + +**Files:** + +- Modify: `crates/trusted-server-core/src/response_privacy.rs:92` +- Modify: `crates/trusted-server-core/src/publisher.rs:4413,6014,6028,6365,17438` +- Modify: `crates/trusted-server-adapter-fastly/src/main.rs:580` + +- [ ] **Step 1: Write failing marker tests for page-bids response paths** + +In `publisher.rs`, add assertions using: + +```rust +assert!( + response + .extensions() + .get::() + .is_some(), + "page-bids response should remain terminal-private after late response effects" +); +``` + +Cover `page_bids_preflight_denied`, `page_bids_unknown_format`, and the successful JSON response returned by `run_page_bids_response`. Extend the invalid-origin-304 test to require the same marker. + +- [ ] **Step 2: Run the focused tests and verify failure** + +Run: + +```bash +cargo test-fastly page_bids -- --nocapture +cargo test-fastly eligible_navigation_rejects_unexpected_origin_304 -- --nocapture +``` + +Expected: the new marker assertions fail because these paths stamp only the header. + +- [ ] **Step 3: Add a generic terminal-private helper** + +In `response_privacy.rs`, add: + +```rust +pub(crate) fn enforce_terminal_private_cache_privacy(response: &mut Response) { + enforce_private_no_store(response); + response.extensions_mut().insert(TerminalPrivateResponse); +} +``` + +Change `enforce_synthesized_html_cache_privacy` to delegate to it. Keep both functions `pub(crate)` so no public API is added. + +- [ ] **Step 4: Route all affected response paths through the helper** + +Replace direct `Cache-Control: private, no-store` insertion for preflight denial, unknown format, successful page-bids JSON, and the invalid-origin-304 rebuild with `enforce_terminal_private_cache_privacy(&mut response)`. Preserve status, content type, body, and deprecated-alias headers. + +- [ ] **Step 5: Add the Fastly late-effects regression test** + +Construct a real page-bids denial response through `trusted_server_core::publisher::page_bids_preflight_denied`, apply `RequestFilterEffects` that sets public `Cache-Control` and CDN cache headers, then call `apply_terminal_response_effects`. Assert the terminal result is exactly `private, no-store`, has no validators/CDN cache headers, and retains the marker-driven behavior. Together with the core successful-JSON marker test, this pins the per-user JSON path without exporting test-only constructors. + +- [ ] **Step 6: Run focused and adapter tests** + +Run: + +```bash +cargo test-fastly page_bids -- --nocapture +cargo test-fastly late_filter_effects_cannot_make -- --nocapture +cargo test-fastly eligible_navigation_rejects_unexpected_origin_304 -- --nocapture +cargo test-axum +cargo test-cloudflare +cargo test-spin +``` + +Expected: all pass. + +- [ ] **Step 7: Commit** + +```bash +git add crates/trusted-server-core/src/response_privacy.rs crates/trusted-server-core/src/publisher.rs crates/trusted-server-adapter-fastly/src/main.rs +git commit -m "Keep page bids responses terminal private" +``` + +### Task 2: Preserve the GPT scheduler latch across handoff + +**Files:** + +- Modify: `crates/trusted-server-core/src/integrations/gpt_bootstrap.js:82` +- Modify: `crates/trusted-server-js/lib/src/core/types.ts:430` +- Modify: `crates/trusted-server-js/lib/src/integrations/gpt/index.ts:648` +- Modify: `crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts:122` +- Modify: `crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts:150` + +- [ ] **Step 1: Mirror the missing bootstrap slot-contract tests** + +Add tests equivalent to the bundle suite: + +```typescript +it('fallback scheduler preserves head-injected slots when initialSlots is omitted', () => { + // Seed ts.adSlots, call scheduleInitialAdInit(bids), assert the same slots remain. +}) + +it('fallback scheduler replaces existing slots when initialSlots is explicitly empty', () => { + // Seed stale slots, call scheduleInitialAdInit({}, []), assert []. +}) +``` + +- [ ] **Step 2: Write a failing bootstrap-to-bundle handoff test** + +In `schedule_initial_ad_init.test.ts`, evaluate the verbatim bootstrap source, call its scheduler once, import the GPT module so it replaces the scheduler, then call the bundle scheduler with different bids/slots. Assert the first payload remains and only one load/double-rAF chain invokes `adInit`. + +- [ ] **Step 3: Run the focused Vitest files and verify the handoff test fails** + +Run: + +```bash +cd crates/trusted-server-js/lib +npx vitest run test/integrations/gpt/gpt_bootstrap.test.ts test/integrations/gpt/schedule_initial_ad_init.test.ts +``` + +Expected: bootstrap slot tests pass against current behavior; handoff test fails because importing the bundle creates a fresh closure latch. + +- [ ] **Step 4: Type the shared internal state** + +Add to `TsjsApi` near `navGeneration`: + +```typescript +/** Internal one-shot state shared by bootstrap and bundle scheduler installs. */ +initialAdInitScheduled?: boolean; +``` + +Use the existing internal-field naming convention; do not expose a new callable API. + +- [ ] **Step 5: Replace both closure latches with shared state** + +In bootstrap JavaScript: + +```javascript +if ((ts.navGeneration || 0) !== 0 || ts.initialAdInitScheduled) return +ts.initialAdInitScheduled = true +``` + +In the TypeScript bundle: + +```typescript +if ((ts.navGeneration ?? 0) !== 0 || ts.initialAdInitScheduled) return +ts.initialAdInitScheduled = true +``` + +Update durable comments to say the state is one-shot per document and survives fallback-to-bundle scheduler replacement. + +- [ ] **Step 6: Run JS tests, build, and formatting verification** + +Run: + +```bash +cd crates/trusted-server-js/lib +npx vitest run test/integrations/gpt/gpt_bootstrap.test.ts test/integrations/gpt/schedule_initial_ad_init.test.ts +npx vitest run +node build-all.mjs +npm run format +``` + +Expected: all pass, generated bundles build successfully, and Prettier reports every JS/TS file already formatted. + +- [ ] **Step 7: Commit** + +```bash +git add crates/trusted-server-core/src/integrations/gpt_bootstrap.js crates/trusted-server-js/lib/src/core/types.ts crates/trusted-server-js/lib/src/integrations/gpt/index.ts crates/trusted-server-js/lib/test/integrations/gpt/gpt_bootstrap.test.ts crates/trusted-server-js/lib/test/integrations/gpt/schedule_initial_ad_init.test.ts +git commit -m "Preserve initial ad scheduler latch across handoff" +``` + +### Task 3: Explicitly cancel invalid Fastly reservations + +**Files:** + +- Modify: `crates/trusted-server-adapter-fastly/src/template_cache.rs:90,242,355` + +- [ ] **Step 1: Write failing obligation-release tests** + +For separate cold keys, obtain `TemplateCacheLookup::Reserved`, then: + +1. call `insert` with mismatched `body_len`; +2. call `insert` with metadata whose `content_type` contains `\n`. + +Assert each error contains the original validation reason. Immediately call `lookup_or_reserve` for the same key and require `Reserved`, proving the first transaction was canceled rather than left pending. + +- [ ] **Step 2: Write an error-composition unit test** + +Extract a private generic result mapper that accepts the validation error and a cancellation result. With an injected `Err("simulated cancellation failure")`, assert the returned `TemplateCacheError` text contains both the original validation reason and simulated cancellation failure. + +- [ ] **Step 3: Run Fastly cache tests and verify failure** + +Run: + +```bash +cargo test-fastly template_cache -- --nocapture +``` + +Expected: re-reservation tests fail or time out under the current implicit-drop behavior; the mapper test fails to compile until implemented. + +- [ ] **Step 4: Implement cancellation with preserved context** + +Create the validation error first, call `self.transaction.cancel_insert_or_update()`, and pass both values through the private mapper: + +```rust +fn invalid_reservation_result( + validation_error: TemplateCacheError, + cancellation: Result<(), E>, +) -> Result<(), TemplateCacheError> { + match cancellation { + Ok(()) => Err(validation_error), + Err(error) => Err(backend_error(format!( + "{validation_error}; cancelling invalid cache reservation also failed: {error:?}" + ))), + } +} +``` + +Use it on both pre-insert validation branches. Do not alter the transaction after `insert` consumes it. + +- [ ] **Step 5: Correct the direct-write partial-entry comment** + +State that `finish()` is deliberately skipped and readers reject partial content through fallible reads and the post-read check against declared `body_len`; do not claim the entry has no known length. + +- [ ] **Step 6: Run Fastly tests** + +Run: + +```bash +cargo test-fastly template_cache -- --nocapture +cargo test-fastly +``` + +Expected: all pass without blocked transaction lookups. + +- [ ] **Step 7: Commit** + +```bash +git add crates/trusted-server-adapter-fastly/src/template_cache.rs +git commit -m "Cancel invalid template cache reservations" +``` + +### Task 4: Close focused coverage and documentation gaps + +**Files:** + +- Modify: `crates/trusted-server-core/src/platform/template_cache.rs:1102` +- Modify: `crates/trusted-server-core/src/html_processor.rs:725,2151` +- Modify: `crates/trusted-server-core/src/publisher.rs:1656,2078,5492,8225,8739,9246` +- Modify: `docs/guide/configuration.md:1430` + +- [ ] **Step 1: Add the missing metadata cases** + +Extend `metadata_encoding_rejects_line_break_injection` with a policy-header name containing `\r` and a `content_type` containing `\n`. Keep all four string-field cases in the same table-driven assertion. + +- [ ] **Step 2: Restore both ESI forms end to end** + +Parameterize `publisher_esi_comment_is_never_stored_or_executed` over: + +```rust +[ + "", + "publisher", +] +``` + +For each form, use a distinct cold cache/stub or distinct URL so each iteration independently proves bypass, byte preservation, no cache entry, and zero assembler calls. + +- [ ] **Step 3: Repair CSP and seam comments** + +Above the `[nonce]` handler, explain that meta `content` matching is supplemental because `lol_html` does not decode entity-encoded quotes; the structural `[nonce]` handler is the load-bearing refusal for any nonce an element can consume. Update the HTML processor assertion and publisher collision fixture rustdoc to describe terminal seam emission, repeated-marker rejection, and cache bypass. + +- [ ] **Step 4: Add the C1/C3 glossary and re-anchor references** + +Near the first surviving publisher cache reference, add a concise glossary: + +```rust +// C1 is Fastly's raw origin/read-through cache. C3 is the forbidden cache of a +// final per-user assembled response. The template cache sits between them. +``` + +Rewrite the remaining references so each is intelligible locally and does not mix the old C2 taxonomy with “template cache.” + +- [ ] **Step 5: Correct request max-age documentation** + +Change the fail-closed inventory to “positive or malformed request `max-age`, `min-fresh`” so it agrees with the `max-age=0` reload paragraph. + +- [ ] **Step 6: Run focused tests and formatting** + +Run: + +```bash +cargo test-fastly metadata_encoding_rejects_line_break_injection -- --nocapture +cargo test-fastly publisher_esi -- --nocapture +cargo test-fastly html_processor -- --nocapture +cargo fmt --all -- --check +cd docs +npx prettier --write guide/configuration.md +npm run format +``` + +Expected: all tests pass and formatters report no changes needed after formatting. + +- [ ] **Step 7: Commit** + +```bash +git add crates/trusted-server-core/src/platform/template_cache.rs crates/trusted-server-core/src/html_processor.rs crates/trusted-server-core/src/publisher.rs docs/guide/configuration.md +git commit -m "Close template cache review gaps" +``` + +### Task 5: Run the complete local CI gate + +**Files:** none unless a verification failure reveals an in-scope defect. + +- [ ] **Step 1: Verify the worktree diff and formatting** + +Run: + +```bash +git status --short +git diff --check main...HEAD +cargo fmt --all -- --check +cd crates/trusted-server-js/lib && npm run format +cd docs && npm run format +``` + +Expected: only planned changes exist; every formatter passes. + +- [ ] **Step 2: Run all target-matched tests** + +Run: + +```bash +cargo test-fastly +cargo test-axum +cargo test-cloudflare +cargo test-spin +cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity +./scripts/test-cli.sh +``` + +Expected: all pass. If only the CLI helper lacks a documented local prerequisite, record that explicitly; parity is required locally or must be confirmed green in CI. + +- [ ] **Step 3: Run all target-matched clippy gates** + +Run: + +```bash +cargo clippy-fastly +cargo clippy-axum +cargo clippy-cloudflare +cargo clippy-cloudflare-wasm +cargo clippy-spin-native +cargo clippy-spin-wasm +``` + +Expected: all pass with warnings denied. + +- [ ] **Step 4: Run final JS verification** + +Run: + +```bash +cd crates/trusted-server-js/lib +npx vitest run +node build-all.mjs +``` + +Expected: all tests and builds pass. + +- [ ] **Step 5: Inspect the final diff** + +Run: + +```bash +git status --short --branch +git diff --stat origin/1009-esi-cacheable-root-spec...HEAD +git log --oneline origin/1009-esi-cacheable-root-spec..HEAD +``` + +Expected: the diff contains only the approved remediation and its spec/plan commits. + +### Task 6: Update the PR and rerun browser integration + +**Files:** none. + +- [ ] **Step 1: Push the reviewed commits** + +Run: + +```bash +git push origin 1009-esi-cacheable-root-spec +``` + +Expected: the PR head advances to the final local commit. + +- [ ] **Step 2: Locate the PR checks and browser workflow run** + +Run: + +```bash +gh pr view --json number,url,headRefOid,statusCheckRollup +gh pr checks +``` + +Identify the new-head `browser integration tests` check and its workflow run ID. Do not rerun an obsolete-head run. + +- [ ] **Step 3: Rerun only the failed browser job if needed** + +Before any rerun, mechanically verify the selected workflow run belongs to the current PR head: + +```bash +PR_HEAD_SHA=$(gh pr view --json headRefOid --jq .headRefOid) +RUN_HEAD_SHA=$(gh run view --json headSha --jq .headSha) +test "$RUN_HEAD_SHA" = "$PR_HEAD_SHA" +gh run rerun --job +``` + +Obtain `` from the selected run's jobs and only run the final command if the SHA comparison succeeds. This reruns the browser job alone rather than every failed job in the workflow. If the new push does not automatically schedule the browser job, locate the new-head workflow run rather than rerunning the old canceled run. Monitor until terminal state. + +Expected: green browser integration tests. If Playwright installation again consumes the job timeout before any browser launches, capture the run URL and exact failure phase as infrastructure evidence; do not change workflow caching or `timeout-minutes` without separate approval. + +- [ ] **Step 4: Report final verification and review mapping** + +Summarize each resolved finding, the local gate results, the browser-check result, commit hashes, and any infrastructure-only limitation. Do not claim completion until all required local gates and the relevant remote check have terminal evidence. diff --git a/docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md b/docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md new file mode 100644 index 000000000..98af3e176 --- /dev/null +++ b/docs/superpowers/specs/2026-06-26-server-side-ad-template-cli-design.md @@ -0,0 +1,842 @@ +# Server-Side Ad Template CLI Design + +**Date:** 2026-06-26 +**Status:** Draft design +**Scope:** Static and browser-backed CLI diagnostics for server-side ad templates + +## 1. Goal + +Add Trusted Server CLI support for server-side ad-template onboarding and +verification without resurrecting the stale standalone `ts-config` design. + +The CLI must answer two operator questions: + +1. Given an effective `trusted-server.toml`, which configured ad-template slots + match this path? +2. Given one or more live publisher URLs, are the configured slots for the + final navigated paths actually present on the page according to DOM, GPT, + and provider evidence, and do any runtime gates explain why Trusted Server + would not inject or auction for that page? + +The command surface is split by whether the command is local-config-only or +browser-backed: + +```bash +ts config ad-templates lint +ts config ad-templates match +ts config ad-templates check +ts config ad-templates explain + +ts audit ad-templates verify ... +``` + +Static commands live under `ts config` because they only load local effective +app config. Browser-backed verification lives under `ts audit` because it loads +public publisher pages in Chrome/Chromium and observes live page behavior. + +## 2. Context + +This design replaces the stale PR #724 direction. + +PR #724 designed a standalone `ts-config` binary around a +`creative-opportunities.toml` file. That is no longer the project shape: + +- Trusted Server configuration now flows through the unified `ts` CLI from PR + #799. +- Server-side ad-template slots live under `[creative_opportunities]` / + `[[creative_opportunities.slot]]` in `trusted-server.toml`. +- Effective config can include EdgeZero app-config environment overlays unless + `--no-env` is passed. +- Operator-owned `trusted-server.toml` is ignored; the repository tracks + `trusted-server.example.toml`. + +PR #799 is the CLI base. It owns the `ts` binary, EdgeZero lifecycle delegates, +and typed app-config validation/push/diff behavior. + +PR #800 is the audit dependency. It adds the generic browser-backed +`ts audit ` collector using local Chrome/Chromium. At the time this spec +was written, PR #800 was stale relative to the latest #799 head, so this work +depends on the #800 audit collector after it is rebased onto the latest #799 +typed blob-config model. + +## 3. Non-Goals + +- Do not add a standalone `ts-config` binary. +- Do not reintroduce `creative-opportunities.toml`. +- Do not implement browser-backed generation in Phase 1. +- Do not mutate `trusted-server.toml` from `verify`. +- Do not probe PBS, GAM, or APS management APIs. +- Do not require EdgeZero platform adapters for local static diagnostics. +- Do not make `ts audit ad-templates verify` push, provision, deploy, or update + platform resources. +- Do not rely on real GPT or APS network calls in tests. + +Browser-backed generation is a later phase: + +```bash +ts audit ad-templates generate ... +``` + +That phase needs separate rules for slot ID derivation, page-pattern inference, +multi-URL merging, TOML ordering, and whether the command emits a patch, a draft +file, or full config blocks. + +## 4. Command Surface + +### 4.1 Shared Config Flags + +All `ts config ad-templates ...` commands and +`ts audit ad-templates verify` accept the same local app-config flags: + +```bash +--app-config +--manifest +--no-env +``` + +Defaults match PR #799: + +| Option | Default | +| -------------- | ------------------------------------------------ | +| `--app-config` | `.toml`, resolved from `edgezero.toml` | +| `--manifest` | `edgezero.toml` | +| `--no-env` | `false`; app-config env overlay is applied | + +If an explicit `--app-config` path is supplied and missing, the command reports +that path as the error. It must not silently fall back to an environment or +manifest-derived path. + +### 4.2 Static Config Diagnostics + +```bash +ts config ad-templates lint [--app-config ] [--manifest ] [--no-env] +``` + +Reports whether `[creative_opportunities]` is configured, how many slots exist, +GAM network ID, auction timeout, auction enablement, configured auction +providers, and whether current EdgeZero routing will fall back to the legacy +path when configured slots are present. + +```bash +ts config ad-templates match [--details] ... +``` + +Normalizes a path or full URL to a path and reports the slots matched by the +runtime `creative_opportunities::match_slots` logic. `--details` includes slot +div ID, GAM unit path, page patterns, formats, and configured providers. + +```bash +ts config ad-templates check \ + (--expected-slot ... | --expect-no-slots) \ + [--allow-extra-slots] ... +``` + +CI-friendly assertion wrapper around the same matching logic. + +```bash +ts config ad-templates explain \ + [--method GET] \ + [--non-navigation] \ + [--prefetch] \ + [--bot] \ + [--consent-denied] \ + [--edgezero-enabled] ... +``` + +Explains the major runtime gates that decide whether the server-side ad stack +would run for a page request. This is a local model, not a live request replay. + +### 4.3 Browser-Backed Verification + +```bash +ts audit ad-templates verify ... \ + [--app-config ] \ + [--manifest ] \ + [--no-env] \ + [--strict] \ + [--json] \ + [--scroll] +``` + +Behavior: + +- Accept one or more `http` or `https` URLs. +- Reject all other schemes before launching a browser. +- Load the effective Trusted Server app config. +- For each URL, navigate first, collect the final URL, normalize the final URL + to a path, and call `creative_opportunities::match_slots`. +- Preserve the requested URL/path separately from the final URL/path. +- Emit a redirect warning when the final path differs from the requested path. +- Expect only the slots matched for the final URL path to be present on that + live page. +- Report live DOM/GPT/APS ad-slot evidence that does not correspond to a + matched configured slot as structured extra evidence. +- Launch Chrome/Chromium through the audit collector from the rebased #800 work. +- Inject a read-only ad-template collector before publisher scripts run. +- Compare configured matched slots against DOM, GPT, and APS evidence. +- Report runtime ad-stack gate evidence separately from placement evidence. +- Print human output by default. +- Emit stable machine-readable output with `--json`. +- Exit `0` by default for missing or partial live evidence; this is an + auditor-assist mode. +- Exit non-zero under `--strict` when a matched configured slot is missing or + only partially confirmed. + +`--scroll` performs a deterministic scroll pass after initial load and settle. +It is opt-in because it is slower and can trigger additional page behavior. +Slots first observed during scroll count as confirmed when the GPT evidence is +otherwise sufficient. + +## 5. Confirmation Model + +The verifier compares configured expected slots to live page evidence. + +It must keep three concepts separate: + +1. **Static slot matching:** which configured slots match a URL path according + to `creative_opportunities::match_slots`. +2. **Runtime ad-stack eligibility:** whether Trusted Server would run its + server-side ad stack for the audited navigation. This mirrors + `should_run_server_side_ad_stack` for the initial publisher request and the + `/__ts/page-bids` kill-switch/consent behavior for SPA route updates. +3. **Live placement evidence:** what the browser actually observes on the + rendered page through DOM, GPT, and APS evidence. + +`verify` is primarily a live placement verifier. `--strict` fails when matched +configured slots for an eligible page are missing or partial. Runtime gates are +reported so operators can distinguish "the slot is not on the page" from "the +current request/config would intentionally suppress Trusted Server ad-template +injection or page-bids slot output". + +### 5.1 Expected Slots + +For each input URL: + +1. Navigate the browser to the requested URL. +2. Record `requested_url`, `requested_path`, `final_url`, and `final_path`. +3. Match configured slots through the core runtime matcher using `final_path`. +4. Build an expected-slot record for each matched slot: + - slot ID; + - resolved div ID; + - resolved GAM unit path; + - configured formats; + - configured providers; + - matching page patterns. + +Only these expected slots are verified for that page. For example, slots whose +only pattern is `/` are expected for the homepage path, not for `/news/story`. + +When a navigation redirects, `verify` uses the final path for expected slots and +reports the requested path in output. This matches runtime behavior: Trusted +Server evaluates the actual publisher request path it handles, not the URL the +operator typed before redirects. + +### 5.2 Runtime Gate Evidence + +For each page result, `verify` reports a local runtime-gate model: + +| Gate | Source | +| ------------------------ | -------------------------------------------------------------------------------------------------------- | +| `method_get` | Browser navigation request; expected to pass for normal `verify`. | +| `navigation` | Browser navigation request; expected to pass for normal `verify`. | +| `not_prefetch` | Browser request headers; expected to pass unless the collector is extended with prefetch simulation. | +| `not_bot` | Browser User-Agent checked against the runtime bot fragments. | +| `matched_slots` | Final-path slot matching. | +| `auction_enabled` | Effective `[auction].enabled` / orchestrator enablement from app config. | +| `consent_allows_auction` | `unknown` unless the collector can prove a consent-allowed or consent-denied state for the live request. | + +`runtime_ad_stack_expected` is a three-state value: `yes`, `no`, or `unknown`. +Known blocking gates produce page warnings and set +`runtime_ad_stack_expected = "no"`. Unknown gates set +`runtime_ad_stack_expected = "unknown"` but do not by themselves fail +`--strict`. + +If `runtime_ad_stack_expected = "no"` because of a known config/request gate +such as `[auction].enabled = false`, strict mode does not fail missing GPT/APS +evidence for that page. The page result is reported as skipped for runtime +verification while still showing the static expected slots and any live +placement evidence that was observed. + +If `runtime_ad_stack_expected = "yes"` or `"unknown"`, strict mode applies the +normal missing/partial placement rules from §5.6. + +For SPA routes, `/__ts/page-bids` returns no slots when the ad-stack kill switch +or consent gate blocks the stack. Browser verification should report observed +page-bids responses when available, but it must not require real partner bids in +tests. + +Live ad-slot evidence that does not map to a matched expected slot is reported +as structured extra evidence. Extra evidence can identify publisher-owned slots +that have not yet moved into server-side ad templates, slots whose +`page_patterns` are too narrow, or slots that should stay outside Trusted +Server. It does not make `--strict` fail in Phase 1. + +### 5.3 DOM Slot Resolution + +The verifier must mirror the runtime GPT bootstrap's slot-root resolution: + +1. Try `document.getElementById(slot.div_id)`. +2. If absent, find the first element with an ID that starts with `slot.div_id`. +3. Ignore elements whose ID ends with `-container`. + +This is required because `div_id` may intentionally be a stable prefix for +framework-generated IDs, for example `ad-header-0-`. + +### 5.4 GPT Evidence + +A slot is confirmed by GPT evidence when the live page exposes a GPT slot whose: + +- ad unit path equals the configured resolved GAM unit path; +- slot element ID equals the resolved DOM element ID or an existing + `${resolved_dom_id}-container` element used by Trusted Server when defining + its own slot; +- configured sizes are compatible with the observed GPT sizes. + +The collector should observe both direct `googletag.defineSlot` calls and +post-load `googletag.pubads().getSlots()` state. + +Size compatibility is defined for Phase 1 as follows: + +- Normalize configured sizes from `CreativeOpportunityFormat` values where + `media_type = "banner"` into `(width, height)` pairs. +- Normalize observed GPT sizes from `defineSlot` input and `getSizes()` output: + - `[300, 250]` becomes one `(300, 250)` pair. + - `[[300, 250], [728, 90]]` becomes two pairs. + - non-numeric values such as `"fluid"` are ignored for numeric matching and + reported as warnings. +- A GPT slot's sizes are compatible when the configured banner size set and the + observed numeric GPT size set have at least one pair in common. +- Extra observed GPT sizes do not block confirmation, but they are reported as + warnings so auditors can decide whether to add formats to config. +- Configured banner sizes that are not observed do not block confirmation when + at least one configured size was observed, but they are reported as warnings. +- If ad unit path and div match but no numeric size overlap exists, the slot is + `partial`, not `confirmed`. +- Configured `video` and `native` formats are not used for Phase 1 GPT size + confirmation. A matched slot with only non-banner formats is `unconfirmable` + with an unsupported-format warning and does not fail `--strict`. +- A sizeless live GPT slot is `partial` when the config declares banner sizes, + because that is observable drift and must fail `--strict`. + +### 5.5 APS Evidence + +Phase 1 does not wrap or collect `apstag.fetchBids`: APS is server-side provider +configuration and client-side calls are neither required nor authoritative for +the runtime ad-template decision. + +### 5.6 Statuses + +| Status | Meaning | +| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `confirmed` | GPT evidence matches the configured GAM unit path, div resolution, and compatible sizes. | +| `partial` | The page has some evidence for the configured slot, but not enough to confirm it. This includes DOM-only evidence, GPT path/div matches with incompatible sizes, GPT path/div matches for unsupported non-banner-only configured formats, and other non-confirming GPT evidence. | +| `missing` | No DOM or GPT evidence confirms the configured slot. | +| `unconfirmable` | The checker cannot evaluate the configured format with Phase 1 evidence, such as a non-banner-only slot. This is reported but does not fail strict mode. | + +In `--strict` mode: + +- `missing` fails. +- `partial` fails. +- `unconfirmable` does not fail. + +Provider issues are not statuses. They are warnings attached to the slot result. +For example, a slot can be `confirmed` and still carry a warning that configured +APS evidence was missing or ambiguous. Provider warnings do not fail `--strict` +unless a future `--strict-providers` flag is added. + +## 6. Architecture + +The architecture should keep command parsing thin and move ad-template behavior +into pure, testable modules. + +```text +crates/trusted-server-cli/src/ + app_config.rs + ad_templates/ + mod.rs + expected.rs + compare.rs + output.rs + config_ad_templates.rs + audit/ + page.rs + browser.rs + ad_templates.rs +``` + +### 6.1 `app_config.rs` + +Shared loader for effective Trusted Server app config. + +Responsibilities: + +- read `edgezero.toml` through EdgeZero manifest helpers; +- resolve the default `.toml` path; +- apply EdgeZero app-config env overlay unless `--no-env`; +- return `TrustedServerAppConfig` / `Settings`; +- report errors in the same terms as #799 config commands. + +This avoids duplicating config path and env-overlay behavior between +`ts config ad-templates ...` and `ts audit ad-templates verify`. + +The current branch already has a private loader in `config_ad_templates.rs`. +Before adding browser-backed verification, move that behavior into this shared +module and route the existing static commands through it so both command +families load the same effective config. + +### 6.2 `ad_templates::expected` + +Pure local expected-slot model. + +Responsibilities: + +- normalize path-or-URL input; +- call `creative_opportunities::match_slots`; +- convert matched slots into stable expected-slot structs; +- preserve deterministic ordering by slot order from config. + +This module must not compile glob patterns independently or duplicate matching +semantics. + +If richer pattern diagnostics are needed, add a small helper to +`trusted-server-core::creative_opportunities` and use it from both runtime and +CLI. + +### 6.3 `ad_templates::compare` + +Pure comparison between expected slots and collected browser evidence. + +Responsibilities: + +- implement DOM prefix matching rules; +- compare GPT path, div, and size evidence; +- compare APS evidence; +- collect unmatched live DOM/GPT/APS ad-slot evidence as structured + `extra_evidence`; +- assign `confirmed`, `partial`, `missing`, and provider warning details; +- decide strict failure status. + +This module should be testable without launching Chrome. + +### 6.4 `ad_templates::output` + +Human and JSON output model. + +Responsibilities: + +- serialize stable JSON output; +- keep arrays ordered by input URL, then configured slot order, then provider + name; +- render concise human summaries; +- avoid leaking page HTML, cookies, local storage, or arbitrary page data. + +### 6.5 `config_ad_templates.rs` + +Thin Clap adapter for `ts config ad-templates ...`. + +Responsibilities: + +- parse command arguments; +- call `app_config` and `ad_templates::expected`; +- delegate formatting to `ad_templates::output`; +- keep no browser-specific logic. + +### 6.6 `audit::browser` + +Shared browser utility extracted from or aligned with the rebased #800 audit +collector. + +Responsibilities: + +- locate Chrome/Chromium; +- launch an isolated profile; +- reject non-HTTP(S) URLs before navigation; +- set bounded navigation and settle timeouts; +- run optional init scripts; +- perform optional deterministic scroll; +- collect final URL, title, rendered scripts, resource entries, and optional + ad-template evidence. + +The generic `ts audit ` command from #800 should continue to work without +ad-template verification enabled. + +### 6.7 `audit::ad_templates` + +Browser-backed verifier orchestration. + +Responsibilities: + +- parse `ts audit ad-templates verify`; +- load effective config through `app_config`; +- compute expected slots for each URL; +- run the browser collector with ad-template evidence enabled; +- call `ad_templates::compare`; +- print human or JSON output; +- apply default auditor-assist exit behavior and `--strict` behavior. + +## 7. Browser Collector + +The ad-template collector is injected before page scripts run. It is read-only: +it records evidence and calls original page functions with unchanged arguments. + +The rebased #800 collector must grow a pre-navigation init-script hook before it +can satisfy this spec. The stale #800 collector only navigates, waits, and reads +post-load page state; that is insufficient for GPT/APS call evidence. + +Instrumentation requirements: + +- install the collector through the browser's "evaluate on new document" / + init-script mechanism before navigation; +- serialize only configured div prefixes and provider IDs needed for matching; +- observe pages that create `window.googletag = { cmd: [] }` after injection; +- wrap `googletag.cmd.push` callbacks without changing callback order; +- record direct `googletag.defineSlot` calls and calls executed from the GPT + command queue; +- read final `googletag.pubads().getSlots()` state after settle and after + scroll; +- observe pages that assign `window.apstag` after injection and wrap + `apstag.fetchBids` when present; +- tolerate pages that never load GPT or APS and report warnings instead of + throwing collector errors. + +Evidence to collect: + +- DOM elements with IDs relevant to configured slot div prefixes; +- calls to `googletag.defineSlot`; +- final `googletag.pubads().getSlots()` state after settle and after scroll; +- calls to `apstag.fetchBids`; +- timestamps or phases indicating whether evidence was observed during + `initial_load` or `scroll`. + +The collector must not: + +- block, rewrite, or suppress publisher scripts; +- override `navigator.webdriver`; +- capture cookies, local storage, session storage, request bodies, or arbitrary + page data; +- require real GPT/APS network calls in test fixtures. + +## 8. JSON Output Contract + +`--json` emits deterministic JSON. Shape: + +```json +{ + "ok": true, + "strict": false, + "pages": [ + { + "url": "https://www.example.com/news/story", + "final_url": "https://www.example.com/news/story", + "requested_path": "/news/story", + "path": "/news/story", + "runtime_ad_stack_expected": "unknown", + "gates": { + "method_get": "pass", + "navigation": "pass", + "not_prefetch": "pass", + "not_bot": "pass", + "matched_slots": "pass", + "auction_enabled": "pass", + "consent_allows_auction": "unknown" + }, + "matched_slot_count": 1, + "slots": [ + { + "id": "atf", + "status": "confirmed", + "phase": "initial_load", + "configured": { + "div_id": "ad-atf-", + "gam_unit_path": "/123/news/atf", + "formats": [ + { "width": 300, "height": 250, "media_type": "banner" } + ], + "providers": ["aps"] + }, + "evidence": { + "dom_id": "ad-atf-0", + "gpt": { + "gam_unit_path": "/123/news/atf", + "div_id": "ad-atf-0", + "sizes": [[300, 250]] + } + }, + "warnings": [] + } + ], + "extra_evidence": [], + "warnings": [] + } + ], + "warnings": [] +} +``` + +Warning entries are objects with stable `code` and human-readable `message` +fields. Human output may print only the message. JSON consumers must not need to +parse warning strings. + +Extra live evidence is structured: + +```json +{ + "kind": "gpt", + "phase": "initial_load", + "dom_id": "ad-right-rail-0", + "gam_unit_path": "/123/publisher/right-rail", + "sizes": [[300, 250]], + "reason": "no_configured_slot_matched" +} +``` + +Allowed `kind` values for Phase 1 are `dom` and `gpt`. + +Strict-mode failures with page results use the same shape and set `ok` to +`false`. Example partial slot: + +```json +{ + "ok": false, + "strict": true, + "pages": [ + { + "url": "https://www.example.com/", + "final_url": "https://www.example.com/", + "requested_path": "/", + "path": "/", + "runtime_ad_stack_expected": "unknown", + "gates": { + "method_get": "pass", + "navigation": "pass", + "not_prefetch": "pass", + "not_bot": "pass", + "matched_slots": "pass", + "auction_enabled": "pass", + "consent_allows_auction": "unknown" + }, + "matched_slot_count": 1, + "slots": [ + { + "id": "homepage-header", + "status": "partial", + "phase": "initial_load", + "configured": { + "div_id": "ad-header-0-", + "gam_unit_path": "/123/homepage/header", + "formats": [{ "width": 728, "height": 90, "media_type": "banner" }], + "providers": ["aps"] + }, + "evidence": { + "dom_id": "ad-header-0-_R_abc123", + "gpt": null + }, + "warnings": [ + { + "code": "dom_without_gpt", + "message": "DOM element matched, but no GPT slot evidence was observed" + } + ] + } + ], + "extra_evidence": [], + "warnings": [] + } + ], + "warnings": [] +} +``` + +For errors that occur before any page result can be produced, the command exits +non-zero and prints the normal CLI error. JSON error output can be added later +if the base CLI standardizes it. + +For multi-URL runs, browser/navigation failures after argument validation are +page-level failures when possible. The command continues to the remaining URLs, +sets top-level `ok` to `false`, and includes a page result: + +```json +{ + "url": "https://www.example.com/broken", + "final_url": null, + "requested_path": "/broken", + "path": null, + "error": { + "code": "navigation_failed", + "message": "failed to read main document navigation response" + }, + "slots": [], + "extra_evidence": [], + "warnings": [] +} +``` + +Invalid schemes are still rejected before browser launch for the whole command, +because they are argument errors rather than page collection results. + +## 9. Error Handling + +Static commands fail when: + +- config cannot be loaded; +- `[creative_opportunities]` is malformed; +- CLI assertions in `check` fail. + +Browser verification fails when: + +- config cannot be loaded; +- any URL is not HTTP(S); +- Chrome/Chromium cannot be found or launched; +- all navigations fail before any page result can be collected; +- at least one page-level error occurs in a multi-URL run; +- command output cannot be written; +- `--strict` is set, runtime verification is not skipped by a known gate, and + at least one matched slot is missing or partial. `unconfirmable` is excluded. + +Browser collection can still produce a page result with warnings when: + +- page settle times out; +- a navigation redirects before final URL matching; +- scroll evidence is incomplete; +- GPT is not loaded; +- extra live DOM/GPT ad-slot evidence has no matched configured slot; +- no slots match the URL. + +## 10. Testing + +Static tests: + +- parse every `ts config ad-templates` command; +- load temp `edgezero.toml` and temp `trusted-server.toml`; +- verify `--app-config`, `--manifest`, and `--no-env` behavior; +- verify `/`, `/news/*`, and full URL normalization behavior; +- verify `check` success and failure output. +- verify the existing static command loader uses the shared `app_config` module. + +Pure comparison tests: + +- exact DOM ID match; +- prefix DOM ID match for framework-generated suffixes; +- ignore `-container` elements; +- GPT confirms by GAM unit path, div ID, and compatible sizes; +- DOM-only creates `partial`; +- no DOM/GPT creates `missing`; +- APS match creates no provider warning; +- APS missing/ambiguous creates provider warnings; +- `--strict` fails only missing and partial slots. + +Browser fixture tests: + +- local HTML fixture with direct `googletag.defineSlot`; +- fixture using `googletag.cmd.push`; +- fixture assigning `window.googletag` after collector injection; +- fixture with delayed/lazy slot observed only with `--scroll`; +- fixture with APS `fetchBids`; +- fixture assigning `window.apstag` after collector injection; +- redirect fixture that matches expected slots on final path; +- multi-URL fixture where one URL fails and one URL returns page results; +- fixture where `[auction].enabled = false` reports runtime skipped instead of + strict missing-slot failure; +- invalid non-HTTP(S) URL rejection before browser launch; +- JSON contract tests for warning codes, `extra_evidence`, page errors, + deterministic ordering, `partial`, `missing`, and strict failures; +- fixture with no real GPT/APS network dependency. + +Verification commands: + +```bash +cargo test --workspace +cargo fmt --all -- --check +cargo clippy --workspace --all-targets --all-features -- -D warnings +cargo test --package trusted-server-cli --target +``` + +## 11. Branch And PR Plan + +The implementation should not be built on stale #724. + +Recommended dependency order: + +1. Land or rebase PR #799 as the CLI base. +2. Rebase PR #800 onto the latest #799 head so `ts audit` uses the current typed + blob app-config model. +3. Harden and refactor the existing static `ts config ad-templates ...` + diagnostics on top of the current server-side ad-template branch and #799: + extract the private config loader into `app_config`, move pure expected-slot + logic into `ad_templates::expected`, and keep existing behavior covered by + tests. +4. Extend the rebased #800 collector with pre-navigation init scripts, + ad-template evidence hooks, optional scroll, page-level errors, and bounded + structured output. +5. Build `ts audit ad-templates verify` on top of that collector and the + server-side ad-template branch. +6. Keep `generate` for a separate Phase 2 spec and PR. + +If delivery needs to be split, static diagnostics can land before browser-backed +verification. Browser-backed verification should not duplicate the #800 browser +collector. + +## 12. CLI Namespace Decision + +`ts audit ad-templates verify` is the final command shape for browser-backed +ad-template verification. + +When this work is combined with the rebased #800 audit command, `ts audit` +should become a subcommand namespace: + +```bash +ts audit page +ts audit generate +ts audit ad-templates verify ... +``` + +The existing #800 `ts audit ` behavior should be preserved as a +compatibility alias for `ts audit generate ` during the transition, +including its artifact output flags. This avoids a successful but silent +behavior change for existing onboarding scripts. + +Parsing contract: + +- `ts audit page ` is the canonical generic page-audit command. +- `ts audit generate ` is the canonical artifact-generation command. +- `ts audit ad-templates verify ...` is the canonical ad-template verifier. +- `ts audit ` is a hidden compatibility alias for + `ts audit generate ` and is accepted only when `` parses as `http` + or `https`. +- `ts audit ad-templates` must never be treated as a legacy URL positional. +- `ts audit page` without a URL must fail with the normal Clap missing-argument + error. + +Implementation shape: + +```rust +#[derive(Debug, clap::Args)] +struct AuditArgs { + #[command(subcommand)] + command: Option, + #[arg(value_parser = parse_http_url, hide = true)] + legacy_url: Option, +} + +#[derive(Debug, clap::Subcommand)] +enum AuditSubcommand { + Page(PageAuditArgs), + #[command(name = "ad-templates", subcommand)] + AdTemplates(AuditAdTemplatesCommand), +} +``` + +If Clap cannot enforce the optional-subcommand plus hidden positional contract +cleanly, implement a small custom dispatcher for the `audit` argv tail and test +it directly. Required parser tests: + +- `ts audit https://www.example.com/` dispatches to artifact generation; +- `ts audit page https://www.example.com/` dispatches to page audit; +- `ts audit ad-templates verify https://www.example.com/` dispatches to + ad-template verification; +- `ts audit ad-templates` does not parse as a URL; +- `ts audit ftp://www.example.com/` fails before browser launch. + +JSON error output is intentionally left to the broader CLI output contract. This +spec only standardizes successful verification result JSON and strict-mode +verification failure JSON where page results exist. diff --git a/docs/superpowers/specs/2026-07-06-cache-control-header-design.md b/docs/superpowers/specs/2026-07-06-cache-control-header-design.md new file mode 100644 index 000000000..1924381f1 --- /dev/null +++ b/docs/superpowers/specs/2026-07-06-cache-control-header-design.md @@ -0,0 +1,145 @@ +# Define Cache-Control strategy for TS edge environment + +**Labels:** `enhancement`, `edge`, `performance`, `caching` +**Area:** Trusted Server runtime (Fetch from Origin / Edge to Clients) + +## Summary + +Trusted Server needs a structured cache policy that separates browser caching from edge/shared caching. Today several TS-owned responses use short, hard-coded cache headers even when the URL is content-versioned. The initial implementation should focus on TS-owned and explicitly fingerprinted assets, especially TSJS bundles. + +Dynamic HTML/template caching, streaming fixes, SSAT compression offload, and Akamai-specific cache behavior are follow-up work and are not part of this cache-header slice. + +## Goals + +- Express cache policy as structured data instead of ad hoc header strings. +- Independently control browser TTL and edge/shared-cache TTL. +- Serve TSJS bundles with immutable caching when the requested URL hash matches the served bytes. +- Keep non-versioned or config-dependent stable URLs out of long-lived immutable browser caches. +- Provide configurable cache-rule presets for known fingerprinted framework paths, starting with Next.js `/_next/static/*`. +- Keep SSAT-assembled ad-stack HTML private and out of shared caches. +- Emit the correct MVP runtime edge-cache headers from the shared policy. + +## Non-goals for this slice + +These are tracked separately: + +- True SSAT publisher streaming and parser-safe assembly: #857. +- SSAT HTML compression offload via `Accept-Encoding: identity` and `X-Compress-Hint`: #858. +- Origin-template caching, transformed-template caching, and dynamic HTML/RSC/API cache-key design: #859. +- Akamai-specific `CDN-Cache-Control` / `Edge-Control` / Property Manager behavior. + +## Background: two cache tiers + +The request path has two cacheable hops: + +```text +Origin ──▶ TS edge/shared cache ──▶ Browser cache +``` + +- **Edge/shared cache:** controlled by `s-maxage` or runtime-specific edge headers. + - Fastly: `Surrogate-Control` + - Cloudflare: `CDN-Cache-Control` / `Cloudflare-CDN-Cache-Control` + - Portable fallback: `s-maxage` inside `Cache-Control` +- **Browser cache:** controlled by `max-age` and related `Cache-Control` directives. + +A single `max-age` cannot express “hold at the edge for a year, but revalidate in the browser daily” or the reverse. TS should model these tiers separately and let adapters render the appropriate headers. Header rendering alone does not enable a runtime cache: Cloudflare Workers Cache must be enabled, and Fastly synthetic/final egress responses require explicit cache integration tracked in [#908](https://github.com/IABTechLab/trusted-server/issues/908). + +## Policy model + +Add a shared cache-policy model along these lines: + +```text +match +edge_ttl +browser_ttl +stale_while_revalidate +stale_if_error +immutable +visibility: public | private +enabled +``` + +Rules should be configurable. Built-in framework presets, such as Next.js `/_next/static/*`, should be represented as default rules in this same model rather than hard-coded in adapters. Operators must be able to disable or override presets and add publisher-specific allowlists. + +## Target behavior by response class + +| Response class | Target policy | Notes | +| --------------------------------------------------------------- | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| TSJS with matching `?v=` | `Cache-Control: public, max-age=31536000, immutable` plus runtime edge header | The serving path must validate that `v` matches the bytes served. | +| TSJS missing/mismatched `?v=` | Short TTL or redirect to canonical hashed URL | Do not mark immutable. | +| TSJS deferred modules, including Prebid | Same as TSJS hash-matching policy | Example: `/static/tsjs=tsjs-prebid.min.js?v=`. | +| Publisher Prebid URL neutralized by TS | `no-store` or very short TTL | The empty compatibility shim is config-dependent and served at a stable publisher URL. Do not cache it for a year. | +| Enabled framework preset static, e.g. Next.js `/_next/static/*` | `Cache-Control: public, max-age=31536000, immutable` | Applied through configurable preset/allowlist rules. | +| Fastly TS-fingerprinted rehosted asset | `Cache-Control: public, max-age=31536000, immutable` | Safe only when TS owns the fingerprinted URL. A matched rehost rule is authoritative over third-party origin cache defaults. | +| Stable Fastly TS-owned/rehosted URL | Conservative short browser TTL, optional longer edge TTL | Do not use `immutable`; later TS/operator `private` or `no-store` finalization remains a veto. | +| Arbitrary publisher-origin CSS/JS/images | Origin-controlled by default | TS may upgrade only via enabled framework preset or publisher allowlist. | +| SSAT-assembled ad-stack HTML | `Cache-Control: private, max-age=0`; strip runtime edge-cache headers | Must never enter shared cache because it can contain per-user slot/bid data. | +| Dynamic HTML/RSC/API | Origin-controlled in this slice | Future dynamic caching belongs to #859. | + +## TSJS-specific requirements + +Current TSJS URLs already include a content hash query string, for example: + +```text +/static/tsjs=tsjs-unified.min.js?v= +/static/tsjs=tsjs-prebid.min.js?v= +``` + +The current serving path still emits a short cache policy. Update it so that: + +- hash-matching requests emit one-year immutable browser caching; +- hash-matching requests emit the runtime edge header with equivalent long edge TTL; +- missing or mismatched hash requests do not receive immutable caching; +- cache-key configuration preserves the `v` query parameter; +- TSJS hashes used in injected URLs are generated at build time or cached so HTML injection does not re-concatenate and re-hash large bundles per pageview; +- `Vary: Accept-Encoding` remains on compressed/static responses; +- ETags may remain as a fallback for clients or intermediaries that revalidate anyway. + +Fastly and Cloudflare include query strings in default cache keys, but TS must still avoid any project-specific query normalization that drops `v` for `/static/tsjs=`. + +## SSAT HTML privacy requirement + +SSAT-assembled ad-stack HTML can contain per-user data such as slot state or bid data. It must remain: + +```http +Cache-Control: private, max-age=0 +``` + +and must strip runtime edge-cache headers, including: + +```http +Surrogate-Control +Fastly-Surrogate-Control +CDN-Cache-Control +Cloudflare-CDN-Cache-Control +``` + +This requirement applies to the browser-facing assembled response. Origin-template caching is separate follow-up work in #859. + +## Runtime header mapping for MVP + +Adapters should render the shared policy as follows: + +| Runtime | Edge/shared-cache header | +| ----------------- | ---------------------------------------------------- | +| Fastly | `Surrogate-Control` | +| Cloudflare | `CDN-Cache-Control` / `Cloudflare-CDN-Cache-Control` | +| Portable fallback | `s-maxage` in `Cache-Control` | + +These mappings define emitted directives, not storage by themselves. The runtime must enable or implement the corresponding shared-cache mechanism. Akamai mapping is deferred until Akamai is on the roadmap. + +## Acceptance criteria + +- [x] Cache policy is represented as structured fields, not hard-coded header strings. +- [x] Built-in framework presets, including a disableable/overrideable Next.js `/_next/static/*` rule, are implemented through the shared cache-policy rule engine. +- [x] TSJS hash-matching requests for unified and deferred modules emit `public, max-age=31536000, immutable` plus the runtime edge header. +- [x] TSJS missing/mismatched hash requests do not get immutable caching. +- [x] TSJS hash generation is build-time or cached enough that HTML injection does not re-concatenate/re-hash the bundle per pageview. +- [ ] Runtime cache-key configuration preserves the `v` query parameter for `/static/tsjs=`. Runtime verification is tracked in #908. +- [x] Neutralized publisher Prebid shim responses use `no-store` or a short TTL, not a year-long policy. +- [x] Arbitrary publisher-origin assets remain origin-controlled unless covered by an enabled framework preset or publisher allowlist. +- [x] Fastly TS-owned rehosted assets have explicit normalized cache policy; immutable is used only for TS-fingerprinted rehosted URLs. +- [x] SSAT-assembled ad-stack HTML continues to emit `private, max-age=0` and strips all runtime edge-cache headers. +- [x] Fastly and Cloudflare adapters emit the correct edge-cache header from the shared policy, with portable `s-maxage` fallback where needed. Actual shared-cache storage remains tracked in #908. +- [x] Dynamic HTML/RSC/API Vary/cache-key normalization is not hard-coded in this PR and is deferred to #859. +- [x] SSAT streaming fixes and compression offload are not included in this PR and remain tracked by #857 and #858. diff --git a/docs/superpowers/specs/2026-07-20-ssat-debug-comment-config-design.md b/docs/superpowers/specs/2026-07-20-ssat-debug-comment-config-design.md new file mode 100644 index 000000000..1c73fd668 --- /dev/null +++ b/docs/superpowers/specs/2026-07-20-ssat-debug-comment-config-design.md @@ -0,0 +1,488 @@ +# SSAT Debug Comment Configuration Design + +**Date:** 2026-07-20 + +**Status:** Proposed security revision; awaiting approval + +**Issue:** [IABTechLab/trusted-server#935](https://github.com/IABTechLab/trusted-server/issues/935) — "For SSAT, make debug comment configurable" + +## Summary + +The server-side auction template (SSAT) can inject a `` +HTML comment before the bids `