Skip to content

[SURF-2672] fix(release): purge after jsDelivr's version list refreshes - #78

Merged
lilnasy merged 3 commits into
mainfrom
fix/release-purge-after-version-list
Nov 11, 2026
Merged

lilnasy merged 3 commits into
mainfrom
fix/release-purge-after-version-list

Conversation

@lilnasy

@lilnasy lilnasy commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Why

The purge loop merged in the first PR purges too early and too often.

  • jsDelivr caches a repo's version list for up to 10 minutes, and a purge does not refresh it. The maintainer advises checking x-jsd-version and purging again 10 minutes later (jsdelivr/jsdelivr#18376).
  • jsDelivr rate-limits purges. A throttled purge still answers HTTP 200, so the workflow counted it as done.
  • The loop could purge each file about 31 times in a few minutes, get throttled, and report success.
  • Re-running a failed run cut a second release.

This follows up the first PR. The surface_forms PR waits on this workflow's run.

What changes

  • The workflow is two jobs: release and purge. "Re-run failed jobs" repeats only the purge.
  • The purge job purges once, checks the version each @latest file reports, and purges once more 10 minutes later: every file still stale, and every file whose first purge failed.
  • A throttled or partly failed second purge fails the run. A failed first purge puts the file on the retry list.
  • The run fails with the stale files named when any file still reports an older version after the second purge.
  • @main is no longer purged. jsDelivr says purging works only for releases.

How it works

Before: one job. It re-purged stale files every 10 seconds.

flowchart LR
  subgraph job["release-and-purge"]
    direction LR
    r["Cut release"] --> p["Purge @latest and @main"] --> w["Check versions every 10 s<br/>(up to 30 rounds)"] --> a["Purge again"]
    w -->|stale file| x["Purge that file"] --> w
  end
Loading

After: two jobs. Stale files and files whose purge failed get one more purge, 10 minutes later.

flowchart LR
  subgraph release
    r["Cut release"]
  end
  subgraph purge
    direction LR
    p["Purge @latest"] --> c["Check versions"]
    p -->|purge failed| c
    c -->|any stale or failed| w["Wait 10 minutes,<br/>purge those files"] --> c2["Check again"]
  end
  release --> purge
Loading
  • The checks decide the result. The first purge step records the files whose purge failed, and the check step merges them with the files this runner sees as stale into one retry list. A file whose purge failed on one CDN provider is retried even when this runner's server already serves it fresh.
  • The purge script reads each purge response. It fails when the response says throttled or when any CDN provider reports failure.

Risk

  • Neither job has run on GitHub. The next Surface Tag release is the first run.
  • A release run takes about 10 minutes longer when any file is stale, or its purge failed, after the first purge.
  • A failed purge job leaves the release cut. Re-run failed jobs; the error says to wait an hour after a throttled purge.
  • @main URLs are no longer purged.
  • Roll back by reverting this PR.

QA

Exercised locally:

  • The workflow file parses and both scripts pass a shell syntax check.
  • The stale check ran against the live CDN. Asked for 1.1.19, it reported no stale files. Asked for 1.1.20, it reported all four.
  • The purge script ran against a stub curl: a clean response passes; a throttled response fails; one provider reporting failure fails; one failing URL fails after the other three are still purged.
  • The purge and check steps ran with stubs: one provider failing the purge of surface_tag.min.js while all four files already serve 1.1.19 puts only that file on the retry list; all clean and fresh gives an empty list; all clean and stale gives all four.

Left for the reviewer: nothing before merge. Watch the first real release after merge.

Known gaps

  • The check reads from one CDN location. One runner cannot reach every CDN provider.
  • The second purge can itself be throttled. It then fails the run.
Implementation details

Self review

The review of the first PR ran after it merged. Its fixes are in this PR.

  • Review of the merged PR: 15 findings, 12 fixed here: the wait window was shorter than jsDelivr's 10-minute version-list cache; re-purging every round trips the purge throttle; re-running a failed run cut another release; the purge response was discarded so throttled purges looked green; the first failed purge aborted the rest; no timeouts; purging @main did nothing useful; the timeout message said 5 minutes for an ~8-minute loop; probes downloaded whole bundles instead of HEAD; checkout ran before cutting the release; the release log claimed @latest already resolved; a comment narrated the steps. 3 declined: freshness is sampled from one CDN provider (one runner can't reach every provider; the second purge covers the rest); longer runs widen pending-run cancellation (needs three pushes within minutes, and Step 1 serializes publishes); the version probe is duplicated in surface_forms' verify (kept as an independent check from the publisher's side).
  • Final round on this PR: 3 findings. 1 fixed: a failed first purge skipped the checks. 2 declined: the second purge 10 minutes later may be throttled (it's the maintainer's recommended pattern, and a throttled purge now fails loudly); a re-run soon after a throttled purge fails again (the error says to re-run after an hour).

Changes

  • .github/workflows/release-and-purge.yml:
    • release (5-minute timeout) cuts the release and outputs version. It no longer checks out the repo. Its log line states the tag and commit.
    • purge (20-minute timeout) needs release. It checks out with persist-credentials: false.
      1. "Purge jsDelivr cache": all four files. Outputs failed, the files whose purge failed, and writes them to the run summary. The step itself never fails.
      2. "Check @latest serves vX": outputs retry, the stale files merged with failed, and writes the stale files to the run summary.
      3. "Wait 10 minutes, then purge those files again": only when retry is not empty. A failed purge fails the job.
      4. "Check @latest serves vX again": only when retry is not empty. Fails with the stale files named.
  • .github/scripts/purge-jsdelivr.sh purges @latest for each file. It reads each response with jq and fails on throttled or a provider flag that is false. It tries every file before it exits 1, prints the names of the files whose purge failed to stdout, and logs to stderr. curl has --max-time 30, --retry 3 and --retry-max-time 60.
  • .github/scripts/stale-jsdelivr-files.sh prints each file whose @latest copy does not report the given version. It sends a HEAD request with --max-time 20 and reads x-jsd-version. It logs each file's version to stderr.

Validation

  • The workflow YAML parses.
  • bash -n passes on both scripts.
  • stale-jsdelivr-files.sh against the live CDN: version 1.1.19 → no files printed; version 1.1.20 → all four printed.
  • purge-jsdelivr.sh against a stub curl: clean → exit 0 with 4 purged; throttled → exit 1; one provider false → exit 1; one URL failing → the other three still purged, exit 1.
  • Purge and check steps with stubs: Fastly fails the purge of surface_tag.min.js and all four HEAD probes return 1.1.19 → retry is surface_tag.min.js; all clean and fresh → empty; all clean and stale → all four.

@notion-workspace

Copy link
Copy Markdown

Comment thread .github/workflows/release-and-purge.yml Outdated

# The checks below decide the outcome; a file this purge missed is purged again.
- name: Purge jsDelivr cache
continue-on-error: true

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve a known purge failure when the local CDN sample is fresh

continue-on-error discards the first purge's failure, while the retry only runs for files the runner sees as stale. If one CDN provider reports a purge failure but the HEAD requests reach a provider already serving the expected version, stale is empty, both remaining steps skip, and the workflow succeeds even though customers on the failed provider can still receive old code. I reproduced this with stubbed purge responses reporting a provider failure and all four HEAD responses reporting the expected version. Preserve the failure unless the affected paths are successfully purged on retry; retry eligibility must account for failed purges as well as locally stale files. This differs from the accepted single-location sampling limitation because the purge failure is already known.

This message is written by Codex (GPT-6) on behalf of Gautam as part of the PR review.

@lilnasy
lilnasy merged commit 6a942e7 into main Nov 11, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants