What Happened
Two Scheduling instances can read the same stored policy revision and digest, then concurrently start with different compatible policy digests. apply_policy serializes both publications but does not compare the locked row with the baseline each process observed. The first writes revision N+1, the second writes N+2, and the second exits because it had activated hooks for N+1. The first remains alive under stale N+1 and refuses new commitments against N+2.
This fails closed for mutations and does not corrupt the capacity ledger, but it can leave the deployment unable to accept bookings until the N+2 instance restarts.
Expected Behavior
Policy publication should act as compare-and-swap. Under the existing scheduling_meta FOR UPDATE lock, a process should publish only when the stored revision and digest still match its observed baseline. Reapplying a digest another process already published should remain idempotent.
Reproduction
- Capture one Scheduling metadata baseline
(revision N, digest A).
- From that baseline, concurrently or sequentially simulate two startup attempts carrying different digests B and C.
- Observe that both publications currently succeed as N+1 and N+2.
Add a PostgreSQL regression proving one divergent publication wins and the loser makes no metadata or retained-policy write. Keep a same-digest concurrency regression.
Environment
Found while reviewing PR #1092 at commit 68e3060fb. This is follow-up operational hardening for overlapping divergent deployments; supervised restart converges on the published policy.
What Happened
Two Scheduling instances can read the same stored policy revision and digest, then concurrently start with different compatible policy digests.
apply_policyserializes both publications but does not compare the locked row with the baseline each process observed. The first writes revision N+1, the second writes N+2, and the second exits because it had activated hooks for N+1. The first remains alive under stale N+1 and refuses new commitments against N+2.This fails closed for mutations and does not corrupt the capacity ledger, but it can leave the deployment unable to accept bookings until the N+2 instance restarts.
Expected Behavior
Policy publication should act as compare-and-swap. Under the existing
scheduling_meta FOR UPDATElock, a process should publish only when the stored revision and digest still match its observed baseline. Reapplying a digest another process already published should remain idempotent.Reproduction
(revision N, digest A).Add a PostgreSQL regression proving one divergent publication wins and the loser makes no metadata or retained-policy write. Keep a same-digest concurrency regression.
Environment
Found while reviewing PR #1092 at commit
68e3060fb. This is follow-up operational hardening for overlapping divergent deployments; supervised restart converges on the published policy.