Skip to content

Make Scheduling policy publication compare-and-swap #1192

Description

@jeremi

What Happened

Two Scheduling instances can read the same stored policy revision and digest, then concurrently start with different compatible policy digests. apply_policy serializes both publications but does not compare the locked row with the baseline each process observed. The first writes revision N+1, the second writes N+2, and the second exits because it had activated hooks for N+1. The first remains alive under stale N+1 and refuses new commitments against N+2.

This fails closed for mutations and does not corrupt the capacity ledger, but it can leave the deployment unable to accept bookings until the N+2 instance restarts.

Expected Behavior

Policy publication should act as compare-and-swap. Under the existing scheduling_meta FOR UPDATE lock, a process should publish only when the stored revision and digest still match its observed baseline. Reapplying a digest another process already published should remain idempotent.

Reproduction

  1. Capture one Scheduling metadata baseline (revision N, digest A).
  2. From that baseline, concurrently or sequentially simulate two startup attempts carrying different digests B and C.
  3. Observe that both publications currently succeed as N+1 and N+2.

Add a PostgreSQL regression proving one divergent publication wins and the loser makes no metadata or retained-policy write. Keep a same-digest concurrency regression.

Environment

Found while reviewing PR #1092 at commit 68e3060fb. This is follow-up operational hardening for overlapping divergent deployments; supervised restart converges on the published policy.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions