Skip to content

air: opt-in GC of v2 snapshot objects with lease and reuse windows - #7417

Closed
stevenchen-db wants to merge 1 commit into
air-snapshot-v2-writerfrom
air-snapshot-v2-gc
Closed

stevenchen-db wants to merge 1 commit into
air-snapshot-v2-writerfrom
air-snapshot-v2-gc

Conversation

@stevenchen-db

Copy link
Copy Markdown
Contributor

PR C: CLI-side GC of v2 snapshot objects (design section 4: databricks-eng/universe PR 3017224). Stacked on #7405 (v2 writer).

Off by default. DATABRICKS_AIR_SNAPSHOT_GC enables it, read with env.GetBool (the repo's flag convention). With it unset, the submit path does nothing: the gate returns before any workspace call.

What it deletes. Only the v2 layout under .internal/: top-level *.air.json manifests and objects/<32 hex>.tar.gz. v1 tarballs, sidecars, and any other name are never listed for deletion.

  • A manifest older than L = 30 d is deleted.
  • An object older than A = 8 d that no manifest younger than L references is deleted.
  • Each delete is rechecked against its age just before it runs, so a manifest or object rewritten since the plan is kept.

Writer changes that GC depends on.

  • An object is reused only while younger than R = 7 d. An older one is written again, which refreshes its mtime.
  • An unchanged resubmit renews its manifest by rewriting it, once all the manifest's objects are present (checked in parallel). A missing object sends the submit down the full path, which repairs it.

Safety argument. A submit reusing an object at time T sees an mtime above T − R. GC deletes it only once its age reaches A, so not before T + (A − R) = T + 1 d. The submit writes its manifest within seconds, inside that day. TestV2GCRaceReusedObjectNotDeletableBeforeMargin checks the exact boundary with a fake clock.

When it runs. After the run is created, at most once per 24 h per snapshot directory (marker .air-gc.marker), time-boxed to 3 s. Every error is logged at debug and never changes the submit's result. A run that cannot finish reading every manifest deletes nothing, because an unread manifest could keep any object.

Interpretation to review. The manager asked that GC never slow a submit. The run is synchronous, since the CLI process exits right after the submit, so an opt-in submit pays one marker stat, and once a day it may pay up to the 3 s box. Running it in the background would almost never finish. Off by default, this costs nothing.

Residual risks.

  • A manifest renewed in the same instant it is deleted, at the 30 d boundary, can dangle. The renewal checks its objects before rewriting, so the result is a loud launch failure, not corruption.
  • The recheck before each delete leaves a window of one stat-to-delete round trip.

Dry run against df1 (.air/repo_snapshots/.internal, listing only, nothing deleted):

snapshot dir: /Workspace/Users/s.chen@databricks.com/.air/repo_snapshots/.internal
top-level: v1 tarballs+sidecars=128  v2 manifests=0  other=0 (untouched)
would delete manifests=0 objects=0 kept objects=0

Tests (fake filer with a fake clock; go test -race ./cmd/air/ -skip TestIndexStrategy passes; golangci-lint on cmd/air and libs/testserver: 0 issues)

  • Off by default: nothing is listed, and a non-true value stays off. Enabled by true.
  • Race boundary: an object reused at T is kept for every time before T + (A − R), and deletable just after.
  • Leased manifest keeps its objects (younger than L); an abandoned manifest and its objects are deleted.
  • Orphans: deleted after A, kept before.
  • v1 files, sidecars, notes, and non-hex object names are untouched, even when ancient.
  • Apply rechecks age: a manifest or object rewritten after the plan is kept.
  • Time box: a listing that blocks past the deadline deletes nothing.
  • Marker throttling: a run within 24 h does nothing; one after 24 h runs.
  • Writer: reuses fresh objects, rewrites older ones, renews the manifest mtime, repairs a missing object.

Not verified

  • Acceptance: the full ./acceptance run fails in this environment before any test runs. The prerequisites check needs jq 1.7+ (this machine has 1.6) and rejects temp dirs inside a git repo. TestAccept/air passes.
  • The unchanged-resubmit path now adds a manifest read, 16 stats, and a rewrite. Not re-measured in this PR; the previous number was 0.05 s.

Off by default; DATABRICKS_AIR_SNAPSHOT_GC enables it (env.GetBool, the repo's flag
convention). When on, a submit runs GC after the run exists, once per 24 h per snapshot
directory (marker file), time-boxed to 3 s, with errors logged at debug.

GC touches only the v2 layout: top-level *.air.json manifests and objects/<32 hex>.tar.gz.
It deletes a manifest older than 30 d, and an object older than 8 d that no manifest
younger than 30 d references. Each deletion is rechecked against the age window just
before it runs.

The writer follows the same windows: an object is reused only while younger than 7 d,
and an unchanged resubmit renews its manifest (a rewrite) once all its objects are
present. Since GC age minus reuse window is one day, an object reused at T cannot be
deleted before T + 1 d, while its manifest is written.

The fake workspace now stamps ModifiedAt on imported files, as the workspace does.

Co-authored-by: Isaac <no-reply@databricks.com>
@stevenchen-db

Copy link
Copy Markdown
Contributor Author

Closing: v2 snapshot format is paused pending a more deliberate design and review. Branch kept for reference. This comment was written by Isaac.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant