Repository navigation
air: opt-in GC of v2 snapshot objects with lease and reuse windows - #7417
Closed
stevenchen-db wants to merge 1 commit into
Closed
stevenchen-db wants to merge 1 commit into
stevenchen-db wants to merge 1 commit into
Conversation
Off by default; DATABRICKS_AIR_SNAPSHOT_GC enables it (env.GetBool, the repo's flag convention). When on, a submit runs GC after the run exists, once per 24 h per snapshot directory (marker file), time-boxed to 3 s, with errors logged at debug. GC touches only the v2 layout: top-level *.air.json manifests and objects/<32 hex>.tar.gz. It deletes a manifest older than 30 d, and an object older than 8 d that no manifest younger than 30 d references. Each deletion is rechecked against the age window just before it runs. The writer follows the same windows: an object is reused only while younger than 7 d, and an unchanged resubmit renews its manifest (a rewrite) once all its objects are present. Since GC age minus reuse window is one day, an object reused at T cannot be deleted before T + 1 d, while its manifest is written. The fake workspace now stamps ModifiedAt on imported files, as the workspace does. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
Author
|
Closing: v2 snapshot format is paused pending a more deliberate design and review. Branch kept for reference. This comment was written by Isaac. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR C: CLI-side GC of v2 snapshot objects (design section 4:
databricks-eng/universePR 3017224). Stacked on #7405 (v2 writer).Off by default.
DATABRICKS_AIR_SNAPSHOT_GCenables it, read withenv.GetBool(the repo's flag convention). With it unset, the submit path does nothing: the gate returns before any workspace call.What it deletes. Only the v2 layout under
.internal/: top-level*.air.jsonmanifests andobjects/<32 hex>.tar.gz. v1 tarballs, sidecars, and any other name are never listed for deletion.Writer changes that GC depends on.
Safety argument. A submit reusing an object at time T sees an mtime above T − R. GC deletes it only once its age reaches A, so not before T + (A − R) = T + 1 d. The submit writes its manifest within seconds, inside that day.
TestV2GCRaceReusedObjectNotDeletableBeforeMarginchecks the exact boundary with a fake clock.When it runs. After the run is created, at most once per 24 h per snapshot directory (marker
.air-gc.marker), time-boxed to 3 s. Every error is logged at debug and never changes the submit's result. A run that cannot finish reading every manifest deletes nothing, because an unread manifest could keep any object.Interpretation to review. The manager asked that GC never slow a submit. The run is synchronous, since the CLI process exits right after the submit, so an opt-in submit pays one marker stat, and once a day it may pay up to the 3 s box. Running it in the background would almost never finish. Off by default, this costs nothing.
Residual risks.
Dry run against df1 (
.air/repo_snapshots/.internal, listing only, nothing deleted):Tests (fake filer with a fake clock;
go test -race ./cmd/air/ -skip TestIndexStrategypasses;golangci-lintoncmd/airandlibs/testserver: 0 issues)true.Not verified
./acceptancerun fails in this environment before any test runs. Theprerequisitescheck needs jq 1.7+ (this machine has 1.6) and rejects temp dirs inside a git repo.TestAccept/airpasses.