Repository navigation
air: Say when --idempotency-key returns an existing run - #7527
Closed
abdulkareem-db wants to merge 3 commits into
Closed
abdulkareem-db wants to merge 3 commits into
abdulkareem-db wants to merge 3 commits into
Conversation
Resubmitting `air run` with an already-used idempotency key (flag or YAML idempotency_token) printed "Submitted workload with Job Run ID" exactly like a fresh submit, so users could not tell a deduplicated submit from a new one (CNXT-2520). runs/submit returns only a run id, so when the token is caller-supplied the CLI now fetches the run and compares its command_path with the one this submit uploaded. Every submit uses a fresh launch directory, so a run pointing at a different command path was started by an earlier submit. Generated tokens skip the lookup. - Text output says "Returned existing workload with Job Run ID: <id> (idempotency key "<key>" was already used)". - JSON results (PENDING envelope, --watch SUBMITTED event and closing envelope) carry an `existing` boolean; dry runs omit it. - The fake workspace's runs/submit now honors idempotency tokens and keeps the submitted ai_runtime_task on the run. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Thanks for the contribution! This pull request was opened from a fork. If you're a member of the Databricks organization, please request access to the If you're an external contributor, no action is needed; a maintainer will review your pull request. |
Co-authored-by: Isaac <no-reply@databricks.com>
Keep submitWorkload's signature unchanged: run opts in through the context, and submitWorkload records whether a caller-supplied idempotency token returned an existing run. Drop `existing` from the --watch JSONL events; the text line and the non-watch -o json result still report it. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Author
|
Superseded by #7529 (same change, opened from a branch in this repo). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
When
air runis given an idempotency key that was already used (via--idempotency-keyor the YAMLidempotency_token) and the API returns the existing run, the CLI now says so:Fresh submits print the same output as before. Every JSON submit result now carries an
existingboolean, including theSUBMITTEDevent and the final result for--watch -o json. Dry-run JSON omits the field because nothing is submitted.--watchon an existing run streams that run's logs as before.Why
The
--idempotency-keyhelp says "Return the existing run if this key was already used", but a repeated submit printed "Submitted workload with Job Run ID: …" and identical JSON, so users couldn't tell a deduplicated submit from a new run.runs/submitreturns onlyrun_id, so the CLI checks the run afterwards. Each submit uploads itscommand.shto a fresh launch directory with a random suffix (cliLaunchDir), andruns/getreturns the run'sai_runtime_task.deployments[].command_path, whichair getalready reads. If that path isn't this submit's path, the run came from an earlier submit. Timestamps were not used, because clock skew makes them unreliable for the quick retries idempotency keys are meant for.runs/getis only made when the key is user-supplied (flag or YAML). Generated tokens skip it, so most submits make no extra call.Known limitation: workloads that use
containersare always reported as new, because their deployment spec carries nocommand_path.libs/testserverdidn't modelruns/submitidempotency. It now returns the earlier run for a repeated token, following the existingrun-nowpattern, and keeps the submittedai_runtime_taskon the run.Tests
existingfield, and--watchon an existing run (text and JSON). Fresh-submit text output is unchanged, and existing fresh JSON tests now assertexisting: false.TestJobsSubmit_IdempotencyTokenReturnsExistingRunfor the test server.acceptance/air/run-idempotency: fresh submit, reuse via the flag (text and JSON), a different key, and reuse via YAML. No existing golden outputs changed.go test ./cmd/air/... ./libs/testserver/...and theair,sshandcmd/workspace/jobsacceptance tests pass. Lint reports 0 issues.This pull request and its description were written by Isaac.