Skip to content

air: Say when --idempotency-key returns an existing run - #7529

Open
abdulkareem-db wants to merge 6 commits into
mainfrom
abdulkareem/air-idempotency-existing-run
Open

abdulkareem-db wants to merge 6 commits into
mainfrom
abdulkareem/air-idempotency-existing-run

Conversation

@abdulkareem-db

@abdulkareem-db abdulkareem-db commented Nov 12, 2026 •

Copy link
Copy Markdown

Changes

When air run is given an idempotency key that was already used (via --idempotency-key or the YAML idempotency_token) and the API returns the existing run, the CLI now says so:

Returned existing workload with Job Run ID: 555 (idempotency key "my-key" was already used)
View job run at: <host>/jobs/runs/555

Fresh submits print the same output as before. The -o json submit result now carries an existing boolean (dry-run JSON omits it because nothing is submitted). The --watch JSONL events are unchanged. --watch on an existing run prints the line above and streams that run's logs as before.

Why

The --idempotency-key help says "Return the existing run if this key was already used", but a repeated submit printed "Submitted workload with Job Run ID: …" and identical JSON, so users couldn't tell a deduplicated submit from a new run.

runs/submit returns only run_id, so the CLI checks the run afterwards. Each submit uploads its command.sh to a fresh launch directory with a random suffix (cliLaunchDir), and runs/get returns the run's ai_runtime_task.deployments[].command_path, which air get already reads. If that path isn't this submit's path, the run came from an earlier submit. Timestamps were not used, because clock skew makes them unreliable for the quick retries idempotency keys are meant for.

  • The extra runs/get is only made when the key is user-supplied (flag or YAML) and the caller opts in. Generated tokens skip it, so most submits make no extra call.
  • submitWorkload's signature is unchanged. air run opts in with withExistingRunReport(ctx), following the context-pointer pattern of armDryRunValidationAttemptBudget, because the command path is built inside submitWorkload and the caller can't recompute it.
  • If the lookup fails, the CLI logs a warning and reports the run as new, so the lookup can't fail a submit that succeeded.
  • Precedence between the flag and the YAML token is unchanged.

Known limitation: workloads that use containers are always reported as new, because their deployment spec carries no command_path.

libs/testserver didn't model runs/submit idempotency. It now returns the earlier run for a repeated token, following the existing run-now pattern, and keeps the submitted ai_runtime_task on the run.

Tests

  • Unit tests in cmd/air/run_test.go:
    • reuse via the flag, reuse via the YAML token, and the JSON existing field; fresh-submit text output is unchanged;
    • --watch text mode with a reused key prints the existing-run line before the streamed logs, and logs still stream;
    • the lookup is made for text (with or without --watch) and non-watch -o json, and skipped under --watch -o json, where existing isn't shown (asserted as zero extra runs/get calls versus a generated token);
    • a failing runs/get, a run with no deployments, or an empty command_path is reported as a new run;
    • containers workloads make no lookup.
  • Every added executable line in cmd/air/run.go and cmd/air/runsubmit.go is covered by go test ./cmd/air/... -coverprofile.
  • TestJobsSubmit_IdempotencyTokenReturnsExistingRun for the test server.
  • New acceptance test acceptance/air/run-idempotency: fresh submit, reuse via the flag (text and JSON), a different key, and reuse via YAML. No existing golden outputs changed.
  • go test ./cmd/air/... ./libs/testserver/... and go test ./acceptance -run TestAccept/air pass. Lint reports 0 issues.

This pull request and its description were written by Isaac.

abdulkareem-db and others added 3 commits November 11, 2026 22:48
Resubmitting `air run` with an already-used idempotency key (flag or YAML
idempotency_token) printed "Submitted workload with Job Run ID" exactly
like a fresh submit, so users could not tell a deduplicated submit from a
new one (CNXT-2520).

runs/submit returns only a run id, so when the token is caller-supplied the
CLI now fetches the run and compares its command_path with the one this
submit uploaded. Every submit uses a fresh launch directory, so a run
pointing at a different command path was started by an earlier submit.
Generated tokens skip the lookup.

- Text output says "Returned existing workload with Job Run ID: <id>
  (idempotency key "<key>" was already used)".
- JSON results (PENDING envelope, --watch SUBMITTED event and closing
  envelope) carry an `existing` boolean; dry runs omit it.
- The fake workspace's runs/submit now honors idempotency tokens and keeps
  the submitted ai_runtime_task on the run.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Keep submitWorkload's signature unchanged: run opts in through the
context, and submitWorkload records whether a caller-supplied
idempotency token returned an existing run. Drop `existing` from the
--watch JSONL events; the text line and the non-watch -o json result
still report it.

Co-authored-by: Isaac <no-reply@databricks.com>
@abdulkareem-db
abdulkareem-db requested review from a team as code owners November 12, 2026 00:06
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 6e31540d1 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (a0a6430) latest (v1.26.0)
TestAccept/air/run-idempotency ✅ ❌ ❌
TestAccept/air/run-idempotency: branch ✅ | main (a0a6430) ❌ | latest (v1.26.0) ❌

main (a0a6430):

acceptance_test.go:1345: Diff:
        --- air/run-idempotency/output.txt
        +++ /tmp/TestAcceptairrun-idempotency1758095907/002/output.txt
        @@ -14,7 +14,7 @@
         
         >>> [CLI] air run -f run.yaml --idempotency-key flag-key
         Submitting experiment: idempotency-smoke
        -Returned existing workload with Job Run ID: [FLAG_KEY_RUN_ID] (idempotency key "flag-key" was already used)
        +Submitted workload with Job Run ID: [FLAG_KEY_RUN_ID]
         View job run at: [DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
        @@ -30,21 +30,20 @@
           "data": {
             "status": "PENDING",
             "run_id": "[FLAG_KEY_RUN_ID]",
        -    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]",
        -    "existing": true
        +    "dashboard_url": "[DATABRICKS_URL]/jobs/runs/[FLAG_KEY_RUN_ID]"
           }
         }
         
         === a different key starts a new run
         
         >>> [CLI] air run -f run.yaml --idempotency-key other-key -o json
        -false
        +null
         
         === a YAML idempotency_token is deduplicated the same way
         
         >>> [CLI] air run -f run-token.yaml
         Submitting experiment: idempotency-smoke
        -Returned existing workload with Job Run ID: [YAML_TOKEN_RUN_ID] (idempotency key "yaml-key" was already used)
        +Submitted workload with Job Run ID: [YAML_TOKEN_RUN_ID]
         View job run at: [DATABRICKS_URL]/jobs/runs/[YAML_TOKEN_RUN_ID]
         
         Tip: use --watch when submitting a run to stream logs to your terminal.
        
FAIL

latest (v1.26.0): same output as main (a0a6430)

abdulkareem-db and others added 2 commits November 12, 2026 00:24
…o json

The --watch JSON events never carry `existing`, so the runs/get that
detects a reused idempotency key was wasted there. Keep it for text
output (with or without --watch) and the non-watch JSON result.

Add tests for --watch text output printing the existing-run line before
the streamed logs, the per-mode runs/get lookup count, the runs/get
failure and missing command path fallbacks, and the containers skip.

Co-authored-by: Isaac <no-reply@databricks.com>
The existing-run report is now tri-state: JSON omits `existing` when the
CLI could not tell (runs/get failed or timed out, no deployment, an
unrecognized command path, or a container run) instead of claiming false.
Generated-token submits still report false, since they can never repeat.

The lookup compares only the unique <run>_<hex> launch directory, so a
different workspace path prefix in the runs/get echo cannot turn a fresh
run into an existing one, and is bounded by a 5s timeout.

Co-authored-by: Isaac <no-reply@databricks.com>
@abdulkareem-db

Copy link
Copy Markdown
Author

End-to-end check against a real workspace, using a CLI built from 6e31540 and a minimal 1×A10 workload (command: echo ..., no code_source):

# Command Result
1 air run --idempotency-key K (first use) Submitted workload with Job Run ID: <A>
2 same key again Returned existing workload with Job Run ID: <A> (idempotency key "K" was already used)
3 same key, -o json "run_id": "<A>", "existing": true
4 different key, -o json new run <B>, "existing": false
5 same key, --watch existing-run line printed before the logs, then run <A>'s logs streamed to completion, exit 0

So the backend echoes the deployment command_path the CLI submitted, and the launch-directory comparison tells a deduplicated submit apart from a fresh one. Steps 2, 3 and 5 started no new runs.

This comment was written by Isaac.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 6e31540

Run: 43130012324

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 16 6:12
💚​ aws-windows-latest-4core-16gb 1 272 14 3:57
💚​ azure linux-2core-8gb 1 269 16 6:16
💚​ azure-windows-latest-4core-16gb 1 271 14 5:02
💚​ gcp linux-2core-8gb 1 270 16 6:46
💚​ gcp-windows-latest-4core-16gb 1 272 14 3:53
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:56 azure-windows-latest-4core-16gb TestAccept
3:55 aws linux-2core-8gb TestAccept
3:54 azure linux-2core-8gb TestAccept
3:51 gcp linux-2core-8gb TestAccept
3:48 gcp-windows-latest-4core-16gb TestAccept
3:45 aws-windows-latest-4core-16gb TestAccept

@pardis-beikzadeh-db pardis-beikzadeh-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description is out of date since 6e31540. Please update it to match the code:

  • When the CLI can't tell whether the run already existed, existing is left out of the JSON. It isn't reported as new.
  • Only the launch directory name is compared.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants