Skip to content

air: Say when --idempotency-key returns an existing run - #7527

Closed
abdulkareem-db wants to merge 3 commits into
databricks:mainfrom
abdulkareem-db:air-idempotency-existing-run
Closed

abdulkareem-db wants to merge 3 commits into
databricks:mainfrom
abdulkareem-db:air-idempotency-existing-run

Conversation

@abdulkareem-db

Copy link
Copy Markdown

Changes

When air run is given an idempotency key that was already used (via --idempotency-key or the YAML idempotency_token) and the API returns the existing run, the CLI now says so:

Returned existing workload with Job Run ID: 555 (idempotency key "my-key" was already used)
View job run at: <host>/jobs/runs/555

Fresh submits print the same output as before. Every JSON submit result now carries an existing boolean, including the SUBMITTED event and the final result for --watch -o json. Dry-run JSON omits the field because nothing is submitted. --watch on an existing run streams that run's logs as before.

Why

The --idempotency-key help says "Return the existing run if this key was already used", but a repeated submit printed "Submitted workload with Job Run ID: …" and identical JSON, so users couldn't tell a deduplicated submit from a new run.

runs/submit returns only run_id, so the CLI checks the run afterwards. Each submit uploads its command.sh to a fresh launch directory with a random suffix (cliLaunchDir), and runs/get returns the run's ai_runtime_task.deployments[].command_path, which air get already reads. If that path isn't this submit's path, the run came from an earlier submit. Timestamps were not used, because clock skew makes them unreliable for the quick retries idempotency keys are meant for.

  • The extra runs/get is only made when the key is user-supplied (flag or YAML). Generated tokens skip it, so most submits make no extra call.
  • If the lookup fails, the CLI logs a warning and reports the run as new, so the lookup can't fail a submit that succeeded.
  • Precedence between the flag and the YAML token is unchanged.

Known limitation: workloads that use containers are always reported as new, because their deployment spec carries no command_path.

libs/testserver didn't model runs/submit idempotency. It now returns the earlier run for a repeated token, following the existing run-now pattern, and keeps the submitted ai_runtime_task on the run.

Tests

  • Unit tests: reuse via the flag, reuse via the YAML token, the JSON existing field, and --watch on an existing run (text and JSON). Fresh-submit text output is unchanged, and existing fresh JSON tests now assert existing: false.
  • TestJobsSubmit_IdempotencyTokenReturnsExistingRun for the test server.
  • New acceptance test acceptance/air/run-idempotency: fresh submit, reuse via the flag (text and JSON), a different key, and reuse via YAML. No existing golden outputs changed.
  • go test ./cmd/air/... ./libs/testserver/... and the air, ssh and cmd/workspace/jobs acceptance tests pass. Lint reports 0 issues.

This pull request and its description were written by Isaac.

Resubmitting `air run` with an already-used idempotency key (flag or YAML
idempotency_token) printed "Submitted workload with Job Run ID" exactly
like a fresh submit, so users could not tell a deduplicated submit from a
new one (CNXT-2520).

runs/submit returns only a run id, so when the token is caller-supplied the
CLI now fetches the run and compares its command_path with the one this
submit uploaded. Every submit uses a fresh launch directory, so a run
pointing at a different command path was started by an earlier submit.
Generated tokens skip the lookup.

- Text output says "Returned existing workload with Job Run ID: <id>
  (idempotency key "<key>" was already used)".
- JSON results (PENDING envelope, --watch SUBMITTED event and closing
  envelope) carry an `existing` boolean; dry runs omit it.
- The fake workspace's runs/submit now honors idempotency tokens and keeps
  the submitted ai_runtime_task on the run.

Co-authored-by: Isaac <no-reply@databricks.com>
@abdulkareem-db
abdulkareem-db requested review from a team as code owners November 11, 2026 23:53
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! This pull request was opened from a fork.

If you're a member of the Databricks organization, please request access to the cli-contrib team and open your pull request from a branch in this repository instead of a fork — fork PRs don't run integration tests automatically and have reduced CI access.

If you're an external contributor, no action is needed; a maintainer will review your pull request.

abdulkareem-db and others added 2 commits November 11, 2026 23:55
Co-authored-by: Isaac <no-reply@databricks.com>
Keep submitWorkload's signature unchanged: run opts in through the
context, and submitWorkload records whether a caller-supplied
idempotency token returned an existing run. Drop `existing` from the
--watch JSONL events; the text line and the non-watch -o json result
still report it.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 7527
  • Commit SHA: 23091166d2dae14be4b6278a057b2a4a5c853991

Checks will be approved automatically on success.

@abdulkareem-db

Copy link
Copy Markdown
Author

Superseded by #7529 (same change, opened from a branch in this repo).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant