Skip to content

air: Skip purged runs in air list --all-status - #7523

Open
abdulkareem-db wants to merge 2 commits into
mainfrom
abdulkareem/air-list-skip-purged-runs
Open

abdulkareem-db wants to merge 2 commits into
mainfrom
abdulkareem/air-list-skip-purged-runs

Conversation

@abdulkareem-db

Copy link
Copy Markdown

Changes

databricks air list --all-status no longer fails when one of the listed runs has been purged. hydrateJobRuns now drops runs whose runs/get call returns apierr.ErrResourceDoesNotExist, alongside the existing 403/404 handling, and returns the per-run error without wrapping it a second time.

Why

runs/get reports a purged run as 400 INVALID_PARAMETER_VALUE "Run does not exist." rather than 404, so hydrateJobRuns treated it as a systemic error and the errgroup failed the whole listing:

$ databricks air list --all-status
Error: failed to get run <id>: failed to get run <id>: Run <id> does not exist.

The Go SDK already maps that response (scoped to GET /api/2.x/jobs/runs/get) to ErrResourceDoesNotExist, so matching the sentinel avoids string-matching the message. Other errors (5xx, other INVALID_PARAMETER_VALUE responses, auth) still fail the batch. The doubled "failed to get run" prefix came from wrapping in both fetchJobRun and hydrateJobRuns; the outer wrap is removed.

Tracked internally as CNXT-2565.

Tests

  • New unit tests in cmd/air/joblist_test.go against an httptest server (so the SDK's error mapping is exercised): purged (400 "does not exist"), 403 and 404 runs are dropped and the rest keep their order; a 500 and an unrelated 400 still fail with a single failed to get run N: prefix. Both tests fail on the previous code.
  • go test ./cmd/air/... and go test ./acceptance -run TestAccept/air/list pass.
  • Manually: with a purged run in the user's history, the released CLI fails as above; this build lists all remaining runs and exits 0.

This pull request and its description were written by Isaac.

runs/get reports a purged run as 400 INVALID_PARAMETER_VALUE "Run <id>
does not exist.", which the SDK maps to ErrResourceDoesNotExist.
hydrateJobRuns only dropped 403/404, so one purged run failed the whole
listing. Drop ErrResourceDoesNotExist too, and stop wrapping the error a
second time ("failed to get run X: failed to get run X: ...").

Fixes CNXT-2565.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: ad3906e

Run: 43123122125

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 15 8:04
💚​ aws-windows-latest-4core-16gb 1 272 13 4:18
💚​ azure linux-2core-8gb 1 269 15 7:57
💚​ azure-windows-latest-4core-16gb 1 271 13 4:47
💚​ gcp linux-2core-8gb 1 270 15 6:57
💚​ gcp-windows-latest-4core-16gb 1 272 13 4:58
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:26 azure linux-2core-8gb TestAccept
3:54 aws-windows-latest-4core-16gb TestAccept
3:51 gcp linux-2core-8gb TestAccept
3:41 aws linux-2core-8gb TestAccept
3:39 azure-windows-latest-4core-16gb TestAccept
3:12 gcp-windows-latest-4core-16gb TestAccept

Comment thread cmd/air/joblist_test.go

// runGetByIDServer serves runs/get per run_id and a stub for the SDK config
// probe; ids missing from responses return a run with that id.
func runGetByIDServer(t *testing.T, responses map[string]runGetResponse) *httptest.Server {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

list_index_test.go already has indexAndGetServer with per-id forbidden/missing maps and TestIndexStrategyDropsForbiddenAndMissing. Can the purged case extend those instead of a second per-id runs/get fake?

Comment thread cmd/air/joblist.go
g.Go(func() error {
run, err := fetchJobRun(gctx, w, id)
if err != nil {
if errors.Is(err, apierr.ErrResourceDoesNotExist) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ErrResourceDoesNotExist inherits ErrNotFound, and a plain 404 maps to ErrNotFound too, so errors.Is(err, apierr.ErrNotFound) || errors.Is(err, apierr.ErrPermissionDenied) would replace both branches (precedent: libs/apps/prompt/prompt.go:861). Worth collapsing, or not worth it?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction: APIError.Unwrap resolves error_code before HTTP status, so collapsing to errors.Is would stop dropping a 403/404 that carries a mapped code. Keep the two branches.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants