Repository navigation
air: Stop showing Max Retries 0 when runs/get omits the retry limit - #7623
Draft
artemy-kolesnikov wants to merge 4 commits into
Draft
artemy-kolesnikov wants to merge 4 commits into
artemy-kolesnikov wants to merge 4 commits into
Conversation
Jobs runs/get leaves max_retries out of every task run, and the SDK decodes the missing field as 0, so `air get` and the `air list` info pane showed "Max Retries 0" even for runs submitted with the default of 3 or an explicit limit . The server value is now used only when the response carries the key. For a one-time submit run, the limit is read from the run's uploaded training config: the configured value when it sets max_retries, and "3 (CLI default)" for ai_runtime_task runs that leave it unset, since the CLI always submits its default. Job runs, multi-container runs, legacy gen_ai_compute_task runs with no explicit value, and runs whose config can't be read show N/A. The derivation is resolveMaxRetries, which returns the limit and its source so `air get -o json` can expose the same value later. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 2 tests: regression=2 unreleased=0 coverage=0 failing=0
TestAccept/air/get: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): TestAccept/air/get-ai-runtime: branch ✅ | main (c4559ed) ❌ | latest (v1.26.0) ❌main (c4559ed): latest (v1.26.0): |
Collaborator
✅ Integration test reportCommit: 9186e20
Top 3 slowest tests (at least 2 minutes):
|
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Title: air: Stop showing Max Retries 0 when runs/get omits the retry limit
Changes
air get(and theair listinfo pane) showedMax Retries 0for AI Runtime runs whatever retry limit they were submitted with. Jobsruns/getleavesmax_retriesout of every task run, and SDK v0.197.0 decodes the missing field as 0.maxRetries(run)incmd/air/format.goprinted that 0.resolveMaxRetries(run, configRaw)now applies these rules in order:max_retrieson the latest task (presence via the SDK'sForceSendFields, so an explicit 0 counts)unlimitedfor -1)SUBMIT_RUN(for example a bundle job deployed withair convert-to-dabs, whose limit lives in the job settings)N/Amax_retries1ai_runtime_taskrun whose config leaves it unset3 (CLI default)gen_ai_compute_taskrun with no explicit value, multi-container run (no deploymentcommand_path, so no config path), config missing or unreadable or unparsableN/Aair getalready downloads for the Configuration box.resolveConfigYAML/configYAMLnow also return the raw bytes, so there is no extra request. Only themax_retrieskey is decoded, leniently, so configs from other CLI versions still parse.air list's info pane now runs the sameai_runtime_taskenrichment asair get(same guard), so it shows the config box, accelerators and the same Max Retries value instead of N/A.resolveMaxRetriesreturns the limit and its source (reported/configured/cli_default) soair get -o jsoncan expose the same value later. JSON output is unchanged in this PR..nextchanges/air/get-max-retries.md.Why
A supervising agent couldn't tell whether a failed workload would be retried: help says the default is 3, the run used 3, and
air getsaid 0. The submit-message half of the report ("Failed attempts will be retried up to 3 times.") was already removed by #7270.Why the default is assumed only for
ai_runtime_tasksubmit runs:ai_runtime_task, andrunsubmit.goalways sendsmax_retries(ForceSendFields), defaulting todefaultMaxRetries = 3. The Python CLI defaulted to 3 before it first emittedai_runtime_task.gen_ai_compute_taskruns may predate the Python CLI's switch from 0 to 3 (2026-05-22), and nothing on the run records which version submitted it, so an unset value is unknown.max_retriesindatabricks.ymlafterconvert-to-dabs, so the syncedtraining_config.yamlcan be stale.Rejected:
jobs/getsettings for job runs and search bundle deployment directories for configs. More code and risk for a less common path. Job runs showN/Ahere; see follow-ups.Tests
TestMaxRetries(15 cases): config 1 / 0 / -1, unset →3 (CLI default), no config, unparsable config, legacy unset → N/A, legacy explicit 2, job run with and without a config → N/A, unknown run type → N/A, server 0 over a config of 2, server 5 on a job run, server -1, no tasks.TestResolveMaxRetriesSource: the source each case reports.TestMaxRetriesFromDecodedRunGet: decodes realruns/getresponse bodies captured from a workspace through the SDK. Absent key + config 1 →1; absent key + unset →3 (CLI default); multi-container body → no config path,N/A; Python-shaped legacy body →N/A; explicit"max_retries": 0→0.TestRunDetailTextReadsTheUploadedConfig(list pane, httptest): configured1, unset3 (CLI default), config download 404 →N/A, job run →N/A.resolveMaxRetriesswapped back to the old behaviour (read the SDK field, 0 when absent), 11 of 15TestMaxRetriescases,TestMaxRetriesFromDecodedRunGetand all 4 list-pane cases fail. The 4 that still pass are the 3 server-reported cases and config0, where the old 0 is right by coincidence.air/getandair/get-ai-runtimeno longer seed"max_retries": 3on the fake task (the real API never returns it) and now set"run_type": "SUBMIT_RUN".get-ai-runtimeadds a second case that re-imports a config withmax_retries: 1. Goldens:get→N/A(legacy, unset),get-ai-runtime→3 (CLI default)then1. With the old code these fixtures fail (0in all three places).go test ./cmd/air/... -count=1passes.go test ./acceptance -run 'TestAccept/air/'passes (20 fixtures)../task lint-q-go-root(0 issues),./task fmt-q-go,./task deadcode(no dead code),./task check-changelogand gofmt are clean.Manual verification (against a workspace)
NO_COLOR=1 databricks air get <run>, base build vs this change:max_retries: 1Max Retries 0Max Retries 1max_retries: 0Max Retries 0Max Retries 0Max Retries 0Max Retries 3 (CLI default)Max Retries 0Max Retries N/Aruns/getfor the submitted run hasrun_type: SUBMIT_RUN, and its task keys contain nomax_retries.air get -o jsonreturns the same keys with both builds.Follow-ups (not in this PR)
air get -o jsonshould exposemax_retries. CallresolveMaxRetriesand report the source as data (reported/configured/cli_default/ null) rather than as a suffix. JSON mode would also need to download the config, which it doesn't today.jobs/get(job_id)settings.tasks[task_key].max_retries, labelled as the current job settings.containers[].command_path. The SDK'sDeploymentSpecdoesn't decodecontainersyet, so this needs a raw read or an SDK bump.runs/getdoesn't echomax_retries/retry_on_timeout/min_retry_interval_millison task runs, although the API model documentsmax_retriesthere.This pull request and its description were written by Isaac.