Repository navigation
acc: also suppress bare "retrying" warnings - #7518
Merged
Merged
Conversation
retry.With logs a bare "retrying" when the error is not an API error, e.g. an unexpected EOF on GET /clusters/get retried by IsTransientRead. The default SuppressLines.warn_retry pattern only matched "retrying after <status>", so these timing-dependent warnings leaked into compared output of cluster tests on cloud. Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 2 tests: regression=0 unreleased=0 coverage=2 failing=0
|
Collaborator
✅ Integration test reportCommit: fe27f9c
Top 6 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 11, 2026 23:18
denik
enabled auto-merge
November 11, 2026 23:23
andrewnester
self-requested a review
November 12, 2026 08:58
andrewnester
approved these changes
Nov 12, 2026
tedjt
pushed a commit
to tedjt/cli
that referenced
this pull request
Nov 12, 2026
…ks#7521) ## Changes - `libs/retry.IsTransient` now also returns true for the SDK's client-side inactivity timeout (`*url.Error` with "request timed out after ... of inactivity"). Callers already use it only for calls that are safe to repeat: overwrite writes and deletes through `retry.Filer`, reads through `IsTransientRead`, and the direct engine's update/delete/wait calls. Non-overwrite writes, such as the deploy lock create, are still not retried. - Retry warnings for non-API errors now read `retrying after <error>` instead of a bare `retrying`, so the warning says what happened and acceptance tests suppress it like the other retry warnings (`SuppressLines.warn_retry`). - `acceptance/bundle/upload/timeout` now records the two retries and the three upload attempts. - Changelog entry in `.nextchanges/bundles/7521.md`. ## Why When the workspace filesystem API sends nothing for `HTTPTimeout` (90s for bundles), the SDK fails the request and never retries it. Its retry check only sees API errors and a short list of transport errors. `libs/retry` ([databricks#7129](databricks#7129)) fills the gaps for 5xx/408 but not for this timeout. This is the most common cloud-test failure: about 10k failures in 120 days on `import-file` alone, mostly on burst days that hit AWS, Azure and GCP together. The same timeout also shows up on `workspace/get-status`, `workspace/import` and GET `workspace-files`. This is not a CLI regression. The stalls are server-side, and a test-level rerun sometimes succeeds (about 1 in 3 affected tests end up flaky rather than failed). A timeout carries the same uncertainty as a 504 (the server may or may not have applied the request), and we already retry 504s at these call sites. Judgment call: a stalled attempt takes up to `HTTPTimeout` (90s) before the retry. With `MaxRetries = 2` and the 15s pause, a deploy that hits a persistent stall can now take about 5 minutes longer before it fails. That matches the intent in `bundle/config/workspace.go` ("do utmost to complete" a deploy). Known edge cases: these call sites are already retried on 504, and a timeout now reaches them too. A UC rename that is applied but times out is re-sent with the old name and fails with a 404 (it fails safely, with no data loss). A lock release that is applied but times out could, in a narrow window, delete a lock that another deployer took in the meantime. Overlap with [databricks#7518](databricks#7518): databricks#7518 widens the acceptance `SuppressLines.warn_retry` filter to also match the bare `retrying` warning. This PR replaces that bare form with `retrying after <error>` at the source, so the current filter already matches it. The two PRs don't touch the same files, and either merge order works; databricks#7518's wider filter stays as a safety net. ## Tests - Unit tests in `libs/retry` for the timeout classification, including the wrapped and cancelled cases. - `acceptance/bundle/upload/timeout` shows the retries. - Full local acceptance suite passes. The only exceptions are tests that record the user agent, which pick up an extra agent-environment token on the machine I ran them on; their only diff is the user-agent string. ## Dashboard: | error | snippet | | -- | -- | | [error/179](http://go/cli/dashboard/error/179?main=1&pr=1&end=2026-11-11T22:45:00Z&last=120d) | `Error: Post "[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request timed out after 1m30s of inactivity` | This pull request and its description were written by Isaac. --------- Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Widen the default
SuppressLines.warn_retrypattern inacceptance/test.tomlfrom^Warn: .*retrying afterto^Warn: (.*: )?retrying( after |$), and add the bare form to theselftest/suppresslinestests (suppressed by default, kept whenwarn_retry = "").Why
retry.WithlogsretryMessage(err), which isretrying after <status> ...for API errors but a bareretryingfor anything else.IsTransientReadalso retries transport errors (unexpected EOF,REFUSED_STREAM), and these are not API errors. One example isGET /clusters/getwhile the deploy waits for a new cluster. The warning depends on timing, but the pattern from #7192 only matched the status-code form, so it leaked into the compared output of cluster tests on Azure and GCP.This is not a CLI regression. The retry is intended (#7129). Only the test-side filter missed this form. I left the CLI message unchanged and fixed the filter, which matches how the other retry warnings are handled.
Tests
selftest/suppresslinesfails with the old pattern (the bare line shows up in output) and passes with the new one.retryingline. The goldens withretrying afterlines all setSuppressLines.warn_retry = "", so they are unaffected.TestAccept.Dashboard
Written by an AI agent (harness_auto, opus5.5, isaac_claude).
This pull request and its description were written by Isaac.