Skip to content

acc: also suppress bare "retrying" warnings - #7518

Merged
denik merged 1 commit into
mainfrom
acc-suppress-bare-retrying
Nov 12, 2026
Merged

denik merged 1 commit into
mainfrom
acc-suppress-bare-retrying

Conversation

@denik

@denik denik commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Changes

Widen the default SuppressLines.warn_retry pattern in acceptance/test.toml from ^Warn: .*retrying after to ^Warn: (.*: )?retrying( after |$), and add the bare form to the selftest/suppresslines tests (suppressed by default, kept when warn_retry = "").

Why

retry.With logs retryMessage(err), which is retrying after <status> ... for API errors but a bare retrying for anything else. IsTransientRead also retries transport errors (unexpected EOF, REFUSED_STREAM), and these are not API errors. One example is GET /clusters/get while the deploy waits for a new cluster. The warning depends on timing, but the pattern from #7192 only matched the status-code form, so it leaked into the compared output of cluster tests on Azure and GCP.

This is not a CLI regression. The retry is intended (#7129). Only the test-side filter missed this form. I left the CLI message unchanged and fixed the filter, which matches how the other retry warnings are handled.

Tests

  • selftest/suppresslines fails with the old pattern (the bare line shows up in output) and passes with the new one.
  • No golden output contains a bare retrying line. The goldens with retrying after lines all set SuppressLines.warn_retry = "", so they are unaffected.
  • Ran the full local TestAccept.

Dashboard

Written by an AI agent (harness_auto, opus5.5, isaac_claude).

This pull request and its description were written by Isaac.

retry.With logs a bare "retrying" when the error is not an API error,
e.g. an unexpected EOF on GET /clusters/get retried by IsTransientRead.
The default SuppressLines.warn_retry pattern only matched
"retrying after <status>", so these timing-dependent warnings leaked
into compared output of cluster tests on cloud.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 11, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Commit: fe27f9c28 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=0 unreleased=0 coverage=2 failing=0
test branch main (78702d5) latest (v1.26.0)
TestAccept/selftest/suppresslines ✅ ✅ ✅
TestAccept/selftest/suppresslines/keep ✅ ✅ ✅

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: fe27f9c

Run: 43115818583

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 17 6:56
💚​ aws-windows-latest-4core-16gb 1 272 15 5:03
💚​ azure linux-2core-8gb 1 269 17 6:33
💚​ azure-windows-latest-4core-16gb 1 271 15 5:54
💚​ gcp linux-2core-8gb 1 270 17 6:14
💚​ gcp-windows-latest-4core-16gb 1 272 15 4:07
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 6 slowest tests (at least 2 minutes):
duration env testname
4:01 aws linux-2core-8gb TestAccept
3:58 aws-windows-latest-4core-16gb TestAccept
3:55 gcp linux-2core-8gb TestAccept
3:51 azure linux-2core-8gb TestAccept
3:39 azure-windows-latest-4core-16gb TestAccept
3:36 gcp-windows-latest-4core-16gb TestAccept

@denik
denik marked this pull request as ready for review November 11, 2026 23:18
@denik
denik requested a review from a team as a code owner November 11, 2026 23:18
@denik
denik enabled auto-merge November 11, 2026 23:23
@andrewnester
andrewnester self-requested a review November 12, 2026 08:58
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit d31dce1 Nov 12, 2026
28 checks passed
@denik
denik deleted the acc-suppress-bare-retrying branch November 12, 2026 10:08
tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
…ks#7521)

## Changes
- `libs/retry.IsTransient` now also returns true for the SDK's
client-side inactivity timeout (`*url.Error` with "request timed out
after ... of inactivity"). Callers already use it only for calls that
are safe to repeat: overwrite writes and deletes through `retry.Filer`,
reads through `IsTransientRead`, and the direct engine's
update/delete/wait calls. Non-overwrite writes, such as the deploy lock
create, are still not retried.
- Retry warnings for non-API errors now read `retrying after <error>`
instead of a bare `retrying`, so the warning says what happened and
acceptance tests suppress it like the other retry warnings
(`SuppressLines.warn_retry`).
- `acceptance/bundle/upload/timeout` now records the two retries and the
three upload attempts.
- Changelog entry in `.nextchanges/bundles/7521.md`.

## Why
When the workspace filesystem API sends nothing for `HTTPTimeout` (90s
for bundles), the SDK fails the request and never retries it. Its retry
check only sees API errors and a short list of transport errors.
`libs/retry` ([databricks#7129](databricks#7129))
fills the gaps for 5xx/408 but not for this timeout. This is the most
common cloud-test failure: about 10k failures in 120 days on
`import-file` alone, mostly on burst days that hit AWS, Azure and GCP
together. The same timeout also shows up on `workspace/get-status`,
`workspace/import` and GET `workspace-files`.

This is not a CLI regression. The stalls are server-side, and a
test-level rerun sometimes succeeds (about 1 in 3 affected tests end up
flaky rather than failed). A timeout carries the same uncertainty as a
504 (the server may or may not have applied the request), and we already
retry 504s at these call sites.

Judgment call: a stalled attempt takes up to `HTTPTimeout` (90s) before
the retry. With `MaxRetries = 2` and the 15s pause, a deploy that hits a
persistent stall can now take about 5 minutes longer before it fails.
That matches the intent in `bundle/config/workspace.go` ("do utmost to
complete" a deploy).

Known edge cases: these call sites are already retried on 504, and a
timeout now reaches them too. A UC rename that is applied but times out
is re-sent with the old name and fails with a 404 (it fails safely, with
no data loss). A lock release that is applied but times out could, in a
narrow window, delete a lock that another deployer took in the meantime.

Overlap with [databricks#7518](databricks#7518): databricks#7518
widens the acceptance `SuppressLines.warn_retry` filter to also match
the bare `retrying` warning. This PR replaces that bare form with
`retrying after <error>` at the source, so the current filter already
matches it. The two PRs don't touch the same files, and either merge
order works; databricks#7518's wider filter stays as a safety net.

## Tests
- Unit tests in `libs/retry` for the timeout classification, including
the wrapped and cancelled cases.
- `acceptance/bundle/upload/timeout` shows the retries.
- Full local acceptance suite passes. The only exceptions are tests that
record the user agent, which pick up an extra agent-environment token on
the machine I ran them on; their only diff is the user-agent string.

## Dashboard:
| error | snippet |
| -- | -- |
|
[error/179](http://go/cli/dashboard/error/179?main=1&pr=1&end=2026-11-11T22:45:00Z&last=120d)
| `Error: Post
"[DATABRICKS_URL]/api/2.0/workspace-files/import-file/[PATH]": request
timed out after 1m30s of inactivity` |

This pull request and its description were written by Isaac.

---------

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants