Repository navigation
Add acceptance test that fails when a package init() is slow - #7406
Conversation
| @@ -0,0 +1 @@ | |||
| OK: package init times are within threshold | |||
There was a problem hiding this comment.
Is it a good idea to record the packages as well (sorted, without timings), to track what's happening in init? or will it add too much churn?
There was a problem hiding this comment.
Good idea. Latest has a copy that should be ~stable and not introduce much churn.
Please take a look.
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 1 tests: regression=0 unreleased=0 coverage=0 failing=1
TestAccept/inittime: branch ❌ | main (9605618) ✅ | latest (v1.25.0) ✅branch: |
✅ Integration test reportCommit: 96b9183
Top 3 slowest tests (at least 2 minutes):
|
5467189 to
c46add8
Compare
go-runewidth v0.0.26/v0.0.27 filled a 2 MB table in init(), adding ~20 ms to every CLI invocation. It arrived through a transitive Charm upgrade and went unnoticed (fixed in #7391). The new acceptance test runs `$CLI --version` with GODEBUG=inittrace=1 and fails if any package's init clock exceeds 5 ms. Each package is judged by its minimum over 5 runs, extended up to 20 runs while a check fails, so load from parallel tests does not cause flakes while a real regression fails on every run. It also records a per-package init time budget in out.initlimits.txt, initialized at 4x the measured minimum (at least 0.1 ms; unlisted packages get 0.1 ms). A package that exceeds its budget gets a new one, so several- fold slowdowns show up as a diff in review without flaking on noise. The test runs on linux/amd64 only: Windows inittrace clocks can be quantized to the system tick, and the set of packages with an init differs per architecture. Co-authored-by: Isaac <no-reply@databricks.com>
c46add8 to
2dc7448
Compare
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
| f"FAIL: package init() takes {total:.2f} ms in total, over {args.max_ms:g} ms; this delays every CLI invocation." | ||
| ) | ||
| print("To investigate:") | ||
| print(" GODEBUG=inittrace=1 databricks --version 2>&1 | sort -k5 -n -r | head") |
There was a problem hiding this comment.
should we just run it here and record output? For historical runs. Can also drop sort and head.
There was a problem hiding this comment.
WDYM for historical runs?
We already capture the timing in the log output.
There was a problem hiding this comment.
Can do it in a follow up if we need to.
…s#7490) ## Changes `check_init_time.py` now judges the **sum of each package's minimum** init() time across runs, instead of the **minimum of the per-run totals**. The 10 ms threshold stays the same, and so do the run counts and output format. The printed `total` is now the sum of the per-package lines listed below it. ## Why `TestAccept/inittime` has been flaky on Linux CI since it was added in databricks#7406: it failed 129 times in its first ~30 hours, on main and in PRs, in both `direct` and `directdms`. This is not a CLI regression. The totals were the same in the first failing run on main (12.04 ms) and the latest one (11.95 ms), and the per-package breakdown didn't change either: the top packages are still `cmd/air` / `encoding/gob` / `bundle/config` at <1 ms each. The problem is the estimator. On GitHub runners under load from parallel tests, almost every run has *some* package hit by noise. So even the fastest of 20 runs stays inflated, while each package's own minimum stays close to its real cost. Here are both metrics, computed from the per-package logs of all 129 failures: | metric | min | median | max | |---|---|---|---| | min over runs of total (current) | 10.17 ms | 11.91 ms | 15.00 ms | | sum over packages of per-package min (this PR) | 5.74 ms | 6.80 ms | 8.78 ms | Locally (linux/amd64, 8 cores), the two metrics are ~3 ms and ~3.5–4 ms with or without 8–16 busy-loop processes running. A slow init() is slow on every run, so its own minimum is high and the new metric still catches it: the +20 ms go-runewidth case that motivated databricks#7406 would still fail. I kept 10 ms rather than raising it. The new metric leaves ~1.2 ms of headroom over the worst CI value seen, and when the total is over the threshold the script keeps adding runs (up to 20), which lowers per-package minimums further. ## Tests - `go test ./acceptance -run 'TestAccept/inittime$'` passes locally. - `check_init_time.py --max-ms 2 -- databricks --version` still fails with `FAIL: package init() takes 3.96 ms in total, over 2 ms`. ## Dashboard - [error/3304](http://go/cli/dashboard/error/3304?main=1&pr=1&end=2026-11-11T17:00:00Z&last=2d), [TestAccept/inittime](http://go/cli/dashboard/test/acceptance.TestAccept/inittime?main=1&pr=1&end=2026-11-11T17:00:00Z&last=2d): ``` FAIL: package init() takes 11.95 ms in total, over 10 ms; this delays every CLI invocation. LOG.inittime: Minimum init() clock time in ms over 20 runs: LOG.inittime: 11.95 total LOG.inittime: 0.87 github.com/databricks/cli/cmd/air LOG.inittime: 0.76 encoding/gob ``` This pull request and its description were written by Isaac. Co-authored-by: Isaac <no-reply@databricks.com>
Changes
New acceptance test
acceptance/inittimeruns$CLI --versionwithGODEBUG=inittrace=1and:init()clock time toLOG.inittime, sorted slowest first, on all OSes;Times are minimums over 5 runs (extended to 20 while the total is over the threshold), so load from parallel tests doesn't cause flakes. The threshold is not enforced on macOS and Windows, where clocks are too coarse or noisy (Windows can quantize to the ~15.6 ms system tick).
Example log (linux/amd64, truncated):
Why
go-runewidth v0.0.27 arrived through a transitive Charm upgrade and added ~20 ms to every invocation, unnoticed until #7391. Checking the total rather than individual packages also catches a slow drip of small regressions, and the log shows which packages contribute.
Tests
--max-ms 3the check fails:FAIL: package init() takes 5.27 ms in total, over 3 ms.This pull request and its description were written by Isaac.