Skip to content

acc: inittime: judge sum of per-package minimum init times - #7490

Merged
denik merged 1 commit into
mainfrom
denik/inittime-sum-of-mins
Nov 12, 2026
Merged

denik merged 1 commit into
mainfrom
denik/inittime-sum-of-mins

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

check_init_time.py now judges the sum of each package's minimum init() time across runs, instead of the minimum of the per-run totals. The 10 ms threshold stays the same, and so do the run counts and output format. The printed total is now the sum of the per-package lines listed below it.

Why

TestAccept/inittime has been flaky on Linux CI since it was added in #7406: it failed 129 times in its first ~30 hours, on main and in PRs, in both direct and directdms. This is not a CLI regression. The totals were the same in the first failing run on main (12.04 ms) and the latest one (11.95 ms), and the per-package breakdown didn't change either: the top packages are still cmd/air / encoding/gob / bundle/config at <1 ms each.

The problem is the estimator. On GitHub runners under load from parallel tests, almost every run has some package hit by noise. So even the fastest of 20 runs stays inflated, while each package's own minimum stays close to its real cost. Here are both metrics, computed from the per-package logs of all 129 failures:

metric min median max
min over runs of total (current) 10.17 ms 11.91 ms 15.00 ms
sum over packages of per-package min (this PR) 5.74 ms 6.80 ms 8.78 ms

Locally (linux/amd64, 8 cores), the two metrics are ~3 ms and ~3.5–4 ms with or without 8–16 busy-loop processes running.

A slow init() is slow on every run, so its own minimum is high and the new metric still catches it: the +20 ms go-runewidth case that motivated #7406 would still fail. I kept 10 ms rather than raising it. The new metric leaves ~1.2 ms of headroom over the worst CI value seen, and when the total is over the threshold the script keeps adding runs (up to 20), which lowers per-package minimums further.

Tests

  • go test ./acceptance -run 'TestAccept/inittime$' passes locally.
  • check_init_time.py --max-ms 2 -- databricks --version still fails with FAIL: package init() takes 3.96 ms in total, over 2 ms.

Dashboard

  • error/3304, TestAccept/inittime:
    FAIL: package init() takes 11.95 ms in total, over 10 ms; this delays every CLI invocation.
    LOG.inittime: Minimum init() clock time in ms over 20 runs:
    LOG.inittime: 11.95 total
    LOG.inittime:  0.87 github.com/databricks/cli/cmd/air
    LOG.inittime:  0.76 encoding/gob
    

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 11, 2026
@denik

denik commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

go/cli/dashboard/error/3304, go/cli/dashboard/test/acceptance.TestAccept/inittime:

short links are broken, should not link to github.com

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: ea253b3

Run: 43066204628

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 15 13:15
✅​ aws-windows-latest-4core-16gb 273 13 6:14
✅​ azure linux-2core-8gb 270 15 10:43
🔄​ azure-windows-latest-4core-16gb 2 270 13 7:43
🔄​ gcp linux-2core-8gb 2 269 15 14:38
✅​ gcp-windows-latest-4core-16gb 273 13 9:04
Test Name azure-windows-latest-4core-16gb gcp linux-2core-8gb
🔄​ TestFsCatOnADir ✅​p 🔄​f
🔄​ TestFsCatOnADir/uc-volumes ✅​p 🔄​f
🔄​ TestSyncFullFileSync 🔄​f ✅​p
🔄​ TestSyncNestedSpacePlusAndHashAreEscapedSync 🔄​f ✅​p
Top 11 slowest tests (at least 2 minutes):
duration env testname
3:51 aws linux-2core-8gb TestAccept
3:43 azure linux-2core-8gb TestAccept
3:35 gcp-windows-latest-4core-16gb TestAccept
3:34 azure-windows-latest-4core-16gb TestAccept
3:30 aws-windows-latest-4core-16gb TestAccept
3:18 gcp linux-2core-8gb TestAccept
2:53 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
2:20 gcp-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:12 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:05 aws linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
2:03 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsStat

@denik
denik marked this pull request as ready for review November 11, 2026 17:43
@denik
denik requested a review from a team as a code owner November 11, 2026 17:43
@denik
denik requested a review from rclarey November 11, 2026 17:43
@denik
denik enabled auto-merge November 11, 2026 17:43
@denik

denik commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

short links are broken, should not link to github.com

Agreed, they were relative links. Fixed in the description: the dashboard links now point to http://go/cli/dashboard/... and show short text (error/3304, TestAccept/inittime).

(Reply from an AI agent: Claude Code, opus5.5, harness_auto.)

@denik denik changed the title acceptance/inittime: judge sum of per-package minimum init times acc: inittime: judge sum of per-package minimum init times Nov 11, 2026
@andrewnester
andrewnester self-requested a review November 12, 2026 08:58
@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 634e6bf Nov 12, 2026
28 checks passed
@denik
denik deleted the denik/inittime-sum-of-mins branch November 12, 2026 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants