Skip to content

Fix flaky TestLogStreamerTailFlushesWithoutFollow - #7505

Merged
denik merged 2 commits into
mainfrom
denik/fix-flaky-logstream-tail-flush
Nov 12, 2026
Merged

denik merged 2 commits into
mainfrom
denik/fix-flaky-logstream-tail-flush

Conversation

@denik

@denik denik commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Changes

TestLogStreamerTailFlushesWithoutFollow now checks the prefetch flush deterministically instead of with a wall-clock window:

  • The server keeps the socket open until the test ends. With follow: false, Run returns only after the prefetch timer flushes the tail, so Run returning proves the flush happened before any server close. The require.Eventually(..., 150ms) is gone.
  • A small waitDialer returns the connection only after the server has written all four entries. The prefetch timer starts after the dial, so the entries are already buffered on the client socket, and a slow runner cannot fire the timer early and flush a partial tail (e.g. line1, line2).
  • Prefetch goes from 50ms to 200ms for extra headroom, and the context timeout from 1s to 10s. A passing run still takes about 0.2s.
  • notifyBuffer lost its only hasWrite user, so it becomes a plain mutex-guarded syncBuffer.

Why

The old check gave the flush 150ms, counted from before the dial. The flush lands about 50ms after the dial plus the search write, and under -race on a busy runner that can exceed 150ms. The test then failed after about 0.25s (when the handler's sleep ends) and passed on retry.

This is a test bug, not a race in the streamer. consumeState flushes on the prefetch read deadline no matter what the server does. The test has not changed since apps logs was introduced (#3908, #4086). The flakiness only became visible around 2026-10-14, most likely from runner load rather than a code change. #6269 fixed the same kind of timing flake in the sibling TestLogStreamerTailBufferFlushes.

Dashboard:

  • error/84: flaky on main and PRs (linux, windows):
    streamer_test.go:443: Error: Condition never satisfied
    Messages: expected tail logs to flush before the server closed the socket
    --- FAIL: TestLogStreamerTailFlushesWithoutFollow (0.25s)
    

Tests

  • Before: reproduced 1 failure in 200 runs (-race, -cpu=1,2, 3x CPU oversubscription with busy loops).
  • After: 1000/1000 passes of -race -count=500 -cpu=1,8 -run 'TestLogStreamerTailFlushesWithoutFollow$' under the same load; the package passes without -race.
  • Out of scope: running the whole package with -race shows pre-existing data races (on main too) in other tests that read a plain bytes.Buffer from Eventually while the streamer writes to it. CI does not run with -race, so this PR leaves them alone.

Review notes

  • Product semantics are intentional and unchanged: prefetch is a fixed window after connect for collecting history before the tail is emitted. Entries a server sends after that window are not part of the tail in non-follow mode. Changing that (e.g. an idle-based timer) would be a behavior change, so it is out of scope here.
  • The test's server sends its entries without first reading the search request, because the client writes the search only after the dial returns. The search write is covered by TestLogStreamerSendsSearchTerm.

This pull request and its description were written by Isaac.

@denik denik added the ai-auto PRs made by AI loops & factories, label Nov 11, 2026
@github-actions github-actions Bot added the test-only-fix PRs that update failing of flaky tests without changing CLI code. label Nov 11, 2026
@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 53efda5

Run: 43102398535

Env 🔄​flaky ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 15 9:18
✅​ aws-windows-latest-4core-16gb 273 13 6:31
✅​ azure linux-2core-8gb 270 15 7:48
🔄​ azure-windows-latest-4core-16gb 1 271 13 6:47
✅​ gcp linux-2core-8gb 271 15 9:23
✅​ gcp-windows-latest-4core-16gb 273 13 7:25
Test Name azure-windows-latest-4core-16gb
🔄​ TestSyncIncrementalFileOverwritesFolder 🔄​f
Top 8 slowest tests (at least 2 minutes):
duration env testname
3:59 gcp linux-2core-8gb TestAccept
3:54 aws linux-2core-8gb TestAccept
3:49 azure-windows-latest-4core-16gb TestAccept
3:36 aws-windows-latest-4core-16gb TestAccept
3:31 gcp-windows-latest-4core-16gb TestAccept
3:15 azure linux-2core-8gb TestAccept
2:09 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:03 gcp linux-2core-8gb TestWorkspaceFilesExtensionsNotebooksAreNotDeletedAsFiles

denik and others added 2 commits November 11, 2026 21:21
The test gave the prefetch flush 150ms (measured from before the dial) to
show up, which a loaded runner can miss. Make it deterministic instead: the
server keeps the socket open until the test ends, so Run returning at all
proves the prefetch timer flushed the tail, and the dial returns only after
the server has written every entry, so the timer cannot fire on a partial
tail.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@denik
denik force-pushed the denik/fix-flaky-logstream-tail-flush branch from 87f861a to 53efda5 Compare November 11, 2026 21:21
@denik
denik marked this pull request as ready for review November 11, 2026 21:43
@denik
denik requested review from a team as code owners November 11, 2026 21:43
@denik
denik requested a review from rugpanov November 11, 2026 21:43
@denik
denik enabled auto-merge November 11, 2026 21:43
@andrewnester
andrewnester self-requested a review November 12, 2026 08:58
close(sent)
// Keep the socket open until the test is done: the only way for the
// streamer to emit the tail is the prefetch timer, not a server close.
<-release

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

t.Context() is canceled before Cleanup runs, so <-t.Context().Done() here (as the stopCtx handlers do) would drop release and the defer-ordering comment. Worth it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, worth it: t.Context() is canceled before the cleanup that waits for the handler, so the handler can wait on it directly. Done in #7543, which drops release and the defer-ordering comment.

AI agent (harness_auto, opus5.5, isaac_claude)

@denik
denik added this pull request to the merge queue Nov 12, 2026
Merged via the queue into main with commit 2df28b1 Nov 12, 2026
27 checks passed
@denik
denik deleted the denik/fix-flaky-logstream-tail-flush branch November 12, 2026 10:08

@rugpanov rugpanov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] The description says "a slow runner cannot fire the timer early and flush a partial tail." The timer starts on the client before the search write and the first read. A client stall of more than 200 ms in that window can still flush a partial tail. waitDialer removes only the server-side delay. Change "slow runner" to "slow server" so that the description matches the code comment.


require.Eventually(t, writer.hasWrite, 150*time.Millisecond, 10*time.Millisecond, "expected tail logs to flush before the server closed the socket")
require.NoError(t, <-done)
// Without follow, Run returns once the prefetch timer flushes the tail.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] This comment repeats the handler comment at line 441 and the failure message at line 472. Three places say that only the prefetch timer can end Run. Reduce this comment to one line: "Without follow, Run returns once the prefetch timer flushes the tail."

tedjt pushed a commit to tedjt/cli that referenced this pull request Nov 12, 2026
…s#7543)

## Changes

Follow-up to [databricks#7505](databricks#7505),
addressing [this review
comment](databricks#7505 (comment)).

The `TestLogStreamerTailFlushesWithoutFollow` server handler now waits
on `<-t.Context().Done()` instead of a `release` channel closed by a
`defer`. `t.Context()` is canceled just before `t.Cleanup` functions
run, so the handler unblocks before `newTestLogServer`'s cleanup waits
for it. This matches the `stopCtx` handlers elsewhere in the file and
removes the channel along with the comment about defer ordering.

## Tests

`go test -race -count=50 -run 'TestLogStreamerTailFlushesWithoutFollow$'
./libs/apps/logstream/` passes, and so does the whole package.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai-auto PRs made by AI loops & factories, test-only-fix PRs that update failing of flaky tests without changing CLI code.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants