Skip to content

Quarantine intermittent HeaderPlusOne split-packet test - #5073

Merged
mdaigle merged 1 commit into
mainfrom
dev/mdaigle/sqlclient-flaky-quarantine-a40
Nov 18, 2026
Merged

mdaigle merged 1 commit into
mainfrom
dev/mdaigle/sqlclient-flaky-quarantine-a40

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

🤖 This pull request was created by a fully automated workflow.

Description

Add [Trait("category", "flaky")] and an observed-error comment to this unparameterized Fact:

Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.HeaderPlusOneTest

The exact Fact timed out consuming the pre-login handshake acknowledgement, then passed in the same ordinary task's retry. Both observations use CI-SqlClient build 192284, main revision 5f74e8f13e38, Windows Server 2022 / SQL22 image / net462 / native SNI / x64 / Release / Project / Set 3, job attempt 1.

Observation Result Observed UTC
Failed 3721949/101751 2026-11-16 01:32:17.457–01:32:32.977
Passed retry 3721949/102143 2026-11-16 01:38:19.587–01:38:20.110

No configuration or revision changed between observations. Method-level quarantine affects this Fact on both frameworks; all test bodies, conditions, other categories and three ordinary siblings remain unchanged. No product, API, pipeline, repair or probation changes. This is quarantine, not a stability claim.

Issues

Related: #5065 quarantined the distinct MARSSplitTest; #5047's existing FullHeaderTest quarantine is untouched. No issue is closed. Fixes and all probation lifecycle work are outside this automation's scope.

Testing

Both framework builds/discovery passed: quarantine selects exactly three split-packet Facts; ordinary selection excludes them and retains three siblings. git diff --check passed. Exact commands/results and the complete evidence/checkpoint record are in the collapsed evidence comment.

Local execution was 0 passed / 1 skipped on each framework because configuration predicates threw TargetInvocationException; skipped-only execution is not passing coverage. Three CI run inventories (13,943 top-level results) were reconciled. The 525-build inventory covers 2026-11-02T16:56:18Z–2026-11-16T16:56:18Z, but broader per-test review remains incomplete after a build-wide MCP transport failure. Discovery checkpoint not advanced.

  • Tests added or updated - category and observed-error comment only
  • Public API changes documented - no API changes
  • Verified against customer repro (if applicable) - CI-only quarantine
  • Ensure no breaking changes introduced - no product or test-body changes

Guidelines

Please review the contribution guidelines before submitting a pull request:

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 16, 2026 17:13
Copilot AI balanced review requested due to automatic review settings November 16, 2026 17:13
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 16, 2026
@mdaigle mdaigle added the Area\Tests Issues that are targeted to tests or test projects label Nov 16, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 16, 2026
@mdaigle

mdaigle commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author
Quarantine evidence, validation and incomplete discovery checkpoint - 2026-11-16

Supported category-only change

Exact identity: Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.HeaderPlusOneTest.
This is an unparameterized conditional Fact. Add only its method-level
[Trait("category", "flaky")] and an immediately preceding observed-error comment:

// SqlException: connection timed out consuming the pre-login handshake acknowledgement.
[Trait("category", "flaky")]

Both existing target frameworks, net462 and net10.0, receive the method category.
The evidence below establishes intermittency on net462, not every platform.
The class's Set=3, all conditions, existing categories, test bodies and pipelines remain unchanged.
Existing FullHeaderTest and MARSSplitTest quarantines are preserved.
The other three split-packet Facts remain ordinary.

Proposal commit: 947d6f9918313f33117777a2d77c34a2211d38d1.
Inspected upstream main: 2bda3c9edb53ce59589cdd401f10ae4415f331a5.

Exact failure and passing control

Both observations are from CI-SqlClient definition 1879, build 192284,
main revision 5f74e8f13e38de44e0a1f184aabf4dad8b0bfbb5.
The run and individual result detail endpoints independently matched build ID, run ID,
result ID, exact test name and outcome. Both results have empty subResults.

Observation Exact result Observed UTC
Failed 192284 / 3721949 / 101751 2026-11-16T01:32:17.457Z - 2026-11-16T01:32:32.977Z
Passed, same ordinary task's retry 192284 / 3721949 / 102143 2026-11-16T01:38:19.587Z - 2026-11-16T01:38:20.110Z

Observed failure: Microsoft.Data.SqlClient.SqlException, connection timeout while
consuming the pre-login handshake acknowledgement; initialization=1 ms,
handshake=15009 ms; inner Win32Exception, "The wait operation timed out".
The source comment describes this symptom, not a speculative cause.

Configuration is identical for the two observations:

  • Stage Win22_Sql22, phase net462_AnyCPU_3, job __default;
    stage/phase/job attempt 1.
  • Unique timeline job 227d917a-c052-5356-3796-5eec3da988b1.
  • Windows Server 2022, native SNI, net462, x64, Release, Project references, Set 3.
  • Runner image windows-2022, version 20260105.167.1;
    hosted image ADO-MMS22-SQL22, version 106.0.0.
  • Same agent allocation, revision, framework, SQL image and selection.
    The only relevant difference is the ordinary task execution before/after its retry.

Mapped evidence:

  • Ordinary task/log 4040.
    Read lines 1-650 through authenticated MCP. Initial manual execution:
    515 passed, 1 failed, 102 skipped. At 2026-11-16T01:34:22.996Z
    RetryHelper reports retry 1 of 2. Identical command reruns at
    2026-11-16T01:34:24.445Z; retry totals are
    516 passed, 0 failed, 102 skipped.
    The selected filter is
    ((category!=failing&category!=flaky&category!=interactive)&category!=signed)&(Set=3).
    This task, not the later quarantine task, contains both decisive observations.
  • Quarantine task,
    separately mapped to log 4152. Its job status is not used as a passing control.
  • Initialization/log 263
    supplies actual OS and image versions.
  • Result publisher/log 4154
    is mapped by the same timeline; the authenticated build helper discovered run 3721949.

Rationale: one exact Fact fails and passes with no revision, job or configuration change.
It exercises a test-owned TCP proxy. The failed ordinary execution's other 515 tests passed;
there is no evidence here of a broad unavailable SQL endpoint or build infrastructure outage.
This supports quarantine, not a diagnosis, repair, stability claim or exclusion of all
possible product defects. The source change since the observed revision only quarantines
the different MARSSplitTest; no intervening HeaderPlusOne repair was found.

Discovery interval, inventories and coverage boundary

No reliable complete tagging checkpoint was found in the prior tagging records.
Rescanned completed-build finish times:
2026-11-02T16:56:18Z through 2026-11-16T16:56:18Z.
Prior fixes/probation records do not advance this checkpoint.

Live definition identity/status checks found the seven allowed definitions enabled.
Disabled sqlclient-ci-stress 2250 was verified and excluded.
Authenticated, bounded MCP build pagination produced 525 unique completed builds:

Definition Builds Succeeded Partially succeeded Failed Canceled
CI-SqlClient 1879 123 0 48 69 6
CI-SqlClient-Package 1917 67 0 29 33 5
PR-SqlClient-Project 2197 121 0 80 39 2
PR-SqlClient-Package 2198 20 0 14 6 0
sqlclient-pr 2281 166 121 0 42 3
sqlclient-ci-package 2300 28 28 0 0 0
sqlclient-ci-unit 2309 0 0 0 0 0
Total 525 149 171 189 16

First actual finish: 2026-11-02T17:08:10.7991101Z.
Last actual finish: 2026-11-16T16:54:33.4355346Z.
The terminal page omits continuationToken. Overlapping pages were deduplicated by build ID.

Build inventory complete; broader per-test review incomplete.
Only these top-level run inventories were completely paginated (skip/top, bounded at 1000)
and reconciled to server statistics:

Build / run Results Passed Failed Not executed
188372 / 3650219 4096 4038 1 57
188372 / 3650122 4351 4275 1 75
192284 / 3721949 5496 5264 1 231
Total 13943 13577 3 363

Result IDs were unique within each run; identities are (buildId, runId, resultId),
not titles. Server statistics include duplicate outcome keys distinguished by metadata:
3650219 Passed=4037 plus Passed=1/resultMetadata=flaky;
3650122 Passed=4273 plus Passed=2/flaky;
3721949 Passed=5262 plus Passed=2/flaky. These are summed, not overwritten.
All three runs were Completed with incompleteTests=0. NotExecuted is never a pass.
Unrelated grouped children were not exhaustively enriched; these are inventory counts,
not a claim that every unrelated parameterized case has been independently reviewed.

Additional build-helper discovery:
188334 (Passed/Failed/Aborted/NotExecuted), 188333 and 188372
(Failed/Aborted/Error/Timeout), and 192284 (Failed/Aborted/Error/Timeout).
188334's successful build exposed actual passes, but its run totals were not independently
reconciled, so it is not counted as complete coverage.
No build has exhaustive per-test review across all of its runs and outcomes.
The remaining builds, successful/canceled jobs, undiscovered outcomes and grouped children
remain unfinished; build success alone is not test evidence.

The build-wide Passed query for 188372 failed:
MCP server 'azure-devops': MCP request failed: Transport closed.
Mapped authenticated publisher log 3299 recovered run 3650122; its full per-run pagination
and individual result verification supplied a usable passing control for the older routing
candidate, but did not recover whole-window coverage.
The permitted public detail/timeline GETs returned HTTP 200 JSON and matching identities.
No anonymous bulk result/run lists, alternate accounts, credentials or REST writes were used.

Checkpoint NOT ADVANCED. The next tagging run must rescan its full 14-day window
unless a later genuinely complete tagging record establishes a checkpoint.
This is not an exhaustive candidate list.

Other candidates and deduplication

  • Unconfirmed/current relevance unresolved:
    Microsoft.Data.SqlClient.UnitTests.SimulatedServerTests.ConnectionRoutingTests.NetworkTimeoutAtRoutedLocation_RetryDisabled_ShouldFail
    (unparameterized Fact).
    On main c1e89abcd960d342f371bf9c9ea9350b61fb9da5,
    188372 / 3650219 / 102630 failed
    2026-11-02T19:05:43.520Z - 19:05:45.867Z:
    Assert.Contains did not find "Connection Timeout Expired" in the SqlException message.
    188372 / 3650122 / 102959 passed
    2026-11-02T19:05:47.103Z - 19:05:48.103Z.
    Exact result details were verified. Both are Win22_Sql19, net10.0 native SNI,
    x64/Release/Project, attempt 1, same image versions; manual Set 2 versus Set 3
    differs, but the unit commands are identical.
    Failure job 92219fbc-623d-57b3-f6fe-520b883a4174, unit task
    8a589fc1-c632-5123-03da-4a5c44b050c2, log 2433;
    passing job b3379570-0f6b-5ffc-fae5-d18fa093cc9f, unit task
    924664b1-6a8a-57f0-859f-9c26b83d158a, log 2457.
    The failing unit task has 2027 passes/1 failure/9 skips; its comparison has
    2028 passes/0 failures/9 skips. The similarly named Azure class is a distinct test
    and is not substituted as a control.
    Current run 3721949 also contains a net462 pass, result 103509, but that is a different
    framework and was not enriched as a decisive control.
    The older failure predates simulator repair #5000.
    Although that repair targets startup/disposal and this Fact's source is unchanged,
    current post-repair failing evidence was not established. Not tagged.
  • Unconfirmed: Microsoft.Data.SqlClient.ManualTesting.Tests.MarsReceiveTeardownTest.FatalError_WhileMarsReaderStreams_DoesNotFaultReceivePump(async: True, pooling: False, delay: 29),
    build 188372 / run 3650195 / result 101368. Helper reports Assert.Throws expected
    SqlException but no exception was thrown. No exact, comparable passing control was
    independently verified; no category change.
  • Unconfirmed: Microsoft.Data.SqlClient.ManualTesting.Tests.PoolIdentityImpersonationTest.NetOnlyImpersonation_Replenishment_PreservesLogin(usePoolV2: False, async: True),
    build 192284 / run 3721940 / result 101826. Helper reports expected impersonated login
    versus actual process login. No exact passing control or exclusion of a deterministic
    functional defect was established; no category change.
  • Unconfirmed/incomplete identity: Microsoft.Data.SqlClient.ManualTesting.Tests.AlwaysEncrypted.ApiShould.TestEnclaveExecutionApis,
    isAsync=True, build 192284 / run 3721980 / result 101629. The helper truncates the
    connection-string argument and omits the error. Recursive detail and comparable
    exact-case control remain unprocessed. No inference from its parent outcome; no tag.
  • TvpColumnBoundariesTests.TvpColumnBoundariesTest, SqlSchemaInfoTest.TestGetSchema
    and TestGetSchemaAsync already have flaky categories on upstream main; preserved.

Live historical inputs #4980 and
#4996 are merged.
Other inspected tagging records include
#5065,
#5038,
#5028,
#5021,
#5016,
#5024,
#5020 and
#5006.
#5047 was read only for context and is
not modified in any way. There was no equivalent open HeaderPlusOne proposal at the
last overlap check. Open PR metadata/body searches and merged source/history were used
for deduplication; unrelated pending fixes are not a global quarantine blocker.

Local validation

Commands ran in this automation's isolated worktree only:

dotnet restore src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj -p:Configuration=Release -v:q
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --no-restore -c Release --list-tests --filter "FullyQualifiedName~Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.&category=flaky" -v:q
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --no-build --no-restore -c Release --list-tests --filter "FullyQualifiedName~Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.&category!=failing&category!=flaky&category!=interactive" -v:q
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --no-build --no-restore -c Release --filter "FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.HeaderPlusOneTest&category=flaky" -v:q --logger 'trx;LogFilePrefix=header-plus-one' --results-directory 'C:\Users\mdaigle\.copilot\session-state\dcaa48f9-f8fc-4e46-8196-2ebd2ef83d07\files\validation' --blame-hang-timeout 2m
git diff --check
  • Restore passed. The first pre-restore list-tests attempt failed with NETSDK1004
    (missing project.assets.json); restoring the existing dependencies resolved it.
  • Build/discovery passed for net462 and net10.0. Quarantine selects exactly
    FullHeaderTest, HeaderPlusOneTest, MARSSplitTest on each framework.
  • Ordinary discovery passed for both frameworks, selecting exactly
    OneByteSplitTest, AlmostFullHeaderTest, MARSReplicateTest; the quarantined
    Facts are absent.
  • Local execution was 0 passed / 0 failed / 1 skipped on each framework.
    Configuration predicates IsTCPConnStringSetup, IsLocalHost and IsNotNamedInstance
    reported TargetInvocationException. Skipped-only execution is not passing coverage.
    TRX output records the exact Fact and skip reasons. No local stability claim.
  • git diff --check passed. The patch is exactly two inserted lines: symptom comment
    and category; no test-body, existing-category or pipeline edits.
  • A local reporting script initially raised KeyError for the omitted terminal
    continuationToken; the corrected inventory aggregation above completed successfully.

Current canonical upstream version resolved to 8.1.0-preview1,
existing open milestone 108. The one-time exclusion of 8.0.0-preview1/#101
does not apply. Upstream was fetched and the exact matching open milestone rechecked
immediately before publication. Live metadata for
#5073 verifies milestone
8.1.0-preview1/#108, the existing Area\Tests label and the exact required
automated-disclosure first line. Source and target are both dotnet/SqlClient;
source branch is dev/mdaigle/sqlclient-flaky-quarantine-a40, target is main.
The latest Validate milestone
and version-match check
both passed at 2026-11-16T17:14:53Z / 17:14:54Z. The initial creation event's
missing-milestone validation failed before the immediate assignment; subsequent
milestone-event and description-update checks passed. Broader PR CI is still pending;
no all-checks-passed claim is made.

Fixes, fix proposals, all probation lifecycle actions and promotions are outside this
automation's scope. No pipeline definitions, queued validation builds, permissions,
auto-merge settings or recurring automation were changed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Policy compliance could not be fully checked because trusted review-policy references from protected configuration or an immutable base were not supplied.

No open issues 🎉

What changed in this PR

Quarantines the intermittently failing HeaderPlusOneTest in SqlClient’s manual split-packet tests without changing test behavior or product code.

Changes:

  • Adds a method-level flaky category to HeaderPlusOneTest.
  • Adds a comment recording the observed pre-login handshake timeout; existing sibling quarantines and pipeline configuration remain unchanged.
File Description
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​SQL/​SplitPacketTest/​SplitPacketTest.cs Marks HeaderPlusOneTest as flaky and documents the observed timeout.

Review effort: Balanced

@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.93%. Comparing base (17bef7b) to head (947d6f9).
⚠️ Report is 8 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5073      +/-   ##
==========================================
+ Coverage   68.99%   69.93%   +0.94%     
==========================================
  Files         314      314              
  Lines       45857    69767   +23910     
==========================================
+ Hits        31638    48793   +17155     
- Misses      14219    20974    +6755     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 69.93% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@paulmedynski paulmedynski changed the title Quarantine intermittent HeaderPlusOne split-packet test Quarantine intermittent HeaderPlusOne split-packet test [workflow replay] Nov 16, 2026
@paulmedynski paulmedynski changed the title Quarantine intermittent HeaderPlusOne split-packet test [workflow replay] Quarantine intermittent HeaderPlusOne split-packet test Nov 16, 2026
@paulmedynski paulmedynski modified the milestone: 8.1.0-preview1 Nov 16, 2026
@mdaigle
mdaigle merged commit 0097df2 into main Nov 18, 2026
49 of 55 checks passed
@mdaigle
mdaigle deleted the dev/mdaigle/sqlclient-flaky-quarantine-a40 branch November 18, 2026 21:41
@github-project-automation github-project-automation Bot moved this from To triage to Done in SqlClient Board Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants