Skip to content

Quarantine intermittent MARS replicate split-packet test - #5125

Open
mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/sqlclient-flaky-quarantine-17
Open

mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/sqlclient-flaky-quarantine-17

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

🤖 This pull request was created by a fully automated workflow.

Description

Quarantine the unparameterized Fact
Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest
with a method-level [Trait("category", "flaky")] and observed-error comment.
Only this method is rerouted on net462/net10.0; its body, conditions, Set=3,
other categories and sibling tests remain unchanged. No APIs or pipelines change.

All controls use main SHA 091d505c9a27, CI-SqlClient,
Win22_Sql17/net10.0/x64/managed SNI/Release/Project, ordinary Set 3.

Outcome Exact build/run/result Observed UTC start - end
Failed: pre-login handshake timeout 193792 / 3745573 / 101502 2026-11-20 01:24:03.203 - 01:24:18.740
Passed, same-task retry 193792 / 3745573 / 102650 2026-11-20 01:29:39.873 - 01:29:41.590
Passed, separate build/job 193799 / 3745619 / 101595 2026-11-20 01:31:43.270 - 01:31:45.800

The upstream SQL Server is verified local, not merely behind a loopback proxy.
The proxy queues blocking forwarding workers, does not terminate on EOF, and
disposes neither their TCP clients nor their lifetimes. That concrete harness
weakness, the isolated handshake failure and comparable actual passes support
quarantine rather than a generic Azure/network timeout. The exact CI
interleaving is not diagnosed or repaired.

Issues

Related: #5065 and #5073 quarantine different split-packet methods; open #5095
and #5118 cover different tests. No issue is closed.
Fixes and probation are outside this automation's scope.

Complete collapsed evidence and run record, including exact identities, configuration comparisons, validation and unfinished coverage.

Testing

Both target frameworks compile. These commands verify quarantine selects exactly
MARSReplicateTest and the ordinary class filter retains only OneByteSplitTest and
AlmostFullHeaderTest:

$p = 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj'
$n = 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest'
foreach ($f in 'net462', 'net10.0') {
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category=flaky" -v:q
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter 'FullyQualifiedName~Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.&category!=failing&category!=flaky&category!=interactive' -v:q
}

Result: all four discovery checks passed. git diff --check passed.
Focused local execution: net10.0 skipped; net462 failed with a proxy handshake
timeout alongside Northwind login/setup failures. Neither provides local passing
or stability coverage; exact execution commands/results are in the evidence record.

Discovery interval: 2026-11-06T14:02:56.290Z - 2026-11-20T14:02:56.290Z.
519 builds enumerated; four run inventories reconcile 21,472 top-level results.
Broader per-test coverage, including release/6.1, /7.0, /7.1 and /8.0, remains
incomplete. Checkpoint not advanced. Release-specific MARS eligibility is
unconfirmed; no Hotfix label/backport is requested.
Current milestone: 8.1.0-preview1; the 8.0.0-preview1 exception no longer applies.

Guidelines

Please review the contribution guidelines before submitting a pull request:

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 20, 2026 14:22
Copilot AI balanced review requested due to automatic review settings November 20, 2026 14:22
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 20, 2026
@mdaigle mdaigle added the Area\Tests Issues that are targeted to tests or test projects label Nov 20, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 20, 2026
@mdaigle

mdaigle commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor Author
Verified quarantine evidence and incomplete discovery checkpoint - 2026-11-20

Supported transition

Add a method-level [Trait("category", "flaky")] and immediately preceding
observed-error comment to the unparameterized Fact
Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest.
Source: src/Microsoft.Data.SqlClient/tests/ManualTests/SQL/SplitPacketTest/SplitPacketTest.cs.
It retains its body, conditional attributes and class-level Set=3.
The change routes this one method on net462/net10.0; other cases, existing
quarantines, APIs and pipelines are unchanged. No probation metadata is added.

All three exact results below report manualtests.dll and the identical fully
qualified name above. They are ungrouped; verified detail responses contain no
child cases.

Outcome Verified build/run/result Main commit SHA Observed UTC start - end
Failed 193792 / 3745573 / 101502 091d505c9a27ff88e1618273d8ce01b3701c057b 2026-11-20T01:24:03.203Z - 01:24:18.740Z
Passed, ordinary-task retry 193792 / 3745573 / 102650 091d505c9a27ff88e1618273d8ce01b3701c057b 2026-11-20T01:29:39.873Z - 01:29:41.590Z
Passed, separate build/job execution 193799 / 3745619 / 101595 091d505c9a27ff88e1618273d8ce01b3701c057b 2026-11-20T01:31:43.270Z - 01:31:45.800Z

The observed failure is SqlException: Connection Timeout Expired while
consuming the pre-login handshake acknowledgement, with initialization=2 ms
and handshake=15024 ms, wrapping Win32Exception: The wait operation timed out.
The mapped log locates it at SplitPacketTest.OpenMarsConnection, during
conn.Open(), before executing the REPLICATE queries. The new source comment
describes that verified symptom, not an inferred root cause.

Both builds are CI-SqlClient, definition 1879, on refs/heads/main.
Run identities verify stage Win22_Sql17, phase
net10_0_AnyCPU_ManagedSNI_3, job __default; stage/phase/job attempts are all 1.
Unique timeline parent chains map to job
fa3e00d2-1f65-56e7-149a-37d19dd1b181 within each respective build.
The same GUID in separate build timelines does not mean the same execution.
Both ordinary tasks use net10.0, x64/AnyCPU, managed SNI, Release,
ReferenceType=Project and TestSet=3. Generated family package suffixes differ
only by build number (32426 versus 32427); source SHA is identical.

Mapped authenticated logs:

Test-specific eligibility, not merely intermittency

The test's upstream IsLocalHost condition requires the original
TCPConnectionString hostname to equal localhost
(DataTestUtility.cs:703-706). This is not an inference from the proxy's
127.0.0.1 listener. The mapped SQL service/readiness log independently verifies
SQL Server on this agent. Both selected executions ran, rather than skipped.

SplitPacketTest.cs:27-40,130-167,183-189 provides a concrete proxy lifecycle
weakness: listener readiness alone releases the constructor; forwarding workers
are queued with Task.Factory.StartNew and perform blocking stream reads;
zero-byte reads do not terminate the forwarding loops; disposal cancels a token
and closes only the listener, without closing the accepted/upstream TcpClients
or joining their workers. Cancellation cannot interrupt those synchronous reads.
This creates an avoidable test-specific dependency on worker scheduling and
leaves forwarding lifetimes unbounded. It is a reasonable suspicion for a
handshake stall through this harness, not proof of the exact CI interleaving.

The failing run has only this one failed top-level result. The same local server
passed OneByteSplitTest (101698, 01:23:47.140Z-01:23:47.667Z) and
AlmostFullHeaderTest (102013, 01:23:47.657Z-01:23:48.173Z) nearby; its ordinary
retry and separate same-revision/configuration job also passed MARSReplicateTest.
Together with the source weakness and verified local upstream, this is stronger
evidence than generic Azure/network disruption or a successful retry alone.
Resource pressure may expose the weakness; it is not ruled out as a contributing
factor. No timing guarantee, repaired implementation or stability is claimed.
The local developer execution below is NOT used to establish eligibility.

Inventory, reconciliation and checkpoint

No reliable fully processed tagging checkpoint was found in the prior tagging
records inspected, including #5118, #5105 and #5073. Fix/probation records are
not checkpoints. This run therefore used the full interval
2026-11-06T14:02:56.290Z through 2026-11-20T14:02:56.290Z.

Authenticated, paginated build enumeration reached a null continuation token:
519 unique completed builds (196 succeeded, 207 failed, 102 partially
succeeded, 14 canceled). All seven requested definitions were rechecked live:
1879, 1917, 2197, 2198, 2281, 2300 and 2309 had the expected names and Enabled
status. Definition 2250 was Disabled and excluded. No completed builds for 2309
were returned in the interval. No branch filter excluded release builds.

Definition Completed builds enumerated
CI-SqlClient / 1879 129
CI-SqlClient-Package / 1917 75
PR-SqlClient-Project / 2197 61
PR-SqlClient-Package / 2198 19
sqlclient-pr / 2281 207
sqlclient-ci-package / 2300 28
sqlclient-ci-unit / 2309 0

Build enumeration is not per-test coverage. Four discovered runs were paged
without outcome exclusions to exhaustion and reconciled by unique result ID
and outcome counts against server run statistics:

Build / run Unique top-level results Passed Failed NotExecuted
189953 / 3676960 4578 4448 1 129
192488 / 3725187 5747 5482 1 264
193792 / 3745573 5890 5625 1 264
193799 / 3745619 5257 5110 0 147

Total: 21,472 top-level records. Duplicate Passed statistic entries were
summed, retaining the separate resultMetadata=flaky count of 2 in each
run. They were not overwritten or treated as unique title counts.
These are run inventories, not complete nested/parameterized-case inspection
or whole-build coverage. Exact failed/passing MARS results were additionally
identity-checked through the allowed single-result endpoint.

Build helpers discovered failure records for main/release builds 189953,
189981, 192488, 192391, 193792, 193799, 193818 and 193899. Build 190099 returned
empty Failed and Passed helper responses; no passing coverage is inferred.
The Passed helper for 189953 failed with
MCP server 'azure-devops': MCP request failed: Transport closed.
Authenticated per-run pagination remained available, and mapped publication
logs supplied the independent MARS passing run instead. The transport failure
does not mean all subsequent access was unavailable.

Checkpoint NOT ADVANCED. Other runs in these builds and per-test evidence
for the remaining scoped builds are unfinished. Grouped/insufficiently detailed
results, canceled builds and missing/expired artifacts remain coverage gaps,
not passes. Future discovery must rescan its full 14-day window unless a
separate trustworthy fully processed tagging checkpoint is established.

Unconfirmed and deduplicated candidates

  • Microsoft.Data.SqlClient.ManualTesting.Tests.TransactionDisposeRaceTest.Dispose_ConnectionClosedAfterUnacknowledgedTimeout_DoesNotThrow:
    192488 / 3725187 / 101820,
    2026-11-16T18:12:31.193Z-18:12:42.500Z, SHA
    2bda3c9edb53ce59589cdd401f10ae4415f331a5, expected SqlException,
    actual InvalidOperationException. Exact retry
    102119
    passed at 18:18:51.390Z-18:18:57.673Z, Win22_Sql16/net10.0/managed SNI.
    The same assertion was discovered in 193799 / 3745737 / 101893.
    Source and merged #4972
    were inspected. The underlying task exception is hidden by the type assertion;
    current evidence cannot distinguish product behavior from a test-specific
    race. Do not tag.
  • Microsoft.Data.SqlClient.Tests.AADAuthenticationTests.IsDummySqlAuthenticationProviderSetByDefault:
    189953 / 3676960 / 100511,
    2026-11-06T14:25:33.027Z-14:25:33.030Z, SHA
    92aa2a1b0ca2318e8390dbd17f7c24779a30c0e7, Assert.NotNull actual null.
    Win22_Sql17/net462; mapped ordinary log 2581. No exact comparable passing
    control or test-specific mechanism established. Do not tag.
  • SqlAuthenticationProviderManagerTests.DefaultAuthenticationProviders_AppConfig,
    AlwaysEncrypted ApiShould.TestEnclaveExecutionApis, and the LocalDBTest
    initializer failures remain unconfirmed. Null provider, invalid XML result,
    changed async implementations, or shared setup failure are not automatically
    defective intermittent tests. Generic connectivity/setup failures are not
    quarantine evidence. Where appropriate, the separate fixes automation can
    investigate bounded transient setup retries, never retry assertions or mask
    deterministic errors.
  • Existing vector schema metadata and pool-worker quarantines are already
    effective in main. MARS concurrency and EventSource listener enumeration
    are covered by open #5095
    and #5118.
    #5119 is repair/probation
    context only. No equivalent pending MARSReplicateTest proposal was found.

Release-branch coverage and routing

Actual CI-SqlClient source branches discovered and checked against upstream refs:

Branch Completed definition-1879 builds Current MARSReplicateTest source Coverage / hotfix decision
release/6.1 7 Exists, unquarantined Per-test evidence unfinished; no label
release/7.0 9 Exists, unquarantined Per-test evidence unfinished; no label
release/7.1 25 Exists, unquarantined Partial failure discovery; no exact branch-specific MARS pass/fail qualification; no label
release/8.0 20 Exists, unquarantined Per-test evidence unfinished; no label

Release/7.1 build 193899 is SHA
a162c938d4d268967f7779368ecaf8f2271b7626.
Authenticated failed-result pagination of run 3747150 and verified detail
identified:

  • Microsoft.Data.SqlClient.ManualTesting.Tests.DistributedTransactionTestWindows.Test_EnlistedTransactionPreservedWhilePooled,
    193899 / 3747150 / 101958,
    2026-11-20T09:36:43.970Z, InvalidOperationException changing the initialized
    process-wide ImplicitDistributedTransaction setting.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.TvpColumnBoundariesTests.TvpColumnBoundariesTest,
    result 101953, 09:36:46.817Z-09:36:48.043Z, Assert.True actual false.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.TransactionPoolTest.TransactionCleanupTest,
    result 102227, 09:36:30.567Z-09:36:31.330Z, grouped/truncated connection case:
    child detail remains unfinished.

The run maps uniquely to Win25_Sql25/net9_0_AnyCPU_ManagedSNI_3, job attempt 1,
job 4260b448-0b2f-5c71-3ce7-6a9f286aa888, ordinary log 6124.
Statistics report 4032 total, 3992 Passed, 3 Failed, 37 NotExecuted; only the
failed page was enumerated, so passing and grouped-case coverage is incomplete.
These distinct release-only failures are NOT main MARS quarantine evidence.
No exact same-test/symptom branch-specific MARS passing/failing pair was verified
on a release branch. No Hotfix label is justified by this record.

The upstream hotfix workflow/extractor were read: version labels backport the
entire merged main PR to release/.. Existing
#5087 and
#5109 concern other tests.
Current labels were read, but no patch target was selected absent release
qualification. No manual backport, build queueing or routing claim is made.

Validation and local execution limitations

The initial command failed with NETSDK1004 because the isolated worktree lacked
NuGet assets:

dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' -f net10.0 --no-restore --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest&category=flaky' -v:q

Repeating without --no-restore restored existing dependencies, compiled the
project, and listed exactly MARSReplicateTest. The net462 command with
--no-restore --list-tests also compiled and listed exactly this method.
No dependencies, versions or feeds were changed.

Final discovery commands and their asserted results:

$p = 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj'
$n = 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest'
foreach ($f in 'net462', 'net10.0') {
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category=flaky" -v:q
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter 'FullyQualifiedName~Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.&category!=failing&category!=flaky&category!=interactive' -v:q
}

Quarantine selection: exactly MARSReplicateTest on both frameworks.
Ordinary class selection: exactly OneByteSplitTest and AlmostFullHeaderTest,
with MARSReplicateTest excluded on both frameworks. Existing quarantines remain.

Focused execution was attempted with:

dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' -f net10.0 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest&category=flaky' -v:q
dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' -f net462 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest&category=flaky' -v:q
dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' -f net462 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.SplitPacketTest.MARSReplicateTest&category=flaky' --logger 'console;verbosity=normal' -v:q

net10.0: 0 passed, 1 skipped. net462: 0 passed, 1 failed, repeated for diagnostics.
The diagnostic attempt reports a proxy pre-login timeout alongside repeated
Northwind database login/setup failures. This local environment is not a clean
control, and the matching timeout is NOT additional proof of a defective test.
No local passing or stability coverage is claimed; no server/configuration
changes or repair attempts were made.

git diff --check passed. The diff adds exactly the observed-error comment and
one flaky trait; removing those two lines restores the exact Git-normalized
upstream file. An initial final-check wrapper had a PowerShell ${f} interpolation
syntax error, and its first byte check did not normalize working-tree CRLF to
Git LF. Both checks were corrected and passed before publication; neither error
was a test/product failure.

Related records, publication and scope

Historical discovery inputs #4980
and #4996 were rechecked live and
are merged. Related split-packet category proposals
#5065 and
#5073 are merged and cover
different methods. Protected PR #5047 and its metadata/comments were not changed.
The final open-PR overlap check found no equivalent proposal for this method.

Upstream main at 091d505c9a27ff88e1618273d8ce01b3701c057b has
SqlClientNextVersion=8.1.0-preview1; the exact open milestone is #108.
The one-time exclusion of 8.0.0-preview1/#101 no longer applies.
Published #5125, targeting main
from dotnet/SqlClient:dev/mdaigle/sqlclient-flaky-quarantine-17, with commit
e6090a6bbb13e8a3665d10e9f449b282bf782dc3. Live metadata verifies milestone
8.1.0-preview1 (#108), the existing literal Area\Tests label, and the exact
visible fully automated disclosure first line. The description links this single
collapsed comment; the live diff contains exactly two added lines and no
deletions. No human milestone choice was overwritten. No Hotfix label is justified
or requested.

The initial milestone check
failed because its creation-event snapshot had no milestone before assignment.
The subsequent automatically triggered
milestone validation
and next-version check
passed after assignment/body update. No workflow was manually rerun.
The PR-triggered Azure DevOps validation and CodeQL were still pending/in progress
at publication verification; publication does not claim CI completion.

Run outcome: one confirmed category-only quarantine proposal, with exact selection
and ordinary exclusion verified on both frameworks. The four reconciled inventories
contain 21,472 top-level results across a complete 519-build inventory for
2026-11-06T14:02:56.290Z - 2026-11-20T14:02:56.290Z. Transaction, authentication,
Always Encrypted, LocalDB and release-only candidates remain unconfirmed as detailed
above. Broader main/PR/release per-test and nested-case coverage remains unfinished;
the Transport closed retrieval error and local skipped/failed executions are
recorded above. The discovery checkpoint is not advanced.

Fixes and all probation lifecycle work are outside this automation's scope.
No product/test-body repairs, category removals, probation changes, unrelated PR
updates, merges, approvals, auto-merge, pipeline changes, automation changes or
manually queued builds were performed.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Trusted review-policy references and current PR feedback were unavailable, and supplied CI evidence was not independently verified to distinguish harness intermittency from a driver regression.

No open issues 🎉

What changed in this PR

Quarantines the intermittent SplitPacketTest.MARSReplicateTest in the SqlClient manual test suite. This changes test routing only; it does not repair the underlying timeout or alter driver behavior.

Changes:

  • Adds a method-level flaky category to route the test into quarantine on net462 and net10.0.
  • Adds a comment recording the observed pre-login handshake timeout.
  • Preserves the test body, eligibility conditions, Set 3 membership, sibling tests, and pipelines.
File Description
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​SQL/​SplitPacketTest/​SplitPacketTest.cs Marks MARSReplicateTest as flaky and documents the observed timeout.

Review effort: Balanced

@codecov

codecov Bot commented Nov 20, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.78%. Comparing base (17bef7b) to head (e6090a6).
⚠️ Report is 25 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5125      +/-   ##
==========================================
+ Coverage   68.99%   72.78%   +3.79%     
==========================================
  Files         314      315       +1     
  Lines       45857    71306   +25449     
==========================================
+ Hits        31638    51902   +20264     
- Misses      14219    19404    +5185     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 72.78% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants