Skip to content

Quarantine intermittent MARS shared-connection concurrency test - #5095

Open
mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/flaky-quarantine-maintenance-1dc
Open

mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/flaky-quarantine-maintenance-1dc

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

🤖 This pull request was created by a fully automated workflow.

Description

Quarantine the unparameterized Fact
Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest
using [Trait("category", "flaky")] and an observed-error comment.
Only that method is affected on net462/net10.0; its body, conditions, Set=3,
other categories and sibling tests are unchanged.

Outcome Exact CI-SqlClient result Main revision Observed UTC
Failed: parallel-command execution timeouts 192583 / 3727645 / 102343 0304c75eb048 Nov 16, 22:47:28.767 - 22:49:00.800
Passed in a separate build/job allocation 192629 / 3728642 / 102024 0304c75eb048 Nov 17, 01:49:27.250 - 01:55:04.187

Both use Windows 11, net10.0, managed SNI, Azure SQL, Release/Project, Set 3,
attempt 1, image 56.0.0. The test shares one connection across parallel command
workers despite MARS operations not being thread-safe.
This concrete test-design weakness supports quarantine, not a claim that every
Azure timeout is a test defect or that the underlying behavior is repaired.

No APIs, product code or pipelines change. No fixes or probation changes.
Release-branch intermittency for this Fact remains unconfirmed; no backport label requested.
Full-window per-test discovery is incomplete and the checkpoint is not advanced.

Issues

#4589 reports this different, already-open-connection scenario but contains no
equivalent repair/quarantine. #5073 quarantines a different Fact.
Complete evidence and run record, including the inspected interval, unconfirmed candidates and incomplete-coverage boundary.

Testing

Actual quarantine selection and ordinary exclusion verified on both target frameworks.
net462 build passed with zero warnings/errors; the net10.0 discovery command also built.
Focused SQL execution was skipped because the isolated worktree lacks test configuration:
zero passed, zero failed, one skipped. This is not execution or stability coverage.
git diff --check passed.

$p = 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj'
$n = 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest'
foreach ($f in 'net462', 'net10.0') {
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category=flaky" -v:q
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category!=failing&category!=flaky&category!=interactive" -v:q
}

Result: exactly one quarantined Fact and intentional ordinary exclusion per framework.
The linked evidence record retains the build/execution commands and limitations.

Guidelines

Please review the contribution guidelines before submitting a pull request:

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 17, 2026 14:27
Copilot AI balanced review requested due to automatic review settings November 17, 2026 14:27
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 17, 2026
@mdaigle mdaigle added the Area\Tests Issues that are targeted to tests or test projects label Nov 17, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 17, 2026
@mdaigle

mdaigle commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author
Quarantine evidence, validation and incomplete discovery checkpoint - 2026-11-17

Supported category-only change

Exact identity: Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest.
This is one unparameterized conditional Fact. Add only its method-level
[Trait("category", "flaky")] and the immediately preceding observed-error comment:

// Assert.Fail: parallel MARS commands reported Execution Timeout Expired.
[Trait("category", "flaky")]

The Fact's conditions, class-level Set=3, other categories and test body are unchanged.
The method-level category applies to net462 and net10.0, not other methods in MARSTest.
No repair, retry implementation, pipeline change, probation transition or category removal.
Inspected upstream main and both decisive CI revisions:
0304c75eb048c6a309cccfc50629b38aca82d53b.
Proposal commit: 93df4ede9b2729448244aa57cd46935b875adee8.

Exact failure and passing controls

All observations below are for that exact, unparameterized Fact, with empty subResults.
Individual result details independently matched result ID, testRun ID, build ID and outcome.
The separate passing build, not a job status or grouped parent outcome, is the decisive control.

Observation Pipeline / build / run / result Observed UTC start - completion
Failed CI-SqlClient 1879 / 192583 / 3727645 / 102343 2026-11-16T22:47:28.767Z - 2026-11-16T22:49:00.800Z
Passed, separate build/job allocation, same SHA CI-SqlClient 1879 / 192629 / 3728642 / 102024 2026-11-17T01:49:27.250Z - 2026-11-17T01:55:04.187Z
Passed, first task retry in failing build CI-SqlClient 1879 / 192583 / 3727645 / 103268 2026-11-16T22:53:46.940Z - 2026-11-16T22:54:14.223Z
Passed, second task retry in failing build CI-SqlClient 1879 / 192583 / 3727645 / 103915 2026-11-16T22:58:49.140Z - 2026-11-16T22:59:16.320Z

Observed failure: the Fact catches an AggregateException from its parallel commands and
calls Assert.Fail with CRITIAL: Test should not fail randomly. Exception occurred:
followed by repeated Execution Timeout Expired errors. The test-source frame is
MARSTest.cs:696. The new source comment describes this observed symptom only.
The server truncates the long error text; no omitted exception detail is inferred.

Both decisive runs independently identify refs/heads/main, Release, definition 1879,
stage Win11_Azure_Sql, phase net10_0_AnyCPU_ManagedSNI_3, job __default,
stage/phase/job attempt 1. Unique timeline job identifier in each build:
1599937a-74da-5c18-7c9a-92d9034abbc4, parent phase
27dc93eb-4957-5b02-236f-894c994ee61c, parent stage
b741d4c7-06c8-53e8-475a-a9e126506e4a.

Configuration: Windows 11, x64, net10.0, managed SNI, Project references, Release,
Set 3, Azure SQL (not a verified local SQL Server). Both allocations report
Standard_D2a_v4, ADO-CI-Win11, image version 56.0.0, westus2.
Different agent allocations and execution times are the meaningful differences.
No code revision, OS, framework, SNI, selection or image-version change.
No local-server claim is made.

Mapped, authenticated MCP logs:

  • Failing ordinary task, build 192583, log 4508:
    lines 1-450 read. Initial ordinary manual execution: 884 passed, 3 failed,
    88 skipped. The other two failures are MaxPoolWaitForConnectionTest scheduler-state
    assertions (Expected: Running, Actual: WaitingToRun), not connectivity errors.
    At 2026-11-16T22:49:37.130Z the task announces retry 1 of 2.
    The effective filter is
    ((category!=failing&category!=flaky&category!=interactive)&category!=signed)&(Set=3).
  • Passing ordinary task, build 192629, log 4491:
    lines 1-250 read. Same TestSqlClientManual command dimensions, 887 passed,
    zero failed, 88 skipped. Skips are not counted as passes.
  • Initialization logs 2836 (192583, lines 1-85) and 2849 (192629, lines 1-25)
    provide the matching image/SKU and distinct allocations.
  • Passing result publisher, build 192629, log 4499:
    lines 1-70 independently discover run 3728642 and its 5,449 published results.
  • The failing build's separate quarantine task is task
    7b5f34e4-4c3b-55c2-50bd-394ffa5ef9c0, log 4510.
    The candidate's decisive observations belong to the ordinary task, not this task.

The UTC ranges in the table are the test-result service's observed fields.
Some result completion times run later than runner/publisher log timestamps;
clock/publication inconsistencies are retained rather than silently reconciled.
Identity/outcome comparisons do not depend on inferred wall-clock ordering across agents.

Test-specific eligibility, not just intermittency

MARSTest.cs:669-690 creates one MARS-enabled connection and shares it between
Parallel.For(0, 300, ...) workers, repeated five times, with each worker executing
its own command on that same connection. It asserts that this must never fail randomly.
The documented MARS thread-safety contract
explicitly states that MARS operations are not thread-safe, and true parallel command
execution requires multiple connections. This is a concrete source-backed test-design
weakness independently of whether a particular Azure SQL timeout could have other causes.

The observed failure occurs executing commands after connection and table setup, not
while acquiring a connection. Other 884 ordinary tests passed in that task; the only
two accompanying failures are the specific worker-scheduling assertions above.
Thus a generic unavailable server, DNS problem or pre-login interruption is not the
better explanation for quarantining this unsupported concurrent-use test.
Agent/server pressure can still exacerbate the weakness; this is not a complete root-cause
diagnosis, exclusion of every possible product defect, or a stability/repair claim.
No source change to this method intervenes between the decisive observations.

Discovery interval, inventory and coverage boundary

Prior tagging records #4980, #4996, #5047 and #5073 were read live.
They do not supply a fully processed discovery checkpoint. Fix/probation records do
not advance this checkpoint. The full completed-build finish-time window rescanned is
2026-11-03T14:02:17.402Z through 2026-11-17T14:02:17.402Z.

All seven specified definitions were rechecked by authenticated MCP for exact
ID/name and enabled status; disabled sqlclient-ci-stress 2250 was excluded.
Seven bounded MCP build pages (50, then five pages of 100, then 14) enumerate
564 unique completed builds, without restricting discovery to main.
The final page has no continuation token.

Definition Builds Succeeded Partially succeeded Failed Canceled
CI-SqlClient 1879 134 1 45 82 6
CI-SqlClient-Package 1917 77 0 30 40 7
PR-SqlClient-Project 2197 113 0 71 41 1
PR-SqlClient-Package 2198 23 0 14 9 0
sqlclient-pr 2281 189 135 0 51 3
sqlclient-ci-package 2300 28 27 0 1 0
sqlclient-ci-unit 2309 0 0 0 0 0

CI-SqlClient inventory includes main (49), release/6.1 (10), release/7.0 (13),
release/7.1 (25), release/8.0 (16), and 21 other source refs.
Release refs were also verified using upstream refs, not a hard-coded discovery filter.
No builds were queued and no definitions were changed.

Build inventory complete; broader per-test review incomplete.
Authenticated failure discovery was performed for builds 192583 and 192629 (main),
192610 (release/6.1), 192218 (release/7.0), 192746 (release/7.1), and
192723 (release/8.0). This is not exhaustive failure discovery across the 564 builds.
Successful, partially successful and canceled builds were inventoried; their
whole-window per-test results have not all been enumerated or reviewed.

Two relevant full runs were paginated with authenticated MCP (1,000-result bounded
pages) and reconciled by unique result ID and outcome to server run statistics:

Build / run Unique results Passed Failed NotExecuted
192583 / 3727645 7,379 7,080 5 294
192629 / 3728642 5,449 5,331 0 118

Both runs are Completed with incompleteTests=0. Passed counts sum duplicate
runStatistics keys, retaining the two resultMetadata=flaky passes in each run
(7,078+2 and 5,329+2). They are not overwritten or interpreted as additional failures.
No skipped/aborted/missing result supplies a passing control.
Individual decisive results were additionally verified through the permitted
read-only single-result detail endpoint with recursive subResults inspection.
Other discovered runs are only partially enriched, not declared fully processed.

Checkpoint NOT ADVANCED. The next tagging run must rescan its full 14-day window
with deduplication unless a later complete tagging record establishes a checkpoint.
In particular, per-branch whole-window review remains unfinished on main and all
four active release branches.

Unconfirmed / excluded observations

  • Microsoft.Data.SqlClient.ManualTesting.Tests.ConnectionPoolTest.MaxPoolWaitForConnectionTest:
    usePoolV2 true/false cases in 192583 and 192629 show a source-backed one-second
    worker-start assumption (Expected: Running, Actual: WaitingToRun).
    In 192629 / 3728643 / 102352 a Failed parent has both Failed and Passed children.
    Not sufficient: the connectionString display is truncated and the provider emits
    both MARS=false and MARS=true with the same displayed prefix. Exact case equivalence
    is unresolved, so this method is not tagged. That run's failed page was enumerated,
    but its remaining outcomes are not reconciled.
    Job Win11_Azure_Sql_ARM64.net10_0_AnyCPU_NativeSNI_3.__default, attempt 1,
    uniquely maps to job 804c0ea4-9f2b-50e2-bc02-8442f504bf2c; ordinary log 4487
    (lines 1-260) confirms the scheduler assertion. MCP artifact enumeration maps
    net10.0WinAz804c0ea4-9f2b-50e2-bc02-8442f504bf2c to that job (42,054,990 bytes).
    The MCP download call returned no usable payload; no unapproved REST endpoint
    was used to bypass it.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.SqlCommandCancelTest.TimeOutDuringRead_Tcp:
    verified Failed result 192629 / 3728604 / 101950,
    2026-11-17T01:30:09.787Z - 2026-11-17T01:30:10.837Z.
    Pre-login handshake reset by remote host, Win22_Sql16 / net10.0 / native SNI /
    Set 3, attempt 1. A proxy alone proves neither local upstream SQL nor defective test
    synchronization. No qualifying exact passing control or test-specific cause was
    established; do not quarantine. The separate fixes automation may evaluate bounded
    retries around transient setup/connectivity only, never assertions or the timeout
    behavior under test.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.PoolIdentityImpersonationTest.NetOnlyImpersonation_Replenishment_PreservesLogin(usePoolV2: False, async: True):
    verified Failed result 192583 / 3727416 / 101925,
    2026-11-16T22:34:02.380Z - 2026-11-16T22:34:03.900Z, Win22_Sql17,
    managed SNI, net10.0, attempt 1. Expected impersonated login, actual process login.
    This may expose product correctness; no evidence-backed test flaw or comparable
    exact pass was established. Not quarantined.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.AlwaysEncrypted.ApiShould.TestEnclaveExecutionApis
    with isAsync=true: truncated connectionString cases in main and release/8.0 remain
    unconfirmed. No case-equivalent passing controls or test-specific defect established.
  • VectorColumnMetadataTests schema observations and
    MetricsTest.ReclaimedConnections_BalanceSoftConnectionCounters(usePoolV2: False, async: True)
    are already covered by merged quarantines Quarantine intermittent vector schema collection tests #5024 and Quarantine intermittent reclaimed-connection metrics theory #5016. No category removal.
  • Release/6.1 build 192610: XEventsTracingTest timeouts and simulator failover failures
    require further case/configuration review; no main eligibility is inferred.
  • Release/7.0 build 192218: 653 discovered failures include widespread
    DiagnosticTest child-process assembly-load failures; not classified as intermittent
    test flaws. No passing-control equivalence or complete result coverage claimed.
  • Release/7.1 build 192746: transaction/failover/TVP failures remain incompletely
    reviewed; existing backport proposal [7.1.3 Cherry-pick] Manage SimulatedServer flaky test failures #5087 is deduplication context.
  • Release/8.0 build 192723: enclave, AsyncTimeoutTest and SplitPacketTest.MARSSplitTest
    failures remain incomplete; the different MARSSplitTest already has main quarantine
    Quarantine intermittent MARS split-packet test #5065. No release pass is substituted for a main pass.

No same-test/symptom release-branch failing/passing pair was established for
MARSTest.MarsConcurrencyTest. Therefore no Hotfix version label is requested,
no backport is reported created, and no patch target is guessed. Release source/diff
compatibility and patch-label resolution would still be required before routing.

Related proposals and state

  • #5073: different HeaderPlusOne Fact,
    open; left unchanged. Live disclosure, Area\Tests and milestone 8.1.0-preview1 verified.
  • #4589: mentions MarsConcurrencyTest
    failures but its current diff explicitly contains no MARS repair or quarantine.
    Read its body/files; not an equivalent pending proposal.
  • #5094: independent shared-transaction
    worker-starvation repair/probation proposal; no overlap, no checkpoint advancement.
  • #5064: exception-clone assertion
    repair; no overlap.
  • #5087: release/7.1 simulator
    quarantine backport; different test, no overlap.
  • Historical tagging records #4980
    and #4996 are merged.
    Main merge history and current open category/fix proposals were rechecked.
    Protected Quarantine intermittent split-packet full-header test #5047 was read only; no branch, body, comment, label or milestone edits.

Local validation and limitations

Exact commands executed from the isolated worktree:

dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --framework net10.0 --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest&category=flaky' --verbosity quiet
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --framework net10.0 --no-build --no-restore --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest&category!=failing&category!=flaky&category!=interactive' --verbosity quiet
dotnet build src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --framework net462 --no-restore --verbosity quiet
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --framework net462 --no-build --no-restore --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest&category=flaky' --verbosity quiet
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --framework net462 --no-build --no-restore --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest&category!=failing&category!=flaky&category!=interactive' --verbosity quiet
dotnet test src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj --framework net10.0 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest&category=flaky' --verbosity quiet
git diff --check

Results: the first command built net10.0 and selected exactly the intended Fact.
The Framework build passed with zero warnings/errors. Both target frameworks'
quarantine listings selected exactly that Fact; both ordinary conjunctions excluded
it (intentional zero selection, not execution coverage).
The execution command had 0 passed, 0 failed, 1 skipped.
A repeat with --logger 'console;verbosity=normal' identified the actual limitation:
FileNotFoundException: Could not find test configuration file, causing
AreConnStringsSetup and IsNotAzureSynapse TargetInvocationException conditions.
No SQL execution, local reproduction, repaired behavior or stability is claimed.
git diff --check passed; the entire source diff is one attribute and its symptom comment.

Operational errors/limitations: the optional signing-config lookup returned unset;
signature inspection reported cannot spawn gpg: No such file or directory.
No signing configuration or access policy was changed or bypassed.
MCP artifact download above returned no usable payload. No account/repository switching,
credential extraction, anonymous bulk result-list requests or validation-build queueing.

Release assignment and publication requirements

Fresh main SqlClientNextVersion is 8.1.0-preview1; the exact matching existing open
milestone is #108. The one-time exclusion of 8.0.0-preview1 #101 no longer applies.
Required label is the existing literal Area\Tests. Required visible first line:
🤖 This pull request was created by a fully automated workflow.
Live REST verification of the created PR confirmed: open; source and target both
dotnet/SqlClient; base main; source dev/mdaigle/flaky-quarantine-maintenance-1dc;
head 93df4ede9b2729448244aa57cd46935b875adee8; exactly one changed source file,
two added lines and no deletions; exact disclosure first line; Area\Tests present;
open milestone 8.1.0-preview1 #108 assigned; auto-merge disabled.
Exactly one collapsed evidence comment for this originating session was verified.
The description links that comment. The isolated worktree is clean after commit.

Fixes and all probation lifecycle work remain outside this tagging automation's scope.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed because no trusted immutable source was established for the required review policies and references, leaving the policy check incomplete.

No open issues 🎉

What changed in this PR

Quarantines the intermittent MarsConcurrencyTest manual test without changing driver behavior or attempting to fix the underlying shared-connection concurrency issue.

Changes:

  • Adds [Trait("category", "flaky")] to this Fact on both target frameworks.
  • Adds a comment recording the observed parallel-command execution timeouts.
  • Preserves the test body, conditions, other categories, and sibling tests.
File Description
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​SQL/​MARSTest/​MARSTest.cs Marks MarsConcurrencyTest as flaky and documents the observed failure.

Review effort: Balanced

@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.50%. Comparing base (17bef7b) to head (93df4ed).
⚠️ Report is 11 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5095      +/-   ##
==========================================
+ Coverage   68.99%   72.50%   +3.51%     
==========================================
  Files         314      314              
  Lines       45857    71062   +25205     
==========================================
+ Hits        31638    51525   +19887     
- Misses      14219    19537    +5318     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 72.50% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants