Skip to content

Quarantine intermittent EventSource listener enumeration test - #5118

Open
mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/flaky-quarantine-maintenance-2c1
Open

mdaigle wants to merge 1 commit into
mainfrom
dev/mdaigle/flaky-quarantine-maintenance-2c1

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

🤖 This pull request was created by a fully automated workflow.

Description

Quarantine the unparameterized Fact
Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll
with a method-level [Trait("category", "flaky")] and the required observed-error
comment. Its body, condition, Set=3, other categories and sibling tests are
unchanged; routing applies on net462 and net10.0. No APIs or pipelines change.

Outcome Exact CI-SqlClient result Main revision Observed UTC
Failed: collection modified during event-ID enumeration 193405 / 3739426 / 101491 6a1ad2ad846f 2026-11-18 23:30:53.257-23:30:53.267
Passed, same-job task retry 193405 / 3739426 / 102625 6a1ad2ad846f 2026-11-18 23:36:20.080-23:36:20.100
Passed, separate build/job allocation 193423 / 3739834 / 101445 981a989dd1b4 2026-11-19 01:05:39.310-01:05:39.320

Both jobs use Windows Server 2022, net10.0 native SNI, SQL22 matrix, Release/Project,
Set 3 and image 106.0.0. The independent pass advances main by async-AE #5080;
this Fact, its listener helper and test-step template are unchanged.

The test enumerates a mutable List<int> while its event listener remains enabled
and can append to that list. This source-backed lifecycle/collection weakness
matches the observed InvalidOperationException; generic connectivity does not
explain that symptom. A successful retry alone is not the eligibility rationale.
This is quarantine, not a repair or stability claim.

No release branch has a verified within-branch qualifying pair for this Fact;
no Hotfix label/backport requested. Fixes and probation are outside scope.

Issues

No equivalent pending proposal found. #5095 quarantines a different MARS Fact;
#5117/#5085 concern an enclave retry product defect, not this listener assertion.
Complete collapsed evidence and run record,
including configuration comparisons, related PRs, unconfirmed candidates and checkpoint status.

Testing

Both framework builds/discoveries succeeded. Actual routing checks below produced
exactly one quarantined Fact and intentional ordinary exclusion per framework:

$p='src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj'
$n='FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll'
foreach ($f in 'net462','net10.0') {
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category=flaky" -v:q
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category!=failing&category!=flaky&category!=interactive" -v:q
}

Focused execution: net10.0 skipped (0 passed/0 failed/1 skipped); net462
failed during database login (0 passed/1 failed/0 skipped: Northwind unavailable
to the configured login). Neither establishes local execution/stability coverage.
The linked record retains exact build/execution commands and results.
git diff --check passed; the diff is two added lines in one file.

Discovery interval: 2026-11-05T16:52:41.935Z–2026-11-19T16:52:41.935Z.
527 builds inventoried; two full run inventories reconcile 11,150 results.
Broader per-test coverage remains incomplete after an MCP transport failure;
checkpoint not advanced.

Guidelines

Please review the contribution guidelines before submitting a pull request:

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 19, 2026 17:10
Copilot AI balanced review requested due to automatic review settings November 19, 2026 17:10
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 19, 2026
@mdaigle mdaigle added the Area\Tests Issues that are targeted to tests or test projects label Nov 19, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 19, 2026
@mdaigle

mdaigle commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author
Quarantine evidence, validation and incomplete discovery checkpoint - 2026-11-19

Category-only scope and exact identity

Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll
is one unparameterized conditional Fact in
src/Microsoft.Data.SqlClient/tests/ManualTests/TracingTests/EventSourceTest.cs.
The method-level addition is exactly the observed-error comment followed by
[Trait("category", "flaky")]. Its body, condition, inherited Set=3, other
methods/categories and both target frameworks remain unchanged. No fixes,
probation, pipelines or public APIs are changed.

Verified failures and actual passing controls

All three results have automatedTestStorage=manualtests.dll, the exact name
above, and no subresults. Public detail responses were HTTP 200 JSON and matched
the discovered build ID, run ID, result ID and outcome.

Outcome Exact result Source SHA Observed UTC start - completion
Failed 193405 / 3739426 / 101491 6a1ad2ad846fad030493842c9b4b62a222be0889 2026-11-18T23:30:53.257Z - 23:30:53.267Z
Passed, same-job task retry 193405 / 3739426 / 102625 6a1ad2ad846fad030493842c9b4b62a222be0889 2026-11-18T23:36:20.080Z - 23:36:20.100Z
Passed, separate build/job allocation 193423 / 3739834 / 101445 981a989dd1b4bc9eed889286d141a8234b8460d5 2026-11-19T01:05:39.310Z - 01:05:39.320Z

Observed failure: System.InvalidOperationException: Collection was modified; enumeration operation may not execute. The stack identifies
System.Collections.Generic.List<int>.Enumerator.MoveNext() and
EventSourceTest.EventSourceTestAll, line 31. This exactly matches the new source
comment. It is not a connection timeout or a failed SQL query.

Both builds are CI-SqlClient / definition 1879, refs/heads/main.
Both run records identify stage Win22_Sql22, phase
net10_0_AnyCPU_NativeSNI_3, job __default, with stage/phase/job attempt 1.
The unique timeline chain in each build is:

  • Stage: 481ba6e4-2a3b-5a77-7c11-2fcac2ba7139.
  • Phase: 1c00573a-0c5b-556d-657e-cf53577f5630, parent = stage.
  • Job: a6f370ef-d5be-5018-d123-8cecddf918cb, parent = phase.
  • Ordinary manual task: 731e81fd-7043-5a7c-0d79-761d529318dd, parent = job.
  • Publish task: addc719e-a050-5502-8cee-b892db55e2e3, parent = job.
Build Job Mapped logs
193405 Win22_Sql22 / native SNI / Set 3 Manual 4059; initialize 355; publish 4241; quarantine manual 4217
193423 Win22_Sql22 / native SNI / Set 3 Manual 3379; initialize 267; publish 3560; quarantine manual 3512

Mapped logs were read with authenticated MCP in bounded ranges. The failed
ordinary task logged the Fact failure at 23:30:53Z, then
Failed: 1, Passed: 529, Skipped: 113, Total: 643, followed by
RetryHelper ... attempt #: 1 out of 2 at 23:30:59.554Z. Thus the same-run pass
is explicitly a task retry, not an independent allocation or a job-status
inference. The additional separate-build result is an actual passing test.
The job eventually succeeded; that does not erase the earlier failing result.

Both jobs use Windows Server 2022 10.0.20348, x64, net10.0, native SNI,
Release/Project, Set 3, ADO-MMS22-SQL22 image 106.0.0, runner image
20260105.167.1, Standard_D2a_v4 in westus2. The matrix configures local SQL Server
2022 and does not enable the enclave configuration. Actual endpoint locality
was not independently established and is not used as quarantine evidence.
The separate passing build has a different allocation/time and generated package
suffix, and advances main by #5080 (async AE parameter discovery). A direct
source diff confirms no change to this test, DataTestUtility or the test-step
template
between these two SHAs. The test executes plain SELECT @@VERSION,
not an Always Encrypted query. The same-revision retry additionally removes
revision differences from the basic intermittency observation.

Test-specific eligibility, not merely intermittency

EventSourceTest.cs at the failing revision
creates its listener with a using declaration, executes and disposes its SQL
objects, then enumerates TraceListener.IDs with Assert.All before the
listener is disposed
.
BaseEventListener
stores IDs in a mutable List<int>, enables all matching event keywords, and
appends directly from OnEventWritten without synchronizing with the reader.
The test never disables events or obtains a synchronized snapshot before
enumeration. Background driver activity can therefore invalidate this test
collection's enumerator even after this test's connection has been disposed.

That concrete listener-lifecycle/collection weakness matches the observed
enumerator exception. General connectivity/shared SQL availability does not
explain a List<int> enumeration failure after the SQL work completed. The
same symptom is not claimed to be reproduced locally, and a complete diagnosis
of which background event interleaved is not claimed. A successful retry alone
would be insufficient; the source-backed weakness is the second eligibility
requirement. This PR isolates the test and does not repair it.

Build interval, coverage and checkpoint

No reliable fully processed tagging checkpoint was found in the prior durable
records, including #4980, #4996, #5073 and #5105. Their merged state was checked
live; incomplete discovery records and fixes/probation work do not advance the
checkpoint. This run therefore rescanned the full interval:
2026-11-05T16:52:41.935Z through 2026-11-19T16:52:41.935Z.

Authenticated MCP build pages were bounded (25 initially, then 100) and followed
through the final empty continuation token. Re-fetched/overlapping pages were
deduplicated by build ID: 527 unique completed builds, with no outcome or
branch exclusion.

Definition Succeeded Partial Failed Canceled Total
sqlclient-pr (2281) 157 0 36 3 196
sqlclient-ci-package (2300) 27 0 1 0 28
sqlclient-ci-unit (2309) 0 0 0 0 0
CI-SqlClient (1879) 2 36 85 7 130
CI-SqlClient-Package (1917) 0 25 45 7 77
PR-SqlClient-Project (2197) 0 46 28 0 74
PR-SqlClient-Package (2198) 0 13 9 0 22

All seven identities were rechecked as Enabled. sqlclient-ci-stress (2250) was
rechecked Disabled and excluded. An enabled definition with no returned builds
does not establish any per-test execution coverage.

CI-SqlClient inventory includes main 46, release/6.1 7, release/7.0 11,
release/7.1 25, release/8.0 20, and other PR/development refs 21.
The release refs came from actual scoped build metadata and were cross-checked
with upstream ls-remote, not a hard-coded discovery branch filter.

Build-helper failure-outcome discovery was performed for main builds
193405, 193423 and 193438, and release builds 192610 (/6.1), 192218 (/7.0),
193579 (/7.1) and 193545 (/8.0). This is only seven builds, not exhaustive
per-test discovery for the 527-build inventory
.

The Passed build-wide helper for 193405 failed exactly:
MCP server 'azure-devops': MCP request failed: Transport closed.
Authenticated per-run pagination remained usable. The mapped publish-results
log for 193423 discovered run 3739834 for the separate passing control.

Both relevant runs were then enumerated without outcome filtering in six
bounded pages each (skip=0,1000,2000,3000,4000,5000, top=1000,
detailsToInclude=SubResults). Results were deduplicated by
(buildId, runId, resultId), never by title:

Build / run Unique top-level results Passed Failed NotExecuted
193405 / 3739426 5,891 5,634 1 256
193423 / 3739834 5,259 5,116 0 143

Counts reconcile exactly to the server run statistics and totalTests; both
runs are Completed with incompleteTests=0. Duplicate Passed statistic entries
were summed, preserving flaky resultMetadata: 5,632+2 and 5,114+2. No skipped,
aborted, missing or inaccessible result was counted as a pass. These 11,150
top-level records
are run-specific inventory coverage; nested/grouped cases
elsewhere are not claimed to be completely investigated.

Checkpoint NOT ADVANCED. Remaining evidence includes the other runs in
193405/193423 and the other 525 builds' complete run/result inventories,
including successful, partial-success,
failed and canceled builds, and complete branch-specific review on main and
each of /6.1, /7.0, /7.1 and /8.0. Even the seven build-helper failure queries
do not establish full per-test coverage for those builds. A future discovery
run must rescan its 14-day window and deduplicate, unless a later tagging
record establishes a trustworthy fully processed checkpoint.

Other observed candidates and existing proposals

  • Microsoft.Data.SqlClient.ManualTesting.Tests.MARSTest.MarsConcurrencyTest
    remains covered by open #5095;
    it was read, not modified. No duplicate tag.
  • Split-packet, metrics and vector-schema failures observed in these builds
    already have merged category proposals including
    #5073,
    #5065,
    #5016 and
    #5024.
    Existing traits/probation are untouched.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.AlwaysEncrypted.ApiShould.TestEnclaveExecutionApis
    (isAsync: True, truncated connection-string cases) and
    ApiShould.TestSqlCommandCancel remain not eligible on this record.
    Mixed grouped parents/truncated configurations are not independent cases.
    Open #5117 reports an XML
    enclave retry product defect and points to #5085
    / #4585.
    That is context, not independently established repair or tagging evidence.
  • Microsoft.Data.SqlClient.ManualTesting.Tests.DiagnosticTest.ConnectionOpenAsyncTest
    (missing WriteConnectionOpenAfter), the LocalDBTest static-initializer
    failure (expected 2, actual 1), and
    Microsoft.Data.SqlClient.Tests.SqlAuthenticationProviderManagerTests.DefaultAuthenticationProviders_AppConfig
    (expected DummySqlAuthenticationProvider, actual null) are unconfirmed:
    no qualifying exact/comparable pass plus test-specific eligibility was
    established here. DiagnosticTest is already class-level quarantined.
  • Release /8.0:
    Microsoft.Data.SqlClient.UnitTests.Microsoft.Data.SqlClient.SqlConnectionFactoryTests.RunPruningPass_DuringSyncPhysicalCreation_RetainsPool
    had Assert.True expected true/actual false. No verified branch-specific
    failure/pass pair or test-specific assessment; unconfirmed.
  • Release /7.1:
    DistributedTransactionTestWindows.Test_EnlistedTransactionPreservedWhilePooled
    reports a process-global ImplicitDistributedTransaction-setting error;
    TransactionPoolTest.TransactionCleanupTest grouped parents lack full detail;
    TvpColumnBoundariesTests.TvpColumnBoundariesTest reports Assert.True failure;
    ParserPrerequisiteTests.ParserFailure_DoesNotPoisonPool reports expected 3,
    actual 4. These identities are under their recorded ManualTesting.Tests or
    UnitTests.SimulatedServerTests namespaces. No exact branch-specific passing
    controls/eligibility were verified, so no new quarantine.
  • Release /6.1 XEvents timeouts and simulator failures, and /7.0 diagnostic
    assembly-loading errors/simulator failures remain unconfirmed, not
    presumed flaky tests. Network/setup timeouts alone do not qualify. If the
    separate fixes/probation automation later verifies transient setup failures,
    bounded retries should be limited to that setup/connectivity, never the
    assertions or tested behavior; no retry implementation/proposal was made here.

Open quarantine/fix/backport proposals and equivalent human titles/scopes were
checked. #5087 and
#5109 concern other release
backports. #5064 repairs an
exception-clone assertion. #4835
matched an EventSource search but its file scope is unrelated. No equivalent
EventSourceTestAll proposal was found. All remain unrelated work, not global
blockers to this category-only proposal.

Release routing and milestone

No release branch qualifies for this new Fact quarantine on this evidence.
No release pass was substituted for a main pass or vice versa. EventSourceTestAll
has not acquired a verified within-branch failure/pass pair on any release
branch; release-only candidates do not qualify main. Thus no Hotfix version
label is requested, and no backport is claimed or manually opened. Release
per-test evidence and target-patch/whole-PR routing assessments remain unfinished.
Available hotfix labels alone are insufficient to choose a patch.

Latest checked upstream main is 981a989dd1b4bc9eed889286d141a8234b8460d5.
Its canonical SqlClientNextVersion is 8.1.0-preview1; the exact existing
open milestone is #108. The temporary 8.0.0-preview1/#101 exclusion no longer
applies. Required publication metadata is Area\Tests and milestone #108, with
the exact automated-disclosure first line. Final live verification is recorded
below after publication.

Validation commands and actual results

The following build/discovery commands both succeeded and each listed exactly
the one Fact:

dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' --framework net10.0 --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll&category=flaky' --verbosity quiet
dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' --framework net462 --list-tests --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll&category=flaky' --verbosity quiet

Actual routing was then asserted on both frameworks with the following
commands; each produced quarantined=1, ordinary=0 (intentional exclusion):

$p='src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj'
$n='FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll'
foreach ($f in 'net462','net10.0') {
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category=flaky" -v:q
    dotnet test $p -f $f --no-build --no-restore --list-tests --filter "$n&category!=failing&category!=flaky&category!=interactive" -v:q
}

Focused execution attempts used these exact commands:

dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' --framework net10.0 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll&category=flaky' --verbosity quiet
dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' --framework net462 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll&category=flaky' --verbosity quiet
dotnet test 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTests.csproj' --framework net462 --no-build --no-restore --filter 'FullyQualifiedName=Microsoft.Data.SqlClient.ManualTesting.Tests.EventSourceTest.EventSourceTestAll&category=flaky' --logger 'console;verbosity=normal' --verbosity quiet
  • net10.0: 0 passed, 0 failed, 1 skipped, no usable test connection
    configuration. This is not execution coverage.
  • net462: 0 passed, 1 failed, 0 skipped, including the diagnostic rerun.
    The selected Fact failed while opening the configured Northwind database:
    Cannot open database "Northwind" requested by the login. The login failed.
    This is an environment/setup blocker, not reproduction of the collection
    exception or evidence that the category change broke behavior. Other
    SQL-backed theory discovery also reported the same database error.
    No database, credentials, primary checkout or shared configuration was changed.
  • git diff --check: passed. Source diff is exactly one file, two added
    lines, no removed lines and no test-body changes. The observed-error comment
    immediately precedes the attribute and matches the verified failure.
  • Ordinary zero-match selection is the expected exclusion and is not
    counted as execution coverage. Neither local failure nor skipped execution
    supports any repair/stability claim.

Fixes and all probation lifecycle work are outside scope. No pipeline was
modified, no build queued, no automation configured, no merge/approval or
auto-merge enabled, no account switched and no fork used.

Publication verification

Created #5118, commit
7ba155e78d22b95deba227ab1e93956cdddc057d, from
dotnet/SqlClient:dev/mdaigle/flaky-quarantine-maintenance-2c1 to
dotnet/SqlClient:main. Origin push succeeded; the isolated worktree was clean
after commit. Immediately before publication, main SHA/version and the exact
open milestone were read fresh, and open PR overlap was rechecked.

Live metadata verified Area\Tests, milestone 8.1.0-preview1 (#108),
and the exact visible first line
"🤖 This pull request was created by a fully automated workflow."
No Hotfix labels were added because release eligibility is incomplete.
Backport routing is not requested, not created/merged.
This comment is the single durable evidence record for this originating session
and is linked from the PR description. Full-window per-test coverage remains
incomplete; checkpoint not advanced.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Policy-specific quarantine eligibility requires human verification because the supplied checkout-relative policy references lack trusted-base provenance.

No open issues 🎉

What changed in this PR

Quarantines EventSourceTestAll in the SqlClient manual test suite to keep its intermittent listener-enumeration failure out of regular test runs. This changes test routing, not the underlying behavior.

Changes:

  • Adds a method-level [Trait("category", "flaky")].
  • Adds a comment describing the observed collection-modification exception.
  • Leaves the test body, conditions, sibling tests, APIs, and pipelines unchanged.
File Description
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​TracingTests/​EventSourceTest.cs Marks EventSourceTestAll as flaky and documents the observed failure.

Review effort: Balanced

@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.40%. Comparing base (17bef7b) to head (7ba155e).
⚠️ Report is 23 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5118      +/-   ##
==========================================
+ Coverage   68.99%   70.40%   +1.41%     
==========================================
  Files         314      315       +1     
  Lines       45857    70026   +24169     
==========================================
+ Hits        31638    49304   +17666     
- Misses      14219    20722    +6503     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 70.40% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area\Tests Issues that are targeted to tests or test projects

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants