Skip to content

[Perf] Capture async-health signals within the performance benchmarks - #5068

Draft
mdaigle wants to merge 15 commits into
mainfrom
dev/mdaigle/automatic-openasync-perf-coverage
Draft

mdaigle wants to merge 15 commits into
mainfrom
dev/mdaigle/automatic-openasync-perf-coverage

Conversation

@mdaigle

@mdaigle mdaigle commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Description

Measure async scheduler health inside the existing BenchmarkDotNet perf flow, only for the four new OpenAsyncBurst.OpenQueryClose cases: Empty, Warm, Oversubscribed (16 pool slots), and NonPooled. There is no standalone sweep, production driver/API change, new pipeline trigger, or default network delay. Existing connection/churn benchmarks remain uninstrumented.

Execution and evidence

  • Every runner supports ProcessIsolation (PerRunner by default, or PerCase) and AdditionalRttMilliseconds (default 0). The burst profile explicitly uses PerCase. Baseline and candidate use the same candidate-owned harness, settings and workload, even with an older source/package baseline.
  • Optional RTT uses owned, SQL-only Linux loopback tc netem; 100ms means 50ms per direction. Unsupported routes/platforms fail explicitly. Ownership, applied delay, traffic, packet drops and cleanup are checked and retained. Other clients of the same endpoint would also be affected, so use an exclusive benchmark environment.
  • Startup includes fixture initialization; ExtraIteration excludes successful setup/cleanup. Neither probe production nor operation recording runs in ordinary timing iterations. Raw probes, resources, queue/open timings, failed/cancelled/pending outcomes and first-worker records are retained.
  • Burst uses one 64-worker wave. Extended keeps Empty as that cold wave and uses 16 waves for the other states, with fixed work per worker. The 100ms profile retains query counts of 20/5/1 for Warm/Oversubscribed/NonPooled. Normalization uses the actual complete work count. No artificial yields were added.
  • Dispatch adequacy requires 25 seconds, 1,000 executed probes and no missing evidence. First-worker adequacy is tracked separately. A cold 64-start p99 remains a sparse observed maximum. These floors are not confidence-interval guarantees.

Open/close-only bursts were removed at the user's request: long synchronous checkout loops can themselves saturate the ThreadPool. Remove their retired OpenClose/... entries from private extended profiles.

Reporting and integration

The compact report pairs matching configurations and displays current maxima across phases/repetitions for dispatch p99, first-worker queue p99, peak ThreadPool threads and peak process threads, with signed baseline deltas. Existing percentiles are not pooled. Rows retain descending dispatch-percentage-delta order.

The leading indicator uses current dispatch p99: green below 1ms, yellow from 1ms through 10ms inclusive, red above 10ms, or unknown without a current value. Coverage/provenance remains explicit. Ordinary timing thresholds, confirmation and CI gates are unchanged. Historical reports still retain retired cases when their raw data is supplied.

Queue runnerProfile=openasync-extended-100ms, platform=linux, and useConnectionPoolV2=true for the opt-in four-case profile. The default profile stays unshaped. Credentials use normal private injection and redacted publication. BDN 0.16.0-preview.1 provides the in-process lifecycle hooks; JitTieringMode.Skip avoids extra automatic preparation without reducing configured measurements. Existing timeout budgets reserve ten minutes for artifact collection.

Reduced diagnostic schema

New artifacts use schema 3. Summary no longer includes Health, ProducerP99Ms, FirstWorkerOfferToOpenP99Ms, SuccessfulOpenP99Ms, FailedOpenP99Ms, CancelledOperationP99Ms, or CompletedOperationsPerSecond. Their calculations and the older 100ms verdict are removed; the four displayed metrics, sample/completeness fields and raw observations/outcome counters remain. Throughput is still available as Operations.CompletedPerSecond. Other latency statistics can be recalculated from retained samples, subject to recorded truncation.

The report reads schemas 2 and 3. It replaces the old verdict counts with resource completeness derived directly from raw samples/drop counters/observer errors. Main-table ordering, values, deltas and colors are unchanged. Async reporting no longer depends on the ordinary timing comparator. This revision removes 20 net runtime-code lines; compatibility tests and documentation add 49, so it is a schema/logic simplification, not a net source-line reduction.

Review corrections and simplification

  • Missing thread peaks now remain nullable/N/A, rather than producing a false 100% improvement. Process-thread peaks include the recorded start count.
  • Resolved the obsolete OpenClose comment and the multi-warmup comment: OpenClose is removed, and burst validation already requires one launch/warmup/invocation.
  • Reuse one frozen operation snapshot per window, derive first-worker completion from retained records, and avoid allocating a separate operation-handle object.
  • Collapse constructor/toolchain wrappers, keep case materialization in the child path, unify artifact-copy loops, and consolidate duplicated documentation. The first cleanup removed 72 net lines independently of the review fix and its tests.
  • The deeper pass removes another 341 net lines: obsolete async-report percentage threshold plumbing, unreachable Windows latency-profile selection, repeated test setup/cleanup, and verbose protocol duplication. Ordinary timing thresholds and Windows' explicit unsupported-profile rejection remain. The async report no longer accepts --threshold; its fixed health bands are unchanged.

A further 53 net lines are removed in f032abe3b: derive probe totals/posted counts from existing samples and loss counters, share operation-completion policy, remove redundant process-launch/wait helpers, use a record for registry entries, and consolidate equivalent collector/test setup. Child cancellation now has one termination path; the network helper keeps its 30-second bound and explicit timeout failure. No workload, report or networking ownership policy was removed.

The protocol, configuration and limitations are documented in async-health.md; the benchmark README links there instead of duplicating the full protocol. Network ownership/cancellation guards and evidence validation are preserved.

Issues

Related to #5014. Experiments and historical evidence remain preserved; this PR does not replace or close it, and its branch was not changed.

Testing

Latest schema simplification: 2a345bcb8, following the output-preserving cleanup commits.

$filter='FullyQualifiedName~AsyncHealth|FullyQualifiedName~ThreadPoolHealthObserverTests|FullyQualifiedName~ThreadPoolProbeRecorderTests|FullyQualifiedName~BenchmarkExecutionPlanTests|FullyQualifiedName~BenchmarkNetworkScopeTests|FullyQualifiedName~BenchmarkConfigTests|FullyQualifiedName~OpenAsyncBurstRunnerJobTests'
dotnet test src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj -c Release -f net10.0 --list-tests --filter $filter --nologo -v quiet
# Passed: intended tests discovered.
dotnet test src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj -c Release -f net10.0 --no-build --filter $filter --nologo -v quiet
# Passed: 53 tests, no skips; includes real in-process BDN lifecycle and failure handling.

python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p 'test_benchmark_health*.py' -q
# Passed: 24 tests, including null peaks, health-band boundaries, historical reports and artifact retention.
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_runner_config.py -q
# Passed: 9 tests; supported configuration paths and Windows latency rejection retained.
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_perf_budget.py -q
# Passed: 9 tests.
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_execution_conditions.py -q
# Passed: 5 tests.
python -B -m unittest discover -s eng\pipelines\perf\scripts\tests -p test_netem.py -q
# Passed: 9 tests.

dotnet build src\Microsoft.Data.SqlClient\tests\PerformanceTests\Microsoft.Data.SqlClient.PerformanceTests.csproj -c Release -f net10.0 -p:ReferenceType=Project --nologo -v quiet
# Passed: 0 warnings/errors.
dotnet build src\Microsoft.Data.SqlClient\tests\PerformanceTests\Microsoft.Data.SqlClient.PerformanceTests.csproj -c Release -f net10.0 -p:ReferenceType=Package -p:MdsPackageVersion=7.1.2 -o artifacts\reduced-schema-baseline --nologo -v quiet
# Passed: 0 warnings/errors.

The earlier thread-peak tests failed before that fix; new schema-3 fixtures also failed against the prior report reader. The real BDN lifecycle test now checks schema-3 output, absence of all seven removed fields, and retained raw timestamps/resource evidence. The compiled producer recomputed summaries from all 32 historical run-193062 windows: every retained summary value stayed identical, exactly the seven fields disappeared, and original files stayed unchanged. Before/after main tables were byte-identical; the new reader produced identical reports from equivalent schema-2 and schema-3 inputs. Resource warnings cover complete/missing/truncated/invalid/unknown evidence. Linux health/condition suites passed from an LF Git export (24/5 tests; 2/0 unavailable-shell skips). Both driver builds still discover four query cases with Extended counts 64/20480/5120/1024. Diff checks passed. The previous SQL-free child-exit/cancellation verification remains applicable; this revision does not change process/network execution.

Earlier runtime evidence: run 193740 passed on 62c90bea7: four query workloads, Linux, V2, 100ms RTT and a matched 7.1.2/current pass. Run 193062 also passed before the open-only removal; its raw evidence verified all 16 case processes and owned netem cleanup. Earlier local checks exercised SQL on Windows/Linux and cancellation cleanup. Historical measurements are not claimed as current-head runtime results.

No long SQL/perf rerun was started for the reporting correction and simplification. The earlier full Windows Python suite had three pre-existing Git Bash temporary-path failures; they were not suppressed. Remaining limitations include single-pass noise, sparse cold cohorts, observer overhead, and lack of coverage for every authentication/failover/transaction/Framework path.

  • Tests added or updated
  • Public API changes documented (not applicable; production API unchanged)
  • Verified against customer repro (earlier local cold-start reproduction; no new production fix claimed)
  • Ensure no breaking changes introduced (production driver unchanged; runner configuration changes documented)

Guidelines

Please review the contribution guidelines before submitting a pull request:

Build one candidate-owned harness against both drivers and retain matched regression, baseline-debt and invalid-evidence outcomes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@mdaigle
mdaigle requested a review from a team as a code owner November 13, 2026 21:36
Copilot AI balanced review requested due to automatic review settings November 13, 2026 21:36
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 13, 2026
@mdaigle mdaigle added this to the 8.1.0-preview1 milestone Nov 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Sequential package comparisons in both scripts can reuse baseline binaries for the current pass, hiding candidate regressions, and the health build fails for the specified preview baseline because required dependency overrides are missing.

3 open issues
What changed in this PR

Adds a bounded OpenAsync health phase before BenchmarkDotNet in SqlClient’s existing performance runners, without changing production driver code or pipeline triggers. This is separate from the broader experiments in #5014.

Changes:

  • Adds 20 workload cases with three paired repetitions, using the same harness against candidate and baseline drivers.
  • Introduces matched regression gating, raw evidence, summaries, and optional Linux network-delay testing; default runs add no delay.
  • Integrates both runners, increases timeout defaults, and adds tests and documentation.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft.Data.SqlClient.UnitTests.csproj Links harness sources into modern .NET unit tests.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​ThreadPoolProbeRecorderTests.cs Tests bounded probe accounting.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​ThreadPoolHealthObserverTests.cs Tests observer failure and incomplete evidence.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​OpenAsyncHealthSettingsTests.cs Tests workload-setting validation.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Measurements/​OpenAsyncHealthRecorderTests.cs Tests outcome retention and counter snapshots.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​OpenAsyncHealth/​Program.cs Adds the standalone command-line entrypoint.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​OpenAsyncHealth/​OpenAsyncHealth.csproj Supports source and package driver references.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Microsoft.Data.SqlClient.PerformanceTests.csproj Excludes standalone harness sources from BenchmarkDotNet.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​ThreadPoolProbeRecorder.cs Records probe timing and missing callbacks.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​ThreadPoolHealthObserver.cs Samples scheduling and process resources.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​OpenAsyncHealthRecorder.cs Retains operation outcomes and counters.
src/​Microsoft.Data.SqlClient/​tests/​PerformanceTests/​Measurements/​OpenAsyncHealthExperiment.cs Executes bounded connection workloads.
eng/​pipelines/​perf/​sqlclient-perf-pr-pipeline.yml Adds 30 minutes to the PR timeout default.
eng/​pipelines/​perf/​sqlclient-perf-pipeline.yml Adds 30 minutes to the standard timeout default.
eng/​pipelines/​perf/​sqlclient-perf-experiment.yml Adds 30 minutes to the experiment timeout default.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health.py Tests evidence validation, comparisons, and orchestration.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health_runners.py Tests both runner integrations and exit handling.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health_process.py Adds opt-in executable lifecycle tests.
eng/​pipelines/​perf/​scripts/​tests/​test_openasync_health_netem.py Tests network-rule ownership and cleanup.
eng/​pipelines/​perf/​scripts/​run-perf-tests.sh Runs health coverage before Linux benchmarks.
eng/​pipelines/​perf/​scripts/​run-perf-tests.ps1 Runs health coverage before Windows benchmarks.
eng/​pipelines/​perf/​scripts/​openasync_health.py Builds, pairs, validates, and reports health runs.
eng/​pipelines/​perf/​scripts/​netem.py Adds optional SQL-only loopback delay.
eng/​pipelines/​perf/​README.md Documents automatic coverage and timeout changes.
eng/​pipelines/​perf/​openasync-health.md Explains workloads, gates, artifacts, and limitations.

Review effort: Balanced

@mdaigle
mdaigle marked this pull request as draft November 13, 2026 22:14
Observe marked BDN workloads in extra iterations, run burst regression cases automatically, and build matched candidate benchmark sources against source baselines.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 17:48
@mdaigle mdaigle changed the title [Perf] Automatically run matched OpenAsync health coverage [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The new performance gates and BDN lifecycle changes need human review of authenticated SQL pipeline runs on Linux and Windows, since SQL-free checks do not establish end-to-end measurement validity for this revision.

1 open issue
2 resolved since last review

Review effort: Balanced

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.47%. Comparing base (17bef7b) to head (2a345bc).
⚠️ Report is 27 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5068      +/-   ##
==========================================
+ Coverage   68.99%   70.47%   +1.48%     
==========================================
  Files         314      315       +1     
  Lines       45857    70150   +24293     
==========================================
+ Hits        31638    49441   +17803     
- Misses      14219    20709    +6490     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 70.47% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Remove the standalone pre-benchmark sweep and network shaping. Record startup/warmup and extra-iteration queue/open/outcome signals in the perf tests, retaining missing work and resource evidence.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:12
@paulmedynski paulmedynski changed the title [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [workflow replay] Nov 16, 2026
@paulmedynski paulmedynski changed the title [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [workflow replay] [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The oversubscribed benchmark may never create pool waiters, leaving the advertised waiter-wakeup regression untested, and the PR description claims health checks and gates that are not implemented.

2 open issues
1 resolved since last review

Review effort: Balanced

Comment thread eng/pipelines/perf/async-health.md Resolved
@mdaigle mdaigle changed the title [Perf] Add async-health diagnoser, automated measurements, and regression benchmarks [Perf] Capture async-health signals within the performance benchmarks Nov 16, 2026
Skip preview-only automatic JIT tier-promotion workload, retaining configured warmups and measurement counts. Reserve collection grace inside the pipeline task budget and retain partial per-unit logs and timings.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 23:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because multi-warmup timing includes setup and cleanup between batches, potentially distorting the reported workload throughput.

2 open issues

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged loopback shaping and cross-process cancellation require human validation on the dedicated performance host because current-head end-to-end behavior and the full extended-profile runtime remain unverified.

3 open issues

Review effort: Balanced

Keep the four OpenQueryClose pool states and their existing workload sizes. Retain historical report compatibility while rejecting retired workload configuration.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 19:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The cross-process loopback-shaping and preview BDN lifecycle changes still need current-head Linux SQL validation of cancellation and cleanup, as supplied runtime evidence covers earlier revisions.

3 open issues

Review effort: Balanced

Use green below 1ms, yellow from 1ms through 10ms inclusive, and red above 10ms. Preserve existing ordering, numeric deltas, raw evidence, and ordinary performance regression policy.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 19:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged network shaping and benchmark cancellation/cleanup require human validation because successful runtime evidence predates the final revisions and current-head SQL/network checks could not be run.

3 open issues

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 22:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged loopback shaping and cancellation cleanup require human review because historical validation does not establish current-head end-to-end behavior or the full extended-profile runtime.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

Remove redundant constructor/toolchain helpers, duplicate snapshot cloning, completion tracking and parent-only case materialization. Share artifact copying and consolidate duplicated documentation without changing the benchmark protocol.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 22:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Privileged host-wide loopback ownership and recovery, plus preview BenchmarkDotNet lifecycle/cancellation integration, warrant human review and current-head Linux runtime validation beyond the available static and historical evidence.

No open issues 🎉

Review effort: Balanced

Drop the unused relative threshold from async reports and both callers, remove unreachable Windows latency-profile selection, share per-test artifact and BDN fixture lifetimes, and condense the protocol. Preserve report output and benchmark behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 18:56

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Cross-process BDN lifecycle, cancellation, and privileged netem cleanup require human validation because the supplied end-to-end SQL/network evidence predates the latest changes.

No open issues 🎉

Review effort: Balanced

Derive posted/total probes from retained samples and loss counters, share completion rules, remove redundant launch/wait helpers, and consolidate equivalent artifact and timeout test setup. Preserve raw evidence, workload selection, and bounded cleanup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 19:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The supplied SQL/performance runs predate the final simplification, leaving process cancellation, BDN preview lifecycle hooks, and host-network cleanup and timeout behavior dependent on current-head human runtime validation.

No open issues 🎉

Review effort: Balanced

Remove seven derived fields and the legacy per-window verdict. Version new artifacts as schema 3 while reading historical schema 2; retain all raw observations, outcomes, and resource completeness. Decouple health reporting from ordinary relative comparison helpers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 19:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human validation is needed because supplied full-run evidence predates the latest lifecycle changes and does not verify current-head SQL-backed execution or cancellation cleanup for privileged network shaping.

No open issues 🎉

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

4 participants