Skip to content

[Async AE 5] Load enclave keys asynchronously and route package retries - #5085

Open
cheenamalhotra wants to merge 33 commits into
dev/cheena/ae-async-metadata-cachefrom
dev/cheena/ae-async-enclave-key-loading
Open

cheenamalhotra wants to merge 33 commits into
dev/cheena/ae-async-metadata-cachefrom
dev/cheena/ae-async-enclave-key-loading

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

Layer 5 of 5 split from #4585. Stacked on #5081.

  • EnclaveDelegate: adds GenerateEnclavePackageAsync and GetDecryptedKeysToBeSentToEnclaveAsync. Keys load one at a time with async key store calls. Sync and async paths share BuildEnclavePackage and CreateColumnEncryptionKeyInfo.
  • SqlCommand.Reader: after describe, ContinueRunExecuteReaderTdsAsync (on a Task.Run worker) generates the enclave package asynchronously, then runs the TDS execution. This replaces the sync GenerateEnclavePackage call. The connection is revalidated after each await. Cancellation propagates unwrapped.
  • SqlCommand.CreateLocalCompletionTask: a retryable enclave package fault before the read stage now goes through the same bounded retry as read-stage failures. This is the first production reader of _asyncExecutionCancellationToken.

Fixes from review feedback:

  • Guard handoff: the layer 4 in-progress guard (_enclaveAttestationInProgress) is held from the post-describe continuation through package key loading, package generation and the RunExecuteReaderTds handoff. It is cleared on failure or cancellation. Changing Connection or starting another execution while package keys load now throws.
  • Database check: ValidateConnectionLifetime gets the database captured before dispatch, after describe and after the package-key await. A database change during key loading throws.
  • XML retry task: BeginExecuteXmlReader now attaches its callback to and returns the global completion task, like Reader and NonQuery. Callers see the retry's outcome, not the faulted first attempt.

Fixes the internal TestEnclaveExecutionApis failure

The XML retry task fix above also fixes the internal pipeline failure in ApiShould.TestEnclaveExecutionApis (builds 193347 and 193466). Its ExecuteXmlReaderAsync step fails with Internal connection fatal error. Error state: 15, Token: 1 (other runs show tokens 0, 70 and 88).

Root cause: on main, BeginExecuteXmlReaderInternal returns localCompletion, which is the first attempt's task. When the enclave session is invalid, the driver runs an internal End and retries. The caller's End then ran on the first attempt at the same time as the retry, using the same state object, which desynced the TDS stream.

The bug is already on main and isn't caused by this stack. #4887 added the test that exposes it, and build 193347, from before #5079 and #5080, already failed the same way. #5117 applies the same fix to main for testing only and won't be merged.

Deviations from #4585:

  • A faulted write already released its async-count reservation, so the retry does not take it over. This avoids a double decrement.
  • Async-count leak after a successful retry: the first attempt's reservation was never released (internal End skips the decrement, the retry takes its own, and the caller's End releases one). CreateLocalCompletionTask now releases it after the retry succeeds, only on the read-stage path; a faulted write already released it.
  • The connection is also revalidated after the describe await, before package generation.
  • EnclaveDelegate.Crypto.cs comments mention the session-lock wait on a cache miss.

Issues

Part of #4585.

Testing

  • New EnclaveDelegateTests (6 tests): async vs sync provider API, all keys resolved, sequential loading, cancellation, provider failure.
  • SqlCommandTests: new CreateLocalCompletionTask_* tests (retryable write fault retries, retry fails, retry canceled, non-retryable fault, canceled write).
  • SimulatedServerTests/SqlCommandEnclaveAsyncTests: Execute_PendingPackageKey_* (revalidation, connection setter, second execution, database change), Prepare_Completed_HandsOffInProgressFlag, Retry_PackageFault_DoesNotTransferReservation, Retry_PackageFaultRetrySucceeds_KeepsOnlyRetryReservation, Retry_AwaitsInvalidationBeforeRestarting (count after successful retry), BeginExecuteXmlReader_RetryablePackageFault_CompletesWithRetry.
  • ManualTests/AlwaysEncrypted/ApiShould.cs: re-enables debug stale-enclave retry coverage.

Local results at aa9fadb:

  • dotnet test src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net10.0 -c Debug|Release --filter "FullyQualifiedName~SqlCommandEnclaveAsyncTests|FullyQualifiedName~SqlCommandQueryMetadataCacheAsyncTests|FullyQualifiedName~EnclaveDelegateTests|FullyQualifiedName~SqlCommandTests|FullyQualifiedName~SqlQueryMetadataCacheTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SymmetricKeyCacheTests|FullyQualifiedName~AsyncKeyStoreProviderTests": passed, 178/178 in each configuration. Simulated-server tests were run with a local-only macOS listener workaround (not committed).
  • dotnet build of UnitTests and ManualTests with -f net462 -c Debug|Release: 0 warnings, 0 errors.
  • dotnet build build.proj -t:BuildSqlClient -p:Configuration=Debug|Release: 0 warnings, 0 errors.

CI (build 192590, aa9fadb):

  • UnitTests: Linux net10 2668 passed / 0 failed; Windows net10 2692 / 0; net462 2588 / 0. Simulated-server tests ran; none of the stack's tests were skipped.
  • AE manual tests: 246 passed / 0 failed. Only enclave tests were skipped (no enclave server in CI).

Pending: enclave end-to-end testing against a real enclave server.

Guidelines

Please review the contribution guidelines before submitting a pull request:

Generate enclave packages with the async key store APIs after the
describe round trip, revalidate the connection after each await, and
route retryable package faults into the bounded enclave retry without
transferring an already released reservation.

Final layer of the #4585 split.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical concurrency gap during the key-store await allows connection reassignment or overlapping execution to redirect pending work or overwrite encryption state.

1 open issue
What changed in this PR

Completes the fifth layer of async Always Encrypted integration in SqlClient, building on #5081’s metadata-cache work to load enclave keys asynchronously and route package failures through the existing retry mechanism.

Changes:

  • Adds sequential async enclave-key loading while sharing validation and package construction with synchronous execution.
  • Awaits package generation before command execution, revalidates the connection, and handles retryable package faults without transferring an already-released reservation.
  • Adds provider, retry, and connection-revalidation tests; re-enables debug stale-enclave retry coverage.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​SimulatedServerTests/​SqlCommandEnclaveAsyncTests.cs Tests connection revalidation during key loading and retry reservation handling.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlCommandTests.cs Covers retry success, failure, cancellation, and non-retryable write faults.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​EnclaveDelegateTests.cs Tests provider dispatch, sequential key loading, cancellation, and errors.
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​AlwaysEncrypted/​ApiShould.cs Re-enables debug package-failure retry scenarios.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Reader.cs Adds async package generation and an await-based post-describe execution path.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.cs Routes retryable package faults through bounded retry with adjusted reservation ownership.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​EnclaveDelegate.cs Adds async key loading and shared key-info validation.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​EnclaveDelegate.Crypto.cs Adds async package generation and shared package assembly; clarifies session-lock behavior.

Review effort: Balanced

Merge dev/cheena/ae-async-metadata-cache (9fee7e7) and reuse its
in-progress guard on the async enclave path. Successful async preparation
now hands the flag to the reader continuation, which holds it through
package key loading and the RunExecuteReaderTds reservation, revalidates
the connection lifetime after each await, and clears it on success,
failure or cancellation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

ExecuteXmlReaderAsync still exposes the original task during retries, so callers can receive an exception and release command state while a retry is using it.

1 open issue
1 resolved since last review

Review effort: Balanced

Pass the captured database to ValidateConnectionLifetime after describe and
after the package key await. Share UseDatabaseQueryEngine and add a
ChangeDatabase test while the package key is pending.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async enclave retry changes affect connection-reservation ownership and command reuse, while unexecuted simulated-server and real enclave stale-session retry tests leave key behavior needing CI/runtime confirmation.

1 open issue

Review effort: Balanced

BeginExecuteXmlReader now attaches its callback to and returns the
global completion task, like Reader and NonQuery, so callers observe
the bounded enclave retry instead of the faulted first attempt.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async enclave changes span command guards, reservation ownership, and XML retry completion, while simulated-server and live SQL Server/enclave validation remain unverified and require human confirmation.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.63780% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.95%. Comparing base (af7f7ae) to head (8f90873).

Files with missing lines Patch % Lines
.../src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs 96.51% 3 Missing ⚠️
Additional details and impacted files
@@                          Coverage Diff                           @@
##           dev/cheena/ae-async-metadata-cache    #5085      +/-   ##
======================================================================
+ Coverage                               70.60%   70.95%   +0.34%     
======================================================================
  Files                                     315      315              
  Lines                                   70176    70201      +25     
======================================================================
+ Hits                                    49551    49813     +262     
+ Misses                                  20625    20388     -237     
Flag Coverage Δ
sqlclient-pr 70.95% <97.63%> (+0.34%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

cheenamalhotra added a commit that referenced this pull request Nov 19, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed for a critical retry-transition race that can allow connection reassignment or overlapping execution while command state is unguarded.

No open issues 🎉

Review effort: Balanced

Ported from 1f99c85, 6792b76 and f47a8c7 (validated on #5117).

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 18:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed for a critical retry-handoff race that allows connection changes or concurrent execution while invalidation is pending, potentially causing retries to use altered command state.

No open issues 🎉

Review effort: Balanced

…nc-enclave-key-loading

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 19:10
cheenamalhotra added a commit that referenced this pull request Nov 19, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Real-enclave end-to-end validation remains pending, and simulated-server tests cannot establish wire/session correctness for the package-timing and retry-ownership changes.

No open issues 🎉

Review effort: Balanced

Adopt the execution-owned enclave in-progress flag (releaseInProgress),
the new PrepareForTransparentEncryption signature, and retarget the
reflection-based enclave tests to the current entry points.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 23:37
cheenamalhotra added a commit that referenced this pull request Nov 19, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A critical gap in the async enclave retry guard allows connection changes or concurrent execution during the retry handoff, risking use of a replacement connection or corruption of another execution’s state.

1 open issue

Review effort: Balanced

cheenamalhotra and others added 2 commits November 19, 2026 16:48
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…om/dotnet/SqlClient into dev/cheena/ae-async-enclave-key-loading

# Conflicts:
#	src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs
Copilot AI balanced review requested due to automatic review settings November 19, 2026 23:49
cheenamalhotra added a commit that referenced this pull request Nov 19, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Enclave-package changes affect TDS handoff, retry cleanup, and async-count ownership, while real enclave end-to-end validation remains pending and simulated tests cannot establish server-side correctness.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

cheenamalhotra and others added 2 commits November 19, 2026 17:52
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The pending-package-key test helper now owns the in-progress flag like
RunExecuteReader, so overlapping executions are rejected instead of
blocking on the pending key. RunExecuteReader sets the flag before the
cache probe so overlaps started there are rejected too. Test awaits on
overlapping executions are bounded.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 20, 2026 01:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Simulated-server tests cannot establish real-enclave package generation and stale-session retry correctness across Reader, NonQuery, and XML, so an enclave-enabled end-to-end run and human review remain necessary.

No open issues 🎉

Review effort: Balanced

cheenamalhotra added a commit that referenced this pull request Nov 20, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: To triage

Development

Successfully merging this pull request may close these issues.

5 participants