Repository navigation
[Async AE 5] Load enclave keys asynchronously and route package retries - #5085
cheenamalhotra wants to merge 33 commits into
Conversation
Generate enclave packages with the async key store APIs after the describe round trip, revalidate the connection after each await, and route retryable package faults into the bounded enclave retry without transferring an already released reservation. Final layer of the #4585 split. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
A critical concurrency gap during the key-store await allows connection reassignment or overlapping execution to redirect pending work or overwrite encryption state.
1 open issue
What changed in this PR
Completes the fifth layer of async Always Encrypted integration in SqlClient, building on #5081’s metadata-cache work to load enclave keys asynchronously and route package failures through the existing retry mechanism.
Changes:
- Adds sequential async enclave-key loading while sharing validation and package construction with synchronous execution.
- Awaits package generation before command execution, revalidates the connection, and handles retryable package faults without transferring an already-released reservation.
- Adds provider, retry, and connection-revalidation tests; re-enables debug stale-enclave retry coverage.
| File | Description |
|---|---|
src/Microsoft.Data.SqlClient/tests/UnitTests/SimulatedServerTests/SqlCommandEnclaveAsyncTests.cs |
Tests connection revalidation during key loading and retry reservation handling. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft/Data/SqlClient/SqlCommandTests.cs |
Covers retry success, failure, cancellation, and non-retryable write faults. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft/Data/SqlClient/EnclaveDelegateTests.cs |
Tests provider dispatch, sequential key loading, cancellation, and errors. |
src/Microsoft.Data.SqlClient/tests/ManualTests/AlwaysEncrypted/ApiShould.cs |
Re-enables debug package-failure retry scenarios. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs |
Adds async package generation and an await-based post-describe execution path. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlCommand.cs |
Routes retryable package faults through bounded retry with adjusted reservation ownership. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/EnclaveDelegate.cs |
Adds async key loading and shared key-info validation. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/EnclaveDelegate.Crypto.cs |
Adds async package generation and shared package assembly; clarifies session-lock behavior. |
Review effort: Balanced
Merge dev/cheena/ae-async-metadata-cache (9fee7e7) and reuse its in-progress guard on the async enclave path. Successful async preparation now hands the flag to the reader continuation, which holds it through package key loading and the RunExecuteReaderTds reservation, revalidates the connection lifetime after each await, and clears it on success, failure or cancellation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pass the captured database to ValidateConnectionLifetime after describe and after the package key await. Share UseDatabaseQueryEngine and add a ChangeDatabase test while the package key is pending. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Async enclave retry changes affect connection-reservation ownership and command reuse, while unexecuted simulated-server and real enclave stale-session retry tests leave key behavior needing CI/runtime confirmation.
1 open issue
Review effort: Balanced
BeginExecuteXmlReader now attaches its callback to and returns the global completion task, like Reader and NonQuery, so callers observe the bounded enclave retry instead of the faulted first attempt. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Async enclave changes span command guards, reservation ownership, and XML retry completion, while simulated-server and live SQL Server/enclave validation remain unverified and require human confirmation.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev/cheena/ae-async-metadata-cache #5085 +/- ##
======================================================================
+ Coverage 70.60% 70.95% +0.34%
======================================================================
Files 315 315
Lines 70176 70201 +25
======================================================================
+ Hits 49551 49813 +262
+ Misses 20625 20388 -237
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed for a critical retry-handoff race that allows connection changes or concurrent execution while invalidation is pending, potentially causing retries to use altered command state.
No open issues 🎉
Review effort: Balanced
…nc-enclave-key-loading Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adopt the execution-owned enclave in-progress flag (releaseInProgress), the new PrepareForTransparentEncryption signature, and retarget the reflection-based enclave tests to the current entry points. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
A critical gap in the async enclave retry guard allows connection changes or concurrent execution during the retry handoff, risking use of a replacement connection or corruption of another execution’s state.
1 open issue
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…om/dotnet/SqlClient into dev/cheena/ae-async-enclave-key-loading # Conflicts: # src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs
There was a problem hiding this comment.
🔵 Needs a closer look
Enclave-package changes affect TDS handoff, retry cleanup, and async-count ownership, while real enclave end-to-end validation remains pending and simulated tests cannot establish server-side correctness.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
The pending-package-key test helper now owns the in-progress flag like RunExecuteReader, so overlapping executions are rejected instead of blocking on the pending key. RunExecuteReader sets the flag before the cache probe so overlaps started there are rejected too. Test awaits on overlapping executions are bounded. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Simulated-server tests cannot establish real-enclave package generation and stale-session retry correctness across Reader, NonQuery, and XML, so an enclave-enabled end-to-end run and human review remain necessary.
No open issues 🎉
Review effort: Balanced

Description
Layer 5 of 5 split from #4585. Stacked on #5081.
EnclaveDelegate: addsGenerateEnclavePackageAsyncandGetDecryptedKeysToBeSentToEnclaveAsync. Keys load one at a time with async key store calls. Sync and async paths shareBuildEnclavePackageandCreateColumnEncryptionKeyInfo.SqlCommand.Reader: after describe,ContinueRunExecuteReaderTdsAsync(on aTask.Runworker) generates the enclave package asynchronously, then runs the TDS execution. This replaces the syncGenerateEnclavePackagecall. The connection is revalidated after each await. Cancellation propagates unwrapped.SqlCommand.CreateLocalCompletionTask: a retryable enclave package fault before the read stage now goes through the same bounded retry as read-stage failures. This is the first production reader of_asyncExecutionCancellationToken.Fixes from review feedback:
_enclaveAttestationInProgress) is held from the post-describe continuation through package key loading, package generation and theRunExecuteReaderTdshandoff. It is cleared on failure or cancellation. ChangingConnectionor starting another execution while package keys load now throws.ValidateConnectionLifetimegets the database captured before dispatch, after describe and after the package-key await. A database change during key loading throws.BeginExecuteXmlReadernow attaches its callback to and returns the global completion task, like Reader and NonQuery. Callers see the retry's outcome, not the faulted first attempt.Fixes the internal
TestEnclaveExecutionApisfailureThe XML retry task fix above also fixes the internal pipeline failure in
ApiShould.TestEnclaveExecutionApis(builds 193347 and 193466). ItsExecuteXmlReaderAsyncstep fails withInternal connection fatal error. Error state: 15, Token: 1(other runs show tokens 0, 70 and 88).Root cause: on main,
BeginExecuteXmlReaderInternalreturnslocalCompletion, which is the first attempt's task. When the enclave session is invalid, the driver runs an internal End and retries. The caller's End then ran on the first attempt at the same time as the retry, using the same state object, which desynced the TDS stream.The bug is already on main and isn't caused by this stack. #4887 added the test that exposes it, and build 193347, from before #5079 and #5080, already failed the same way. #5117 applies the same fix to main for testing only and won't be merged.
Deviations from #4585:
CreateLocalCompletionTasknow releases it after the retry succeeds, only on the read-stage path; a faulted write already released it.EnclaveDelegate.Crypto.cscomments mention the session-lock wait on a cache miss.Issues
Part of #4585.
Testing
EnclaveDelegateTests(6 tests): async vs sync provider API, all keys resolved, sequential loading, cancellation, provider failure.SqlCommandTests: newCreateLocalCompletionTask_*tests (retryable write fault retries, retry fails, retry canceled, non-retryable fault, canceled write).SimulatedServerTests/SqlCommandEnclaveAsyncTests:Execute_PendingPackageKey_*(revalidation, connection setter, second execution, database change),Prepare_Completed_HandsOffInProgressFlag,Retry_PackageFault_DoesNotTransferReservation,Retry_PackageFaultRetrySucceeds_KeepsOnlyRetryReservation,Retry_AwaitsInvalidationBeforeRestarting(count after successful retry),BeginExecuteXmlReader_RetryablePackageFault_CompletesWithRetry.ManualTests/AlwaysEncrypted/ApiShould.cs: re-enables debug stale-enclave retry coverage.Local results at aa9fadb:
dotnet test src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net10.0 -c Debug|Release --filter "FullyQualifiedName~SqlCommandEnclaveAsyncTests|FullyQualifiedName~SqlCommandQueryMetadataCacheAsyncTests|FullyQualifiedName~EnclaveDelegateTests|FullyQualifiedName~SqlCommandTests|FullyQualifiedName~SqlQueryMetadataCacheTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SymmetricKeyCacheTests|FullyQualifiedName~AsyncKeyStoreProviderTests": passed, 178/178 in each configuration. Simulated-server tests were run with a local-only macOS listener workaround (not committed).dotnet buildof UnitTests and ManualTests with-f net462 -c Debug|Release: 0 warnings, 0 errors.dotnet build build.proj -t:BuildSqlClient -p:Configuration=Debug|Release: 0 warnings, 0 errors.CI (build 192590, aa9fadb):
Pending: enclave end-to-end testing against a real enclave server.
Guidelines
Please review the contribution guidelines before submitting a pull request: