Repository navigation
[Async AE 4] Complete query metadata cache asynchronously - #5081
cheenamalhotra wants to merge 70 commits into
Conversation
Extract the cache foundation from #4585 with per-key async loading, cancellation, and shared sync/async publication. Relocate cache regressions into focused cache tests. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split layer 2 from #4585: decrypt CEKs and verify CMK signatures asynchronously, preserving provider fallback and cancellation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Avoid repeating decryption when a synchronous caller publishes the key while an async caller waits for the provider gate. Cover the race with a deterministic regression test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Layer 3/5 of #4585. Defer CMK signature verification and CEK decryption from the describe-parameter-encryption parser so the async path awaits the async key-store utilities with the caller's token. Capture and reset the caller token across Reader/NonQuery/XML async setup and cleanup. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split the metadata cache lookup into an in-memory probe and an async key load. Async execution loads cached column encryption keys on a worker, falls back to describe parameter encryption when keys are stale, and keeps good metadata cached when the load is canceled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical concurrency and connection-lifetime gaps during cached-key loading can let execution use changed connection state or fallback RPCs corrupt an active reader’s results.
3 open issues
What changed in this PR
This fourth layer of async Always Encrypted support extends asynchronous key loading to query metadata cache hits, while preserving synchronous provider dispatch.
Changes:
- Splits cache lookup into an in-memory probe and asynchronous key-loading completion.
- Dispatches async cache completion and stale-key discovery fallback to a worker.
- Adds coverage for cache hits, stale keys, cancellation, and worker dispatch.
| File | Description |
|---|---|
src/Microsoft.Data.SqlClient/tests/UnitTests/Utilities/QueryMetadataCacheTestCommands.cs |
Adds isolated commands and cached encryption metadata for tests. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft/Data/SqlClient/SqlQueryMetadataCacheTests.cs |
Tests async completion, cancellation, and sync compatibility. |
src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft/Data/SqlClient/SqlCommandTests.cs |
Tests worker dispatch for cache hits and stale-key fallback. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlQueryMetadataCache.cs |
Separates metadata probing from key loading and completion. |
src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlCommand.Encryption.cs |
Integrates worker-dispatched async cache completion into encryption preparation. |
Review effort: Balanced
Add isolated global-provider coverage for canceled inline and suspended verification results, retry, and positive/negative result reuse. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a gated provider that holds async CEK decryption pending and then returns the real key. Cover successful completion after release and cancellation while the provider is pending, followed by command reuse. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #5081 +/- ##
==========================================
+ Coverage 68.99% 72.73% +3.74%
==========================================
Files 314 315 +1
Lines 45857 71469 +25612
==========================================
+ Hits 31638 51984 +20346
- Misses 14219 19485 +5266
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Keep _enclaveAttestationInProgress set while cached keys load and until execution reserves the connection, so connection changes and overlapping executions are rejected. Reject a close/reopen or connection replacement before the stale-key fallback and before execution, and revalidate the connection before the fallback describe. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🟡 Changes recommended
Critical races allow command state changes before pending-operation publication and database changes during cached-key loading, potentially executing with altered state or writing to the wrong database.
2 open issues
3 resolved since last review
Review effort: Balanced
Capture the connection database before cached-key completion and reject execution if ChangeDatabase ran while keys loaded, because the query metadata cache key includes the database. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Replace the deferred-failure and dispatched flags with one holdsInProgress result. Cache hits and miss paths that complete synchronously return directly; only work that awaits runs in ContinueParameterEncryptionAsync. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Async cache-hit execution changes guard ownership and depends on #5085 for retries, requiring coordinated landing and maintainer validation against a real enclave server because CI skipped enclave tests.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
mdaigle
left a comment
There was a problem hiding this comment.
Still reviewing latest changes, leaving a placeholder.
| /// </summary> | ||
| private static Task GetPreparationTask(ValueTask preparation, bool holdsInProgress) | ||
| { | ||
| if (holdsInProgress || !preparation.IsCompletedSuccessfully) |
There was a problem hiding this comment.
We should be able to choose between getting the result and returning the task purely based on the isAsync value from PrepareForTransparentEncryption. Tying resource cleanup to the presence of a returnTask is confusing. My agent is saying that we should be able to manage _enclaveAttestationInProgress at the level above PrepareForTransparentEncryption. Then we don't need special signaling down at this level indicating when/how the flag should be released.
Acquire preparation guard
try
Complete encryption preparation
Revalidate connection and transaction
Start command execution
Establish normal pending-operation protection
finally
Release preparation guard
There was a problem hiding this comment.
Addressing my other comment may make this one irrelevant.
There was a problem hiding this comment.
Done in d1dd1bc. RunExecuteReader acquires the in-progress flag for async preparation (when it isn't already set) and passes releaseInProgress to execution. Execution releases it on success, failure and cancellation, and in the inline branch's finally. If preparation throws, it's released straight away. Preparation no longer touches the flag. isAsync selects between awaiting the task and GetAwaiter().GetResult(). An async preparation that already completed still runs execution inline, per #4887.
There was a problem hiding this comment.
The shared preparation method is a good step, but the caller still uses preparation != default to choose between inline execution and the continuation. Please remove that signaling contract and the custom ThreadPoolHop. We're not at the optimization stage: first establish a straightforward structure, then optimize inline completion separately.
Please make the coordinator own scheduling and the preparation guard, with thin sync/async entry paths around one shared preparation/execution sequence:
- Probe the metadata cache synchronously, keeping the probe result available for retry setup.
- For synchronous execution, run the shared sequence with
isAsync: falseand consume its result synchronously. Continue selecting synchronous provider APIs in this mode. - For asynchronous execution, acquire the guard before dispatch and use an ordinary
Task.Runboundary for the shared sequence withisAsync: true. Return the task tracking that sequence; don't infer routing from task identity, completion timing, or whether theValueTaskequalsdefault. - In the shared sequence, await preparation, revalidate connection/transaction state, generate the enclave package, and initiate execution. Hold the guard until the required write/setup completes and normal pending-operation protection has taken over. Release it in one owning
finally, including failure and cancellation, rather than passing cleanup ownership through preparation/continuation signals.
It's fine for async plaintext-only hits to take the same dispatched path initially. Please don't retain a special inline fast path at the cost of this structure. Keep asyncWrite independent of isAsync for legacy Begin/End compatibility, and preserve cache eviction, cancellation, bounded retry, and connection-lifetime checks. Coordinate the execution/package portion with #5085 so the two layers compose, without duplicating orchestration or expanding this into an unrelated execution-pipeline rewrite.
Please retain the existing regression coverage for pending key loads, stale-cache fallback, cancellation/reuse, and connection/transaction changes, and cover immediate and genuinely pending preparation completion under the same ownership model.
Auto-replied by the GitHub Copilot app
There was a problem hiding this comment.
Implemented this in d19ebc4. The synchronous cache probe stays separate; both modes now use one Task-based prepare/execute core. The async coordinator explicitly dispatches with Task.Run, acquires the guard before dispatch, and releases it in one finally after TDS write/setup or failure. Removed default(ValueTask) routing, ThreadPoolHop, and the plaintext inline fast path.
Always dispatching also moves package-generation failures into the task pipeline, so this includes the deferred-retry/reservation handling and XML global-completion fixes needed to preserve retries. Asynchronous enclave package generation itself remains in #5085.
Auto-replied by the GitHub Copilot app
| /// flag-holding work in <paramref name="returnTask"/>, and null when execution can start | ||
| /// inline. | ||
| /// </remarks> | ||
| private void PrepareForTransparentEncryption( |
There was a problem hiding this comment.
This method has a really awkward signature. I think we could get rid of a ton of the complexity here by splitting it out. It only has one caller, so it's not a significant refactor.
Instead of checking the metadata cache and retrieving/decrypting in the same call, we can have one method to check the cache and another to retrieve/decrypt. That will help avoid the careful return shapes we're constructing to guide the caller's retry flow.
ResetEncryptionState
Probe cached metadata → foundCachedMetadata + metadata
Prepare encryption(metadata, isAsync) → task
Execute after preparation
There was a problem hiding this comment.
Split in d1dd1bc. TryGetCachedQueryMetadata(isRetry, out metadata) now does the probe. PrepareForTransparentEncryption(metadata?, timeout, asyncWrite, isRetry, method, isAsync, ct) returns a ValueTask and has no out params. The flow is probe → prepare → task → execute.
TryGetCachedQueryMetadata probes the cache. PrepareForTransparentEncryption takes the probed metadata and returns a ValueTask. RunExecuteReader acquires the in-progress flag for async preparation and hands its release to execution, which releases it on every path. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because accessing command.Transaction after transaction completion during cached-key loading can clear the binding and allow the pending command to execute outside its original transaction.
No open issues 🎉
1 resolved since last review
Review effort: Balanced
| /// <see cref="ContinueParameterEncryptionAsync"/>. The caller holds the in-progress flag | ||
| /// for asynchronous preparation. | ||
| /// </remarks> | ||
| private ValueTask PrepareForTransparentEncryption( |
There was a problem hiding this comment.
Much of this method is still being duplicated in ContinueParameterEncryptionAsync. Let's combine them into one single method.
There was a problem hiding this comment.
Done in 636e39c. ContinueParameterEncryptionAsync is removed; PrepareForTransparentEncryption is now one async ValueTask for both modes. It branches on isAsync only for the worker hop, the async attestation, and the revalidation points. It returns default only when it finishes without yielding; any other task goes to execution, which releases the guard.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reading SqlCommand.Transaction during a pending cached key load cleared the binding, so execution skipped the TransactionZombied check. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🔵 Needs a closer look
Maintainer review is needed to coordinate landing with #5085 for deferred cache-hit retry handling and validate enclave runtime behavior, since supplied CI skipped real-enclave tests.
No open issues 🎉
Review effort: Balanced
Separate cached metadata probing from a shared preparation and execution sequence. Dispatch async work at the coordinator boundary and keep guard ownership and failure cleanup there, without completion-shape signals or a custom awaiter. Preserve retry results and connection reservations when deferred enclave package failures restart execution, including XML Begin/End. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| bool cacheHit = await cache.CompleteCachedQueryMetadata( | ||
| this, metadata, isAsync, cancellationToken) | ||
| .ConfigureAwait(false); |


Description
Layer 4 of 5 split from #4585. Stacked on #5080.
SqlQueryMetadataCache: splits the lookup into an in-memory probe (TryGetCachedQueryMetadata) and one sharedCompleteCachedQueryMetadata(..., isAsync, ct)that loads keys one by one, using async key store calls whenisAsyncis true.SqlCommand.PrepareForTransparentEncryption: on an async cache hit, keys load on a thread-pool worker (explicitThreadPoolHop). A stale key falls back to describe parameter encryption on that same worker, never on the caller thread. The sync path is unchanged.Review fixes, for the window while cached keys load:
_enclaveAttestationInProgressstays set until execution starts and is cleared on success, failure and cancellation. Until then, changingConnectionorTransaction, or starting a second execution, throws.ChangeDatabase. This is checked before the stale-key fallback and again before execution.ValidateConnectionForExecuteruns before the stale-key fallback describe.ChangeDatabaseduring the key load can't evict the new database's entry.ValidateCommandrun at both points, so a transaction begun meanwhile throwsTransactionRequired, and a bound transaction that completes meanwhile throwsTransactionZombied, instead of executing outside it. Readingcommand.Transactionmeanwhile returns null but keeps the binding, so the zombied check still runs.TryGetCachedQueryMetadataprobes the cache once.PrepareForTransparentEncryption(metadata?, ..., isAsync, ct)returns aValueTask. A hit completes in the selected mode; otherwise it runs attestation and describe (RunDescribeParameterEncryption).RunExecuteReadersets the in-progress flag before preparation and hands its release to execution (releaseInProgress), which releases it on success, failure and cancellation. Preparation never touches the flag.PrepareForTransparentEncryptionis oneasync ValueTaskfor sync and async callers. It returnsdefaultonly when it finished without yielding, and execution then runs inline; any other task, even a completed or faulted one, is handed to execution.Out of scope: the gap between the write completing and the read stage publishing
PendingAsyncOperationalready exists on main for every async path (open thread).Deferred to layer 5: async enclave package and key loading, and retry changes. Retry for deferred enclave-package faults on cache hits is added in #5085; land both together.
Issues
Part of #4585.
Testing
UnitTests/Microsoft/Data/SqlClient/SqlQueryMetadataCacheTests.cs: async hit, stale and cancel cases, plus sync hit and stale cases.SqlCommandTests: worker dispatch on an async cache hit, stale-key fallback, inline plaintext-only hit, and worker dispatch for a hit with cached enclave keys.SimulatedServerTests/SqlCommandQueryMetadataCacheAsyncTests.cs: 9 guard tests using a gated key store provider. They cover setter and second-execution rejection, the guard being released on cancel, reopen before execution and before the fallback, a reader opened before the fallback,ChangeDatabase(rejection and stale-key eviction), and a transaction begun during the key load. The simulated server has no transaction manager, so that test records the transaction on the session directly.dotnet test src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net10.0 -c Debug|Release --filter "FullyQualifiedName~SqlQueryMetadataCacheTests|FullyQualifiedName~SymmetricKeyCacheTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SqlCommandTests|FullyQualifiedName~SqlCommandQueryMetadataCacheAsyncTests|FullyQualifiedName~SqlCommandEnclaveAsyncTests": 168/168 passed in each configuration at af7f7ae. On macOS this ran with a local, uncommitted TDS listener patch, because the simulated server hangs on shutdown there.dotnet build src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net462 -c Debug|Release: succeeded, 0 warnings, 0 errors.Guidelines
Please review the contribution guidelines before submitting a pull request: