Skip to content

[Async AE 4] Complete query metadata cache asynchronously - #5081

Open
cheenamalhotra wants to merge 70 commits into
mainfrom
dev/cheena/ae-async-metadata-cache
Open

cheenamalhotra wants to merge 70 commits into
mainfrom
dev/cheena/ae-async-metadata-cache

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

Layer 4 of 5 split from #4585. Stacked on #5080.

  • SqlQueryMetadataCache: splits the lookup into an in-memory probe (TryGetCachedQueryMetadata) and one shared CompleteCachedQueryMetadata(..., isAsync, ct) that loads keys one by one, using async key store calls when isAsync is true.
    • Stale key: the entry is evicted and the lookup is a miss.
    • Canceled: the exception propagates and good metadata stays cached.
    • Other errors: the entry is evicted and the error is rethrown.
  • SqlCommand.PrepareForTransparentEncryption: on an async cache hit, keys load on a thread-pool worker (explicit ThreadPoolHop). A stale key falls back to describe parameter encryption on that same worker, never on the caller thread. The sync path is unchanged.

Review fixes, for the window while cached keys load:

  • _enclaveAttestationInProgress stays set until execution starts and is cleared on success, failure and cancellation. Until then, changing Connection or Transaction, or starting a second execution, throws.
  • Execution is rejected if the connection was replaced, closed and reopened, or switched with ChangeDatabase. This is checked before the stale-key fallback and again before execution.
  • ValidateConnectionForExecute runs before the stale-key fallback describe.
  • A stale key evicts the cache entry that was probed, so a ChangeDatabase during the key load can't evict the new database's entry.
  • The transaction binding checks of ValidateCommand run at both points, so a transaction begun meanwhile throws TransactionRequired, and a bound transaction that completes meanwhile throws TransactionZombied, instead of executing outside it. Reading command.Transaction meanwhile returns null but keeps the binding, so the zombied check still runs.
  • Split preparation: TryGetCachedQueryMetadata probes the cache once. PrepareForTransparentEncryption(metadata?, ..., isAsync, ct) returns a ValueTask. A hit completes in the selected mode; otherwise it runs attestation and describe (RunDescribeParameterEncryption).
  • Guard ownership: for async execution, RunExecuteReader sets the in-progress flag before preparation and hands its release to execution (releaseInProgress), which releases it on success, failure and cancellation. Preparation never touches the flag. PrepareForTransparentEncryption is one async ValueTask for sync and async callers. It returns default only when it finished without yielding, and execution then runs inline; any other task, even a completed or faulted one, is handed to execution.
  • A cache hit with no keys to load completes inline only if it has no cached enclave keys either. Otherwise it stays on the worker, so the enclave package is never built on the caller thread.

Out of scope: the gap between the write completing and the read stage publishing PendingAsyncOperation already exists on main for every async path (open thread).

Deferred to layer 5: async enclave package and key loading, and retry changes. Retry for deferred enclave-package faults on cache hits is added in #5085; land both together.

Issues

Part of #4585.

Testing

  • New UnitTests/Microsoft/Data/SqlClient/SqlQueryMetadataCacheTests.cs: async hit, stale and cancel cases, plus sync hit and stale cases.
  • SqlCommandTests: worker dispatch on an async cache hit, stale-key fallback, inline plaintext-only hit, and worker dispatch for a hit with cached enclave keys.
  • New SimulatedServerTests/SqlCommandQueryMetadataCacheAsyncTests.cs: 9 guard tests using a gated key store provider. They cover setter and second-execution rejection, the guard being released on cancel, reopen before execution and before the fallback, a reader opened before the fallback, ChangeDatabase (rejection and stale-key eviction), and a transaction begun during the key load. The simulated server has no transaction manager, so that test records the transaction on the session directly.
  • dotnet test src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net10.0 -c Debug|Release --filter "FullyQualifiedName~SqlQueryMetadataCacheTests|FullyQualifiedName~SymmetricKeyCacheTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SqlCommandTests|FullyQualifiedName~SqlCommandQueryMetadataCacheAsyncTests|FullyQualifiedName~SqlCommandEnclaveAsyncTests": 168/168 passed in each configuration at af7f7ae. On macOS this ran with a local, uncommitted TDS listener patch, because the simulated server hangs on shutdown there.
  • dotnet build src/Microsoft.Data.SqlClient/tests/UnitTests/Microsoft.Data.SqlClient.UnitTests.csproj -f net462 -c Debug|Release: succeeded, 0 warnings, 0 errors.
  • CI for this PR at an earlier head (8dbd424): all 39 checks passed, including sqlclient-pr 192584.
  • Full-stack CI ([Async AE 5] Load enclave keys asynchronously and route package retries #5085, build 192590, aa9fadb):
    • UnitTests: Linux net10 2668 passed, Windows net10 2692 passed, net462 2588 passed, 0 failed. Simulated-server tests ran and none of the stack's tests were skipped.
    • AE manual tests: 246 passed, 0 failed. Only the enclave tests were skipped, because CI has no enclave server.

Guidelines

Please review the contribution guidelines before submitting a pull request:

cheenamalhotra and others added 9 commits November 16, 2026 11:59
Extract the cache foundation from #4585 with per-key async loading, cancellation, and shared sync/async publication. Relocate cache regressions into focused cache tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split layer 2 from #4585: decrypt CEKs and verify CMK signatures asynchronously, preserving provider fallback and cancellation behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Avoid repeating decryption when a synchronous caller publishes the key while an async caller waits for the provider gate. Cover the race with a deterministic regression test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Layer 3/5 of #4585. Defer CMK signature verification and CEK decryption
from the describe-parameter-encryption parser so the async path awaits the
async key-store utilities with the caller's token. Capture and reset the
caller token across Reader/NonQuery/XML async setup and cleanup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split the metadata cache lookup into an in-memory probe and an async key
load. Async execution loads cached column encryption keys on a worker,
falls back to describe parameter encryption when keys are stale, and
keeps good metadata cached when the load is canceled.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:19
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 16, 2026
@cheenamalhotra cheenamalhotra added this to the 8.1.0-preview1 milestone Nov 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical concurrency and connection-lifetime gaps during cached-key loading can let execution use changed connection state or fallback RPCs corrupt an active reader’s results.

3 open issues
What changed in this PR

This fourth layer of async Always Encrypted support extends asynchronous key loading to query metadata cache hits, while preserving synchronous provider dispatch.

Changes:

  • Splits cache lookup into an in-memory probe and asynchronous key-loading completion.
  • Dispatches async cache completion and stale-key discovery fallback to a worker.
  • Adds coverage for cache hits, stale keys, cancellation, and worker dispatch.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Utilities/​QueryMetadataCacheTestCommands.cs Adds isolated commands and cached encryption metadata for tests.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlQueryMetadataCacheTests.cs Tests async completion, cancellation, and sync compatibility.
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlCommandTests.cs Tests worker dispatch for cache hits and stale-key fallback.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlQueryMetadataCache.cs Separates metadata probing from key loading and completion.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Encryption.cs Integrates worker-dispatched async cache completion into encryption preparation.

Review effort: Balanced

cheenamalhotra and others added 5 commits November 16, 2026 13:20
Add isolated global-provider coverage for canceled inline and suspended verification results, retry, and positive/negative result reuse.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a gated provider that holds async CEK decryption pending and then
returns the real key. Cover successful completion after release and
cancellation while the provider is pending, followed by command reuse.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.79167% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.73%. Comparing base (17bef7b) to head (d19ebc4).
⚠️ Report is 27 commits behind head on main.

Files with missing lines Patch % Lines
.../Microsoft/Data/SqlClient/SqlQueryMetadataCache.cs 89.47% 6 Missing ⚠️
.../src/Microsoft/Data/SqlClient/SqlCommand.Reader.cs 94.91% 3 Missing ⚠️
.../Microsoft/Data/SqlClient/SqlCommand.Encryption.cs 98.36% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5081      +/-   ##
==========================================
+ Coverage   68.99%   72.73%   +3.74%     
==========================================
  Files         314      315       +1     
  Lines       45857    71469   +25612     
==========================================
+ Hits        31638    51984   +20346     
- Misses      14219    19485    +5266     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 72.73% <94.79%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Keep _enclaveAttestationInProgress set while cached keys load and until
execution reserves the connection, so connection changes and overlapping
executions are rejected. Reject a close/reopen or connection replacement
before the stale-key fallback and before execution, and revalidate the
connection before the fallback describe.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 21:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical races allow command state changes before pending-operation publication and database changes during cached-key loading, potentially executing with altered state or writing to the wrong database.

2 open issues
3 resolved since last review

Review effort: Balanced

Capture the connection database before cached-key completion and reject
execution if ChangeDatabase ran while keys loaded, because the query
metadata cache key includes the database.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 22:05
Copilot AI balanced review requested due to automatic review settings November 19, 2026 19:47
benrr101
benrr101 previously approved these changes Nov 19, 2026
Replace the deferred-failure and dispatched flags with one holdsInProgress
result. Cache hits and miss paths that complete synchronously return
directly; only work that awaits runs in ContinueParameterEncryptionAsync.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async cache-hit execution changes guard ownership and depends on #5085 for retries, requiring coordinated landing and maintainer validation against a real enclave server because CI skipped enclave tests.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 19, 2026 19:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a transaction completing during asynchronous key loading can cause the pending command to execute outside its original transaction.

No open issues 🎉

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 19, 2026 20:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a transaction-lifecycle race during cached-key loading can allow execution outside the original transaction.

No open issues 🎉

Review effort: Balanced

@mdaigle mdaigle left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still reviewing latest changes, leaving a placeholder.

@github-project-automation github-project-automation Bot moved this from In review to Waiting for customer in SqlClient Board Nov 19, 2026
/// </summary>
private static Task GetPreparationTask(ValueTask preparation, bool holdsInProgress)
{
if (holdsInProgress || !preparation.IsCompletedSuccessfully)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to choose between getting the result and returning the task purely based on the isAsync value from PrepareForTransparentEncryption. Tying resource cleanup to the presence of a returnTask is confusing. My agent is saying that we should be able to manage _enclaveAttestationInProgress at the level above PrepareForTransparentEncryption. Then we don't need special signaling down at this level indicating when/how the flag should be released.

Acquire preparation guard
try
    Complete encryption preparation
    Revalidate connection and transaction
    Start command execution
    Establish normal pending-operation protection
finally
    Release preparation guard

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressing my other comment may make this one irrelevant.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in d1dd1bc. RunExecuteReader acquires the in-progress flag for async preparation (when it isn't already set) and passes releaseInProgress to execution. Execution releases it on success, failure and cancellation, and in the inline branch's finally. If preparation throws, it's released straight away. Preparation no longer touches the flag. isAsync selects between awaiting the task and GetAwaiter().GetResult(). An async preparation that already completed still runs execution inline, per #4887.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The shared preparation method is a good step, but the caller still uses preparation != default to choose between inline execution and the continuation. Please remove that signaling contract and the custom ThreadPoolHop. We're not at the optimization stage: first establish a straightforward structure, then optimize inline completion separately.

Please make the coordinator own scheduling and the preparation guard, with thin sync/async entry paths around one shared preparation/execution sequence:

  1. Probe the metadata cache synchronously, keeping the probe result available for retry setup.
  2. For synchronous execution, run the shared sequence with isAsync: false and consume its result synchronously. Continue selecting synchronous provider APIs in this mode.
  3. For asynchronous execution, acquire the guard before dispatch and use an ordinary Task.Run boundary for the shared sequence with isAsync: true. Return the task tracking that sequence; don't infer routing from task identity, completion timing, or whether the ValueTask equals default.
  4. In the shared sequence, await preparation, revalidate connection/transaction state, generate the enclave package, and initiate execution. Hold the guard until the required write/setup completes and normal pending-operation protection has taken over. Release it in one owning finally, including failure and cancellation, rather than passing cleanup ownership through preparation/continuation signals.

It's fine for async plaintext-only hits to take the same dispatched path initially. Please don't retain a special inline fast path at the cost of this structure. Keep asyncWrite independent of isAsync for legacy Begin/End compatibility, and preserve cache eviction, cancellation, bounded retry, and connection-lifetime checks. Coordinate the execution/package portion with #5085 so the two layers compose, without duplicating orchestration or expanding this into an unrelated execution-pipeline rewrite.

Please retain the existing regression coverage for pending key loads, stale-cache fallback, cancellation/reuse, and connection/transaction changes, and cover immediate and genuinely pending preparation completion under the same ownership model.

Auto-replied by the GitHub Copilot app

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Implemented this in d19ebc4. The synchronous cache probe stays separate; both modes now use one Task-based prepare/execute core. The async coordinator explicitly dispatches with Task.Run, acquires the guard before dispatch, and releases it in one finally after TDS write/setup or failure. Removed default(ValueTask) routing, ThreadPoolHop, and the plaintext inline fast path.

Always dispatching also moves package-generation failures into the task pipeline, so this includes the deferred-retry/reservation handling and XML global-completion fixes needed to preserve retries. Asynchronous enclave package generation itself remains in #5085.

Auto-replied by the GitHub Copilot app

/// flag-holding work in <paramref name="returnTask"/>, and null when execution can start
/// inline.
/// </remarks>
private void PrepareForTransparentEncryption(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method has a really awkward signature. I think we could get rid of a ton of the complexity here by splitting it out. It only has one caller, so it's not a significant refactor.

Instead of checking the metadata cache and retrieving/decrypting in the same call, we can have one method to check the cache and another to retrieve/decrypt. That will help avoid the careful return shapes we're constructing to guide the caller's retry flow.

ResetEncryptionState
Probe cached metadata → foundCachedMetadata + metadata
Prepare encryption(metadata, isAsync) → task
Execute after preparation

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split in d1dd1bc. TryGetCachedQueryMetadata(isRetry, out metadata) now does the probe. PrepareForTransparentEncryption(metadata?, timeout, asyncWrite, isRetry, method, isAsync, ct) returns a ValueTask and has no out params. The flow is probe → prepare → task → execute.

TryGetCachedQueryMetadata probes the cache. PrepareForTransparentEncryption
takes the probed metadata and returns a ValueTask. RunExecuteReader acquires
the in-progress flag for async preparation and hands its release to
execution, which releases it on every path.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A cache-load race can leave the task handoff null, causing synchronous enclave key loading on the caller thread and skipping required connection revalidation.

1 open issue

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because accessing command.Transaction after transaction completion during cached-key loading can clear the binding and allow the pending command to execute outside its original transaction.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

/// <see cref="ContinueParameterEncryptionAsync"/>. The caller holds the in-progress flag
/// for asynchronous preparation.
/// </remarks>
private ValueTask PrepareForTransparentEncryption(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Much of this method is still being duplicated in ContinueParameterEncryptionAsync. Let's combine them into one single method.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 636e39c. ContinueParameterEncryptionAsync is removed; PrepareForTransparentEncryption is now one async ValueTask for both modes. It branches on isAsync only for the worker hop, the async attestation, and the revalidation points. It returns default only when it finishes without yielding; any other task goes to execution, which releases the guard.

cheenamalhotra and others added 2 commits November 19, 2026 17:46
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Reading SqlCommand.Transaction during a pending cached key load cleared the
binding, so execution skipped the TransactionZombied check.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a critical race during cached-key loading can allow a pending command to execute outside its original transaction.

No open issues 🎉

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Maintainer review is needed to coordinate landing with #5085 for deferred cache-hit retry handling and validate enclave runtime behavior, since supplied CI skipped real-enclave tests.

No open issues 🎉

Review effort: Balanced

benrr101
benrr101 previously approved these changes Nov 20, 2026
Separate cached metadata probing from a shared preparation and execution sequence. Dispatch async work at the coordinator boundary and keep guard ownership and failure cleanup there, without completion-shape signals or a custom awaiter.

Preserve retry results and connection reservations when deferred enclave package failures restart execution, including XML Begin/End.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Provider task continuations can resume inline on a non-pool completion thread, allowing synchronous fallback discovery or enclave key loading to bypass the intended worker dispatch.

1 open issue

Review effort: Balanced

Comment on lines +545 to +547
bool cacheHit = await cache.CompleteCachedQueryMetadata(
this, metadata, isAsync, cancellationToken)
.ConfigureAwait(false);

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Waiting for customer

Development

Successfully merging this pull request may close these issues.

6 participants