Skip to content

[Async AE 3] Load keys asynchronously during parameter discovery - #5080

Merged
cheenamalhotra merged 36 commits into
mainfrom
dev/cheena/ae-async-parameter-discovery
Nov 19, 2026
Merged

cheenamalhotra merged 36 commits into
mainfrom
dev/cheena/ae-async-parameter-discovery

Conversation

@cheenamalhotra

@cheenamalhotra cheenamalhotra commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Description

Layer 3/5 split from #4585. Depends on #5079 (base branch).

  • The describe-parameter-encryption parser now records CMK signature checks and CEK decryptions, then runs them after parsing completes. The sync path calls the sync utilities; the async path awaits the async utilities with the caller's token. Metadata is cached only after the key operations succeed (CacheQueryMetadataIfNeeded). The parser documents its exception ordering and why keys load sequentially.
  • ExecuteReaderAsync, ExecuteNonQueryAsync and ExecuteXmlReaderAsync save the caller's token, then clear it on cleanup and on failed setup. On failed setup, the async call context or the token registration is disposed.
  • No public API changes.

Review feedback fix: added gated-provider manual tests for key-store I/O that is still pending (cdb22c8).

Integration is partial until later layers land:

  • Layer 4 makes the query metadata cache path async. The existing sync cache lookup stays for now.
  • Layer 5 makes enclave package loading async and is the first production reader of the saved token. Sync package loading stays for now.

Issues

Relates to #3672.

Testing

  • Unit tests added in tests/UnitTests/Microsoft/Data/SqlClient/SqlCommandTests.cs, moved there from the source PR's SqlCommandAsyncShould:
    • End*Async_CanceledTask_ThrowsOperationCanceled (Reader, NonQuery, XML)
    • *Async_FailedSetup_ClearsCancellationToken (Reader, NonQuery, XML; with and without a connection)
  • Manual tests added in ManualTests/AlwaysEncrypted/AsyncKeyStoreProviderTests.cs:
    • Sync execution uses the sync provider API.
    • Reader/NonQuery/XML async execution uses the async API.
    • The caller's token reaches the provider when it cancels.
    • With a gated provider, the command stays pending until the provider is released, then returns the result.
    • Cancelling while the provider is pending throws, and the command stays reusable.

Local validation (macOS, head 86b2cdc):

  • dotnet test -f net10.0 -c Debug|Release --filter "FullyQualifiedName~Microsoft.Data.SqlClient.UnitTests.SqlCommandTests|FullyQualifiedName~SqlSecurityUtilityTests|FullyQualifiedName~SymmetricKeyCache": passed, 74/74, in both configurations.
  • dotnet build -f net10.0 and -f net462 for ManualTests: 0 errors.
  • The manual AE tests were not run locally because they need SQL Server.

CI from the top of the stack (#5085, build 192590, commit aa9fadb, which contains this head):

  • UnitTests: Linux net10 passed 2668 with 0 failures, Windows net10 passed 2692 with 0 failures, and net462 passed 2588 with 0 failures. Simulated-server tests ran, and no stack tests were skipped.
  • AE manual tests: 246 passed and 0 failed, including AsyncKeyStoreProviderTests (not skipped). Only the enclave tests were skipped, because CI has no enclave server.

Guidelines

Please review the contribution guidelines before submitting a pull request:

cheenamalhotra and others added 7 commits November 16, 2026 11:59
Extract the cache foundation from #4585 with per-key async loading, cancellation, and shared sync/async publication. Relocate cache regressions into focused cache tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Split layer 2 from #4585: decrypt CEKs and verify CMK signatures asynchronously, preserving provider fallback and cancellation behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Avoid repeating decryption when a synchronous caller publishes the key while an async caller waits for the provider gate. Cover the race with a deterministic regression test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Layer 3/5 of #4585. Defer CMK signature verification and CEK decryption
from the describe-parameter-encryption parser so the async path awaits the
async key-store utilities with the caller's token. Capture and reset the
caller token across Reader/NonQuery/XML async setup and cleanup.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 19:14
@github-project-automation github-project-automation Bot moved this to To triage in SqlClient Board Nov 16, 2026
@cheenamalhotra cheenamalhotra added this to the 8.1.0-preview1 milestone Nov 16, 2026
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The tests use a provider that throws synchronously, leaving pending key-store I/O, successful metadata-cache publication, and cancellation followed by command reuse unverified.

1 open issue
What changed in this PR

This third layer of the Always Encrypted async integration connects parameter discovery to asynchronous key-store utilities without changing public APIs. Async query-metadata cache and enclave-package key loading remain deferred to later layers.

Changes:

  • Shares describe-result parsing, defers signature verification and key decryption, and caches metadata only after key operations succeed.
  • Captures caller cancellation tokens and improves cleanup for Reader, NonQuery, and XML async execution.
  • Adds tests for provider dispatch, cancellation propagation, command reuse, and failed-setup cleanup.
File Description
src/​Microsoft.Data.SqlClient/​tests/​UnitTests/​Microsoft/​Data/​SqlClient/​SqlCommandTests.cs Adds cancellation and failed-setup token-cleanup tests.
src/​Microsoft.Data.SqlClient/​tests/​ManualTests/​AlwaysEncrypted/​AsyncKeyStoreProviderTests.cs Adds provider-dispatch, cancellation, and command-reuse integration tests.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Xml.cs Captures and clears XML execution tokens; handles failed-setup disposal.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Reader.cs Captures and clears reader execution tokens; refines failed-setup disposal.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.NonQuery.cs Captures and clears non-query execution tokens; handles failed-setup disposal.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.Encryption.cs Defers key operations, selects sync or async utilities, and delays metadata caching until success.
src/​Microsoft.Data.SqlClient/​src/​Microsoft/​Data/​SqlClient/​SqlCommand.cs Adds and documents the saved async-execution cancellation token.

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

SQL Server/key-store integration remains untested and CI is pending, leaving deferred provider completion, cancellation, command reuse, and enclave execution in need of human validation.

1 open issue

Review effort: Balanced

cheenamalhotra and others added 2 commits November 16, 2026 13:20
Add isolated global-provider coverage for canceled inline and suspended verification results, retry, and positive/negative result reuse.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Add a gated provider that holds async CEK decryption pending and then
returns the real key. Cover successful completion after release and
cancellation while the provider is pending, followed by command reuse.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 20:20
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async Always Encrypted discovery changes parser/key-loading ordering and cancellation lifetimes, requiring human review and SQL Server/key-store validation because integration tests for success, cancellation, and command reuse were not run.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 16, 2026 20:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Async Always Encrypted discovery changes key-loading and cancellation lifetimes, requiring human validation because SQL Server/key-store integration tests for pending-provider completion, cancellation/reuse, and enclave signature verification have not run.

No open issues 🎉

Review effort: Balanced

@codecov

codecov Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.17647% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.48%. Comparing base (17bef7b) to head (8979391).
⚠️ Report is 22 commits behind head on main.

Files with missing lines Patch % Lines
.../Microsoft/Data/SqlClient/SqlCommand.Encryption.cs 74.50% 13 Missing ⚠️
...ent/src/Microsoft/Data/SqlClient/SqlCommand.Xml.cs 78.57% 3 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5080      +/-   ##
==========================================
+ Coverage   68.99%   70.48%   +1.49%     
==========================================
  Files         314      315       +1     
  Lines       45857    70026   +24169     
==========================================
+ Hits        31638    49360   +17722     
- Misses      14219    20666    +6447     
Flag Coverage Δ
CI-SqlClient ?
sqlclient-pr 70.48% <81.17%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Deferred CMK signature verification changes security-sensitive enclave session ordering, while CI skipped enclave tests, leaving enclave-enabled integration validation and human review outstanding.

No open issues 🎉

Review effort: Balanced

@github-project-automation github-project-automation Bot moved this from In review to Waiting for customer in SqlClient Board Nov 18, 2026
- Rename CompleteAsyncExecuteReader to CompleteAsyncExecutionSynchronously.
- Set the in-progress flag at the start of CompleteParameterEncryptionAsync.
- Remove the unused _asyncExecutionCancellationToken field and its tests.
- Make End*Async and CreateEnclaveSession internal and call them directly in tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 22:12
Base automatically changed from dev/cheena/async-ae-crypto-utilities to main November 18, 2026 22:12
…arameter-discovery

# Conflicts:
#	src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlCommand.Encryption.cs

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a critical MARS database-switching race can cache encryption metadata under the wrong database and allow execution to proceed there.

No open issues 🎉

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 18, 2026 22:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Stack-owner review is still needed to resolve the cross-layer parser design and confirm key-verification and decryption ordering around enclave-session creation, since CI skipped real enclave scenarios.

No open issues 🎉

Review effort: Balanced

Pass isAsync and the cancellation token through the describe parser so CMK
signature checks and CEK decryption run inline. Remove the deferred
PendingColumnEncryptionKeyOperations and ColumnMasterKeySignatureVerification
types. Add sync-parity manual tests for ExecuteNonQuery and ExecuteXmlReader.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 22:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Policy-dependent review checks require human verification because trusted immutable copies of the required review references and repository policies were unavailable.

No open issues 🎉

Review effort: Balanced

@cheenamalhotra
cheenamalhotra merged commit 981a989 into main Nov 19, 2026
40 checks passed
@cheenamalhotra
cheenamalhotra deleted the dev/cheena/ae-async-parameter-discovery branch November 19, 2026 00:33
@github-project-automation github-project-automation Bot moved this from In review to Done in SqlClient Board Nov 19, 2026
cheenamalhotra added a commit that referenced this pull request Nov 19, 2026
Main now contains layers 1-3 (#5077, #5079, #5080). Conflicts are resolved
to the reviewed final form of the stack (#5085 b8f9d12), which includes
layers 4 and 5. Old *AsyncShould tests are removed because they target APIs
the stack replaced; their coverage lives in the stack's tests.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 82aca8ca-2078-4a47-860c-6f6ee78cbb02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants