Repository navigation
direct: retry vector search endpoint create on transient errors - #7599
Merged
Merged
Conversation
The vector search endpoints API sometimes answers POST with 504 TEMPORARILY_UNAVAILABLE. DoCreate now retries transient errors and, if a retry is rejected with ALREADY_EXISTS, reads the endpoint the earlier attempt created and uses it. The testserver now returns 409 ALREADY_EXISTS with the cloud's message for a duplicate endpoint name (checked on AWS). aws: TestAccept/bundle/resources/vector_search_endpoints/basic Co-authored-by: Isaac <no-reply@databricks.com>
After a transient error and a 409, use the endpoint only if it was created after the first create attempt started, so a pre-existing endpoint or the old endpoint of a recreate still fails the deploy. Log the adoption as a warning. aws: TestAccept/bundle/resources/vector_search_endpoints/basic Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 2 changed tests (limit=20, 0 not selected) 3 tests: regression=1 unreleased=0 coverage=2 failing=0
TestAccept/bundle/resources/vector_search_endpoints/create-504/DMS=: branch ✅ | main (66d86c1) ❌ | latest (v1.26.0) ❌main (66d86c1): latest (v1.26.0): same output as main (66d86c1) |
Collaborator
✅ Integration test reportCommit: da05427
Top 6 slowest tests (at least 2 minutes):
|
denik
marked this pull request as ready for review
November 13, 2026 10:16
andrewnester
self-requested a review
November 13, 2026 10:52
andrewnester
left a comment
Contributor
There was a problem hiding this comment.
No blocking findings from a full pass; I'll take a manual look at the direct changes before approving.
andrewnester
approved these changes
Nov 13, 2026
Collaborator
❌ Integration test reportCommit: 8e70c8b
66 interesting tests: 45 FAIL, 9 RECOVERED, 9 flaky, 3 KNOWN
Top 50 slowest tests (at least 2 minutes):
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
The direct engine now retries a vector search endpoint create on transient errors (for example 504 TEMPORARILY_UNAVAILABLE). Until now the endpoint create was never retried, so one gateway timeout failed the deploy.
A retried create can be rejected with 409 ALREADY_EXISTS when the timed-out attempt had in fact created the endpoint. In that case DoCreate reads the endpoint and uses it, but only if it was created after the first attempt started (with one minute allowed for clock skew). An older endpoint still fails the deploy as before: one owned by someone else, or the old endpoint of a recreate that is still being deleted. The adoption is logged as a warning.
The testserver now returns what the cloud returns for a duplicate endpoint name: 409 ALREADY_EXISTS, "AI Search endpoint X already exists. Please try again with a different endpoint name." (checked on AWS; it used to return RESOURCE_ALREADY_EXISTS). DoCreate accepts both codes.
Decisions:
Why
Flaky bundle/invariant tests on main (aws and azure):
Error: cannot create resources.vector_search_endpoints.[NAME]: The service at /api/2.0/vector-search/endpoints is taking too long to process your request. Please try again later or try a faster operation. [TraceId: [TRACE_ID]] (504 TEMPORARILY_UNAVAILABLE)Dashboard:
Tests
bundle/resources/vector_search_endpoints/create-504injects a 504 on the create twice: once before the endpoint is created (the retry creates it) and once after (the retry gets 409 and uses the endpoint). Without the fix both deploys fail with the error above.vector_search_endpoints/basic.This PR was written by an AI agent (harness_auto, isaac_claude, opus5.5).
This pull request and its description were written by Isaac.