[improvement](build) Add merged lance-c multi-vector API patch - #68056
Merged
Merged
Conversation
### What problem does this PR solve? Related PR: apache#67741; consumer integration: apache#68028 Add merged [lance-format/lance-c#83](lance-format/lance-c#83) to the Lance-C third-party patch chain on master. This provides the C/C++ multi-vector query API and its upstream scoring, validation and regression tests as a prerequisite for Doris multi-vector search integration. The patch records upstream commit `0a30ee6c5a9d1455ceb36f4745acb79e53a00461`. Overlapping context is adapted to the existing patch chain while retaining PR apache#79's scalar-segment execution path and tests. The multi-vector implementation and integration tests are byte-identical to the merged upstream commit. The release archive and dependency versions remain unchanged from apache#67741. Apply PR apache#83 with a separate completion marker so source trees already carrying the existing patches also receive the new API, and repeat runs do not reapply it. Rebuild Lance-C before consuming the new API. This PR changes only third-party patch integration and does not expose new Doris SQL syntax. ### Validation - Applied the actual Lance-C patch-driver block to clean v0.1.9 sources and to sources already patched through apache#67741; both produce identical source trees. Repeat execution leaves the sources unchanged. All patches apply with `--fuzz=0`. - Compared the resulting source tree with the previously validated integrated chain and checked the multi-vector implementation/tests against the merged upstream commit. - `cargo test --locked`: 410 tests passed, including 14 multi-vector integration tests; three opt-in C/C++ compile/run and static-transport tests were ignored in this run. - `cargo fmt --check` and `bash -n thirdparty/download-thirdparty.sh` passed. - Completed local self-review of patch provenance, conflict adaptation, existing API behavior, ownership/resource bounds and cached-source application before committing. A complete Doris build and SQL regression run are not part of this local dependency-only validation. ### Release note None ### Check List (For Author) - Test - [x] Unit Test - [x] Manual test: patch application, cached-source upgrade, repeat execution and source equivalence - Behavior changed: - [x] Yes: the third-party library gains the upstream multi-vector API; existing API signatures remain unchanged. - Does this need documentation? - [x] No: upstream API documentation is included in the patch. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
Contributor
Author
|
run buildall |
Contributor
Author
|
/review |
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Static exact-head review completed for b43d176. I found two blocking P2 issues, both left inline.
Critical checkpoints:
- Goal and tests: the patch adds the merged lance-c multi-vector C/C++ API and covers ordinary exact, indexed, refined, hybrid, nested-projection, metric, validation, offset, and strict-batch behavior. The implementation does not fully meet that goal for finite values that derive non-finite f32 scores, and the tests miss that numeric domain.
- Scope and compatibility: the Doris change is focused on the third-party patch chain. The ABI addition is additive; no FE/BE protocol, configuration, transaction, persistence, or storage-format change is involved. The independent PR83 cache marker is ordered safely under set -e and the embedded implementation matches the upstream merge context.
- Concurrency and lifecycle: scoring uses owned Arrow data across bounded blocking tasks; no new shared mutable state or locking is introduced. Scanner mutation is validation-before-commit, caller buffers are copied, callbacks/batch options are preserved, and FTS/scalar-segment coexistence remains fail-closed.
- Parallel paths and correctness: exact, indexed, mandatory-refinement, appended/hybrid, fragment-scoped, nested, and selected-segment paths were traced. Scorer replacement occurs below the relevant TopK nodes and its schema/distribution contracts are sound. Numeric error handling is not sound for derived overflow/NaN, as noted inline.
- Performance: candidate counts are bounded, but moving the result window above remaining-column Take defeats limit-driven late materialization and can read/sort up to 100,000 arbitrarily large payload rows for a one-row result.
- Observability and coverage: existing execution-stat callbacks are preserved and no separate new metric is required. Add focused tests for finite numeric overflow/underflow and for payload reads with k much larger than limit.
User focus: no additional review focus was supplied. Review status: complete after two rounds converged with no new valuable findings. This was static-only as required; no builds or tests were run.
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
morningman
approved these changes
Sep 16, 2026
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Related PR: #67741; consumer integration: #68028
Add merged lance-format/lance-c#83 to the Lance-C third-party patch chain on master. This provides the C/C++ multi-vector query API and its upstream scoring, validation and regression tests as a prerequisite for Doris multi-vector search integration.
The patch records upstream commit
0a30ee6c5a9d1455ceb36f4745acb79e53a00461. Overlapping context is adapted to the existing patch chain while retaining PR #79's scalar-segment execution path and tests. The multi-vector implementation and integration tests are byte-identical to the merged upstream commit. The release archive and dependency versions remain unchanged from #67741.Apply PR #83 with a separate completion marker so source trees already carrying the existing patches also receive the new API, and repeat runs do not reapply it. Rebuild Lance-C before consuming the new API. This PR changes only third-party patch integration and does not expose new Doris SQL syntax.
Validation
--fuzz=0.cargo test --locked: 410 tests passed, including 14 multi-vector integration tests; three opt-in C/C++ compile/run and static-transport tests were ignored in this run.cargo fmt --checkandbash -n thirdparty/download-thirdparty.shpassed.A complete Doris build and SQL regression run are not part of this local dependency-only validation.
Release note
None
Check List (For Author)
Check List (For Reviewer who merge this PR)