SOLR-18424: add oversampling + raw dense vector reranking

[liangkaiwen]

https://issues.apache.org/jira/browse/SOLR-18424

Description

Solr 10 introduced Binary Quantization, however the recall drop-off is fairly sharp when used as-is. This can be partially offset by oversampling (retrieving more total documents from HNSW graph) and reranking (using raw non-quantized vector values from .vec). While reranking is possible through the standard Solr reranking feature, it is cumbersome syntactically and easy to get wrong. This aims to offer a cleaner and performant oversampling/reranking natively in the KNN query parser

Solution

Code mostly written by Claude Opus 5

• Add a param 'rerankOversample' to the KNN query parser. This param accepts a positive integer and is optional (default value=1 when not provided). Behaviorally, this value is a multiplier on the existing topK * efSearch, and if it is > 1 do a rerank stage after search down to topK only.
• Validation of rerankOversample. The provided value should be >= 1 and oversample should not be applied to BYTE encoded vectors
• Uses generic RescoreTopNQuery and passes the similarity function explicitly, because the lucene rerank vector function can hit a NullPointerException when similarity function is not explicitly defined (as a binary quantized field definition might not).

Latency / Recall Benchmarks
To be added soon

Tests

• Manually querying
• Unit tests
• Benchmarking

Checklist

Please review the following and check all that apply:

• I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
• I have created a Jira issue and added the issue ID to my pull request title.
• I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
• I have developed this patch against the main branch.
• I have run ./gradlew check.
• I have added tests for my changes.
• I have added documentation for the Reference Guide
• I have added a changelog entry for my change

[liangkaiwen]

Exploring a bug where Lucene's RescoreTopNQuery can get an index out of bounds exception when the inner query returns 0 documents