Skip to content

SOLR-18466: Support _route_ in the streaming expression search function - #4921

Open
tsornin14 wants to merge 1 commit into
apache:mainfrom
tsornin14:streaming-search-route/SOLR-18466
Open

tsornin14 wants to merge 1 commit into
apache:mainfrom
tsornin14:streaming-search-route/SOLR-18466

Conversation

@tsornin14

Copy link
Copy Markdown

https://issues.apache.org/jira/browse/SOLR-18466

Description

The search() function in streaming expressions currently ignores _route_ for shard selection and queries every active slice of the target collection. Users with known routing keys therefore incur unnecessary request and response overhead, which can be significant on large clusters. This change adds support for _route_ param in the streaming expression search function, reusing docCol.getRouter().getSearchSlices() which is used for shard selection in normal distributed search.

Solution

Two files in solr/solrj-streaming:

  1. the CloudSolrStream class:

    the existing method:

    List<Slice> getSlices(
        String collectionName,
        CloudSolrClient cloudSolrClient,
        boolean checkAlias)

    is kept for existing calls from TopicStream, TextLogitStream, FeaturesSelectionStream, and StatementImpl (all out of scope for this change); these are unaffected and continue to receive all active slices

    A new overloaded helper method is added:

    List<Slice> getSlices(
        String collectionName,
        CloudSolrClient cloudSolrClient,
        boolean checkAlias,
        SolrParams solrParams)

    When solrParams contains a nonempty _route_, each collection's slices are resolved via docCol.getRouter().getSearchSlices(routeKeys, params, docCol) and when _route_ is absent (or empty), all active slices are returned exactly as before.

  2. the TupleStream class:

    CloudSolrStream.getSlices(collection, cloudSolrClient, true, solrParams) is called after the request params are assembled and passed to the route aware getSlices.

Example usages:

expr=search(collection1,
       zkHost="localhost:9983",
       qt="/export",
       q="*:*",
       fl="id,a_s,a_i,a_f",
       sort="a_f asc, a_i asc", _route_="routeKeyA")
expr=search(collection1,
       zkHost="localhost:9983",
       qt="/export",
       q="*:*",
       fl="id,a_s,a_i,a_f",
       sort="a_f asc, a_i asc", _route_="routeKeyA,routeKeyB,routeKeyC")

One note on backward compatibility:

Previously, _route_ was ignored during shard selection for streaming expression search(). With this change, nonempty route values are passed to docCol.getRouter().getSearchSlices(routeKeys, params, docCol). For collections using the implicit router, an invalid _route_ value now causes the request to fail. Previously, the same request searched all active shards despite the invalid _route_. This should be documented in the release/ upgrade notes.

Tests

The following four tests cover single and multiple route keys, composite id routing, missing and empty route values, and invalid implicit router shard names.

  1. testCloudSolrStreamWithRouteParam
  2. testCloudSolrStreamWithCompositeIdRoute
  3. testCloudSolrStreamWithEmptyRoute
  4. testCloudSolrStreamWithInvalidRoute

Checklist

Please review the following and check all that apply:

  • I have reviewed the guidelines for How to Contribute and my code conforms to the standards described there to the best of my ability.
  • I have created a Jira issue and added the issue ID to my pull request title.
  • I have given Solr maintainers access to contribute to my PR branch. (optional but recommended, not available for branches on forks living under an organisation)
  • I have developed this patch against the main branch.
  • I have run ./gradlew check.
  • I have added tests for my changes.
  • I have added documentation for the Reference Guide
  • I have added a changelog entry for my change

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant