Skip to content

SOLR-18417: Have a proper page on healthchecks - #4886

Open
epugh wants to merge 2 commits into
apache:mainfrom
epugh:introduce_healthcheck_focused_docs
Open

epugh wants to merge 2 commits into
apache:mainfrom
epugh:introduce_healthcheck_focused_docs

Conversation

@epugh

@epugh epugh commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

https://issues.apache.org/jira/browse/SOLR-18417

Description

Inspired by the upcoming removeal of "ping.adoc", this is a page on how to perform healthchecks.

Solution

A single page covering our Healthcheck API. I chose to only document the V2 one since that is our future. Did add docs on the healthcheck cli command too.

I think there might be room to add richer healthchecks in the future to our healthcheck api, like ones that run a query on a collection..

Tests

Docs.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 8, 2026
The optional `maxGenerationLag=<n>` parameter specifies the maximum number of Lucene commit generations by which a follower is allowed to trail its leader; the endpoint returns HTTP 503 if any core exceeds this threshold.
See xref:deployment-guide:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details and examples.
Health:: Report the health of the node, via the `api/node/health` endpoint ({solr-javadocs}/core/org/apache/solr/handler/admin/api/NodeHealth.html[NodeHealth]).
See xref:deployment-guide:healthcheck.adoc[] for details, parameters, and examples.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] Is there a reason you dropped the references to the v1 endpoint here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, we don't want v1, so why explain it? Maybe it should be listed until we remove V1? Happy to go whichever way you think...


See xref:user-managed-index-replication.adoc#monitoring-follower-replication-lag[Monitoring Follower Replication Lag] for details.

== Collection Data Healthcheck

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[0] I found this section-title a bit confusing, as it didn't appear in the summary table at the top of the page. Might be worth aligning that terminology.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah... glad you didn't like it, cause after i re-read it, I didn't lvoe it either, and was like... what the heck is "Collection Data" ;-)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, reworked the section title, and hoenstly, the whole paragraph.

== Collection Data Healthcheck

Where the Node Health endpoint asks "is this process alive?", `bin/solr healthcheck` asks "does this collection actually work?"
It queries the collection for a total document count, then queries each individual replica directly (non-distributed) for its own document count, and reports whether every shard has a leader and every replica is `ACTIVE`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] Is bin/solr healthcheck specific to SolrCloud? What would it do in stand-alone mode?

[Q] What is a user supposed to do with this per-replica docCount info? Healthy, well-functioning replicas often disagree on doc count just by nature of their autoCommit periods not being perfectly aligned. It might be worth flagging that for users, so they don't treat mismatches as "issues".

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I just learned about the colour coding stuff that is in one of the cluster apis:

ClusterStatus.postProcessCollectionJSON()computes ahealthfield per shard and per collection —GREEN/YELLOW/ORANGE/RED, based on the fraction of ACTIVE` replicas and whether a leader exists. This computation exists nowhere else
in the codebase.

So that is interesting.....

I will check if hte CLI is cloud only.

And fair point on the counts. Also, I kind of want bin/solr healthcheck to optionally take in a query to deterimien "good/bad" ness... Maybe like the old ping did. That might be out of scope or a new thing to think trhoguh.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You know what, I think I won't hold this PR up on the GREEN/YELLOW/ORANGE/RED. That can be for the future beasue it wrequires coding.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] Is bin/solr healthcheck specific to SolrCloud? What would it do in stand-alone mode?

What is "stand-alone" mode? I've heard of "user managed" mode. I actually don't know anything about how bin/solr heathcheck works in "stand-alone" mode. Probably the same way it works with Solr "embedded" mode? 🤷

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[Q] What is a user supposed to do with this per-replica docCount info? Healthy, well-functioning replicas often disagree on doc count just by nature of their autoCommit periods not being perfectly aligned. It might be worth flagging that for users, so they don't treat mismatches as "issues".

Humm, interesting... I don't know how far into the minutia of solr cloud we want to go here... let me see...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I tried User Managed mode and:

➜  dev git:(introduce_healthcheck_focused_docs) ✗ bin/solr healthcheck -c techproducts
Neither --solr-connection, --zk-host or --solr-url parameters, nor SOLR_CONNECTION, ZK_HOST env var provided, so assuming solr url is http://localhost:8983.
Healthcheck tool only works in Solr Cloud mode.


== Coming From `/admin/ping`?

Solr's older `/admin/ping` endpoint (`PingRequestHandler`) was deprecated in Solr 10.1 and removed in Solr 11.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[-1] IMO it does not make sense to continue to carry around documentation for features that have been removed from Solr. We don't do it for DIH, etc.

These sort of "If you used to use 'X', switch to 'Y'" notices are one of the main benefits of our changelog. IMO this should just go there.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree... After having writtne the text, my first thought was "does this live until solr 11 or 12 or when do we remove it". We do have a policy that we don't do this, so yeah, agreed.

Solr's older `/admin/ping` endpoint (`PingRequestHandler`) was deprecated in Solr 10.1 and removed in Solr 11.
Unlike the Node Health endpoint, Ping executed a real, configurable query against a specific core -- so a passing ping meant the query path was actually functional, not just that the node was alive.

If you configured a load balancer against `/admin/ping` specifically to catch a broken query path (not just a dead node), the Node Health endpoint is *not* a drop-in replacement -- it never touches the index.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[0] I personally don't know what a "broken query path" means. Maybe I'm just blanking, but if you end up retaining this section (see my comment on L174 above) then might be worth word-smithing that a bit.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so, I konw that some poeple would have a loadbalancer that would take a node out of rotation if sya it didn't have all the docs.. if you expected 10,000 docs, and your nightly job deletes and recreates the index, and it only go to 9,999 docs then it would be taken out by the load balancer.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this section was removed

@epugh
epugh requested a review from gerlowskija September 16, 2026 18:37
@epugh

epugh commented Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Okay, I have done rewrites based on your feedback, would love one more review.

@epugh

epugh commented Sep 21, 2026

Copy link
Copy Markdown
Contributor Author

Going to merge on Monday.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants