[redis-ha] Self-heal a masterless set instead of waiting for sentinel

[ShmuelOps]

What this PR does / why we need it:

Fixes the masterless deadlock in #412.

When the master pod is lost and its replacement rejoins as a replica behind the same announce address, sentinel keeps advertising that address as master (it answers pings), every member's local view agrees with sentinel, and the set ends up with no master at all. Replicas loop on -NOMASTERLINK Can't SYNC while not connected with my master; automatic failovers abort with -failover-abort-no-good-slave / -failover-abort-not-elected, and an explicit SENTINEL FAILOVER returns -NOGOODSLAVE, because sentinel considers every remaining candidate stale. We hit this in production and it persisted for 7.5h until manual intervention.

The split-brain-fix container cannot help as written, for two reasons:

1. It only reinitializes a pod whose local view disagrees with sentinel — here everyone agrees, on a master that is really a replica.
2. It is blind in this state anyway: it reaches sentinel through the headless service, which drops all endpoints once every pod fails its readiness probe, so identify_master cannot resolve the name. MASTER comes back empty and the existing quorum-reset is equally a no-op.

The change. The announce-N services set publishNotReadyAddresses, so they keep resolving throughout the outage. Poll those for the one fact that matters — does any member hold role:master. After MASTERLESS_CONFIRMATIONS consecutive observations of none (default 5, MASTERLESS_CONFIRM_INTERVAL seconds apart, default 5), promote the reachable member with the highest replication offset via REPLICAOF NO ONE, repoint the others, and rewrite every sentinel's monitor entry to the promoted member (SENTINEL REMOVE + SENTINEL MONITOR — a bare sentinel reset rediscovers from the stale configured address, so the wrong entry survives it and the healer re-triggers).

The reinit crashloop branch. The branch where sentinel names this pod as master while it runs as a replica used to call reinit, which re-derives slaveof and shuts the pod down — it restarts straight back into a replica while sentinel keeps naming it, an endless shutdown loop (#383). That branch now heals from data-plane truth: it scans the announce services for whichever member actually holds role:master — if one does, sentinels are repointed at it (no failover, so a healthy master behind a stale sentinel answer is never demoted); only a confirmed masterless set goes through SENTINEL FAILOVER and the offset-aware direct promotion. The misnamed pod is the fastest observer of this state, so it heals on its first detection cycle instead of waiting for its peers. The stale-master escalation path uses the same healer.

Safety properties:

• Any member reporting role:master aborts the sequence, so concurrent containers never double-promote, and ordinary sentinel failovers are never interfered with.
• Promotion is offset-aware: the healer promotes the reachable member with the highest replication offset, never a specific pod merely because sentinel names it. A freshly replaced (empty) pod behind a stale sentinel record is therefore never promoted, which would otherwise resync every replica from an empty dataset.
• The scan retries up to PROMOTE_SCAN_ATTEMPTS times, so a member that happens to be mid-restart doesn't postpone recovery.
• All knobs are env-only, matching the existing MAX_QUORUM_FAILURES pattern — no new chart values.

Relation to #410

#410 addresses the same family of failure (sentinel naming an address whose real role is slave) by having the named pod promote itself with replicaof no one. This PR deliberately does not do that, for three reasons:

1. Unconditional self-promotion risks data loss. MASTER == ANNOUNCE_IP + role:slave is exactly what a freshly replaced pod looks like behind a stale sentinel record — sentinel never re-elected it, it just kept the address. Promoting that pod without comparing replication offsets can promote an empty node, and the other replicas (which already point at it) then full-resync from the empty dataset. Here the healer always promotes the highest-offset reachable member — which is the named pod whenever it genuinely is the best candidate.
2. fix(redis-ha): recover no-master cycle when sentinel names a replica as master #410's path is blind in the production outage. It still resolves sentinel through the headless service, which has no endpoints once every pod is unready (the [chart/redis-ha][BUG] Masterless deadlock: sentinel keeps naming a master whose actual role is replica; split-brain container never heals it #412 state), so its new branch never executes exactly when it is needed. This PR polls the announce-N services, which publish not-ready addresses.
3. No escalation or confirmation. fix(redis-ha): recover no-master cycle when sentinel names a replica as master #410 promotes after a single re-check; this PR tries SENTINEL FAILOVER first and promotes directly only after N consecutive masterless confirmations, aborting the moment any master appears.

#410's valid core idea — stop the reinit shutdown loop in that branch (#383) — is folded in here, with the promotion decision kept offset-aware.

Which issue this PR fixes

• fixes [chart/redis-ha][BUG] Masterless deadlock: sentinel keeps naming a master whose actual role is replica; split-brain container never heals it #412

Validation

Verified on an isolated 3-replica release. Fault: point every member at an unreachable address, then SENTINEL RESET — this reproduces the signature above exactly.

chart	outcome
4.39.0	no recovery after 5m41s
this patch	recovered in 64s

The named pod detects the state on its next detection cycle and heals directly (previously 1m57s via the peers' stale-master counters):

Redis role is slave but sentinel names this pod as master; checking whether the set is masterless
No member held the master role across 5 consecutive checks.
ERROR: no member holds the master role. Promoting redis-announce-0 (replication offset 39704).
Promotion complete: redis-announce-0 is now master.

Peers, confirming only one member acts:

A member holds the master role; standing down after 2/5 confirmations.

No false positives: deleting the master pod produces an ordinary sentinel failover — new master within seconds, the deleted pod rejoins as a replica, and no healer on any pod promotes or enters the confirmation loop. It acts only in the state sentinel cannot leave on its own.

helm lint clean; the rendered script passes sh -n with defaults and with auth + sentinel.auth enabled.

Checklist

• DCO signed
• Chart Version bumped
• Title of the PR starts with chart name (e.g. [stable/mychartname])

[ShmuelOps]

Pushed ac939b0 with two hardenings found while running this healer in production (PR body updated to match):

1. sentinel reset → explicit repoint. After a direct promotion, sentinel reset rebuilds the topology from the sentinel's configured (stale) master address — the replica table comes back empty, every later failover returns -NOGOODSLAVE, and the healer re-triggers forever. The healer now rewrites each sentinel's monitor entry (SENTINEL REMOVE + SENTINEL MONITOR at the promoted member), which converges deterministically. Reproduced on a live 3-replica release by pointing every sentinel at an address that answers role:slave with zero known replicas: the previous revision never recovers; with this commit it converges in ~85s, and concurrent healer containers on the other pods correctly stand down.

2. No blind forced failover. Escalation used to force SENTINEL FAILOVER on the strength of a single sentinel's answer (the script reaches sentinel through the service, i.e. a random endpoint). If that answer is merely stale while a healthy master exists, the forced failover demotes it. Both escalation paths now first scan the announce-N services for whichever member actually holds role:master: if one does, sentinels are repointed at it and no failover happens; only a confirmed masterless set fails over. This also closes a gap where sentinel names a pod that runs as a replica while the real master lives elsewhere — previously the masterless check aborted (a master exists) and nothing ever corrected sentinel.