[spalen0]

Closes #465

Problem

On 2026-11-07 14:05 UTC the freshness check logged Failed to fetch block 47317489 timestamp on KATANA: 'NoneType' object is not subscriptable. The raw eth_getBlockByNumber call returns the JSON-RPC body as is, so a null block or an RPC error never raises inside make_request. Provider rotation never ran, the diagnostic said nothing useful, and Katana stayed unmeasured for that run without any notice.

Changes

• Validated parsing (parse_block_timestamp): RPC error (code and message), null result, a non-dict body and a missing or non-hex timestamp each raise BlockTimestampUnavailableError with a specific reason.
• Bounded retry and fallback: response-level failures are retried up to 3 times, 5s apart, moving to the next configured provider between attempts (new Web3Client.rotate_provider(), which does nothing when only one provider is set). Transport failures are not retried again because make_request has already gone through every provider with backoff.
• Credential-safe diagnostics: provider URLs usually carry the API key. The new utils.formatting.redact_urls reduces URLs to scheme and host and strips the scheme-less url: /path in urllib3 errors. The shared provider logs and ProviderConnectionError in utils/web3_wrapper.py now use it, and invalid provider URLs are no longer logged. Transport-failure alerts name only the exception type, so exception text never reaches Telegram.
• Unknown lag counts as degraded coverage, separate from fresh and stale:
  • A single unmeasurable run is only logged. The observed incident cleared on the next run, so it would not alert.
  • After --unknown-alert-after-runs consecutive unmeasurable runs (default 2, env INDEXER_UNKNOWN_ALERT_AFTER_RUNS), a silent Envio-channel alert goes out with the streak, the last indexed block and the reason. It uses the existing per-chain cooldown.
  • When the chain can be measured again, a one-time "lag measurable again" note goes out and the state resets.
  • The stale alert state is left alone. A stale chain that becomes unmeasurable gets no "caught up" note.
• The other chains are always checked, and a missing RPC client for one chain no longer stops the rest.
• README and monitoring.yaml are updated.

Testing

• New tests cover every malformed or error response shape, a null result followed by success on a retry, running out of attempts, URL redaction on transport failure, a missing client, a one-off unknown run (no alert), unknown → alert → cooldown → recovery, and an unmeasurable stale chain (no false recovery).
• uv run pytest: 2117 passed, 4 skipped. ruff check, ruff format --check and ty check are clean.
• Ran the script against the live indexer with an isolated cache: all 5 expected chains were measured and fresh. I also confirmed that Katana RPC returns {"result": null} for a block it does not have.

🤖 Generated with Claude Code