[spalen0]

Summary

Fixes from a security and stability review of the monitoring box. I reproduced the first two issues before fixing them.

Secret leakage (high)

• ProviderConnectionError put full RPC URLs (/v2/<alchemy-key>) in its message. run_with_alert then sent str(exc) to Telegram and stored it in alert_events under the protocol key, which the API and public site query. The automation digest did the same with stderr tails.
• New utils/redact.py masks configured secrets: PROVIDER_URL_* becomes scheme://host/***, and the values of *_KEY, *_TOKEN, *_SECRET and *_PASSWORD vars become ***. It also has pattern fallbacks for bot tokens, ?apikey= query params and keys in RPC URL paths.
• Redaction is applied in the logger formatter (scripts and the automation orchestrator), in send_telegram_message before the message is stored or sent, in crash alerts, in the digest error tail, in web3 provider errors and logs (host only), and in the dispatched message.

Emergency dispatch (high)

• The hook used to run only after a successful Telegram send. A 429 or Telegram outage therefore skipped the emergency withdrawal. The hook now runs before the Telegram send.
• Telegram 429s are retried once after retry_after (capped at 30s).
• New Alert.dispatch flag. Unconfirmed signals still alert but don't move funds:
  • Stables depeg: one DeFiLlama sample alerts as "Unconfirmed". Dispatch waits until the next run confirms it (2 consecutive breaches). A missing sample neither confirms nor clears.
  • 3Jane USD3/sUSD3 PPS: dispatch only for drops of at least 0.01%. Float and rounding noise still alerts.
  • Cap stcUSD assets-per-share: dispatch only for drops of at least 1 bp.
• Built on main's durable dispatch queue (Persist emergency deliveries and bypass HIGH cooldown for CRITICAL #457): dispatch=False alerts are never enqueued, and the message is redacted before it is written to the outbox DB or sent. The signed body has no sent_at, because retries resend the exact persisted bytes and the liquidity envelope stays unchanged. Vaults-mommy dedupes on event_id.

Stability (medium)

• Per-task timeout: in jobs.yaml: default 3600s, 300s for the stables sync-heartbeat task, 600s for the per-minute emergency-delivery-retries task. A timed-out task is killed and shows up in the digest. Before this, a hung task held its flock forever, and if that was ten_minute, code sync stopped too.
• git fetch (120s), git reset (60s) and uv sync (600s) now have timeouts.
• Gauntlet requests.get calls now have timeouts.

Deploy / CI

• monitoring-api.service: no longer loads an EnvironmentFile (the API needs no credentials), and /etc/monitoring is InaccessiblePaths.

• Caddyfile: rejects any bearer value that isn't 64 hex characters before comparing tokens. An empty ALERTS_API_TOKENS previously reduced the token check to ^Bearer ()$; it now fails closed. I checked this against a local Caddy 2.10.2:

  Token list	No/empty/wrong token	Correct token
  empty	401	401
  set	401	passed to upstream

  The docs now include an ExecStartPre check so Caddy refuses to start without tokens.

• install.sh: no longer copies the operator's personal gh auth token into ~/.git-credentials, a file the service can read. It requires an explicit GITHUB_TOKEN and the docs recommend a fine-grained, read-only token.

• CI: actions pinned to commit SHAs, permissions: contents: read, and persist-credentials: false.

• Runbook: new "Security baseline" section and a note on rotating keys after redaction.

LLM explainer (low)

• The system prompt now treats supplied context as untrusted data.
• The Telegram header reads AI Summary (unverified).
• No alert routing depends on the model's risk tag (checked).

Smaller fixes

• Telegram truncation now counts UTF-16 units.
• The test-chat override now gets the plain-text retry after a Markdown parse error.
• Fixed the stale jobs.yaml comment about manual uv sync.

Behaviour changes to check

• Stables depeg dispatch is delayed by one run (~10 min). The first alert still fires immediately.
• 3Jane/Cap PPS drops below the thresholds no longer dispatch. Adjust PPS_DISPATCH_MIN_DECREASE_BPS and ASSETS_PER_SHARE_DISPATCH_MIN_DECREASE_BPS if you want different thresholds.
• Long tasks: anything legitimately over 60 minutes needs an explicit timeout:.
• API unit: the new unit only takes effect when install.sh re-renders it.

Ops follow-ups (not code)

• Rotate RPC/API keys that may have appeared in past crash alerts or logs. Consider deleting affected alert_events rows.
• Branch protection on main: required review and CI, no force push, signed commits if possible.
• Replace the box's git credential with a fine-grained read-only token or a deploy key.
• Add the Caddy ExecStartPre token check to the caddy drop-in.
• liquidity-monitoring receiver: decide on replay protection. Its envelope has no ID or timestamp, so the 127.0.0.1-only binding is the current guard.
• Not done here: give the subscriber unit its own minimal env file. That needs coordination because it changes which env file the unit reads.
• Decide whether Wavey Gist reports for queued, not-yet-executed Safe transactions should stay at public URLs.

Test plan

• uv run pytest tests/: 2318 passed, 4 skipped (after merging main). New tests cover redaction, 429 retry, hook-on-failure, dispatch gating, task/git/uv timeouts, stables confirmation, and the 3Jane/Cap thresholds.
• ruff check, ruff format --check, ty check
• Caddyfile adapts, and the auth behaviour above was confirmed with a local Caddy run.
• bash -n deploy/install.sh

🤖 Generated with Claude Code