Skip to content

feat(morpho): audit live governance against daily jobs - #469

Merged
spalen0 merged 4 commits into
mainfrom
feat/morpho-live-audit
Nov 9, 2026
Merged

spalen0 merged 4 commits into
mainfrom
feat/morpho-live-audit

Conversation

@spalen0

@spalen0 spalen0 commented Nov 9, 2026

Copy link
Copy Markdown
Collaborator

Implements the audit in #449 and the fixes needed before the #450 cutover (epic #437). MORPHO_LIVE_DELIVERY is unchanged, so this does not switch anything to production.

Audit (#449)

#453 made the live path re-read on-chain state, so there is no event-derived state to compare with the daily snapshots. The audit compares outcomes instead: each daily alert that implies an indexed event is recorded and checked against the live path's decision for that event, matched on market id, calldata, new role holder, or adapter.

Verdict Meaning
matched The live path alerted the event, or evaluated it after the daily job had alerted it
divergent Reconciliation passed the block the daily job read with no such event, or the live path had already decided not to alert
waiting The indexer has not reached that block or the event is undecided, so lag never counts as agreement
out_of_window Within 3 days of the stream's first run, the event may predate the inbox

Divergences show in report and health, and post once to the errors chat as [morpho-live-audit]. Recording is best effort and cannot fail a daily job. It is skipped when the stream has no inbox.

Reminders, resolutions, escalations of an operation already seen, and guardian, sentinel, and allocator changes are not audited. The last three role types stay daily-only by decision (recorded on #437).

Fixes before production

  • Delivery latency: production alerts now store sent_at, and report adds decision-to-Telegram and block-to-Telegram delays.
  • Reorgs: if a reorg removes an event behind a sent production alert, one LOW correction goes to the Morpho channel. A rewritten event is the same change, so it gets none.
  • Long messages: a line longer than one Telegram message is now wrapped. Before, Telegram cut it at 4096 characters while the send still counted as delivered.

Docs

monitoring.yaml, the Morpho README, the runbook (audit, cutover criteria, the CACHE_DIR requirement for report), and outdated "sends no alerts" docstrings.

Testing

  • uv run ruff format, ruff check, ty check, and prek pass.
  • tests/test_morpho_live_audit.py is new, and 153 Morpho and subscriber tests pass.
  • The last full-suite run had 5 failures outside this change: 4 TestSafeApiQuota tests caused by a real SAFE_API_KEY in the local .env, and test_sd_notify_sends_datagram (socket path too long on the local machine).
  • Not tested against live data: the audit has not run on the server, so Observe and validate Morpho event delivery before production cutover #450's measurements are still open.

🤖 Generated with Claude Code

spalen0 and others added 4 commits November 9, 2026 18:49
Record each daily alert that implies an indexed event and settle it
against the live path's decisions, so a missed event is reported instead
of hidden by the daily job. Indexer lag stays waiting, never agreement.

Before production cutover, also record when Telegram accepted production
alerts, send one LOW correction when a reorg removes an event behind a
sent alert, and wrap lines too long for one Telegram message.

Guardian, sentinel and allocator changes stay daily-only by decision.

Refs #437, #449

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@spalen0
spalen0 merged commit b1501f3 into main Nov 9, 2026
3 checks passed
@spalen0
spalen0 deleted the feat/morpho-live-audit branch November 9, 2026 20:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant