Skip to content

Add hourly Tenderly usage spike alerts - #470

Merged
spalen0 merged 2 commits into
mainfrom
feat/tenderly-usage-spikes
Nov 10, 2026
Merged

spalen0 merged 2 commits into
mainfrom
feat/tenderly-usage-spikes

Conversation

@spalen0

@spalen0 spalen0 commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

Checks organization-wide Tenderly TU usage in the existing hourly profile. A Telegram alert fires when a settled hour consumes at least 3x the busiest hour in the preceding 24 hours and at least 100,000 additional TUs. Complete hourly buckets and the prior peak tolerate normal hourly and daily cron bursts.

The monitor checkpoints its first pending hour before fetching data and processes missed hours chronologically. Normal hours and successfully delivered spikes advance the checkpoint; API failures, missing data, and skipped or failed delivery retain it for a later run. Whole-valued float timestamps are normalized without rounding, while malformed timestamps and values are rejected.

Messages go to Yearn maintenance with the existing Yearn fallback and are stored under yearn-internal. --dry-run evaluates the latest settled hour without reading or writing monitor state or sending messages. The hourly schedule, monitoring metadata, and Tenderly README are included.

Uses the existing Tenderly credential and the dashboard's undocumented usage endpoint. Catch-up uses one weekly history response; if required history is unavailable, the monitor reports an error and retains the checkpoint for investigation. Small leaks can be hidden by normal organization traffic, and sustained high usage enters subsequent baselines.

Validation:

  • Full suite: 2236 passed, 4 skipped, 117 subtests passed. Local Safe credentials were isolated and a short temporary path was used for macOS socket tests.
  • prek run --all-files passed; full Ruff lint/format and type checks passed.
  • Dependency audit found no known vulnerabilities or adverse project statuses.
  • Live dry-run against the configured Yearn account succeeded without alerting.
  • Regression tests failed before the fixes and now cover hour rollover, multi-hour catch-up, failed/skipped delivery, API/data failures, and timestamp normalization.
  • Codex review found no actionable regressions; Gemini was unavailable locally.

@spalen0
spalen0 marked this pull request as ready for review November 10, 2026 17:23
- P2: persist the next pending hour before fetching usage, process pending
  hours chronologically, and advance only after evaluation and delivery.
- P3: normalize whole-valued numeric timestamps without accepting invalid,
  fractional, non-finite, or misaligned values.
- Cover hour rollovers, API/data/delivery failures, catch-up progress, and
  invalid state. Document retry and retained-history behavior.

Validation: 2236 tests and 117 subtests passed, with 4 skips; lint,
formatting, types, hooks, dependency audit, and live dry-run passed.
The existing hourly schedule is unchanged.
@spalen0
spalen0 merged commit e2344d0 into main Nov 10, 2026
3 checks passed
@spalen0
spalen0 deleted the feat/tenderly-usage-spikes branch November 10, 2026 17:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant