Skip to content

Stop limiter from opening every events collection each hour - #587

Open
Kuchizu wants to merge 4 commits into
masterfrom
fix/limiter-mongo-restarts
Open

Kuchizu wants to merge 4 commits into
masterfrom
fix/limiter-mongo-restarts

Conversation

@Kuchizu

@Kuchizu Kuchizu commented Sep 14, 2026

Copy link
Copy Markdown
Member

MongoDB restarts every hour at ~:01 while the limiter runs regular-workspaces-check: 12 restarts on 14.09, each a full 30-60 s
outage.

Cause: every project has its own events, repetitions and dailyEvents collections (~9k in total). In WiredTiger each collection and index is a separate
table, and reading one opens a data handle that stays open for ~10-15 min. Since the limiter queries all three collections for every project each
hour. On dailyEvents the planner also trial-runs all 3 groupingTimestamp indexes. That adds up to ~24k handles within a minute against a ~3k baseline, and
mongod dies during this surge.

Fix: dailyEvents is read with a $natural hint, which opens one table per project. These collections are tiny, and an index hint is not an option because
60 collections lack the index. events and repetitions are queried only if the boundary-day dailyEvents bucket is non-empty: grouper counts every event
there, so an empty bucket means no raw events that day. Counting results are unchanged.

Checked on prod: ~5.8k handles per run instead of ~24k, and counts on the 120 most active workspaces match the old code.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Non-atomic grouper writes can cause the new boundary-bucket gate to transiently undercount usage.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Optimizes limiter event counting to reduce MongoDB handle spikes.

Changes:

  • Uses $natural scans for dailyEvents.
  • Skips raw collections when the boundary bucket is empty.
  • Updates fixtures and adds optimization coverage.
File summaries
File Description
workers/limiter/src/dbHelper.ts Optimizes event-count queries.
workers/limiter/tests/dbHelper.test.ts Tests counting and skipped raw queries.
workers/limiter/tests/index.test.ts Adds daily-event fixtures.
Review details
  • Files reviewed: 3/3 changed files
  • Comments generated: 1
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread workers/limiter/src/dbHelper.ts Outdated
Comment thread workers/limiter/src/dbHelper.ts
neSpecc
neSpecc previously approved these changes Sep 16, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The unbounded validation rate can accidentally restore the database load spike this change intends to prevent.

Get a fresh assessment by requesting another Copilot review.

Review details
  • Files reviewed: 5/5 changed files
  • Comments generated: 1
  • Review effort level: Balanced

Comment thread workers/limiter/src/index.ts Outdated
const updatedWorkspaces: WorkspaceWithTariffPlan[] = [];

/** share of workspaces recounted with raw boundary day, 0 disables */
const validationRate = Number(process.env.LIMITER_COUNTER_VALIDATION_RATE ?? DEFAULT_COUNTER_VALIDATION_RATE);
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants