Skip to content

fix(hw): discard impossible cgroup v2 CPU samples - #7103

Open
Alphaxiaoteng wants to merge 3 commits into
livekit:mainfrom
Alphaxiaoteng:fix/cgroup-v2-impossible-cpu-samples
Open

fix(hw): discard impossible cgroup v2 CPU samples#7103
Alphaxiaoteng wants to merge 3 commits into
livekit:mainfrom
Alphaxiaoteng:fix/cgroup-v2-impossible-cpu-samples

Conversation

@Alphaxiaoteng

@Alphaxiaoteng Alphaxiaoteng commented Sep 3, 2026

Copy link
Copy Markdown

Summary

Fixes #7102.

  • Prefer the process's cgroup CPU usage, falling back to the mounted root when its path is unavailable or escapes the mount.
  • Read effective cpu.max limits along that same cgroup's ancestor chain. A child with max can still inherit a restricted parent.
  • Use measured elapsed time and normalize against effective capacity, clamping to [0, 1].
  • Start the elapsed-time window after the first counter read. Cap effective capacity by known host CPUs and the Linux scheduler affinity mask (which already reflects cpuset restrictions), preserving NUM_CPUS override precedence and unavailable-affinity fallbacks.
  • Reject negative or host-capacity-impossible deltas. Quota-period/burst usage is not a physically impossible sample: applying a quota-rate ceiling to every short window incorrectly reported a saturated worker as idle.
  • Give the regression file its required pytest.mark.unit category, addressing the failed hosted collection gate.

Verification

  • Archived the previously submitted 099dcbb with only the new tests retained: 4 failed, 4 passed. The failures reproduce both full/fractional quota bursts, nested cgroup normalization, and escaping-path behavior.
  • The first follow-up 2e81a70, archived with only the first additional edge-case tests overlaid, produced 4 failed, 9 passed. These reproduce delayed first reads, quotas exceeding host capacity, tighter affinity, and unknown host capacity with known affinity.
  • Current CPU + worker configuration + connection tests, with normal imports, conftest, and repository --unit selection: 30 passed, 1 warning. This includes 19 CPU regression cases. The test fixture also emitted an unretrieved SystemExit(0) task diagnostic. Unchanged baseline worker tests passed 11 tests with the same synthetic short-JWT-key warning; the output is not claimed to be warning-free.
  • Repository-pinned Ruff formatting/lint passed, including the full repository checks (976 files); git diff --check passed. Focused Linux-target mypy passed for the changed module (--follow-imports=silent).
  • make check in the isolated environment passed formatting/lint but could not complete all-plugin type checks: optional plugin packages/typing dependencies are absent. Fresh hosted CI and tests passed for final commit 6f14da7: full Python 3.10/3.13 type checks, all three blockguard platforms, and 2457 passed / 5 skipped / 34 warnings in the unit-test job. This hosted result was verified from the actual job log.

Limits and disclosure

The tests use tiny local file/time mocks plus actual package imports and repository test configuration. I have not run the reporter's Xen guest or an overnight production experiment. CLA confirmation still belongs to the account owner.

The initial patch was made with Cursor. Codex performed the follow-up repairs and tests, with independent agent review and main-agent reruns. This updates the same PR rather than creating a duplicate.

Prefer the process cgroup cpu.stat, divide by measured elapsed time,
drop non-monotonic/oversized deltas, and clamp to [0, 1] like v1 so an
idle worker cannot flip to FULL on a torn root-cgroup read.
Fixes livekit#7102.
@Alphaxiaoteng
Alphaxiaoteng requested a review from a team as a code owner September 3, 2026 05:40
@CLAassistant

CLAassistant commented Sep 3, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CGroupV2CPUMonitor trusts a non-monotonic root-cgroup cpu.stat → worker flips to FULL on an idle host and the server refuses room jobs (lost calls)

2 participants