fix: lock CPU power history swap - #1335
davidberenstein1957 wants to merge 3 commits into
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1335 +/- ##
==========================================
+ Coverage 91.70% 91.74% +0.04%
==========================================
Files 49 49
Lines 5157 5160 +3
==========================================
+ Hits 4729 4734 +5
+ Misses 428 426 -2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d70b9ae to
ed2b386
Compare
00f2a1e to
b813d76
Compare
The 1 Hz monitor scheduler thread appends to `CPU._power_history` while the measurement scheduler thread drains it. The read-then-rebind in `total_power()` is not atomic, so any sample appended between the list comprehension and the rebinding was written to the discarded list and lost, biasing the reported `cpu_power`. Take the swap under a lock, held only for the O(1) rebinding so a slow `_get_power_from_cpus()` backend never blocks the monitor thread. Also drop the unreachable empty-history branch: a sample is always appended before the average. The test asserts the drain invariant rather than the drain shape, so it does not pin the implementation. Closes #1315 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
b813d76 to
f72f432
Compare
Verdict: ✅ Approve with nitsThe race is real. The monitor scheduler can append to the old list between the list comprehension and the
56 tests pass on this PR merged with master. Nits:
|
…window Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
Made the changes in 44ef635: merged master, added a test that a post-swap sample lands in the next window. Not done: updating the description's test paragraph (my PR-edit was blocked; the diff's tests are the lock-blocking one plus the new next-window one, no list subclass). |
Description
Fixes a lost-update race on
CPU._power_history. The 1 Hz monitor scheduler thread (CPU.monitor_power) appends to it, and themeasure_power_secsscheduler thread (CPU.total_power) drains it. The read-then-rebind intotal_powerwas not atomic, so a sample appended between the list comprehension andself._power_history = []was dropped.CPUnow has athreading.Lock(_power_history_lock), held only for the O(1) list swap._get_power_from_cpus()still runs outside the lock in both paths, so a slow backend (for example theIntelPowerGadgetsubprocess) never blocks the monitor thread. The unreachableif not power_history_in_W:branch is removed, since a sample is always appended before the average is taken.Related Issue
Fixes #1315
Motivation and Context
In
cpu_loadmode the reportedcpu_poweris the mean of the buffered samples. The samples lost in the unlocked swap window are the ones taken while a slow measurement is in flight, so the error is small but systematic and silent.How Has This Been Tested?
Two tests in
tests/test_cpu_load.py::TestCPULoad:test_cpu_total_power_keeps_samples_added_while_draining: holdscpu._power_history_lockwhilemonitor_power()runs on another thread, checks the monitor waits instead of appending, and that the sample is in the history once the lock is released.test_cpu_sample_appended_right_after_the_swap_goes_to_next_window: wraps the real lock so a 100 Wmonitor_power()sample fires right after the swap, and checks the firsttotal_power()averages the old window (5.5 W) and the second picks up the 100 W sample (50.5 W).CI is green on this branch.
Screenshots (if appropriate):
N/A
Types of changes
AI Usage Disclosure
Checklist: