Parallelize coherence-based bad channel detection - #4750
Open
JESUSROYETH wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The default
coherence+psdmethod first materialises 100 scaled chunks and then processes them in serial. On a 384-channel recording, profiling shows that the Welch and median computations dominate the call, while extracting the random traces takes a small part of the total time.This change sends the same random slices through
TimeSeriesChunkExecutor. The default stays atn_jobs=1, but the chunks are streamed instead of kept together in memory. A finaljob_kwargsparameter lets callers select a thread or process pool without changing positional arguments.I measured the default 100 × 0.3 s chunk path on an Intel Core i9-13900HX, with numerical thread pools pinned to one thread. The generated row uses the 30 s / 384-channel shape reported in the issue. The second row uses real 384-channel SpikeGLX traces from the public Noise4Sam fixture; the public file is short, it was repeated in memory to cover the same chunk path.
n_jobs=8, processes)The channel ids and labels matched exactly between both implementations in each run. On the serial real-trace path, three fresh-process peak RSS measurements gave a median of 1,688,604 KiB before (range 1,688,500-1,689,352) and 353,452 KiB after (range 352,984-354,896), around 79% less. This comes from processing one random slice at a time instead of retaining the scaled chunks.
Validation:
pytest -q src/spikeinterface/preprocessing/tests/test_detect_bad_channels.py: 11 passed.highpass_filterwrapperdetect_bad_channelsbuilds internally also gets exercised across a process boundary; it matches the serial result on the same data.mainrevision and pass with this patch.blackand the trailing-whitespace/end-of-file checks are clean.Fixes #2869.