Read KIT data in cache-sized blocks - #14255
Conversation
Ahh yes I see how this could be dependent on the number of channels (and/or number of samples per stored block) for some data formats... if it makes sense, instead of this being a module-level thing, it could be computed in the |
The reader asked for up to 100 MB per block, then cast the whole block to
float64 and scaled it in place, so the working set was several hundred MB and
none of it stayed in cache.
raw.get_data(), interleaved cross-process medians:
136 MB file (161ch x 425000) 258.5 -> 145.6 ms 1.78x
A sweep over 32/16/8/4/2/1 MiB puts the optimum at 2 MiB (32 MiB is only 1.14x,
8 MiB 1.38x, 4 MiB 1.66x, 1 MiB 1.60x).
No shipped fixture regresses; two of them improve because they were already
being split differently:
ArtificalSignalData_Yokogawa_1khz.con 2.87 -> 2.26 ms 1.27x
ArtificalSignalData_RICOH_1khz.con 3.06 -> 2.77 ms 1.10x
Example_PQA160C_1001-export.con 0.99x
010409_Motor_task_coregist-export.con 0.98x
Output is bit-identical on every readable KIT fixture. The large file was
synthesised by tiling the raw-data section of data_berlin.con -- it is the last
section in the file -- and patching n_samples in the acquisition header.
Same treatment as mne-tools#14241 and mne-tools#14246; the constant differs because the optimum
tracks time points per block, so it moves with the channel count.
8eca0cc to
5615a65
Compare
|
Updated this to compute block lengths in time samples during reader initialization and store them in raw_extras. The same small change is included for BrainVision, Artemis123, Curry, EEGLAB, Eximia, FIL, NSx, Persyst, and Nihon Kohden; the shared reader now accepts max_block_samples. EGI is unchanged because its event-block calculation already occurs once during initialization. The affected reader suites pass locally. |
|
Any big speedups from those or did they end up being subtle? |
|
hey @larsoner,
|
|
Love it! Thanks @bruAristimunha |
github-project-automation
Bot
moved this from Ideas
to Done
in 2026 MNE-Python Maintainers Sprint
3 participants
What does this implement/fix?
The reader asked for up to 100 MB per block, then cast the whole block to
float64 and scaled it in place (
.astype(float)thenblock *= conv_factor), sothe working set ran to several hundred MB and none of it stayed in cache.
Same treatment as #14241 and #14246.
Numbers
Median of 5 interleaved process pairs against a pristine
mainworktree:A sweep puts the optimum at 2 MiB — 32 MiB is only 1.14x, 8 MiB 1.38x,
4 MiB 1.66x, 1 MiB 1.60x.
No shipped fixture regresses, and two improve because they were already being
split differently:
ArtificalSignalData_Yokogawa_1khz.conArtificalSignalData_RICOH_1khz.conExample_PQA160C_1001-export.con010409_Motor_task_coregist-export.conCorrectness
mainon every readable KIT fixture (np.array_equal).pytest mne/io/kit: 15 passed.Caveats
The constant differs from the other readers (2 MiB here, 8 MiB BrainVision,
2–16 MiB in #14246) because the optimum tracks time points per block, so it
moves with the channel count. As on #14246, I think a
max_block_samples=parameter would express this better than a per-reader byte budget, and I am
happy to go that way instead.
No fixture in the testing dataset is large enough to show the effect, so CI
cannot demonstrate it. The large file was synthesised by tiling the raw-data
section of
data_berlin.con— verified to be the last section in the file — andpatching
n_samplesin the acquisition header.Additional information
AI disclosure: I directed the work and reviewed and tested every change; Claude
Code (Claude Opus 5) ran the block-size sweep and the A/B measurements and made
the edits under my direction.