Reuse the MEF session across reads - #14254
Merged
Merged
Conversation
_read_segment_file built a fresh MefSession on every call. Parsing the session metadata takes ~38 ms on the 194-channel testing recording and does not depend on the requested range, so a short window cost as much as reading the whole file: get_data(start=0, stop=100, picks=[one channel]) was 39.0 ms, of which 38.0 ms was metadata and 0.02 ms was data. Caching it in _raw_extras, which _ReadSegmentFileProtector already exposes: full read (194ch x 98000) 490.5 -> 451.5 ms 1.1x 1000 samples, all channels 47.2 -> 7.2 ms 6.6x 1000 samples, 4 channels 39.7 -> 0.19 ms 206x 100 samples, 1 channel 39.0 -> 0.07 ms 590x 100 epochs of 2 s 4895.6 -> 914.5 ms 5.4x The epoching row is the one that matters in practice -- every windowed read paid the full metadata parse, so epoching, plotting and scrolling were all dominated by it. MefSession and RawMef are both picklable, so caching in _raw_extras keeps raw.copy(), copy.deepcopy() and pickling (n_jobs) working; all four are checked to return bit-identical data, as are crop and preload=True. Not fixed here: the remaining 451 ms of a full read is pymef decoding channel by channel. pymef does not release the GIL -- reading the channels across 4-14 threads measured 0.99x -- so that floor cannot be moved from MNE.
bruAristimunha
requested review from
agramfort,
drammock and
larsoner
as code owners
August 29, 2026 20:17
larsoner
reviewed
Aug 30, 2026
Contributor
Author
|
very very good suggestion @larsoner :) |
github-project-automation
Bot
moved this from Ideas
to Done
in 2026 MNE-Python Maintainers Sprint
larsoner
added a commit
to larsoner/mne-python
that referenced
this pull request
Sep 2, 2026
* upstream/main: (22 commits) Avoid copying all epochs data in GetEpochsMixin._getitem (mne-tools#14262) Add Report.save(only_if_changed=True) (mne-tools#14261) Add jamica to related software [ci skip] (mne-tools#14260) Remove debugging cruft (mne-tools#14259) ENH: add Raw annotation span conversion (mne-tools#14240) Interactive dipole fitting: add STC mesh controls (mne-tools#14256) Document code principles in AGENTS.md (mne-tools#14239) MAINT: Update dependency specifiers (mne-tools#14257) [dependabot]: Bump the actions group with 2 updates (mne-tools#14258) ENH: Add JAMICA as an ICA method (mne-tools#14247) Reuse the MEF session across reads (mne-tools#14254) Read KIT data in cache-sized blocks (mne-tools#14255) Read EGI simple-binary event channels in blocks (mne-tools#14250) Decode Persyst and Nihon Kohden data in cache-sized blocks (mne-tools#14251) Normalize byte order before calibrating strided integer buffers (mne-tools#14252) Speed up EDF and BDF reading (mne-tools#14237) ENH: Add Forward-based projection reconstruction (mne-tools#14235) Read Artemis123, Curry, EEGLAB, Eximia, FIL and NSx in cache-sized blocks (mne-tools#14246) Remove rotating dipoles capability from interactive dipole fitting GUI (mne-tools#14243) Read BrainVision data in cache-sized blocks [ci skip] (mne-tools#14241) ...
larsoner
added a commit
to natinew77-creator/mne-python
that referenced
this pull request
Sep 3, 2026
* upstream/main: (37 commits) Speed up evoked time plotting (mne-tools#14249) Persist the numba JIT cache, sysmon coverage, refleak freeze (mne-tools#14265) Fix interpolate_to spline target positions (mne-tools#14266) CI reduce package builds on pull requests (mne-tools#14264) Avoid copying all epochs data in GetEpochsMixin._getitem (mne-tools#14262) Add Report.save(only_if_changed=True) (mne-tools#14261) Add jamica to related software [ci skip] (mne-tools#14260) Remove debugging cruft (mne-tools#14259) ENH: add Raw annotation span conversion (mne-tools#14240) Interactive dipole fitting: add STC mesh controls (mne-tools#14256) Document code principles in AGENTS.md (mne-tools#14239) MAINT: Update dependency specifiers (mne-tools#14257) [dependabot]: Bump the actions group with 2 updates (mne-tools#14258) ENH: Add JAMICA as an ICA method (mne-tools#14247) Reuse the MEF session across reads (mne-tools#14254) Read KIT data in cache-sized blocks (mne-tools#14255) Read EGI simple-binary event channels in blocks (mne-tools#14250) Decode Persyst and Nihon Kohden data in cache-sized blocks (mne-tools#14251) Normalize byte order before calibrating strided integer buffers (mne-tools#14252) Speed up EDF and BDF reading (mne-tools#14237) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this implement/fix?
_read_segment_filebuilt a freshMefSessionon every call. Parsing thesession metadata takes ~38 ms on the 194-channel testing recording and does not
depend on the requested range, so a short window cost as much as reading the
whole file:
This caches the session in
_raw_extras, which_ReadSegmentFileProtectoralready exposes to readers.
Numbers
Median of 3 interleaved process pairs against a pristine
mainworktree:The epoching row is the one that matters in practice — every windowed read paid
the full metadata parse, so epoching, plotting and scrolling were all dominated
by it.
Safety
MefSessionandRawMefare both picklable, so caching in_raw_extrasdoesnot break
n_jobs. Checked explicitly, all returning bit-identical data:raw.copy(),copy.deepcopy(raw),pickleround-tripcrop()after a cached read,preload=True, repeated readspytest mne/io/mef mne/io/tests: 104 passedWhat this does not fix
The remaining 451 ms of a full read is pymef decoding channel by channel
(
read_mef_ts_data, 194 calls). pymef does not release the GIL — I measuredreading the channels across 4, 8 and 14 threads at 0.99x, and with per-thread
sessions it gets worse — so that floor cannot be moved from MNE.
Additional information
AI disclosure: I directed the work and reviewed and tested every change; Claude
Code (Claude Opus 5) profiled the reader, measured the GIL behaviour and the
pickling edge cases, and made the edits under my direction.