Read NeuroExplorer files above 2 GB and fix spike waveform offsets - #1898
Conversation
Two defects in neuroexplorerrawio, both returning wrong data without raising. Waveform data starts after the variable's timestamps, which the specification stores as 4 bytes each. _get_spike_raw_waveforms skipped only n * 2 bytes, so it began reading halfway through the timestamp array. On the gin file File_neuroexplorer_2.nex, sig01i_wf started [-26086, 22, -24757, 22] where the file holds [-60, -13, 37, 138]. DataOffset is declared signed in the specification, but NeuroExplorer keeps writing past 2 GB and stores the low 32 bits, so a variable beyond that point reads back negative and the reader indexes its memmap from the wrong end of the file. Reading the field unsigned recovers the true offset exactly for any file below 4 GB; above that the information is genuinely lost and the file has to be re-exported as .nex5, which uses 64-bit offsets. Verified on a 2.5 GB recording with 196 variables, 69 of them past the boundary: every array then lands where the headers say, and the first samples of a continuous variable read -208, -241, -282 instead of 193, 215, 224. The counts come out of the header as numpy int32, so mixing them with a python int in the offset arithmetic raises OverflowError under numpy 2. Every value entering that arithmetic is now a python int.
zm711
left a comment
zm711
left a comment
There was a problem hiding this comment.
Looks good to me. Just a light request for a couple comments for ease in case I have to review this in the future. Not a blocker for merging, though.
| # offset2 = entity_header['offset'] + n*4 | ||
| # fragment_starts = self._memmap[offset2:offset2+n*4].view('int32') | ||
| offset3 = entity_header["offset"] + n * 4 + n * 4 | ||
| offset3 = int(entity_header["offset"]) + n * 4 + n * 4 |
There was a problem hiding this comment.
could we get a comment here for the offset logic. Maybe more for my benefit. whenever I have to go back and read one of these readers I always have to re-teach myself why the values are the way they are. A quick comment about the 4*n would be super useful.
There was a problem hiding this comment.
Also why offset and offset3. What is the difference (if you can fit that in a comment I mean).
There was a problem hiding this comment.
I used variables to try to make this more clear, check out the last discussion.
zm711
left a comment
zm711
left a comment
There was a problem hiding this comment.
This looks good to me. Thanks for the better variable names.
2 participants
Rodrigo Paz from the Nelson lab can't convert their NeuroExplorer sessions as most of their files are above 2 GB.
This PR reads
DataOffsetas unsigned and that recovers the true offset for any file below 4 GB. Above that the low bits are really gone and the file has to be re-exported as.nex5, which uses 64 bit offsets. I also fixed the spike waveforms which were read fromoffset + n * 2when the timestamps that precede them are 4 bytes each.I tested both on a 2.5 GB recording of theirs with 196 variables, 69 of them past the boundary. Every array lands where the headers say and the first samples of
AD18read-208, -241, -282instead of193, 215, 224.