Skip to content

Import files over 2 GiB by mounting them instead of copying - #89

Merged
wassgha merged 1 commit into
mainfrom
fix-large-file-import
Aug 29, 2026
Merged

Import files over 2 GiB by mounting them instead of copying#89
wassgha merged 1 commit into
mainfrom
fix-large-file-import

Conversation

@wassgha

@wassgha wassgha commented Aug 29, 2026

Copy link
Copy Markdown
Owner

Fixes #87.

Cause

ensureInput handed the file to @ffmpeg/util's fetchFile, which slurps the whole thing through FileReader.readAsArrayBuffer before ffmpeg.writeFile copies it into MEMFS. A Uint8Array cannot exceed 2 GiB in Chrome, so that read fails outright at exactly that size.

Measured in Chromium against slices of a single file:

slice readAsArrayBuffer
2040 MiB ok
2048 MiB FAIL Code=-1
2560 MiB FAIL Code=-1

That's the wall exactly: the reporter's 472 MB file worked, their 3–8 GB files didn't. It reads as Code=-1 because a modern DOMException has no legacy .code for fetchFile's template to interpolate, so it falls through to the || -1 — which is why the error says nothing useful.

Worth noting the size limit is not ffmpeg's 1 GiB wasm heap, which is what I'd have guessed: MEMFS stores file contents in a plain JS-heap Uint8Array (MEMFS.expandFileStorage), not in wasm memory. The 8h20m case below writes 1831 MB of PCM into MEMFS quite happily.

Fix

Mount the File through WORKERFS, which is already compiled into the core we ship (FS.filesystems = {"MEMFS":…,"WORKERFS":…}). It serves reads straight off the Blob — one slice plus a FileReaderSync per avio block — so the media is never materialised as a single ArrayBuffer, and seeks past 2 GiB work.

Copying stays the default below 256 MiB: it's ~1.7× faster end to end on a 300 MB file, and those are the sizes it already handled fine. Above that the input is mounted, falling back to copying if the mount is rejected (older core, etc.). Switching inputs now unmounts or deletes the previous one instead of leaving it resident.

Testing

Headless Chromium (COOP/COEP, crossOriginIsolated: true), driving the real lib/ffmpeg.ts via esbuild rather than a reimplementation, against synthesised WAVs:

case result
3.00 GiB / 76 min — pre-fix File could not be read! Code=-1 in 0.3s ✅ reproduces #87
3.00 GiB / 76 min — post-fix extracts all 4565.2s in 68s
2.46 GiB / 8h20m extracts 30000.0s, 1831 MB PCM
300 MB (mount) / 50 MB (copy) correct durations, non-silent audio
mount → copy → mount → copy, one instance all correct
exportAudio from mounted 3 GiB, ranges at 4000s/4500s valid RIFF/WAVE, exactly 40.0s out for 40s of ranges

The last two matter most: the first covers clearInput's unmount/remount, the second confirms the full import → edit → export round trip works with a read-only mounted input and that seeks past the 2 GiB byte offset resolve correctly. tsc --noEmit, eslint, test:timeline and test:i18n all clean.

Not covered

  • Tested with WAV, not H.264 MP4. No ffmpeg on this machine to synthesise multi-GB video, so I generated WAVs instead. WORKERFS is container-agnostic and llseek handles SEEK_END, so an MP4 with a trailing moov should be fine — but a real 3 GB video import is worth one manual check before release.
  • Not tested in Electron. The desktop app takes the same code path with a real File, so it should behave identically.
  • Very long media still has a ceiling, unchanged by this PR: extraction writes 16 kHz mono f32 PCM at ~230 MB/hour, so an 8-hour file needs ~1.8 GB of MEMFS plus the same again for the Float32Array copy. It worked here, but it's the next thing to give. Halving it with s16le would be a separate change.

🤖 Generated with Claude Code

Fixes #87: "File could not be read! Code=-1" on any file at or above 2 GiB.

The message comes from @ffmpeg/util's fetchFile, which slurps the whole File
through FileReader.readAsArrayBuffer before handing it to ffmpeg.writeFile. A
Uint8Array cannot exceed 2 GiB in Chrome, so that read fails outright at
exactly that size — measured in Chromium against slices of one file: 2040 MiB
reads, 2048 MiB does not. The reporter's 472 MB file worked and their 3-8 GB
files did not, which is the wall exactly. The error reads as "Code=-1" because
a modern DOMException has no legacy .code for fetchFile's template to
interpolate, so it falls through to the || -1.

Mount the File through WORKERFS instead, which is compiled into the core we
already ship. It serves reads straight off the Blob — one slice plus a
FileReaderSync per avio block — so the media is never materialised as a single
ArrayBuffer, and seeks past 2 GiB work.

Copying stays the default below 256 MiB: it is ~1.7x faster end to end on a
300 MB file, and the sizes it handles are the ones it already handled. Above
that the input is mounted, with a fall back to copying if the mount is
rejected. Switching inputs now unmounts or deletes the previous one rather
than leaving it resident.

Verified in headless Chromium against synthesised WAVs, driving the real
lib/ffmpeg.ts through esbuild rather than a replica:

- 3.00 GiB / 76 min: pre-fix "File could not be read! Code=-1" in 0.3s;
  post-fix extracts all 4565.2s of audio in 68s
- 2.46 GiB / 8h20m: extracts 30000.0s, 1831 MB of PCM
- 300 MB (mount) and 50 MB (copy): correct durations, non-silent audio
- mount -> copy -> mount -> copy in one ffmpeg instance: all correct
- exportAudio from a mounted 3 GiB input with keep ranges at 4000s/4500s
  (past the 2 GiB byte offset): valid RIFF/WAVE, exactly 40.0s out for
  40s of ranges

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
app.rescript Ready Ready Preview Aug 29, 2026 8:11pm

@wassgha
wassgha merged commit 473324f into main Aug 29, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

File could not be read! Code=-1

1 participant