Import files over 2 GiB by mounting them instead of copying - #89
Merged
Conversation
Fixes #87: "File could not be read! Code=-1" on any file at or above 2 GiB. The message comes from @ffmpeg/util's fetchFile, which slurps the whole File through FileReader.readAsArrayBuffer before handing it to ffmpeg.writeFile. A Uint8Array cannot exceed 2 GiB in Chrome, so that read fails outright at exactly that size — measured in Chromium against slices of one file: 2040 MiB reads, 2048 MiB does not. The reporter's 472 MB file worked and their 3-8 GB files did not, which is the wall exactly. The error reads as "Code=-1" because a modern DOMException has no legacy .code for fetchFile's template to interpolate, so it falls through to the || -1. Mount the File through WORKERFS instead, which is compiled into the core we already ship. It serves reads straight off the Blob — one slice plus a FileReaderSync per avio block — so the media is never materialised as a single ArrayBuffer, and seeks past 2 GiB work. Copying stays the default below 256 MiB: it is ~1.7x faster end to end on a 300 MB file, and the sizes it handles are the ones it already handled. Above that the input is mounted, with a fall back to copying if the mount is rejected. Switching inputs now unmounts or deletes the previous one rather than leaving it resident. Verified in headless Chromium against synthesised WAVs, driving the real lib/ffmpeg.ts through esbuild rather than a replica: - 3.00 GiB / 76 min: pre-fix "File could not be read! Code=-1" in 0.3s; post-fix extracts all 4565.2s of audio in 68s - 2.46 GiB / 8h20m: extracts 30000.0s, 1831 MB of PCM - 300 MB (mount) and 50 MB (copy): correct durations, non-silent audio - mount -> copy -> mount -> copy in one ffmpeg instance: all correct - exportAudio from a mounted 3 GiB input with keep ranges at 4000s/4500s (past the 2 GiB byte offset): valid RIFF/WAVE, exactly 40.0s out for 40s of ranges Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #87.
Cause
ensureInputhanded the file to@ffmpeg/util'sfetchFile, which slurps the whole thing throughFileReader.readAsArrayBufferbeforeffmpeg.writeFilecopies it into MEMFS. AUint8Arraycannot exceed 2 GiB in Chrome, so that read fails outright at exactly that size.Measured in Chromium against slices of a single file:
readAsArrayBufferCode=-1Code=-1That's the wall exactly: the reporter's 472 MB file worked, their 3–8 GB files didn't. It reads as
Code=-1because a modernDOMExceptionhas no legacy.codeforfetchFile's template to interpolate, so it falls through to the|| -1— which is why the error says nothing useful.Worth noting the size limit is not ffmpeg's 1 GiB wasm heap, which is what I'd have guessed: MEMFS stores file contents in a plain JS-heap
Uint8Array(MEMFS.expandFileStorage), not in wasm memory. The 8h20m case below writes 1831 MB of PCM into MEMFS quite happily.Fix
Mount the
Filethrough WORKERFS, which is already compiled into the core we ship (FS.filesystems = {"MEMFS":…,"WORKERFS":…}). It serves reads straight off theBlob— onesliceplus aFileReaderSyncper avio block — so the media is never materialised as a single ArrayBuffer, and seeks past 2 GiB work.Copying stays the default below 256 MiB: it's ~1.7× faster end to end on a 300 MB file, and those are the sizes it already handled fine. Above that the input is mounted, falling back to copying if the mount is rejected (older core, etc.). Switching inputs now unmounts or deletes the previous one instead of leaving it resident.
Testing
Headless Chromium (COOP/COEP,
crossOriginIsolated: true), driving the reallib/ffmpeg.tsvia esbuild rather than a reimplementation, against synthesised WAVs:File could not be read! Code=-1in 0.3s ✅ reproduces #87exportAudiofrom mounted 3 GiB, ranges at 4000s/4500sThe last two matter most: the first covers
clearInput's unmount/remount, the second confirms the full import → edit → export round trip works with a read-only mounted input and that seeks past the 2 GiB byte offset resolve correctly.tsc --noEmit,eslint,test:timelineandtest:i18nall clean.Not covered
llseekhandlesSEEK_END, so an MP4 with a trailingmoovshould be fine — but a real 3 GB video import is worth one manual check before release.File, so it should behave identically.Float32Arraycopy. It worked here, but it's the next thing to give. Halving it withs16lewould be a separate change.🤖 Generated with Claude Code