Forward stream options to S3 download operation - #16
Merged
Conversation
stream!/2 accepted options but only used them to resolve the bucket, so ExAws applied its defaults of 1 MB chunks at a concurrency of 8. A caller wanting only the first bytes of an object could not ask for less, and would fetch up to 8 MB across eight concurrent requests. Options are now passed straight to download_file, which ignores keys it does not recognise — consistent with how put/2 reads opts from the same flat list.
tfwright
force-pushed
the
feat/s3-stream-opts
branch
from
August 25, 2026 22:17
729e73e to
1c06822
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Capsule.Storages.S3.stream!/2accepts options, but only uses them to resolve the bucket:Nothing reaches
download_file/4, soop.optsis always[]and ExAws falls back to its defaults — 1 MB chunks atmax_concurrency: 8.That matters for a caller who only wants the beginning of an object.
ExAws.Operation.stream!/2forS3.Downloadwraps the chunk stream inTask.async_stream(max_concurrency: 8), so evenStream.take(1)can spin up eight concurrent ranged GETs and pull ~8 MB before the stream halts. There is currently no way to ask for less.The concrete use case that prompted this: checking that an object is readable before persisting a locator that references it. That wants a HEAD plus the smallest possible read, not several megabytes.
Change
Pass the caller's options straight through:
ExAws.S3.Downloadonly reads:chunk_size,:max_concurrencyand:timeoutfromop.optsand ignores anything else, so forwarding the whole list is safe — a:bucketkey that a caller passed forconfig/2is simply inert. This also keepsstream!/2consistent withput/2in the same module, which already readsopts[:key]andopts[:s3_options]from the same flat list.Callers can now do:
which is a HEAD followed by a single one-byte ranged GET.
Note this differs from
Capsule.Storages.Disk.stream!/2, which nests under:stream_opts. Disk has to, because it forwards toFile.stream!/2and an unrecognised key there would raise; ExAws tolerates extra keys, so the flat form is both simpler and consistent with the rest of this adapter.Compatibility
Backwards compatible. Existing callers either pass no options, or pass
:bucketforconfig/2— both behave exactly as before, since ExAws applies the same defaults when the relevant keys are absent.Testing
Adds a test asserting the options reach the
%ExAws.S3.Download{}operation. Without the change it fails withright: [[]], showing the options being dropped.Full suite: 25 tests, 0 failures.
mix format --check-formattedclean.Note on the behaviour docs
Capsule.Storagealready declares@callback stream!(locator_id, [option]), so the option contract exists upstream — the S3 adapter was simply discarding it. But the behaviour doesn't say anything about which keys an adapter should accept, which is how S3 and Disk came to disagree. Might be worth a follow-up incapsuleto write that down so third-party adapters have something to follow.