Skip to content

Forward stream options to S3 download operation - #16

Merged
tfwright merged 1 commit into
mainfrom
feat/s3-stream-opts
Aug 25, 2026
Merged

Forward stream options to S3 download operation#16
tfwright merged 1 commit into
mainfrom
feat/s3-stream-opts

Conversation

@tfwright

@tfwright tfwright commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Problem

Capsule.Storages.S3.stream!/2 accepts options, but only uses them to resolve the bucket:

def stream!(id, opts \\ []) do
  opts
  |> config(:bucket)
  |> Client.download_file(id, :memory)
  |> ex_aws_module().stream!()
end

Nothing reaches download_file/4, so op.opts is always [] and ExAws falls back to its defaults — 1 MB chunks at max_concurrency: 8.

That matters for a caller who only wants the beginning of an object. ExAws.Operation.stream!/2 for S3.Download wraps the chunk stream in Task.async_stream(max_concurrency: 8), so even Stream.take(1) can spin up eight concurrent ranged GETs and pull ~8 MB before the stream halts. There is currently no way to ask for less.

The concrete use case that prompted this: checking that an object is readable before persisting a locator that references it. That wants a HEAD plus the smallest possible read, not several megabytes.

Change

Pass the caller's options straight through:

|> Client.download_file(id, :memory, opts)

ExAws.S3.Download only reads :chunk_size, :max_concurrency and :timeout from op.opts and ignores anything else, so forwarding the whole list is safe — a :bucket key that a caller passed for config/2 is simply inert. This also keeps stream!/2 consistent with put/2 in the same module, which already reads opts[:key] and opts[:s3_options] from the same flat list.

Callers can now do:

S3.stream!(id, chunk_size: 1, max_concurrency: 1)

which is a HEAD followed by a single one-byte ranged GET.

Note this differs from Capsule.Storages.Disk.stream!/2, which nests under :stream_opts. Disk has to, because it forwards to File.stream!/2 and an unrecognised key there would raise; ExAws tolerates extra keys, so the flat form is both simpler and consistent with the rest of this adapter.

Compatibility

Backwards compatible. Existing callers either pass no options, or pass :bucket for config/2 — both behave exactly as before, since ExAws applies the same defaults when the relevant keys are absent.

Testing

Adds a test asserting the options reach the %ExAws.S3.Download{} operation. Without the change it fails with right: [[]], showing the options being dropped.

Full suite: 25 tests, 0 failures. mix format --check-formatted clean.

Note on the behaviour docs

Capsule.Storage already declares @callback stream!(locator_id, [option]), so the option contract exists upstream — the S3 adapter was simply discarding it. But the behaviour doesn't say anything about which keys an adapter should accept, which is how S3 and Disk came to disagree. Might be worth a follow-up in capsule to write that down so third-party adapters have something to follow.

stream!/2 accepted options but only used them to resolve the bucket, so
ExAws applied its defaults of 1 MB chunks at a concurrency of 8. A
caller wanting only the first bytes of an object could not ask for
less, and would fetch up to 8 MB across eight concurrent requests.

Options are now passed straight to download_file, which ignores keys it
does not recognise — consistent with how put/2 reads opts from the same
flat list.
@tfwright
tfwright force-pushed the feat/s3-stream-opts branch from 729e73e to 1c06822 Compare August 25, 2026 22:17
@tfwright
tfwright merged commit a866770 into main Aug 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant