Skip to content

Decode Transfer-Encoding: gzip response bodies incrementally - #96

Open
oalders wants to merge 1 commit into
masterfrom
transfer-encoding
Open

Decode Transfer-Encoding: gzip response bodies incrementally#96
oalders wants to merge 1 commit into
masterfrom
transfer-encoding

Conversation

@oalders

@oalders oalders commented Aug 14, 2026

Copy link
Copy Markdown
Member

Summary

Net::HTTP::Methods::read_entity_body handled a Transfer-Encoding: gzip
response body by accumulating every de-chunked byte in a closure-local buffer
and gunzipping the whole thing in a single pass once the terminating
zero-length chunk arrived, then returning a scalar reference to the result.

That has two problems:

  1. The read cap is never consulted while the body streams. Each read
    returned "" (surfaced upstream as -1, "read again"), so the buffer grew
    with no upper bound until the stream ended. A caller's read/size cap — e.g.
    LWP::UserAgent's max_size, enforced in LWP::Protocol::collect — only
    sees the body after it has already been fully buffered and decompressed,
    which defeats the point of the cap. Compression ratios stack on top of this.

  2. Size accounting is wrong. Because the closure returned a reference,
    length() measured the stringified reference (~19 bytes) rather than the
    decoded payload.

Fix

Decode incrementally with Compress::Raw::Zlib::Inflate->new(WindowBits => WANT_GZIP),
mirroring the deflate branch immediately above it. Each read now returns a
plain string of just-decoded bytes, so:

  • length() is truthful,
  • a caller's read cap applies on every read, and
  • peak memory tracks the read size instead of the whole response size.

Tests

  • New t/te-gzip.t builds a gzip, chunked response split across many small
    chunks and asserts correct decoding, truthful per-read lengths, and
    incremental delivery. It fails against the previous code (returns a
    SCALAR(0x…)) and passes with this change.
  • Existing t/http.t and t/socket-class.t continue to pass.

🤖 Generated with Claude Code

@oalders
oalders force-pushed the transfer-encoding branch from f7868ac to 5d408b3 Compare August 14, 2026 00:59
The gzip transfer-encoding handler buffered every de-chunked byte and
gunzipped the whole body in one pass at end of stream, returning a scalar
reference. This meant per-read returns were "" (surfaced as -1), so a
caller's read cap was never consulted while the body streamed and the
buffer grew without bound; the returned reference also made length()
report the stringified ref rather than the payload.

Decode incrementally with Compress::Raw::Zlib::Inflate (WANT_GZIP),
mirroring the deflate branch. Each read now returns a plain string of
just-decoded bytes, so length() is truthful, a caller's cap applies on
every read, and peak memory tracks the read size rather than the
response size.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 14, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 68.09%. Comparing base (dc1a5d2) to head (3cd7927).

Files with missing lines Patch % Lines
lib/Net/HTTP/Methods.pm 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master      #96      +/-   ##
==========================================
+ Coverage   62.43%   68.09%   +5.65%     
==========================================
  Files           4        4              
  Lines         402      398       -4     
  Branches      108      107       -1     
==========================================
+ Hits          251      271      +20     
+ Misses        104       80      -24     
  Partials       47       47              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@oalders
oalders force-pushed the transfer-encoding branch from 5d408b3 to 3cd7927 Compare August 14, 2026 01:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants