Add chunked upload support to legacy SWORD routes - #70
Merged
Merged
Conversation
Large file set payloads can now be uploaded in chunks without sending the whole file in one HTTP request, staying under edge limits like Cloudflare's ~100MB body cap. The protocol uses standard SWORD semantics on existing file set URLs: POST with In-Progress: true to initiate staging, PUT with Content-Range to send chunks, final PUT with In-Progress: false to assemble and create the Hyrax FileSet. Non-chunked uploads are unchanged. Staging uses tmp/network_files/willow_sword by default so multi-pod Hyku deploys share chunk state. CleanupStaleUploadsJob and a rake task handle expiry. Defines WillowSword::SwordError for structured error handling from the chunked upload handler.
The same chunked upload protocol now works on legacy routes. The only
difference is the URL structure: legacy uses nested
/collections/{CID}/works/{WID}/file_sets/{STAGING_ID} while V2
uses flat /file_sets/{STAGING_ID}.
Includes ChunkedUploadHandler in the base FileSetsController and
adds staging detection to create/show/update. The V2 controller
inherits this include so the redundant include there is removed.
Uses rescue_from SwordError (narrowly scoped) to avoid changing
existing legacy error handling. Moves the staging status view to a
shared location so both controllers render the same SWORD XML. Adds
collection_id to the staging manifest for legacy response URL
generation. Fixes with_manifest_lock to raise SwordError instead of
ENOENT for unknown upload IDs.
The V2 controller completely overrides create/show/update, so
these base controller changes do not affect V2 behavior.
End-to-end request specs hitting legacy routes: /collections/:cid/works/:wid/file_sets. Tests the full SWORD chunked upload lifecycle: staging initiation, chunk upload, finalization, and status checks.
Was seeing some file sets being missed the old way, this way seems to be more accurate.
Allow Content-Range "bytes X-Y/*" per RFC 9110 §14.4 so clients can stream chunks without a known complete-length up front. Termination is signaled by In-Progress: false on the final PUT: total_size is inferred from bytes_received when "*", or must equal bytes_received when numeric. Ref: - https://www.rfc-editor.org/rfc/rfc9110.html#section-14.4-8
Pull staging/finalize/chunk-append flow up to the legacy file_sets controller. Break append_chunk into named helpers and split validate_chunk! into four phase methods. Add sword_error helper. Method-level `ensure` where applicable. initiate_staging takes metadata_body directly, dropping the temp-dir hop. Sweep orphan .manifest.*.tmp older than expiry.
kirkkwang
force-pushed
the
i540-chunk-uploads-legacy-support
branch
2 times, most recently
from
May 6, 2026 21:30
c6ff7a3 to
8866db8
Compare
Change chunk_sequence_error: 416 to 409. RFC 9110 §15.5.17 reserves 416 for Range request-header failures (response-side partial-content), not Content-Range request-header failures (request-side partial PUT). tus.io mandates 409 Conflict for offset mismatches; adopting that. Ref: - https://www.rfc-editor.org/rfc/rfc9110#section-15.5.17 - https://tus.io/protocols/resumable-upload#patch Update staging_status view: ws: namespace for extension fields, atom:title/summary, expand sword:treatment to all states, omit nil total_size/filename. Drop dead returns in validate_staging_owner!.
kirkkwang
force-pushed
the
i540-chunk-uploads-legacy-support
branch
from
May 7, 2026 00:03
8866db8 to
2bdd6b5
Compare
kirkkwang
approved these changes
May 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Extends the chunked upload feature (V2) to also work on the legacy SWORD routes. The protocol is identical —
In-Progress: trueto initiate staging,PUTwithContent-Rangeto send chunks,In-Progress: falseto finalize — only the URLs differ.POST /sword/v2/works/{WID}/file_setsPOST /sword/collections/{CID}/works/{WID}/file_setsPUT /sword/v2/file_sets/{STAGING_ID}PUT /sword/collections/{CID}/works/{WID}/file_sets/{STAGING_ID}GET /sword/v2/file_sets/{STAGING_ID}GET /sword/collections/{CID}/works/{WID}/file_sets/{STAGING_ID}Non-chunked
POST/PUTbehavior for works and file sets is unchanged on both legacy and V2.What changed
WillowSword::SwordError— newStandardErrorsubclass wrappingWillowSword::Error. Required byChunkedUploadHandler(was referenced but never defined). Also added missing error types::chunk_sequence_error(416),:upload_not_found(404),:upload_incomplete(409).Shared staging status view — moved
staging_status.xml.builderfromv2/file_sets/toshared/so both legacy and V2 controllers render the same SWORD staging XML.collection_idin staging manifest — legacy routes are nested under/collections/{CID}/works/{WID}/, so the collection ID is stored in the manifest alongsidework_idfor use in response URL generation. The newcollection_id:keyword arg defaults tonil, so V2 callers are unaffected.Legacy
FileSetsController— includesChunkedUploadHandlerand adds chunking support tocreate,show, andupdate. The V2 controller inherits this include so the redundant include there is removed. Usesrescue_from SwordError(narrowly scoped) rather than the V2HandleErrorconcern to avoid changing existing legacy error handling. Renders legacy Atom feed XML on finalization, not HykuCrosswalk.with_manifest_lockfix — whenappend_chunkis called with a nonexistent upload ID, the staging directory doesn't exist, soFile.openon the.lockfile raisedErrno::ENOENT. Now checks directory existence first and raises the expectedSwordError(:upload_not_found).The V2 controller completely overrides
create/show/update, so base controller changes do not affect V2 behavior.Kirk's review shortcut
12 of the 17 files are unchanged from your PR #59 review. The new work is 5 files:
app/controllers/willow_sword/file_sets_controller.rb— legacy chunking support (main change)app/views/willow_sword/shared/staging_status.xml.builder— moved fromv2/file_sets/, same contentapp/controllers/concerns/willow_sword/chunked_upload_handler.rb— 2 small additions:collection_id:param + directory check inwith_manifest_locklib/willow_sword/sword_error.rb— new 12-line filespec/request/legacy/chunked_deposit_spec.rb— new legacy integration testTicket Number
Test Results
Tested all four scenarios on
pittir-staging.hykucommons.org(2026-05-05):Expected Behavior
POSTto create a work or attach a file set, withoutIn-Progress/Content-Rangeheaders, behaves exactly as before on both legacy and V2.POSTwithIn-Progress: trueto initiate staging,PUTwithContent-Rangeto send chunks, finalPUTwithIn-Progress: falseto finalize. Legacy URLs are nested (/collections/{CID}/works/{WID}/file_sets/{STAGING_ID}), and finalization renders the legacy Atom feed XML format.tmp/network_files/...works across pods;flockon.lockdoes not error on EFS/NFS.CleanupStaleUploadsJoband the rake task work the same as before.