Skip to content

Add chunked upload support to legacy SWORD routes - #70

Merged
kirkkwang merged 7 commits into
mainfrom
i540-chunk-uploads-legacy-support
May 8, 2026
Merged

kirkkwang merged 7 commits into
mainfrom
i540-chunk-uploads-legacy-support

Conversation

@ShanaLMoore

Copy link
Copy Markdown

Summary

Extends the chunked upload feature (V2) to also work on the legacy SWORD routes. The protocol is identical — In-Progress: true to initiate staging, PUT with Content-Range to send chunks, In-Progress: false to finalize — only the URLs differ.

Step V2 route Legacy route
Initiate staging POST /sword/v2/works/{WID}/file_sets POST /sword/collections/{CID}/works/{WID}/file_sets
Send chunks PUT /sword/v2/file_sets/{STAGING_ID} PUT /sword/collections/{CID}/works/{WID}/file_sets/{STAGING_ID}
Check status GET /sword/v2/file_sets/{STAGING_ID} GET /sword/collections/{CID}/works/{WID}/file_sets/{STAGING_ID}

Non-chunked POST/PUT behavior for works and file sets is unchanged on both legacy and V2.

What changed

  1. WillowSword::SwordError — new StandardError subclass wrapping WillowSword::Error. Required by ChunkedUploadHandler (was referenced but never defined). Also added missing error types: :chunk_sequence_error (416), :upload_not_found (404), :upload_incomplete (409).

  2. Shared staging status view — moved staging_status.xml.builder from v2/file_sets/ to shared/ so both legacy and V2 controllers render the same SWORD staging XML.

  3. collection_id in staging manifest — legacy routes are nested under /collections/{CID}/works/{WID}/, so the collection ID is stored in the manifest alongside work_id for use in response URL generation. The new collection_id: keyword arg defaults to nil, so V2 callers are unaffected.

  4. Legacy FileSetsController — includes ChunkedUploadHandler and adds chunking support to create, show, and update. The V2 controller inherits this include so the redundant include there is removed. Uses rescue_from SwordError (narrowly scoped) rather than the V2 HandleError concern to avoid changing existing legacy error handling. Renders legacy Atom feed XML on finalization, not HykuCrosswalk.

  5. with_manifest_lock fix — when append_chunk is called with a nonexistent upload ID, the staging directory doesn't exist, so File.open on the .lock file raised Errno::ENOENT. Now checks directory existence first and raises the expected SwordError(:upload_not_found).

The V2 controller completely overrides create/show/update, so base controller changes do not affect V2 behavior.

Kirk's review shortcut

12 of the 17 files are unchanged from your PR #59 review. The new work is 5 files:

  • app/controllers/willow_sword/file_sets_controller.rb — legacy chunking support (main change)
  • app/views/willow_sword/shared/staging_status.xml.builder — moved from v2/file_sets/, same content
  • app/controllers/concerns/willow_sword/chunked_upload_handler.rb — 2 small additions: collection_id: param + directory check in with_manifest_lock
  • lib/willow_sword/sword_error.rb — new 12-line file
  • spec/request/legacy/chunked_deposit_spec.rb — new legacy integration test

Ticket Number

Test Results

Tested all four scenarios on pittir-staging.hykucommons.org (2026-05-05):

# Test Route HTTP Codes Result
1 Non-chunked file set upload V2 201 (work), 201 (file set) PASS
2 Chunked file set upload (2 chunks) V2 201 (staging), 200 (chunk), 201 (finalize) PASS
3 Non-chunked file set upload Legacy 201 (work), 201 (file set) PASS
4 Chunked file set upload (2 chunks) Legacy 201 (staging), 200 (status), 200 (chunk), 201 (finalize) PASS

Expected Behavior

  • Default deposits unchanged: single POST to create a work or attach a file set, without In-Progress/Content-Range headers, behaves exactly as before on both legacy and V2.
  • Chunked file set uploads (legacy): same protocol as V2 — POST with In-Progress: true to initiate staging, PUT with Content-Range to send chunks, final PUT with In-Progress: false to finalize. Legacy URLs are nested (/collections/{CID}/works/{WID}/file_sets/{STAGING_ID}), and finalization renders the legacy Atom feed XML format.
  • Staging: shared tmp/network_files/... works across pods; flock on .lock does not error on EFS/NFS.
  • Cleanup: CleanupStaleUploadsJob and the rake task work the same as before.

ShanaLMoore and others added 6 commits May 5, 2026 12:00
Large file set payloads can now be uploaded in chunks without sending
the whole file in one HTTP request, staying under edge limits like
Cloudflare's ~100MB body cap.

The protocol uses standard SWORD semantics on existing file set URLs:
POST with In-Progress: true to initiate staging, PUT with
Content-Range to send chunks, final PUT with In-Progress: false to
assemble and create the Hyrax FileSet. Non-chunked uploads are
unchanged.

Staging uses tmp/network_files/willow_sword by default so multi-pod
Hyku deploys share chunk state. CleanupStaleUploadsJob and a rake
task handle expiry. Defines WillowSword::SwordError for structured
error handling from the chunked upload handler.
The same chunked upload protocol now works on legacy routes. The only
difference is the URL structure: legacy uses nested
/collections/{CID}/works/{WID}/file_sets/{STAGING_ID} while V2
uses flat /file_sets/{STAGING_ID}.

Includes ChunkedUploadHandler in the base FileSetsController and
adds staging detection to create/show/update. The V2 controller
inherits this include so the redundant include there is removed.
Uses rescue_from SwordError (narrowly scoped) to avoid changing
existing legacy error handling. Moves the staging status view to a
shared location so both controllers render the same SWORD XML. Adds
collection_id to the staging manifest for legacy response URL
generation. Fixes with_manifest_lock to raise SwordError instead of
ENOENT for unknown upload IDs.

The V2 controller completely overrides create/show/update, so
these base controller changes do not affect V2 behavior.
End-to-end request specs hitting legacy routes:
/collections/:cid/works/:wid/file_sets. Tests the full SWORD
chunked upload lifecycle: staging initiation, chunk upload,
finalization, and status checks.
Was seeing some file sets being missed the old way, this way seems to be
more accurate.
Allow Content-Range "bytes X-Y/*" per RFC 9110 §14.4 so clients can
stream chunks without a known complete-length up front.  Termination is
signaled by In-Progress: false on the final PUT: total_size is inferred
from bytes_received when "*", or must equal bytes_received when numeric.

Ref:
- https://www.rfc-editor.org/rfc/rfc9110.html#section-14.4-8
Pull staging/finalize/chunk-append flow up to the legacy file_sets
controller.  Break append_chunk into named helpers and split
validate_chunk! into four phase methods.  Add sword_error helper.
Method-level `ensure` where applicable.  initiate_staging takes
metadata_body directly, dropping the temp-dir hop.  Sweep orphan
.manifest.*.tmp older than expiry.
@kirkkwang
kirkkwang force-pushed the i540-chunk-uploads-legacy-support branch 2 times, most recently from c6ff7a3 to 8866db8 Compare May 6, 2026 21:30
Change chunk_sequence_error: 416 to 409.  RFC 9110 §15.5.17 reserves 416
for Range request-header failures (response-side partial-content), not
Content-Range request-header failures (request-side partial PUT).
tus.io mandates 409 Conflict for offset mismatches; adopting that.

Ref:
- https://www.rfc-editor.org/rfc/rfc9110#section-15.5.17
- https://tus.io/protocols/resumable-upload#patch

Update staging_status view: ws: namespace for extension fields,
atom:title/summary, expand sword:treatment to all states, omit nil
total_size/filename.

Drop dead returns in validate_staging_owner!.
@kirkkwang
kirkkwang force-pushed the i540-chunk-uploads-legacy-support branch from 8866db8 to 2bdd6b5 Compare May 7, 2026 00:03
@kirkkwang
kirkkwang merged commit e4add59 into main May 8, 2026
1 check passed
@kirkkwang
kirkkwang deleted the i540-chunk-uploads-legacy-support branch May 8, 2026 21:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants