Serve data links as an S3 bucket so they browse in Neuroglancer - #436
Open
krokicki wants to merge 1 commit into
Open
Serve data links as an S3 bucket so they browse in Neuroglancer#436krokicki wants to merge 1 commit into
krokicki wants to merge 1 commit into
Conversation
Neuroglancer shows a folder tree over plain https by reading the first
path segment of a URL as an S3 bucket and asking that bucket root for
everything under the rest as a prefix:
GET /files/?list-type=2&prefix={sharing_key}/{link name}/&delimiter=/
Nothing answered at /files/, so the probe 404'd and browsing was off for
every data link. Reads were unaffected, which is why this only showed up
as "the folder tree doesn't work".
The whole /files space is now one bucket. A key is
{sharing_key}/{link name}/{path}, which is exactly what a share URL
already looks like, so existing links gain browsing without changing.
Since the first two segments have no counterpart on disk, they are
handed to x2s3 1.5.1's new virtual_prefix option, which strips them from
requests and puts them back on the keys it reports.
Sharing keys are secret, so the bucket root never enumerates them: a
missing, partial or unknown key all produce the same empty listing, and
the only key that can appear in a response is one the caller already
supplied in full. Empty rather than an error is deliberate -- Neuroglancer
remembers per-server whether S3 listing works at all, and an error there
would turn off browsing for the valid links too.
The existing per-key list branch is fixed the same way. It reported keys
without the link name, so they didn't resolve as URLs, and it 404'd at
the bucket root because a bucket has no path of its own to match.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #435 — review the last commit only.
Folder browsing didn't work for data links in Neuroglancer. Verified working against a real Neuroglancer after this change.
What was wrong
Neuroglancer shows a folder tree over plain
https://by reading the first path segment of the URL as an S3 bucket and asking that bucket root for everything under the rest as a prefix:Nothing answered at
/files/— the only route was/files/{sharing_key}/{path}— so the probe 404'd and Neuroglancer recorded the server as unable to list. Reads were unaffected, which is why this only ever showed up as "the folder tree doesn't work".What changed
The whole
/filesspace is now one S3 bucket. A key is{sharing_key}/{link name}/{path}, which is exactly what a share URL already looks like, so existing links gain browsing with no change to their URLs. The first two segments have no counterpart on disk, so they are handed to x2s3 1.5.1's newvirtual_prefixoption (JaneliaSciComp/x2s3#28), which strips them from incoming requests and puts them back on the keys it reports.GET /files/— the bucket root.list-type=2only; no list-type isNoSuchKey, another list type isInvalidArgument, and neither falls through to the SPA./files/{key}/?list-type=2because a bucket has no path of its own to match. Both readings of a share URL now resolve to the same files, which matters because Neuroglancer races them and keeps whichever answers first.encoding-type=urllistings (Fix encoding-type=url in file listings x2s3#29) — Neuroglancer sends that on every listing request, and it used to 500.Sharing keys stay secret
The bucket root never enumerates them. A missing, partial or unknown key all produce the same empty listing, so the endpoint can't be probed, and the only key that can appear in a response is one the caller already supplied in full (exact lookup, no prefix matching; keys are 96 bits).
Empty rather than an error is deliberate: Neuroglancer caches "this server can't list" per origin, so an error at the root would turn off browsing for the valid links too.
This does bend S3 — a real
ListObjectsV2with no prefix lists the bucket. Nothing in the toolchain cares: Neuroglancer, zarr, tensorstore and s3fs all list under a path, andaws s3 ls s3://files/{key}/{name}/works whileaws s3 ls s3://files/shows nothing.Tests
Five new tests in
tests/test_endpoints.py: bucket-root listing whose keys resolve back as share URLs; the link name browsing like a folder; no enumeration from missing/partial/unknown keys; the root not being an object; and both bucket readings naming identical URLs. Full backend suite passes (926).Names with spaces
A file or directory whose name contains a space was listed with
+in place of the space, which no client turns back into a space, so entries showed up but wouldn't open. That was in x2s3's key encoding and is fixed by JaneliaSciComp/x2s3#30 (1.5.2), which this branch will pin. Covered here by a test that reads a key back out of a listing and fetches it, for both the stored and the decoded form of a link name.🤖 Generated with Claude Code