feat(indexing): index-url — index a document set from an allow-listed http(s) URL in both transports (#208)

[adityamparikh]

Implements #208: an index-url tool that indexes a JSON, CSV, XML or Markdown document set from an http(s) URL into a Solr collection without the payload passing through the model, registered in both the STDIO and HTTP transports.

Design record with every decision and its reason: docs/superpowers/specs/2026-09-15-url-ingestion-design.md (in this PR).

Why

The four inline indexing tools take the payload as a tool-call argument, so the model emits every byte. Measured when #197 was closed: 61 records took over two minutes, of which Solr took under a second. The server's differentiator is the onboarding loop (look at data, design schema, index, verify, search) and it broke at the indexing step, with no workaround in chat-only clients such as Claude Desktop. A URL argument is ~20 tokens regardless of payload size.

What

index-url(collection, url, format?)

• Allow-listed hosts, checked on the requested URL and on every redirect hop. SOLR_INDEX_URL_ALLOWED_HOSTS takes exact hosts, *.suffix patterns or *. Default: raw.githubusercontent.com,*.githubusercontent.com,github.com, so the tutorial URL works with no configuration and an HTTP deployment is not an SSRF primitive on day one.
• Link-local and cloud-metadata addresses refused on every hop even with *.
• No credentials or caller headers, ever; embedded user:pass@ is rejected. Only Accept and User-Agent are sent.
• Redirects: five followed, the sixth is an error, https→http refused.
• Non-2xx is an error before any body is read (a GitHub 404 body would otherwise index as a CSV row).
• Size cap SOLR_INDEX_URL_MAX_BYTES, default 10 MB, checked from Content-Length and while reading. Over the cap, the error points at bin/solr post / the /update handler.
• Format: explicit format > requested URL extension (query string ignored) > final URL extension > Content-Type; text/plain and text/html never resolve.
• Body read in memory and handed to the existing document creators through a new shared IndexingService.indexPayload, which the four inline tools now also use (IndexingServiceTest passes unmodified). Nothing is indexed unless the whole document parses, so every fetch or parse error truthfully says so.
• HTTP client: Spring RestClient over HttpURLConnection (already on the classpath), whose read timeout covers the body; redirect following disabled at the connection so the policy sees each hop. SOLR_INDEX_URL_CONNECT_TIMEOUT 10s, SOLR_INDEX_URL_READ_TIMEOUT 30s.
• Guidance: the index-data prompt and the server instructions route URLs under the cap to index-url, larger datasets to bin/solr post or curl against /update (run where the file is, which also covers Claude Desktop local files), and pasted data to the inline tools.

Relationship to earlier decisions

• feat(indexing): stream local JSON, CSV, XML and Markdown files #194 closed a STDIO-only index-file for giving the transports different surfaces. This PR adds no transport-specific tool and reverses nothing recorded there; the local-file case is served by bin/solr post guidance.
• feat(indexing)!: fold per-format inline tools into index-documents #197 kept per-format inline tools for token shape. index-url carries no payload, so one tool with an optional format suffices.
• refactor(indexing): forward CSV and XML to Solr's own update handlers #205 (forward CSV/XML to Solr's handlers): no conflict. This PR adds no parser code and touches the four inline tool methods only to extract indexPayload. After refactor(indexing): forward CSV and XML to Solr's own update handlers #205 merges, a follow-up can proxy the fetched body straight to Solr's /update for JSON/CSV/XML and drop the cap for those formats with the same tool contract.
• A first version that streamed through server-side parsers with a hand-written watchdog, kept index-file, and allowed any host by default is preserved at wip/index-url-streaming and is not proposed.

Threat model

THREAT_MODEL.md §8.5 said the client "cannot inject a target URL" at critical severity; this tool exercises that deliberately behind an allow-list. The PR narrows §8.5 to the Solr backend and its credentials, adds the allow-listed fetch and the DNS-rebinding window as a §9 bounded property, adds the *-on-an-internal-network misuse to §11, the index-url non-finding to §11a, a dated entry to §12, qualifies §13, adds the three knobs to §5a, corrects the tool counts in §1/§5a, and updates docs/security/stdio.md and http.md.

Two defaults are the author's choice and need maintainer confirmation: the allow-list contents and the 10 MB cap. Both are single constants.

Tests

• New, all running natively: UrlTargetPolicyTest (21), IndexFormatsTest (14), UrlFetcherTest (17, real JDK HttpServer), UrlIndexingIntegrationTest (5, Testcontainers Solr).
• UrlIndexingServiceTest (14, Mockito, the only class skipped natively).
• Both MCP client transports assert index-url and its hints through the shared base and add a URL round trip plus a refused metadata address.
• IndexingServiceTest unmodified.

Review pass (three independent reviewers: security, correctness, tests/docs)

No critical findings. Fixed in 94897cd:

• Refused responses were still downloaded. Spring's response close() drains an unread body for connection reuse, so an over-cap or non-2xx response was pulled in full after being refused. UrlFetcher now closes the body stream before returning on every non-body path, which makes the JDK drop the connection. Proven by a 2 MB body streamed at 10 KB/s that must fail inside a 5-second preemptive timeout.
• "Cloud-metadata addresses" was broader than the code. Only link-local plus the AWS IPv6 literal were refused; Alibaba Cloud (100.100.100.200) and Azure WireServer (168.63.129.16) now are too, and every document says "the known cloud-metadata addresses".
• A port above 65535 is an invalid URL instead of a raw HttpURLConnection message; a non-numeric Content-Length counts as unknown; syntactic failures on a redirect hop report the redirect message.
• Coverage: every redirect status, 3xx without Location, quoted charset, SolrException, csv/markdown media types, blank explicit format, exact IPv6 allow-list entry, exact messages in the integration test, and the field-name summary that pins the indexPayload refactor.
• THREAT_MODEL.md §9 now also records that the read timeout is per read (a slow allow-listed sender can hold one call open), that response headers are not size-capped, and that the JDK DNS cache narrows the rebinding window. FAQ and AGENTS.md tool counts and service lists updated.

Accepted and documented rather than built: a total deadline per fetch (it would break legitimate slow downloads, and the default allow-list is not attacker-controlled), and a concurrency limit on fetches (same exposure class as the inline tools).

Verification

./gradlew build          BUILD SUCCESSFUL — 499 tests, 0 failures, 7 skipped
                         (all 7 in OtlpExportIntegrationTest, @Disabled on main since 375a710)
./gradlew nativeTest -Pnative   BUILD SUCCESSFUL — 336 successful, 0 failed, 160 skipped
                         (skipped = the pre-existing @DisabledInNativeImage Mockito classes and the
                          @Disabled OTLP class, plus UrlIndexingServiceTest, the one new Mockito class)

Manual STDIO run of the fat jar against a local Solr, indexing the raw GitHub shows.json:

[tools/list] index-url present
[index-url] isError=False :: Successfully indexed 61 of 61 documents into collection 'shows-url-manual'. Indexed field names ...
[search]    isError=False :: {"numFound":61,...}
[index-url http://example.invalid/x.json] isError=True :: The URL's host is not on this server's allow-list. Allowed by default: raw.githubusercontent.com, *.githubusercontent.com, github.com. ...

🤖 Generated with Claude Code