test(data): ship the shows sample dataset as CSV, XML and Markdown too - #201
Closed
adityamparikh wants to merge 10 commits into
Closed
adityamparikh wants to merge 10 commits into
adityamparikh wants to merge 10 commits into
Conversation
shows.json is the dataset the tutorial and the hackathon card index. The same 61 shows now exist as shows.csv, shows.xml and one Markdown file per show under shows-markdown/, so the indexing walkthrough can be followed in whichever format a reader has. CSV carries multi-valued fields as repeated column headers, which the CSV creator turns into one value per non-empty cell under the same field name. Markdown puts every field in YAML front matter and the description in the body. XML uses one <show> element per record with repeated children for lists; the XML creator prefixes those fields with the record name (show_title) and carries no plain id, which ShowsSampleDataTest pins along with the field-for-field equality of the other three formats. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
RAT audits XML; CSV, JSON and Markdown data files are exempt. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
…est binary Spring AOT's resource hints cover every *.json file, which is why shows.json was already available to native tests; the new flavours were not, and ShowsSampleDataTest failed natively with missing-resource NPEs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
The XML creator flattened every element by path from the record element, so <show><id>x</id><title>T</title></show> produced show_id and show_title and no id at all; Solr then minted a UUID per document and re-indexing duplicated rather than replaced. XML was the one format whose documents did not match the same data in JSON, CSV or Markdown, and the only one where a reader had to learn different field names for the tutorial's later prompts. The record element (the root for a single document, each repeated child for many) is now a wrapper: its child elements are the fields, named after themselves, nested elements still flatten below that with underscores, and attributes keep the _attr suffix: unqualified on the record (id_attr), qualified by the element on a child (name_lang_attr). XmlIndexingTest's expectations drop the record prefix. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
Depends on the XML record-mapping fix; with it, the record element is a wrapper and shows.xml yields id, title and multi-valued genres like the other three formats. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
Solr already parses CSV and its own update XML; the server's CSV and XML document creators re-implemented that with conventions of their own (a generic-XML mapping that prefixed every field with the record element and dropped the id). Both creators are gone. index-csv-documents reads only the header, sanitizes the column names, and forwards the payload to Solr's CSV handler with those names as fieldnames; repeated columns are multi-valued and empty cells skipped, as before. index-xml-documents takes Solr update XML (<add><doc><field name=...>) and forwards it. The one thing the server still checks on XML is the root element: Solr's update grammar also carries <delete>, <commit>, <optimize> and <rollback> on the same endpoint, so an indexing tool that forwarded blindly would be a delete tool. UpdatePayloads.inspectXml does that check with a hardened StAX read (no DTD, no external entities) and counts documents and field names for the response, without building a document model. Solr accepts or rejects each payload as a whole; the per-document retry no longer applies to these two formats. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
CSV and XML are now forwarded to Solr rather than parsed here, so the sample XML is Solr's own <add><doc><field name=...> format and the check that the same 61 shows come out identical from JSON, CSV and XML moves to ShowsSampleDataIntegrationTest, which indexes all three through the tools into a real Solr and compares the documents. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
Two titles contained ': ' unquoted, which a YAML parser rejects and only the line-by-line reader tolerated. The files are now emitted by a YAML library, so they hold under apache#206 and under the current reader alike. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com>
adityamparikh
added a commit
to adityamparikh/solr-mcp
that referenced
this pull request
Sep 14, 2026
…egration tests Port shows.csv, shows.xml, and ShowsSampleDataIntegrationTest from PR apache#201. The integration test verifies that JSON, CSV, and XML datasets index the exact same 61 documents into Solr. Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com> Co-authored-by: Junie <junie@jetbrains.com>
adityamparikh
added a commit
to adityamparikh/solr-mcp
that referenced
this pull request
Sep 14, 2026
Port shows-markdown/ (61 show records) and ShowsSampleDataTest from PR apache#201. Validates that SnakeYAML front matter parsing produces documents matching shows.json across all 61 records. Signed-off-by: Aditya Parikh <aditya.m.parikh@gmail.com> Co-authored-by: Junie <junie@jetbrains.com>
Contributor
Author
|
Closing as superseded and distributed into format-specific PRs:
All datasets, tests, and native resource hints are preserved in their respective PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
shows.jsonis the dataset the tutorial and the hackathon card index. The same 61 shows now ship in every format the indexing tools accept, so the walkthrough can be followed in whichever one a reader has:src/test/resources/shows.csvgenres,creators,cast,tags) are repeated column headers; the CSV creator adds one value per non-empty cell under the same field name, so the documents match JSON field for field.src/test/resources/shows.xml<add>with one<doc>per record and repeated<field name="genres">elements for lists, the formatindex-xml-documentsforwards to Solr (#205).src/test/resources/shows-markdown/<id>.md# Titleheading. One Markdown document is one Solr document, hence 61 files.ShowsSampleDataTestpins that the formats the server parses itself (JSON and Markdown) yield the same 61 documents;ShowsSampleDataIntegrationTestindexes the JSON, CSV and XML files through the tools into a real Solr and asserts the three collections hold identical documents. CSV and XML go through Solr's own handlers (#205), so this branch carries #205 as a merge until it lands; merge #205 first and this diff shrinks to the data files, the tests and the native include.The native test binary needs an explicit
-H:IncludeResourcesfor the three new flavours; Spring AOT only registers*.jsonresources, which is whyshows.jsonalready worked natively.Verification
./gradlew buildon Java 25: green,ShowsSampleDataTest4/4../gradlew nativeTest -Pnativeon GraalVM CE 25.0.2: green including the four new tests (the first run failed on the missing resources, which is what the build change fixes)../gradlew buildgreen, including the end-to-end equality test.🤖 Generated with Claude Code
https://claude.ai/code/session_01CiUHyyXLTo9ATdgg8eRFZJ