feat(indexing): parse Markdown front matter with SnakeYAML and split multi-record files

[adityamparikh]

Summary

Two changes to Markdown indexing, in one PR because the second builds directly on the first. Supersedes #206, now closed.

Front matter is parsed by a YAML parser. Front matter is YAML, but its values did not come from a YAML parser: the commonmark front-matter extension reads the block line by line, and MarkdownDocumentCreator added a hand-rolled splitter for flow sequences ([a, b]). A quoted title containing a comma or a colon, a quoted list element, or a nested mapping came out wrong or was lost. The block is now sliced out verbatim using commonmark's source spans and handed to SnakeYAML, which Spring Boot already ships. Implicit type resolution is switched off, so every scalar stays the text as written (2026-01-01 is not turned into a Date, 8.4 not into a Double); Solr's schema guessing types it. Sequences become multi-valued fields, nested mappings flatten with underscores (ratings_imdb), nulls are skipped. The flow-sequence splitter is deleted.

One file can hold many documents. create() always returned a single document, so a 61-show sample dataset would have needed 61 separate files. A file may now hold several records, each its own front matter block followed by its body, and the dataset ships as one shows-markdown.md.

Why the split rule is narrow

Markdown has no record separator of its own, and --- already means two other things — a thematic break, and a setext heading underline. Naive concatenation fails twice over: commonmark only recognises front matter at position 0, so later blocks become body text, and a mid-file --- promotes the preceding paragraph into a heading.

Splitting therefore engages only when all of the following hold:

• the file itself opens with a front matter block — ordinary prose containing a thematic break is never considered;
• a boundary is a --- line preceded by a blank line and followed by a YAML key — a rule followed by prose is not a boundary;
• that block is closed by a later --- — an unterminated block is not a boundary.

Anything that is not unambiguously several records still yields exactly one document. The residual ambiguity is documented on splitRecords: a document that both opens with front matter and uses a thematic break immediately followed by a key: value line will split when it should not.

What changed

• MarkdownDocumentCreator: parser built with includeSourceSpans(BLOCKS); addFrontMatterFields loads the block with SnakeYAML's SafeConstructor and a resolver with no implicit tags. create() splits into records and maps each through the existing per-document logic, extracted as createOne(); adds splitRecords() / isRecordStart().
• MarkdownDocumentCreatorTest (plain JUnit): quoted scalars with delimiters, flow and block sequences, nested mapping, typed-looking scalars kept as text, invalid YAML reported, no-front-matter document unchanged — plus two blocks split, one record unchanged, a thematic break does not split, and a document without front matter is never split.
• ShowsSampleDataTest: parses shows-markdown.md and checks its 61 documents against shows.json, field for field.
• build.gradle.kts: -H:IncludeResources on the nativeTest binary, which is the one that reads the fixture. nativeImageBuildArgs feeds the shipped binary and the published *-native-* images, which never load a test resource.

Verification

./gradlew spotlessApply build on Java 25: BUILD SUCCESSFUL, 412 tests, 0 failures. The only skips are the 7 pre-existing OtlpExportIntegrationTest cases.

The regression guard is that the eleven MarkdownIndexingTest cases asserting hasSize(1) pass unedited — if the split rule were even slightly too eager, several would go red.

MarkdownDocumentCreatorTest   7/7 pass
ShowsSampleDataTest           2/2 pass
MarkdownIndexingTest         11/11 pass (unedited)

Independent of #202, #203 and #205; any merge order.

🤖 Generated with Claude Code

https://claude.ai/code/session_019qWfXJUJ7TU3HJgtNZZq2E