feat: page boundaries, heading inference, and lattice tables in pdfToMarkdown; abort-before-open in readPdf - #754
Merged
Merged
Conversation
Mearman
marked this pull request as ready for review
August 21, 2026 00:03
…arkdown output A pageBreak block is the one block kind every PDF-to-wordprocessing reconstruction emits that markdown-codec's writer silently drops (that package's own model: markdown has no page construct to lose fidelity from, so no named mapping gap either). The page boundary is exact, free information a text-extraction consumer can chunk on, so buildMarkdownText now maps the block to an HTMLPreformatted paragraph carrying '<!-- page break -->' -- HTMLPreformatted so the emitter writes the text verbatim rather than backslash-escaping its punctuation, and an HTML comment because it is the one markdown spelling that carries metadata without claiming to be content: no renderer displays it, and markdown-codec's reader lowers it back as literal text. pdfToMarkdown on a multi-page PDF now emits one marker per page boundary, and none for a single page.
The inverse of the write-side marker: markdown-codec lowers an '<!-- page break -->' HTML comment to an HTMLPreformatted paragraph carrying that literal text, and readMarkdownContent now promotes exactly that paragraph -- whole-paragraph text equal to the marker, nothing less -- back to the pageBreak block it spelled. The marker therefore MEANS a page break rather than decorating one: markdownToPdf re-renders a real page boundary instead of printing the marker as literal text, and a pdfToMarkdown -> markdownToPdf round trip regenerates markers from real boundaries rather than accumulating the previous round's as prose. A comment merely containing the marker stays a paragraph: the match is on the whole paragraph, not a substring.
…ties A heading sits tight above the body it names at ordinary line spacing, so the vertical-gap and indent break signals alone glue the two into one paragraph -- the observed pdfToMarkdown failure where a bold heading rendered as '**Part 1 Scope **This is body paragraph 0...'. A font-size discontinuity between adjacent lines is a third break signal, reusing the presentation direction's own fontSizesClose merge condition (clusterIntoBlocks already refuses to merge across one), so both clusterers now apply the same rule rather than the wordprocessing path lacking half of it. Lines within the close tolerance (superscripts, rounding jitter) still merge.
…econstruction
A rendered PDF carries no semantic headings, but relative font size
against the document's modal body size is a well-worn signal, and the
reconstruction was discarding it: every heading came back as a bold
body paragraph, so pdfToMarkdown emitted '**Title**' instead of
'# Title'. A document-wide census now ranks every distinct size at
least 2pt above the modal body size largest-first into Heading1..6,
and a clustered paragraph whose dominant size lands on a census bucket
is emitted with that Heading{N} styleId -- rank-based, which inverts
the layout engine's own 28/22/18/14pt heading render sizes exactly, so
markdown heading levels survive the markdownToPdf -> pdfToMarkdown
round trip, and generalises to foreign PDFs as 'largest text is the
title'. Sizes within 2pt of the body (including this engine's own
Heading5/6 render sizes) carry no signal and stay paragraphs.
An inferred heading's run-level bold is dropped: the weight is
structural, carried by the styleId every consumer resolves, and keeping
it would render markdown as '# **Title**' -- the bold-run noise this
inference exists to replace.
Where the table recovery's gridline-lattice gate succeeds, the recovered ContentTable already flows through buildMarkdownText into a real GFM pipe table; this holds that wiring at the markdown surface, the same way the pdfToDocx lattice test holds it at the docx one. The fixture is a spreadsheet printed with gridlines (the ordinary odsToPdf path), because a markdown-authored table renders no lattice at all -- markdown carries no border concept, so its cells arrive as tab-separated prose and the lattice gate correctly refuses to invent structure from alignment alone.
The abort check lived only inside the page loop, so it was never consulted for a document with an empty page tree -- an already-aborted signal returned a parsed document instead of throwing -- and an abort arriving during openPdfDocument's xref resolution could not take effect until the first page iteration. The check now runs before the open phase; the per-iteration page-loop check (pinned here by a fixture whose first page aborts the controller through its own diagnostic sink) remains the granularity boundary: signal is page-granular, a single page's content-stream interpretation and the document-open phase are not interruptible.
The signal is page-granular in both pipelines and the reconstructors, checked once per page-loop iteration plus once before readPdf's document-open phase; a single page's content-stream interpretation and the document-open phase itself are the two uninterruptible spans, and cost is roughly linear in decompressed content length. Stated in both READMEs so a CPU-metered runtime (Cloudflare Workers) can budget statically, and the pdfToMarkdown fidelity notes now record what the structure-carrying work delivers: exact page-break markers, rank- inferred heading levels, and GFM tables wherever the drawn-lattice gate succeeds.
…ng styleId
The font-size heading inference set only styleId: Heading{N}, but that is
producer-specific spelling -- headingLevel is the canonical signal the
schema documents. decompose groups on headingLevel alone (a Heading
styleId without it never becomes a HeadingGroupNode, so the onDocument
tree and document-outline's TOC projection were blind to inferred
headings), and the docx writer emits w:outlineLvl from headingLevel, so
every output surface except markdown (whose emitter parses the styleId)
lost the level. Both fields now ride together, matching markdown-codec's
own lowerHeading convention.
…icit undefined
Dropping an inferred heading's run-level bold wrote { ...run, bold:
undefined }, leaving an explicitly-present-undefined key: it survives
'bold' in run and trips toStrictEqual against a key-absent object, so a
de-bolded heading run had a different shape from a run that was never
bold -- and textItemToContentRun gave never-bold/never-italic runs the
same present-undefined shape. Both sites now omit absent keys outright,
so every run the reconstruction produces has one shape.
The marker write transform recurses into table cells, but its read-side inverse cannot: GFM cell content is inline-only, so markdown-codec never lowers a cell's <!-- page break --> to the block-scoped HTMLPreformatted paragraph the promotion matches -- a cell marker always reads back as literal text. Both halves now say so, so the cell marker is documented as a one-way annotation rather than an invertible directive.
buildDocxPackage copied only a paragraph's styleId, so every caller (markdownToDocx, pdfToDocx) wrote headings with no outline level at all -- the depth signal Word's navigation pane and TOC fields read. DocxParagraph.headingLevel now maps the schema's 1-based level onto 0-based w:outlineLvl (the same +1 mapping ooxml.js's own docx reader applies reading it back), ordered per CT_PPrGeneral between w:jc and w:rPr in PPR_ORDER, and populateParagraph writes it. A pdfToDocx of an inferred heading now round-trips its level through the real reader instead of leaving a dangling w:pStyle Heading1 with no outline.
Mearman
force-pushed
the
feat/pdf-structure
branch
from
August 21, 2026 00:22
a4c71f6 to
0c9e0ab
Compare
Contributor
|
🎉 This PR is included in version 3.1.2 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 4.2.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 3.1.17 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 2.1.29 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 1.26.35 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 4.6.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 1.0.7 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 4.0.14 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 5.0.9 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 4.4.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
Contributor
|
🎉 This PR is included in version 1.1.1 🎉 The release is available on: Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #584
Closes #585
Two clusters, both on the PDF read path: the structural information
pdfToMarkdownwas discarding (#584), and the honesty gap in what the synchronous parse promises about cancellation and cost (#585).Page boundaries reach markdown (#584 ask 1). A
pageBreakblock now writes as an<!-- page break -->HTML comment marker — anHTMLPreformattedparagraph so markdown-codec's emitter writes the text verbatim rather than backslash-escaping its punctuation — instead of being dropped by the writer. The read side promotes exactly that paragraph back to apageBreakblock, so the marker means a break rather than decorating one: re-rendering the output honours the boundary and lands on the same page count, and a pdfToMarkdown → markdownToPdf round trip regenerates markers from real boundaries instead of accumulating the previous round's as literal text. One documented asymmetry: inside a table cell the marker is a one-way annotation, because GFM cell content is inline-only and can never reparse as the block-scoped paragraph the promotion matches.Headings inferred from font size (#584 ask 2). Reconstruction now splits clustered paragraphs at font-size discontinuities — the same rule the presentation direction's own clusterer already refused to merge across — so a heading stops being glued to the body it names, and classifies a paragraph as a heading by rank: every distinct text size sitting at least 2pt above the document's modal body size, ranked largest-first into levels 1–6. Ranking is what inverts this package's own write side exactly (Heading1–4 render at 28/22/18/14pt against a 12pt body), and generalises to foreign PDFs as "largest is the title, next are sections"; sizes within the delta of the body carry no signal and stay paragraphs. An inferred heading carries the canonical
headingLevelalongside theHeading{N}styleId — decompose groups onheadingLevelalone, so the tree form and document-outline's TOC projection see inferred headings, and the docx writer can emit their outline level — and drops its run-level bold (the weight is structural now; '# Title' was the failure mode this replaces). The edit side got the matching fix:buildDocxPackagewritesheadingLevelasw:outlineLvl, sopdfToDocx/markdownToDocxheadings carry the depth signal Word's navigation pane and TOC fields read, mapped 1-based to 0-based exactly as ooxml.js's own reader inverts it.Lattice tables pinned as GFM (#584 ask 3). Where the table recovery's gridline-lattice gate succeeds, the recovered
ContentTablealready flowed through to a real GFM pipe table — that wiring is now pinned end to end throughodsToPdf(a spreadsheet actually printed with gridlines, since markdown-authored tables render no lattice), so the path can't silently regress to tab-separated prose.Cancellation and cost stated honestly (#585).
readPdfnow honours an aborted signal before the document-open phase runs — previously a signal aborted before the call, on a document with an empty page tree, never entered the page loop at all and returned a parsed document instead of throwing. Both PDF READMEs now state the granularity and cost model: the pipelines are synchronous end to end, the signal is checked once per page-loop iteration (and now once before document-open), so a mid-parse abort takes effect at the next page boundary; the document-open phase and a single page's content-stream interpretation are the two non-interruptible spans, and cost is roughly linear in decompressed content length — budget for the worst single page, not the page count. An async page-at-a-time API is recorded as a deliberate non-goal of the current surface.Testing
pnpm exec turbo run _build _lint _typecheck _test --filter=documents.js --filter=pdf-codec— all green.convert.test.ts: one marker per page boundary and none for a single-page document; pdfToMarkdown output re-rendered at the same page count;#/##recovered end to end through markdownToPdf; the lattice fixture reaching markdown as a GFM pipe table with a reparseable delimiter row.reconstruct.test.ts: rank-based level assignment against the layout engine's own render sizes, level clamping at six, the font-size paragraph split, one shape for every produced run (absent bold/italic omitted, never present-undefined).markdown/read+write.test.ts: the marker paragraph promoted back to apageBreakblock and a longer containing comment left as a genuine comment.pdf-codec/read.test.ts: an already-aborted signal throws before the document-open phase.edit/docx/paragraph.test.ts+convert.test.ts:headingLevelwritten asw:outlineLvlin schema order and read back through the real reader.