Derive the reading order of a page laid out in columns: each column threaded to its end, full width rows kept whole, running heads, folios and footnotes lifted out of the flow.
import { threadDocument } from 'column-thread';
// Boxes come from your extractor. Origin is top left, so y grows downward.
const pages = [
{ page: 47, width: 612, height: 792, runs: [
{ text: 'The', x: 72, y: 100, width: 15, height: 10 },
...
]},
...
];
const doc = threadDocument(pages);
doc.text;
// column one of page 47, then column two, then the next page
doc.segments[2];
// { kind: 'span', page: 47, band: 1, column: null,
// text: 'Table 3. Harbour works and expenditure by year to September\n
// Segment 2020 2019 2018 Notes\n
// Harbour works 1,204 980 774 E revised\n...' }
doc.headers[0].text; // 'The Harbour Commission 47'
doc.footnotes[0].text; // '1. See the minute book for the full text of the'
doc.pages[0]; // { columnCount: 2, gutters: [...], spanLines: 2, bands: 3 }
doc.diagnostics; // choices made conservatively, each with a reasonZero runtime dependencies. TypeScript, strict, Node 18 and up.
On a two column page, every emitted line alternates between the columns:
The commission reported that the harbour A second survey was ordered in the spring
works were incomplete before the season and the engineer sent a revised drawing
closed and that no further funds had been showing the breakwater carried forty feet
Nothing errors. The output is token complete, the character count matches the page, chunk sizes look normal, and the document still retrieves because the bag of words is intact. It is spliced at every single line and there is no signal anywhere that says so.
Split the page at the gutter and the ordinary text comes out right. Then the page carries one full width table.
A wide row's cells are laid out on the full page grid, and the gutter falls in the gap between two of them. That row has no ink where the gutter is, so it looks exactly like two lines of ordinary column text sitting level with each other. Cut there, and the row's labels go into column one while its figures go into column two, about forty lines later:
column one: Harbour works 1,204 980
Dredging 884 1,010
...
column two: 774 E revised
902 F restated
Every figure is present. Every one is quotable. Not one of them is attached to the row it came from any more.
Page furniture compounds it. A running head prepends itself to the first paragraph of each page, and a footnote lands inline at the column break, so a page produces a sentence that is fluent, extractable, citable, and was never written by anyone.
A gutter is not an empty band, it is a band that almost nothing crosses. The detector builds a coverage profile from the ink boundaries themselves, with no fixed grid to quantise the answer, and looks for a wide interval that at most gutterSpanTolerance of the page's lines reach into. The lines that do reach into it are not evidence against the gutter. They are the full width material, and they go to the span pass instead of being thrown away.
The boundary is set in two passes. A ragged right margin also produces a wide low coverage region, and it runs continuously into the gutter, so a single pass puts the column edge several units inside the measure. That is enough to throw the last word of every long line into the wrong column: the same splice, one word at a time. So the first pass finds a provisional band, the lines that genuinely reach across it are taken out of the profile, and the second pass re-measures what is left. The final boundary sits on the last piece of real column text.
Full width regions are grown, not classified line by line. Deciding one row at a time is what cuts the table in half, because only some of its rows cross the gutter. A wide table almost always crosses somewhere: a title, a head with a cell that straddles the gap, a long label. Those rows seed a region, and the region grows:
- inward, taking every line between two crossing rows, provided they all look tabular
- outward, one line at a time, over lines that look tabular and sit within a normal line advance
A line looks tabular when it holds more separated pieces of text than the page has columns. After gaps narrower than a gutter are bridged, a line of prose contributes exactly one interval per column it appears in, because every gap inside a measure is word spacing. Six intervals on a two column page is a row built out of cells. A paragraph of prose below the table has two, so it stops the growth and stays in the flow.
Full width material breaks the page into bands, and each band is read column by column. Emitting the whole of column one and then the whole of column two would carry the reader past the table and back again. A page with a table in the middle comes out as: column one above, column two above, the table, column one below, column two below. This is how the page was set.
A running head is identified by its normalized text and its height, never by its horizontal position. Matching exact strings finds nothing, because the head carries the folio and no two pages print the same one, so digits and roman numeral folios are reduced to a placeholder. Stripping whatever sits in the top band instead takes every chapter opener with it, so a candidate has to repeat at the same height on at least minFurnitureRepeats pages and on at least half the document. Position is matched on height alone because a folio alternates between the outer corners, verso left and recto right, and keying on x would split one head into two groups of half the size.
A footnote block needs both a size drop and a step down. Size alone takes a small caption or a run-in signature at the foot of a column. A gap alone takes the last paragraph after any generous space. The body size is the mode across the whole document, not the page, because a page set larger than the rest of the book would elect its own display size and then read its ordinary last lines as notes.
The module throws rather than guess when the geometry has two readings and they disagree about the text:
- A band that between
gutterSpanToleranceandgutterRejectFloorof the lines cross. Too many for a gutter interrupted by full width rows, too few for ordinary single column text. Read as one column the page is spliced line by line, read as two the crossing lines are cut in half. Both are silent, so neither is chosen for you. - A line that crosses some gutters but not all of them on a page with three or more columns. It is either a heading that interrupts part of the page or a table that reaches part way across, and those read in opposite orders. Set
partialSpans: 'band'to take the second reading deliberately.
Everything the module decides conservatively is reported in diagnostics with the reason, and the conservative choice always leaves the text in the flow rather than removing it. strict: true turns every diagnostic into a thrown error.
The one case geometry cannot settle is a tabular row that leaves the gutter clean and has no crossing row anywhere near it to seed a region. That row is split, and a tabular-line-split diagnostic names the page and quotes the row.
A margin note next to a single column of text is not modelled. If a band clears the column tests it becomes a gutter and the note becomes a second column read after the body. If it does not, the band is rejected and the note is read inline, at whatever vertical position it sits. Neither is right. There is no option that fixes this because the module has no signal that separates a note from a narrow column.
Bands are horizontal, so a full width element reorders everything level with it. A figure that occupies the lower half of column one only, with column two running past it, is read as a band break across the whole page if it crosses the gutter.
Column order is left to right or right to left across the whole document. A page that mixes directions is not supported.
Reading order is derived, paragraphs are not rejoined. Segments end at column and band boundaries, and a sentence broken across a column break stays broken across two segments. Lines within a segment are joined with a newline, not with a space, so hyphenation and word rejoining are left to the caller.
Rotated text is not handled. Runs are treated as axis aligned boxes, so a sideways table caption contributes ink across the width it occupies on the page and can suppress a gutter.
Footnote detection needs a size difference. Notes set at body size, separated only by a rule that carries no text, are not found. The rule itself is not text and never reaches this module.
npm install
npm test # 91 tests: gutters, spans, tables, running heads, footnotes, refusalsThe fixtures include a two column page carrying one full width table whose data rows leave the gutter clean, and the tests assert that the naive y then x order produces the spliced text while this one does not.
MIT