feat(table): detect table regions from text alignment, opt-in - #27
Merged
hallelx2 merged 1 commit intoSep 17, 2026
Conversation
camelot reaches 0.762 recall on ICDAR 2013 where we reach 0.422, and reading its source explained why: its stream flavour runs Anssi Nurminen's text-edge detector, which won the 2013 competition. We were running pdfplumber's text strategy, which was never a detector at all. The idea is that a table is not primarily a thing with lines around it. It is a region where text stays vertically aligned over several consecutive rows. Prose does not do that — a paragraph shares a left margin with the whole page, but its interior positions wander. So find x-coordinates where four or more vertically adjacent lines start, end or centre together, and their extent is the table. That threshold is the entire method. It is the filter prose cannot pass, and it is why this can raise recall without inventing tables the way page-wide text derivation does. Measured over 125 documents, against the identical pipeline without it: precision 0.187 -> 0.281 recall 0.547 -> 0.635 F1 0.245 -> 0.337 (+38% relative) ms/doc 288 -> 227 Faster as well as better, because gridding a few bounded regions is less work than gridding a page. It is NOT the default, and the result is only half a success. Against camelot running the same algorithm, recall came across (0.635 vs 0.762) and precision did not (0.281 vs 0.514). Recall is a detection property; precision, once you hold a region, is a gridding property. So the detector transferred and our cell inference inside a known region is now the bottleneck — a different problem, and one the oracle result already bounds at 0.935. lines stays the default at 0.442. DetectRegions is worth enabling on corpora known to carry unruled tables, where lines returns nothing at all and 0.337 beats zero. StrategyAuto is the precedent for shipping a detection change behind a flag until it is measured. Implemented independently from the published algorithm, not translated from camelot's MIT source. Pure geometry over words already extracted — no model, no network, no new dependency, no CGo. Both full benchmark runs agree to every decimal place across all 13 systems, and the detector carries its own determinism test.
Reviewer's GuideThis PR adds an opt-in, independently implemented Nurminen-style text-edge detector that bounds text-based table extraction to regions with sustained row alignment, integrates it without changing default behavior, and supplies tests, benchmark wiring, and evaluation documentation showing improved text-strategy results but unresolved precision limitations in cell gridding. Sequence diagram for opt-in text-edge table extractionsequenceDiagram
participant Page
participant Settings as TableSettings
participant Detector as DetectTextEdgeRegions
participant Extractor as baseEdges
participant Grid as CellExtraction
Page->>Settings: DetectRegions = true
Page->>Detector: DetectTextEdgeRegions(words, TextEdge)
Detector-->>Page: regions or nil
alt regions detected
Page->>Page: wordsInBBox(words, region)
Page->>Extractor: baseEdges(text strategy, words in region)
Extractor-->>Page: bounded vertical and horizontal edges
else no regions or detection disabled
Page->>Extractor: baseEdges(text strategy, page words)
Extractor-->>Page: page-wide edges
end
Page->>Grid: infer cells from edges
Flow diagram for text-alignment region detectionflowchart TD
A[Extracted words] --> B[wordsToTextLines]
B --> C[Register line left, centre, and right coordinates]
C --> D[buildTextEdges]
D --> E{Coordinate match and vertical adjacency?}
E -->|yes| F[Extend alignment edge]
E -->|no| G[Start alignment edge]
F --> H{At least 4 lines?}
G --> H
H -->|no| I[Discard alignment]
H -->|yes| J[dominantAlignment]
J --> K[regionsFromEdges]
K --> L[Bounded table regions]
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Review limit reachedNext included review available in 26 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Advanced Run ID: 📒 Files selected for processing (9)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="detect_textedge.go" line_range="314-333" />
<code_context>
+ best := textAlign(0)
+ bestScore := -1
+
+ for a := textAlign(0); a < numAligns; a++ {
+ score := 0
+ for _, e := range edges[a] {
+ if e.count >= minLines {
+ score += e.count
+ }
+ }
+ if score > bestScore {
+ best, bestScore = a, score
</code_context>
<issue_to_address>
**issue (bug_risk):** A prose paragraph with four or more lines sharing the normal left margin produces a valid `alignLeft` edge and is returned as a table region, because the detector selects a single strongest alignment without requiring multiple independent column alignments. `wordsInBBox` then grids that prose region instead of falling back to page-wide detection.
**Triggers:** When ordinary body text contains at least four consecutive lines with the same left margin.
**Suggested fix:** Require evidence from at least two distinct alignments or add a rule that rejects a region supported only by a page-wide margin alignment.
</issue_to_address>
### Comment 2
<location path="detect_textedge.go" line_range="314-333" />
<code_context>
+ best := textAlign(0)
+ bestScore := -1
+
+ for a := textAlign(0); a < numAligns; a++ {
+ score := 0
+ for _, e := range edges[a] {
+ if e.count >= minLines {
+ score += e.count
+ }
+ }
+ if score > bestScore {
+ best, bestScore = a, score
</code_context>
<issue_to_address>
**issue (bug_risk):** Only one alignment class is retained for the entire page, so a page containing tables whose strongest evidence differs between left, center, and right alignment loses every table outside the globally dominant class. The detector consequently returns incomplete regions even though each individual table satisfies `MinLines`.
**Triggers:** When multiple tables on one page use different alignment styles and no single alignment class has the highest total count across the page.
**Suggested fix:** Select dominant alignment per candidate vertical band or retain valid alignment classes when their regions do not overlap.
</issue_to_address>
Comment on lines
+314
to
+333
| for a := textAlign(0); a < numAligns; a++ { | ||
| score := 0 | ||
| for _, e := range edges[a] { | ||
| if e.count >= minLines { | ||
| score += e.count | ||
| } | ||
| } | ||
| if score > bestScore { | ||
| best, bestScore = a, score | ||
| } | ||
| } | ||
| if bestScore <= 0 { | ||
| return nil | ||
| } | ||
|
|
||
| valid := make([]*textEdge, 0, len(edges[best])) | ||
| for _, e := range edges[best] { | ||
| if e.count >= minLines { | ||
| valid = append(valid, e) | ||
| } |
There was a problem hiding this comment.
issue (bug_risk): A prose paragraph with four or more lines sharing the normal left margin produces a valid alignLeft edge and is returned as a table region, because the detector selects a single strongest alignment without requiring multiple independent column alignments. wordsInBBox then grids that prose region instead of falling back to page-wide detection.
Triggers: When ordinary body text contains at least four consecutive lines with the same left margin.
Suggested fix: Require evidence from at least two distinct alignments or add a rule that rejects a region supported only by a page-wide margin alignment.
Comment on lines
+314
to
+333
| for a := textAlign(0); a < numAligns; a++ { | ||
| score := 0 | ||
| for _, e := range edges[a] { | ||
| if e.count >= minLines { | ||
| score += e.count | ||
| } | ||
| } | ||
| if score > bestScore { | ||
| best, bestScore = a, score | ||
| } | ||
| } | ||
| if bestScore <= 0 { | ||
| return nil | ||
| } | ||
|
|
||
| valid := make([]*textEdge, 0, len(edges[best])) | ||
| for _, e := range edges[best] { | ||
| if e.count >= minLines { | ||
| valid = append(valid, e) | ||
| } |
There was a problem hiding this comment.
issue (bug_risk): Only one alignment class is retained for the entire page, so a page containing tables whose strongest evidence differs between left, center, and right alignment loses every table outside the globally dominant class. The detector consequently returns incomplete regions even though each individual table satisfies MinLines.
Triggers: When multiple tables on one page use different alignment styles and no single alignment class has the highest total count across the page.
Suggested fix: Select dominant alignment per candidate vertical band or retain valid alignment classes when their regions do not overlap.
hallelx2
deleted the
halleluyaholudele/hal-1362-pdfgrab-port-nurminen-text-edge-detection
branch
September 17, 2026 22:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports Anssi Nurminen's text-edge table detector — the algorithm that won the ICDAR 2013 structure track and that camelot's
streamflavour runs. Opt-in, and a partial result.Why
camelot reaches 0.762 recall where pdfgrab reaches 0.422. Reading its source explained it:
_nurminen_table_detection(). We were running pdfplumber'stextstrategy, which was never a detector — it infers columns across the whole page with no notion of where a table is, which is why its precision is 0.188.The idea: a table is not primarily a thing with lines around it, it is a region where text stays vertically aligned over several consecutive rows. Prose shares a left margin with the whole page but its interior positions wander. Find x-coordinates where ≥4 vertically adjacent lines start, end or centre together — that threshold is the entire method, and the filter prose cannot pass.
Measured — 125 docs, against the identical pipeline without it
text, page-wide (control)text+ region detection+38% relative F1, and faster — gridding a few bounded regions is less work than gridding a page.
It is half a success, and the half that failed is informative
Against camelot running the same algorithm:
streamRecall transferred. Precision did not. Recall is a detection property; precision, once you hold a region, is a gridding property. So the detector works and our cell inference inside a known region is now the bottleneck — a different problem, and one the oracle result already bounds at 0.935.
Not the default
linesstays default at 0.442 vs this at 0.337.StrategyAutois the precedent: it also found more regions and scored slightly worse, because regions found but gridded badly cost more precision than they buy in recall.DetectRegionsis worth enabling on corpora known to carry unruled tables — wherelinesreturns nothing at all (22% of this corpus) and 0.337 beats zero.Notes
Local:
gofmt -lclean,go build,go vet,go test ./...green. CI red for HAL-1354 (no runner assigned account-wide).Closes HAL-1362
Summary by Sourcery
Add opt-in text-alignment table-region detection to improve extraction of unruled tables while preserving the existing default strategy.
New Features:
Enhancements:
Documentation:
Tests:
Chores: