Skip to content

Address researcher review of the paper - #32

Open
antonkayai wants to merge 1 commit into
mainfrom
docs/researcher-paper-review
Open

antonkayai wants to merge 1 commit into
mainfrom
docs/researcher-paper-review

Conversation

@antonkayai

Copy link
Copy Markdown
Contributor

Summary

  • Address the annotated PDF review: clarify the extraction task, domain terms, stressors, OCR checks, and artifact availability; tighten the abstract and surrounding prose.
  • Introduce metrics before construction, define micro/macro F1 and strict completeness, and document the existing claim normalization rules. Dataset, predictions, scorer behavior, and published scores are unchanged.
  • Remove Author Contributions, put dataset/code links on separate first-page footnote lines, fit the appendix stressor map on one page, and exclude macOS metadata from the arXiv archive.
  • Update manuscript/report consistency assertions for the revised wording.

Verification

.venv/bin/python -m pytest -q
make -C paper arxiv
git diff --check
.venv/bin/python benchmarks/check_evaluation_report.py --claims-dir data --results-dir benchmarks/results/codex_gpt56_sol_full_current_ocr_v2 --require-full-corpus
.venv/bin/python benchmarks/check_evaluation_report.py --claims-dir data --results-dir benchmarks/results/codex_full_current_ocr_v2 --require-full-corpus
.venv/bin/python benchmarks/check_evaluation_report.py --claims-dir data --results-dir benchmarks/results/claude_fable5_full_current_ocr_v2 --require-full-corpus
.venv/bin/python benchmarks/check_evaluation_report.py --claims-dir data --results-dir benchmarks/results/claude_opus48_full_current_ocr_v2 --require-full-corpus

203 tests pass; all four reports reproduce. The 29-page PDF was visually checked, and the extracted arXiv source archive compiles independently with its included bibliography.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants