Purpose
IB needs the same hostile-web boundary being characterized for Common Crawl/syllabi rather than inventing a second, incompatible notion of successful retrieval and parsing.
Computer-science / language capability issue: walnut-burgundy/computer-science#62
AICI receipt issue: isomorphisms/ai-ci#46
Syllabi / Common Crawl issue: isomorphisms/syllabi#2
Related IB/computer-science design work: walnut-burgundy/computer-science#46
Reuse one language and one checked boundary
The long-term target is the same language across IB and the ingestion stack, not merely a shared interchange format between separate language implementations.
Use external HTML parsers, curl/libcurl, WARC libraries, browser engines, and Haskell/Python/Java implementations as side-by-side oracles, fixture sources, and code to study. They may coexist during bootstrap, but the production path being developed should progressively own each layer itself and must never silently fall back to an oracle.
The shared stage model remains:
network/input → bytes → decompression → text decoding → HTML recovery → document representation → IB indexing/rendering/extraction.
Keep distinct:
- not attempted
- retrieval failed
- partial/truncated retrieval
- decompression failed
- encoding unknown/failed
- HTML recovered with caveats
- document representation produced
- resource bound reached
- interrupted
DOM/document semantics
Do not call the internal representation a browser DOM unless its semantics are actually tested. IB may use the file-backed document-log idea from computer-science #46 if that is the better bounded-memory representation, while standards-aware external parsers remain differential oracles until equivalent required semantics are implemented in the target language.
Acceptance
- same hostile-input fixtures can be reused where relevant;
- AICI exposes
PASS / FAIL / SKIP at each boundary;
- a failed network or parser stage cannot appear to IB as an empty successful page;
- source/provenance survives far enough to explain what bytes/document a later index entry came from;
- external implementations remain reproducible comparison targets rather than hidden runtime dependencies;
- progress is measured by how much of the stack the target language owns correctly end to end.
Purpose
IB needs the same hostile-web boundary being characterized for Common Crawl/syllabi rather than inventing a second, incompatible notion of successful retrieval and parsing.
Computer-science / language capability issue: walnut-burgundy/computer-science#62
AICI receipt issue: isomorphisms/ai-ci#46
Syllabi / Common Crawl issue: isomorphisms/syllabi#2
Related IB/computer-science design work: walnut-burgundy/computer-science#46
Reuse one language and one checked boundary
The long-term target is the same language across IB and the ingestion stack, not merely a shared interchange format between separate language implementations.
Use external HTML parsers, curl/libcurl, WARC libraries, browser engines, and Haskell/Python/Java implementations as side-by-side oracles, fixture sources, and code to study. They may coexist during bootstrap, but the production path being developed should progressively own each layer itself and must never silently fall back to an oracle.
The shared stage model remains:
network/input → bytes → decompression → text decoding → HTML recovery → document representation → IB indexing/rendering/extraction.
Keep distinct:
DOM/document semantics
Do not call the internal representation a browser DOM unless its semantics are actually tested. IB may use the file-backed document-log idea from computer-science #46 if that is the better bounded-memory representation, while standards-aware external parsers remain differential oracles until equivalent required semantics are implemented in the target language.
Acceptance
PASS/FAIL/SKIPat each boundary;