The directory collection: the public half of the OutreachGraph crawl - #68
Merged
Merged
Conversation
Anthony: "from all our public data put it all in nichedb.dev too from outreachgraph.com crawler/scanner". nichedb ingests only by pulling, so this reads the keyless public directory outreachgraph.com now serves at /api/v1/public/directory (outreachgraph PR #73): companies and sites by their domain, and people only when they publish their own profile. The line is drawn on the OutreachGraph side; this adapter adds nothing to the row, tags only what the directory stated, and keeps the row verbatim under data.row with the attribution. Collection `directory`, source `outreachgraph-directory` (hourly, enabled, cursor-paged 200 a page, 50 pages a run), feeds: all, companies, sites, people, self-served OpenProfile.md files. Thirteen tests: page URLs, the three row kinds, absent stays absent, rows without id/name/known kind dropped, no private field can arrive, cursor walk, page cap, failed page, and the 404 before the endpoint deploys. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cmNRtR2vL1p89dbVQ7FZJ
ralyodio
added a commit
that referenced
this pull request
Sep 13, 2026
Both sides add an adapter and a collection; kept both. The directory collection lists OutreachGraph's public companies, sites and self-published people as rows; the profiles collection merges the people into one entry each. They read the same site and do different jobs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S7yeJUHGxA4P5N74xnsRPQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthony: "from all our public data put it all in nichedb.dev too from outreachgraph.com crawler/scanner". nichedb ingests only by pulling, so this reads the keyless public directory outreachgraph.com serves at
/api/v1/public/directory(profullstack/outreachgraph#73, not yet merged: the first run 404s until that deploys, and after a deploy a new source waits behind the older ones).What arrives: companies and sites by their domain (industry and stack as
topic:tags), and people only when they publish their own profile: an OpenProfile.md they serve (taggedopenprofile, linked indata.openprofile) or a profile and home page that vouch for each other. The privacy line is drawn on the OutreachGraph side; this adapter adds nothing to a row, tags only what the directory stated, and keeps the row verbatim with attribution.Seeded: collection
directory; sourceoutreachgraph-directory(hourly, enabled, cursor-paged 200 a page, 50 pages a run); feedsdirectory-all,directory-companies,directory-sites,directory-people,directory-openprofiles. README rows added.Checks:
bun test1077 pass, 0 fail (13 new); biome clean.🤖 Generated with Claude Code
https://claude.ai/code/session_014cmNRtR2vL1p89dbVQ7FZJ