Public directory: the crawler's public half, keyless, for nichedb.dev - #73
Merged
Merged
Conversation
Anthony: "from all our public data put it all in nichedb.dev too from outreachgraph.com crawler/scanner". nichedb ingests only by pulling, so this is the endpoint it pulls: GET /api/v1/public/directory, above the session guard, paged by since + cursor (keyset on updated, id), cached five minutes, sixty requests a minute per caller. The line it holds, in one query: a company or site by its domain; a person only when they publish their own profile, meaning an OpenProfile.md they serve (published_url) or a profile and a home page that point at each other with rel=me, which the openprofile job now records as openprofiles.corroborated (migration 0036). Never an email, a phone, a location, a bio, a score, a signal, a campaign, a note or a workspace id; a person known from a scraped handle alone is not listed. Topics come from the industry and technologies on a company, and only from the ## Topics line of a person's profile. Tests: 15 for the route, the withholding rules, paging, the cursor and the limiter; the full suite is 1506 passing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014cmNRtR2vL1p89dbVQ7FZJ
ralyodio
added a commit
that referenced
this pull request
Sep 13, 2026
Both blocks sit above the session guard and both stay. #73 took migration 0036, so the settings table is 0037. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S7yeJUHGxA4P5N74xnsRPQ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthony: "from all our public data put it all in nichedb.dev too from outreachgraph.com crawler/scanner". nichedb ingests only by pulling, so this is the endpoint it pulls.
GET /api/v1/public/directory, keyless, above the session guard. Paged bysinceand an opaquecursor(keyset on updated, id),cache-control: public, max-age=300, sixty requests a minute per caller.What is listed: a company or site by its domain (
sitewhen the crawler named the row after the domain alone); a person only when they publish their own profile: an OpenProfile.md they serve (published_url), or a profile and a home page that point at each other with rel=me, which the openprofile job now records asopenprofiles.corroborated(migration 0036, applied at boot).What is never listed: email, phone, location, bio, score, signal, campaign, note, workspace id, avatar. A person known from a scraped handle alone is not listed. Topics come from a company's industry and technologies, and only from the
## Topicsline of a person's profile. The rule lives inapps/api/src/public-directory.tsand nowhere else.Row shape:
{ id, kind, name, url, description, topics, country, openprofile, updated }.Checks: 15 new tests (route, withholding, paging, cursor, limiter); full suite 1506 passing;
tsc --noEmitclean in apps/api. The nichedb adapter that reads this is a separate PR in niche-db.🤖 Generated with Claude Code
https://claude.ai/code/session_014cmNRtR2vL1p89dbVQ7FZJ