Follow nichedb.dev for a campaign: its open collections become crawls, on a clock - #75
Merged
Merged
Conversation
…, on a clock nichedb.dev's webrings, sites and profiles collections are lists of people's own sites, with who makes each one and what it publishes about itself. Anyone in there publishes on the open web in a machine-readable way, which is who an open-standards offering wants to talk to. The `discover_nichedb` job reads each collection since a cursor through nichedb's keyless items API (sort=updated, order=asc, since=), turns items into candidate sites (one per host; platforms, our own hosts and the l.ink pages skipped; a profile's site from its accounts or its OpenProfile), queues a `crawl_site` per new host under the campaign, and queues itself again with the newest updated_at as the next cursor, everyMs from now. Two alternating dedupe keys per campaign keep one next run outstanding while a run is still running. Needs no model. POST /campaigns/:id/discover/nichedb starts or re-tunes it (collections, everyHours, limit, since), GET reads the pending and running runs, DELETE stops it; a running job whose row was deleted under it finishes without a successor. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XYae2mH3khdwiXUVzcVMDw
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthony: "automate the shit maybe a nichedb.dev/all.json or something and we crawl that shit with a daemon to discover people to outreach in our profile db for selling promoting sharing later."
discover_nichedb(packages/pipeline/src/nichedb-discovery.ts)webrings(members),sites(OpenSite pages) andprofiles(people with a home page) since a cursor, via the keyless items API, up to five pages per collection per run.github.com,x.com,linkedin.comand the other platforms, our own hosts, and*.l.inkare skipped.crawl_siteper new host withdedupeKey crawl:<campaign>:<host>andbatchId= the run, so the intake screen reports it like any batch.everyMslater (default 6 h) with the newestupdated_atassince. Two alternating dedupe keys (nichedb:<campaign>:a/:b) keep exactly one next run outstanding while the current one is running.discoverevents; needs no model.API
POST /api/v1/campaigns/:id/discover/nichedb{collections?, everyHours?, limit?, since?}→ 202. Replaces any pending schedule.GET …/discover/nichedb→{following, runs[]}.DELETE …/discover/nichedb→ removes the pending run; a running one finishes without a successor.Worker dispatch, pipeline exports and the activity label are wired. Tests: candidate extraction, host rules, a full run against a stubbed nichedb (3 crawls queued, next run queued with the cursor), the incremental run and the double-schedule guard, stop.
First use: the "LogicSRC open standards" campaign in Anthony's workspace, so the people publishing OpenWebring, OpenSite and OpenProfile data end up as prospects for the specs.
🤖 Generated with Claude Code
https://claude.ai/code/session_01XYae2mH3khdwiXUVzcVMDw