health: every healthcare provider in the United States (NPPES)

[ralyodio]

An NPI is the number on every insurance claim filed in the US. Everyone who bills insurance has one, and NPPES is the register: ~8 million clinicians and organizations with their specialty, licence state and location. CMS publishes it in full monthly, public domain, no key, no terms.

That's why this is ingested rather than scraped. A provider directory is normally assembled out of somebody's licensed database — this one is given away by the agency that maintains it. /c/health previously had openFDA recalls and clinical trials, ~3,200 items.

The file was measured, not assumed

I read the zip's central directory and inflated only the first member's opening — so its shape was established without downloading 1.16 GB:

member	size
npidata_pfile_*.csv	4.29 GB, 330 columns
pl_pfile_*.csv	0.12 GB (practice locations)
endpoint_pfile_*.csv	0.13 GB
othername_pfile_*.csv	0.05 GB

V1 was retired 2026-03-03; V2 is the only one published.

The walk

CourtListener's shape, because the problem is identical: http.download with Range resume (already in the repo), unzip once (already in the Dockerfile), then walk a batch at a time saving the line offset after each, across as many runs as it takes.

Columns by index, checked on every run

The header is 330 columns of prose (Provider Business Practice Location Address City Name) so matching on it is slow and fragile. All sixteen indices were verified against the real header before being written down.

A government file of that width is exactly the sort that gains a column, so the header is re-checked every run and a shift throws — reading a first name out of a position that now holds a middle name would otherwise be silently wrong in every row.

Two things the real data taught

• The archive ships a _fileheader.csv next to the data with the same stem, holding only the header row. Picking it would look like an empty but successful walk.
• 2,353 of the first 24,000 rows have a blank entity type and no name — NPIs deactivated and stripped. Dropped rather than stored nameless.

Both are tested.

The privacy decision, defaulting to cautious

Most records are organizations, but a large minority are sole practitioners who registered a home address as their practice location. It's public by law and CMS publishes it for reuse, so there's no legal question — but republishing several million people's home addresses in a searchable index is a different act from republishing a hospital's, and it should be a decision rather than a side effect.

	individuals	organizations
city / state	✅	✅
ZIP	5-digit only	full
street lines	withheld	✅
phone	withheld	✅

The file carries ZIP+4, which narrows a home to roughly a building, hence the truncation. includeIndividualStreet turns it off and defaults to no. A test asserts a built item never carries an individual's street.

Also

Taxonomy codes resolve through NUCC's published code set, fetched per run rather than vendored — a stale copy would quietly degrade every row to a bare 207X00000X.

Checks

• 34 new tests, built on two real records pulled from the September 2026 file
• 2,042 tests across the suite; lint clean
• The usual 5 Redis/Postgres integration failures this dev box can't run

🤖 Generated with Claude Code