Shrink the country-named region, and count what is left in it - #2
Open
navjottomer wants to merge 6 commits into
Open
Shrink the country-named region, and count what is left in it#2navjottomer wants to merge 6 commits into
navjottomer wants to merge 6 commits into
Conversation
A settlement whose P131 says nothing cannot be reached by any walk of the containment graph in either direction. 863 of India's 966 unplaced rows are that case: a country, a class and a coordinate, and nothing else. "74 GB", a village in Sri Ganganagar district, is one of them. Wikidata has no statement about where it is; it does have a point, and divisions have boundaries. Natural Earth is the only boundary source that clears the bar this project holds itself to. It is public domain -- no attribution, no share-alike, nothing that attaches an obligation to a table of city names -- which is the same test the IANA time zone database passes. OpenStreetMap-derived data is ODbL and GeoNames is CC-BY, and either would put a licence on the output. No dependency. The shapefile and dBASE formats are fixed-width binary and the whole reader is a hundred lines; a GIS stack for two rectangles' worth of parsing would be a much larger thing to take on. The only subtlety is winding order, which is how a division's outer rings are told from its holes. It answers with an ISO 3166-2 code and never with a place, so a caller can check the answer against the divisions it already selected. Where the code names a level or an ISO edition this dataset does not ship -- Natural Earth gives France its departements and this ships regions -- learn_code_map reads what the code means here off the settlements already placed inside it, and refuses anything under 90% agreement over at least five of them.
Issue #1: build-stats.json reports 52,734 for no_division while 57,250 settlements ship in a country-named region. The gap is real -- a settlement whose division resolved and was then removed by the region exclusions fires no counter -- but the more useful answer was to make the region smaller. Five changes, in the order the evidence is consulted. Containment now reads P150 as well. Wikidata records containment from both ends, and where the child says nothing the parent often does. Lithuania is the extreme: all 60 municipalities point P131 straight at the country, because the county administrations were abolished in 2010 and the county edge carries an end date, while the ten counties still list their municipalities under P150. The walk runs twice -- P131 alone first, then again with P150 read backwards alongside it, seeded from the divisions only. A settlement takes the second answer only where the first got it no further than its own country, and only for a division of the country it already belongs to, so a statement made by a parent can never outrank one made by the child or move a place across a border. That places 22,070 of Lithuania's 23,193 and 553 of Greece's 708. What still reaches no division is placed by its coordinate, against Natural Earth. Over the last release that is 44,558 by a code this dataset already ships and 6,685 more by reading what a code means here, leaving 5,953. France goes from 4,220 unplaced to 85, India from 966 to 64. Tier 4 asks a question that was never asked. Tier 2 selects the country's administrative P131 children, and the Faroe Islands cannot answer it: its six sysslur state no P131 at all, so nothing is a child of the country and the whole hierarchy -- 147 of 149 settlements filed under one of 33 municipalities, every municipality inside a syssla -- was invisible. So for a country no other tier resolves, the divisions are taken from what its own settlements say contains them, and keep_root_most picks the level as it does everywhere else: six sysslur, not 33 municipalities, for the same reason France gets regions rather than departements. Twenty-one countries ship one region named after themselves and 237 of their 326 settlements have a container recorded upstream. Asking what the country claims instead gave the Cook Islands 26 regions, most of them its own villages, and gave the Vatican a charity; a municipality is a settlement by this pipeline's rule, so the chain is walked through it rather than stopping at it, which is what made Avarua a region holding its own suburb. Country-as-its-own-region moves to tier 5. P460 removes 93 rows that were one place recorded twice. A settlement that reaches no division and that upstream says is the same thing as one that does is that one: "Warszawa" beside Warsaw, "Cochin" beside Kochi, an item labelled "do not use" two hundred metres from Stuttgart. Distance is deliberately not consulted -- P460 links confusable places as readily as identical ones, Hoya in Lower Saxony to La Hoya in Salamanca 1,778 km apart, and every such pair has a division on both sides, so asking which side is placed refuses them without measuring anything. None of the 93 has a population and 89 have no P131 at all. It runs before placement: 92 of them fall inside a Natural Earth polygon, so otherwise each would be filed beside the row it duplicates under a different name, where nothing would merge them. Finally build-stats.json carries country_region per country, counted on the rows that ship rather than on the containment failures alone, so it can be checked against the published file. no_division is untouched and still counts what it always counted. All of it is optional and additive. A scan taken before P150 and P460 builds exactly as it did, and so does a build without --boundaries; the fixture build is byte-identical either way.
UNPLACED-SETTLEMENTS.md measured the region against the last release and suggested four directions. Three were taken, so the note now records what the pipeline does rather than what it might: 57,196 settlements in the region, 44,558 placed by a code that matches one shipped here, 6,685 by reading what a code means, 5,953 left. It also has the numbers for what was left after each step and why, including Kiribati, whose 105 unplaced rows all fall in the longitude band of one of the three island groups the dataset already ships. The correction that matters is to what "regionless" means. Almost none of the 5,953 are places with no division; they are places whose division exists and that no evidence available here reaches. Genuinely division-free is Sint Maarten, the Vatican, Gibraltar, Christmas Island, the Cocos Islands and Pitcairn -- on the order of a hundred rows. README gains a stated guarantee, one row per place, and the licence section gains the one field Wikidata alone no longer decides. That claim was "no other source contributes a single field", and it had to change: admin1_id for a settlement that states no containment is decided by a public-domain boundary. It says so, with the three fences that keep it narrow. The fourth direction, a vote of the nearest already-placed settlements, is recorded and not taken. It would answer for 10,962 of the 12,638 left before the code mapping ran and reproduce the known division 96% of the time, and 96% over six thousand rows is 240 settlements confidently filed in the wrong division -- which is the argument the issue made in the first place.
Tier 4 was unreachable. It ran only when tier 2 found nothing, and the countries that need it are exactly the ones where tier 2 finds something useless -- so a full build selected it for no country at all and the Faroe Islands still shipped one region. The Faroes are the case. A handful of administrative items do point P131 at Q4628, so tier 2 answers and the ladder stops; then 0 of 155 settlements attach to them, and the attachment fallback flattened the country to tier 5 without ever asking what its settlements say contains them. So the question moves to where the evidence is. A tier that was selected and then attached nothing is the same situation as no tier at all, and the attachment fallback is the one place that can see it. Tier 4 is tried there, and taken only if it places more than the tier it replaces. Containment propagation has already run by then, so the assignment comes back with the divisions rather than being left to a second pass. It can: everything in the chain is in the country's own shard, and the walk is a few hundred settlements rather than fifteen million edges. The selection itself is now one function both call sites use. Measured on a real build from the 11 August dump: the Faroe Islands go from one region holding all 149 settlements to five sysslur holding 120, and the country region falls from 57,250 to 5,477 across the release. Niue, Sint Maarten and Western Sahara still flatten, which is right -- the first two have no subdivisions at all. The regression test is the shape that shipped broken: a decoy division that points P131 at the country and that no settlement can reach.
Every figure in these notes was measured against the published release and reasoned forward. A build over the same 11 August dump that produced 2026-08-15T0426Z has now run, so they are replaced with what it did, and the differences are worth having on the record. The country-named region holds 5,477 settlements rather than 57,250, across 209 countries rather than 249. P150 placed 27,476 -- more than the 22,600 projected, because the projection could only see what a released file exposes. Coordinates placed 38,675 and the learned code mapping 6,991. P460 dropped 111 rows rather than 93. Two claims were wrong and are corrected. The Faroe Islands do not come out with nothing left in their country region: five sysslur hold 120 of the 149 and 29 remain. And tier 4 did not run at all on the first full build, for a reason now described where it is relevant -- it was asked only where tier 2 found nothing, and the countries that need it are the ones where tier 2 finds something that turns out to reach nothing. validate.py passes against the published baseline: 100% coordinate coverage, no country below the floor, 1,668,628 cities against 1,674,947. The 6,319-row difference is duplicates surfacing rather than data lost -- 3,121 of France's 4,220 unplaced rows already had a same-named row within 2 km of them, invisible while they sat in the country region and merged the moment placement put each beside its twin.
The P460 rule reached the README's guarantees and the note on the country-named region, but not LIMITATIONS.md, which is where a consumer looks for what is deliberately not shipped. It belongs beside the archaeology and prison-camp exclusions: it is the same kind of statement, a row that exists upstream and does not ship here, and it should not have to be inferred from the guarantee list.
Member
Author
|
Built and validated over the same 11 August dump that produced Measured, not projected
Placement: 27,476 by P150, 38,675 by coordinate, 6,991 by reading
The −6,319 cities is duplicates surfacingFrance is −3,163 of it and Germany −2,629. Of France's 4,220 unplaced rows, Tier 4 shipped broken and is fixedIt was unreachable. It ran only where tier 2 found nothing, and the countries It is now asked where a tier is seen to have attached nothing, and taken only The regression test is the shape that shipped broken: a decoy division that AlsoThe docs carried projections; they now carry what the build did, including two 171 tests. Still nothing published: the build sits at |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the measurable half of #1, and most of the rest.
build-stats.jsonreported 52,734 forno_divisionwhile 57,250 settlementsshipped in a country-named region. The counting is fixed, but the more useful
answer was to make the region smaller.
What lands where now
Measured over release
2026-08-15T0426Z, all 57,196 settlements that ship in acountry-named region across 254 countries:
France goes from 4,220 unplaced to 85, India from 966 to 64, Lithuania from
23,193 to 293 — and Lithuania's are border cases that P150 places before any
boundary is consulted.
The five changes
Containment reads P150. Wikidata records containment from both ends, and
where the child says nothing the parent often does. All 60 Lithuanian
municipalities point P131 at the country — the county administrations were
abolished in 2010, so the county edge carries an end date — while the ten
counties still list their municipalities under P150. The walk runs twice, and a
settlement takes the second answer only where the first got it no further than
its own country, and only for a division of the country it already belongs to.
22,070 of Lithuania's, 553 of Greece's.
Coordinates place what containment cannot.
boundaries.pyreads NaturalEarth's admin-1 shapefile with no GIS dependency. It answers with an ISO 3166-2
code, never a place, and the code must name a division already shipped for that
country. Where it names another level or an older ISO edition — Natural Earth
gives France its départements, this ships régions — what the code means here is
read off the settlements already placed inside it, and refused under 90%
agreement over at least five of them.
Tier 4 asks what the settlements say. The Faroe Islands' six sýslur state
no P131 at all, so tier 2 found nothing and the country became its own region,
hiding a complete hierarchy: 147 of 149 settlements under one of 33
municipalities, every municipality inside a sýsla. Divisions now come from what
a country's own settlements say contains them, with the same root-most rule —
six sýslur, not 33 municipalities. Country-as-its-own-region moves to tier 5.
P460 removes 93 rows that were one place twice. "Warszawa" beside Warsaw,
"Cochin" beside Kochi, an item labelled "do not use" 200 m from Stuttgart. None
of the 93 has a population; 89 have no P131 at all. Distance is deliberately
not consulted: P460 links confusable places as readily as identical ones — Hoya
in Lower Saxony to La Hoya in Salamanca, 1,778 km apart — and every such pair
has a division on both sides, so asking which side is placed refuses them
without measuring anything. It runs before placement, because 92 of the 93 fall
inside a polygon and would otherwise be filed beside the row they duplicate
under a different name, where nothing merges them.
country_regionper country, counted on the rows that ship rather than oncontainment failures, so it can be checked against the published file.
no_divisionis untouched.What is left, and what is not being done about it
5,953 rows, 0.36% of the dataset: countries whose divisions Natural Earth has
not caught up with (BI, KR, NP, CI, MA), points in no polygon at 1:10m, border
cases, and countries that genuinely have no divisions. Almost none are places
without a division — they are places whose division exists and that no evidence
here reaches.
A vote of the nearest already-placed settlements would answer for 10,962 of the
12,638 left before the code mapping ran, at 96% accuracy on held-out data. It
is not taken. 96% over six thousand rows is 240 settlements confidently filed
in the wrong division, which is the failure #1 objects to.
Compatibility
Additive and optional throughout. A scan taken before P150 and P460 builds
exactly as it did; a build without
--boundariesbehaves as every release has.The fixture build is byte-identical either way (
s_version 6885c02741c68b91).The one contract change is in the README: "no other source contributes a single
field" is no longer true, and now says so —
admin1_idfor a settlement thatstates no containment is decided by a public-domain boundary set.
Testing
170 tests. New coverage drives the P150 rescue, the boundary placement, the
learned code map, tier 4 and the P460 rule end to end, over synthetic scans and
a shapefile written by the test rather than committed. Two assertions are worth
naming: no region contains the same name twice after rows move between regions,
and a Natural Earth code naming a division this build does not ship is refused
rather than approximated.
This is not a release. The graphs it needs do not exist in any scan on
disk; every number above is measured against the published release and becomes
real at the next
tools/refresh.py --rescan, wherevalidate.pygets a say.