Reduce non-intersecting map archive downloads

[FrogAi]

Summary

• Add compact 2-degree archive ranges so country and state downloads avoid archive groups that do not intersect their selected boundary geometry.
• Preserve the existing bounding-box behavior for custom menus and entries whose generated archive set already matches their bounding box.
• Add a standalone Go command to check or regenerate the ranges from pinned country and state boundary sources.
• Reduce archive requests across the 229 default regions from 17,238 to 8,613 (50.03%) and their summed server-reported Content-Length from 65.41 GB to 43.54 GB (33.44%).

Root cause

Map downloads currently round each region's rectangular bounding box outward to the 2-degree archive grid, then request every archive in that rectangle. Large or irregular regions such as Alaska and Canada therefore request many archive groups that do not intersect the selected region.

The archives already contain road data rather than terrain or ocean layers. The unnecessary cost comes from selecting irrelevant archive groups, so this change narrows the requested coordinates without changing the contents of any individual archive.

Change

• Add an optional archive_ranges field to download-menu entries. Each row is [latitude, inclusive minimum longitude, exclusive maximum longitude] for one 2-degree latitude band.
• Apply the same range-first, bounding-box-fallback selection rule to progress totals and archive downloads.
• Load the download menu once per download instead of re-reading it while resolving each path.
• Add go run ./cmd/update-download-regions to check the committed ranges and go run ./cmd/update-download-regions --write to regenerate them from pinned Natural Earth and Census sources.
• Preserve menu ordering, unknown metadata, numeric formatting, and newline style when updating the inline ranges.
• Keep the existing menu authoritative for catalog entries and detached-territory scope instead of automatically making geopolitical policy decisions.
• Document the range format, validation, precedence, fallback behavior, and regeneration commands.

Download impact

Metric	Base 7201c6b4	Candidate f538b5e1	Difference
Region-summed archive requests	17,238	8,613	-8,625 (-50.03%)
Region-summed Content-Length	65.41 GB	43.54 GB	-21.88 GB (-33.44%)
Unique archive coordinates	9,850	6,703	-3,147 (-31.95%)
Unique-union Content-Length	23.95 GB	23.91 GB	-45.55 MB (-0.19%)
Canada archive requests	765	527	-238 (-31.11%)
Canada Content-Length	1.44 GB	504.51 MB	-931.95 MB (-64.88%)
Alaska archive requests	1,980	143	-1,837 (-92.78%)
Alaska Content-Length	2.42 GB	11.56 MB	-2.40 GB (-99.52%)

The region-summed rows reflect independently downloading each menu region, which matches the current downloader's behavior. The unique-union rows count each archive coordinate once and describe total catalog coverage.

Compatibility and scope

• The archive URL format and per-coordinate archive contents are unchanged. Only the selected coordinates change.
• Existing custom menus containing only bounding_box continue to use the previous behavior. Omitted or empty archive_ranges also fall back to the bounding box.
• Archive rows are validated as exactly three ordered, world-bounded, 2-degree-aligned integers when the menu is loaded.
• The existing 229 catalog entries, names, bounding boxes, submenus, and ordering are preserved. Of those entries, 184 use explicit ranges and 45 retain bounding-box selection.
• Existing menu bounds continue to control which disconnected source components belong to an entry. New countries and newly relevant detached territories remain intentional menu changes.
• The updater is a standalone Go command and is not invoked by the mapd runtime. Its go-shp parser is only linked into the updater; orb was already in mapd's dependency closure through osm.
• This does not change archive extraction, cancellation, progress publication, file formats, concurrency, cross-location deduplication, retries, or installation behavior.

Engineering record and validation

Design decisions

• Add optional range rows instead of rewriting stored bounding boxes so existing custom menus and fallback behavior remain unchanged.
• Keep the generated ranges inline with each menu entry so the catalog and its archive selection remain one authoritative artifact.
• Represent contiguous longitude archives as one latitude row because that matches the downloader's iteration directly without storing every coordinate.
• Apply the same range precedence to both progress totals and downloads so the reported file count and requested coordinates use the same selection contract.
• Generate the menu from pinned boundaries instead of maintaining thousands of coordinates manually.
• Use existing menu bounds to preserve established disconnected-territory scope while retaining selected components whole for ordinary boundary corrections.
• Keep the updater as a separate cmd target so regeneration-only source parsing is not linked into the mapd binary.

Boundary-generation policy

The updater pins:

• Natural Earth 10m Admin 0 countries at commit f1890d9f152c896d250a77557a5751a93d494776, SHA-256 239eec57ac17f100a11e2536cffc56752c318b50ae765b0918ff7aab4ce8f255.
• Census TIGER/Line 2025 states, SHA-256 59a220888a8d9be8117c4fcd38f542bd02d81abf0d198c78113595ad540dd957.

Country GeoJSON is decoded with orb; Census ZIP/SHP/DBF data is decoded with go-shp and converted to orb polygons. Downloaded source bytes are accepted only when they match the pinned SHA-256.

Each existing menu bounding box is rounded to the archive grid and used as the policy seed for selecting source polygon components. Selected components are retained whole so ordinary boundary corrections are not clipped by stale decimal bounds. Their intersecting 2-degree archives are then compacted into latitude rows.

Natural Earth's nonstandard country identifiers are handled explicitly: FR, NO, and TW are selected through ADM0_A3 values FRA, NOR, and TWN. mapd's historical us_state.GM entry maps to Census STUSPS=GU.

The Go updater evaluates source rings directly and does not add a general geometry-repair dependency. The pinned Natural Earth Egypt self-intersection was explicitly checked against the expected archive set during the port. Unsupported antimeridian jumps are rejected instead of producing ambiguous longitude ranges.

The updater never adds or removes region definitions in the download catalog. New countries and newly relevant detached territories require an intentional menu change and review of the generated diff.

The candidate union removes 3,159 legacy coordinates and adds 12 coordinates reached by retaining selected components whole, producing a net reduction of 3,147 coordinates.

Measurement method

The live sweep ran from 2026-08-22 16:03:13 through 16:15:53 UTC. Base coordinates were derived from each menu bounding box using mapd's 2-degree floor/ceiling behavior. Candidate coordinates used non-empty archive_ranges when present and the same bounding-box fallback otherwise.

Each coordinate maps to https://map-data.pfeifer.dev/offline/{latitude}/{longitude}.tar.gz. The combined base/candidate union contained 9,862 unique archive coordinates, each mapping to one URL. The audit queried every URL exactly once with HEAD using 24 workers, a 30-second timeout, and up to four attempts with exponential backoff. A parseable positive Content-Length was required. Every URL passed on its first attempt, so no retries were used and no archive payloads were downloaded.

One coordinate-to-size map was then summed two ways:

• Region-summed results count an archive each time it belongs to a separately selected region.
• Unique-union results count each archive coordinate once.

Unrounded Content-Length totals:

• All regions: 65,410,800,451 -> 43,535,791,511
• Unique union: 23,952,699,874 -> 23,907,153,765
• Canada: 1,436,458,288 -> 504,505,102
• Alaska: 2,415,287,103 -> 11,561,464

The displayed MB and GB values use decimal units. The totals exclude HTTP/TLS overhead, retries, and extracted on-disk size.

The size sweep was originally recorded against eb23a2ab25557cb4aa62384878cf589a2b4512bf. The Go port at the current head reproduces the same ranges and effective archive coordinates byte-for-byte, so the measurements still apply to the current selection behavior.

The one-off measurement harness and per-URL manifest were not retained. Boundary and menu generation remain reproducible from pinned sources, but the historical size figures are contemporaneous evidence and cannot be independently recalculated or diagnosed per coordinate from the retained record. Because hosted archive sizes can change, rerunning this method verifies the current hosted data rather than the exact historical totals above.

Validation

Compared:

• Base: 7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee
• Candidate: f538b5e13dd0755939b8839c9c4e7640ebd4e90b

The candidate changes only:

• cmd/update-download-regions/geometry.go
• cmd/update-download-regions/main.go
• cmd/update-download-regions/menu.go
• cmd/update-download-regions/sources.go
• docs/overriding-internal-defaults.md
• go.mod
• go.sum
• settings/download.go
• settings/download_menu.json

Exact-head checks:

• go run ./cmd/update-download-regions
  • download menu is up to date (229 regions, 184 with explicit ranges; 17238 legacy archive occurrences -> 8613 selected)
• go test ./...
• go vet ./...
• docker buildx build --platform linux/arm64 --load .
• git diff --check 7201c6b4b4ec1b0b9ea21daa8c05b80fdd7e01ee...f538b5e13dd0755939b8839c9c4e7640ebd4e90b

Audit-only reconstruction and differential checks:

• Reproduced the previous pinned-source archive ranges and effective coordinates exactly with the Go updater.
• Reproduced the committed menu byte-for-byte while preserving LF and CRLF newline styles.
• Confirmed every selected archive intersects its pinned source geometry.
• Confirmed no intersecting archive within the retained component scope was omitted.
• Confirmed archive rows are aligned, ordered, non-overlapping, compact, and within world bounds.
• Confirmed names, bounding boxes, submenus, catalog entries, metadata, numeric formatting, and ordering remain unchanged.
• Confirmed malformed range lengths, types, bounds, ordering, and grid alignment are rejected while omitted or empty ranges retain bounding-box fallback.
• Confirmed go-shp is linked only into the standalone updater and not into the production mapd binary.

The exact-head Go tests and vet ran in a Linux/amd64 container. The Docker build compiled the candidate for Linux/ARM64 and produced a static binary.

No exact-head race test or on-device/end-to-end map download was performed. No permanent tests for the new selection behavior are included.

The one-off independent geometry and formatting checkers were not retained. Their results are contemporaneous audit evidence rather than durable regression coverage; the committed updater's check mode remains the reproducible check that pinned sources generate the committed menu.