History: paging, span, and not sleeping through the budget

CHANGELOG.md

@@ -5,10 +5,73 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [8.1.0] - 2026-09-23
 
 ### Added
 
+- **`entity(id).history.span()`, `.days()` and `.changeRows()`** — the loop
+  above the three history calls.
+
+  ```js
+  const history = tp.entity(DISNEYLAND).history;
+  const span = await history.span();
+  for await (const { entityId, row } of history.days({
+    from: span.archiveFrom,
+    to: span.retrievableThrough,
+  })) {
+    // ...
+  }
+  ```
+
+  - `span()` returns `archiveFrom`, `recordedTo` and `retrievableThrough` in
+    one shape. The underlying coverage documents do not: a park nests them
+    under `summary`, an entity carries them at the top level under different
+    names, so without this every caller writes that branch first.
+    `retrievableThrough` is the end date to bound a backfill by, because it is
+    what the key may read rather than what the archive holds.
+  - `days()` pages until the server stops offering a `next`, following that URL
+    verbatim, and yields `{ entityId, row }` as rows arrive rather than
+    collecting them. A park's daily call is the one paged call in the family,
+    so without this a park backfill silently stopped at the first 31 days.
+  - Both flatten a park envelope and an entity envelope to the same stream, so
+    a caller writes one loop and does not branch on `'entities' in res`.
+  - `BudgetExhaustedError` (a `RateLimitError`) is thrown when the history
+    budget is spent and the server asks for longer than `maxWaitMs` (120000 by
+    default). It carries `retryAfterMs`, so a backfill can checkpoint and
+    resume rather than hold a process open for most of an hour.
+
+- **`examples/backfill.mjs`** — a complete backfill with resume and NDJSON or
+  CSV output. It pulled Disneyland Resort's whole daily archive, 98,452 rows,
+  in one run.
+
+### Fixed
+
+- **A 429 could park the client for hours.** The transport honoured any
+  `Retry-After` up to `retry.max` times. That is right for a REST 429, which
+  asks for seconds, and wrong for a history 429: that budget is hourly, so a
+  spent one can ask for most of an hour, and three of those is roughly two and
+  a half hours of a silent process. `RetryConfig` gains `maxRetryAfterMs`
+  (120000 by default): past it the client does not sleep at all and throws
+  `RateLimitError` with `retryAfterMs` set.
+
+- **`EntityHistoryCoverage` was missing the park shape.**
+  `/entity/{id}/history/coverage` answers a PARK with
+  `HistoryParkCoverageDocument`, the same way `/history` and `/history/daily`
+  do, and the type named only `HistoryCoverageDocument`. The two do not
+  overlap where it counts: a park carries `summary` and `fields`, an entity
+  carries `firstRecordedAt`, `lastRecordedAt` and `kinds`. A TypeScript user
+  read `.kinds` off a park's coverage, got `undefined` at runtime, and the
+  compiler said nothing. The fixture that covered this was hand-written in the
+  entity shape and named after a park, so it agreed with the code for the same
+  reason the code was wrong; both coverage fixtures are now captured from
+  production, and the live smoke test asserts the park shape it actually gets.
+
+- **The user agent announced the wrong version.** `PACKAGE_VERSION` was still
+  `7.0.0-alpha.0` in a package at `8.0.0`, so every request announced a version
+  a major old and nothing failed. A gate test now asserts the `User-Agent` the
+  server actually receives carries the version `package.json` declares, so
+  forgetting the bump is a red test rather than a quiet lie in a header.
+
 - **`apiKey` client option.** Sent as the `X-API-Key` header on every
   request. Every endpoint still answers without one; a key raises the limits,
   which matters for the history endpoints (30 days of history and 600

README.md

@@ -49,15 +49,15 @@ Jungle Cruise                                      40 min
 
 `new ThemeParks(options)` takes the following keyword options:
 
-| Option      | Type                                | Default                          | Purpose                                                                                               |
-| ----------- | ----------------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------- |
-| `baseUrl`   | `string`                            | `https://api.themeparks.wiki/v1` | API base URL (point at a mock / staging if you need to).                                              |
-| `userAgent` | `string`                            | `themeparks-sdk-js/<version>`    | Sent as the `User-Agent` header. Set this to identify your app.                                       |
-| `apiKey`    | `string`                            | none                             | API key from api.themeparks.wiki, sent as `X-API-Key`. Optional; a key raises the limits.             |
-| `fetch`     | `typeof fetch`                      | `globalThis.fetch`               | Custom fetch implementation. Useful for logging, mocking, or older runtimes.                          |
-| `timeoutMs` | `number`                            | `10000`                          | Per-request timeout in milliseconds.                                                                  |
-| `retry`     | `Partial<RetryConfig>`              | `{ max: 3, on429: true }`        | Retry/backoff behavior. `max` counts retries **beyond** the initial attempt (so `3` = up to 4 total). |
-| `cache`     | `Cache \| false \| { maxEntries? }` | in-memory LRU                    | See [Caching](#caching) below. `false` disables caching entirely.                                     |
+| Option      | Type                                | Default                                            | Purpose                                                                                                                                                                                                                                        |
+| ----------- | ----------------------------------- | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `baseUrl`   | `string`                            | `https://api.themeparks.wiki/v1`                   | API base URL (point at a mock / staging if you need to).                                                                                                                                                                                       |
+| `userAgent` | `string`                            | `themeparks-sdk-js/<version>`                      | Sent as the `User-Agent` header. Set this to identify your app.                                                                                                                                                                                |
+| `apiKey`    | `string`                            | none                                               | API key from api.themeparks.wiki, sent as `X-API-Key`. Optional; a key raises the limits.                                                                                                                                                      |
+| `fetch`     | `typeof fetch`                      | `globalThis.fetch`                                 | Custom fetch implementation. Useful for logging, mocking, or older runtimes.                                                                                                                                                                   |
+| `timeoutMs` | `number`                            | `10000`                                            | Per-request timeout in milliseconds.                                                                                                                                                                                                           |
+| `retry`     | `Partial<RetryConfig>`              | `{ max: 3, on429: true, maxRetryAfterMs: 120000 }` | Retry/backoff behavior. `max` counts retries **beyond** the initial attempt (so `3` = up to 4 total). `maxRetryAfterMs` is the longest `Retry-After` the client will sleep through; past it you get `RateLimitError` instead of a silent wait. |
+| `cache`     | `Cache \| false \| { maxEntries? }` | in-memory LRU                                      | See [Caching](#caching) below. `false` disables caching entirely.                                                                                                                                                                              |
 
 Example:
 
@@ -189,9 +189,13 @@ if (!('entities' in week)) {
   for (const d of week.days) console.log(d.date, d.operatingMinutes, d.standby?.p50);
 }
 
-// Which days, and which live-data fields, are held at all.
+// Which days, and which live-data fields, are held at all. A PARK answers the
+// park document here too (summary + fields + entities), so narrow, or use
+// span() below, which reads both to one shape.
 const coverage = await barnstormer.history.coverage();
-console.log(coverage.firstRecordedAt, coverage.retrievableThrough, Object.keys(coverage.kinds));
+if (!('summary' in coverage)) {
+  console.log(coverage.firstRecordedAt, coverage.retrievableThrough, Object.keys(coverage.kinds));
+}
 ```
 
 Sample output of the first loop:
@@ -212,17 +216,83 @@ Three things to know before polling these:
   days back and 60 history requests an hour; a free key sees 30 days and 600.
   A day outside the window is a 403 `ApiError` whose
   `body.error.earliestAllowedDate` names the first day you may ask for. Over
-  the budget is a 429 whose `Retry-After` can be most of an hour; with the
-  default `retry.on429` the client sleeps that long before trying again, so a
-  poller that would rather fail fast passes `retry: { on429: false }` and reads
-  `err.retryAfterMs`.
+  the budget is a 429 whose `Retry-After` can be most of an hour. The client
+  will not sleep that long: past `retry.maxRetryAfterMs` (120000) it stops
+  retrying and throws `RateLimitError` with `retryAfterMs` set, so a poller
+  fails fast by default rather than looking hung. A REST 429, which asks for
+  seconds, is still ridden out.
 - **Today is not final.** The default cache leaves `changes` and `daily`
   uncached and keeps `coverage` for an hour. A completed day never changes, so
   cache it yourself for as long as you like.
 
 `tp.raw.getEntityHistory(id, query)`, `getEntityHistoryDaily(id, query)` and
 `getEntityHistoryCoverage(id)` are the underlying calls.
 
+### Backfilling: span, paging, and the budget
+
+The three calls above are one request each. A backfill is not one request, and
+the three things it needs are here rather than in your code.
+
+```js
+import { BudgetExhaustedError, ThemeParks } from 'themeparks';
+
+const DISNEYLAND = '7340550b-c14d-4def-80bb-acdb51d49a66';
+const tp = new ThemeParks({ apiKey: process.env.THEMEPARKS_API_KEY });
+const history = tp.entity(DISNEYLAND).history;
+
+// What exists, and what your key may read. The same three fields whether the
+// id is a park or a single ride.
+const span = await history.span();
+// -> { archiveFrom: '2021-07-03', recordedTo: '2026-09-22', retrievableThrough: '2026-09-23' }
+
+// Pages until the server stops offering a `next`, yielding as it goes.
+for await (const { entityId, row } of history.days({
+  from: span.archiveFrom,
+  to: span.retrievableThrough,
+})) {
+  console.log(entityId, row.date, row.operatingMinutes, row.standby?.p50);
+}
+```
+
+**Ask the park, not the rides.** Both history endpoints answer every entity in
+a park in one request. Pulling the same data ride by ride is around a hundred
+times more calls against the same budget. Pass a park id and `days()` takes the
+cheap path; every row is tagged with the entity it came from, which is the only
+thing you give up.
+
+**Bound the range with `retrievableThrough`, not `recordedTo`.** The first is
+what your key may read, the second is what the archive holds. They differ on
+every plan below the top one, and asking past the entitlement is how a long run
+ends in 403s.
+
+**`days()` yields, it does not collect.** Nothing accumulates, so the only thing
+that grows is whatever you write the rows to.
+
+**The budget is hourly.** When it runs out the server asks for a wait the client
+will not sit through, and `days()` throws `BudgetExhaustedError` carrying
+`retryAfterMs`, so you can write down where you got to:
+
+```js
+let lastDay = null;
+try {
+  for await (const { entityId, row } of history.days({ from, to })) {
+    write(entityId, row);
+    lastDay = row.date;
+  }
+} catch (error) {
+  if (!(error instanceof BudgetExhaustedError)) throw error;
+  checkpoint(lastDay);
+  console.error(`resume in ${Math.round(error.retryAfterMs / 1000)}s`);
+}
+```
+
+`history.changeRows(query)` is the same treatment for `changes`: one flattened
+stream of `{ entityId, row }` whether you asked a park or a ride.
+
+A complete backfill with resume and CSV output is in
+[`examples/backfill.mjs`](examples/backfill.mjs). It pulled Disneyland Resort's
+whole daily archive, 98,452 rows, in one run.
+
 ## Low-level escape hatch
 
 Every ergonomic helper is built on top of `tp.raw`, which is a thin, typed 1:1 wrapper over the OpenAPI operations. Use it directly when you want the raw response shape:

examples/backfill.mjs

@@ -0,0 +1,174 @@
+#!/usr/bin/env node
+/**
+ * Pull a park's whole daily history into a file, and survive the budget.
+ *
+ *   node examples/backfill.mjs 7340550b-c14d-4def-80bb-acdb51d49a66
+ *   node examples/backfill.mjs --format csv PARK_ID_A PARK_ID_B
+ *
+ * The key comes from --api-key or the THEMEPARKS_API_KEY environment variable.
+ *
+ * Three things this shows that are easy to get wrong by hand:
+ *
+ * 1. It asks the PARK, not the rides. Both history endpoints answer every
+ *    entity in a park in one request, so a park-level backfill of a large
+ *    resort is around a hundred times fewer calls than the same data pulled
+ *    ride by ride.
+ *
+ * 2. It bounds the range with span().retrievableThrough, not with what the
+ *    archive holds. Those are different dates on every plan below the top one,
+ *    and asking past the entitlement is how a long backfill ends in 403s.
+ *
+ * 3. It checkpoints. The history budget is hourly, so a spent one can be most
+ *    of an hour from resetting. The SDK raises BudgetExhaustedError rather
+ *    than sleeping through that; this writes down the last day it wrote and
+ *    exits 75 (EX_TEMPFAIL), the code that makes a cron or a systemd timer
+ *    retry rather than alert.
+ *
+ * Re-running picks up from the checkpoint. It re-reads the last day on
+ * purpose: a page can end mid-day, and one duplicate day is cheaper to
+ * de-duplicate than a missing one is to notice.
+ */
+
+import {
+  appendFileSync,
+  existsSync,
+  mkdirSync,
+  readFileSync,
+  rmSync,
+  statSync,
+  writeFileSync,
+} from 'node:fs';
+import { join } from 'node:path';
+import { parseArgs } from 'node:util';
+import { BudgetExhaustedError, ThemeParks } from 'themeparks';
+
+const EX_TEMPFAIL = 75;
+
+const CSV_COLUMNS = [
+  'entityId',
+  'date',
+  'firstOperatingAt',
+  'lastClosedAt',
+  'operatingMinutes',
+  'downMinutes',
+  'showCount',
+  'changes',
+  'standbyMin',
+  'standbyP50',
+  'standbyMean',
+  'standbyP90',
+  'standbyMax',
+  'singleRiderP50',
+  'singleRiderMax',
+];
+
+/** Flatten the nested standby/singleRider statistics into one wide row. */
+function csvRow(entityId, row) {
+  const s = row.standby;
+  const sr = row.singleRider;
+  const cells = [
+    entityId,
+    row.date,
+    row.firstOperatingAt ?? '',
+    row.lastClosedAt ?? '',
+    row.operatingMinutes,
+    row.downMinutes,
+    row.showCount ?? '',
+    row.changes,
+    s?.min ?? '',
+    s?.p50 ?? '',
+    s?.mean ?? '',
+    s?.p90 ?? '',
+    s?.max ?? '',
+    sr?.p50 ?? '',
+    sr?.max ?? '',
+  ];
+  // No field here can contain a comma or a quote, so this stays a join rather
+  // than pulling in a CSV writer.
+  return cells.join(',') + '\n';
+}
+
+async function backfillPark(tp, parkId, outDir, format) {
+  const history = tp.entity(parkId).history;
+  const span = await history.span();
+
+  const outPath = join(outDir, `${parkId}.${format === 'csv' ? 'csv' : 'ndjson'}`);
+  const checkpointPath = join(outDir, `${parkId}.checkpoint`);
+
+  const resuming = existsSync(checkpointPath);
+  const hasRows = existsSync(outPath) && statSync(outPath).size > 0;
+  const from = resuming ? readFileSync(checkpointPath, 'utf8').trim() : span.archiveFrom;
+  const to = span.retrievableThrough;
+
+  console.error(`${parkId}: ${from} .. ${to}${resuming ? ' (resumed)' : ''} -> ${outPath}`);
+
+  if (format === 'csv' && !hasRows) {
+    writeFileSync(outPath, CSV_COLUMNS.join(',') + '\n');
+  }
+
+  let written = 0;
+  let lastDay = null;
+  try {
+    for await (const { entityId, row } of history.days({ from, to })) {
+      appendFileSync(
+        outPath,
+        format === 'csv' ? csvRow(entityId, row) : JSON.stringify({ entityId, ...row }) + '\n',
+      );
+      written++;
+      lastDay = row.date;
+      if (written % 5000 === 0) console.error(`  ${written} rows, at ${lastDay}`);
+    }
+  } catch (error) {
+    if (!(error instanceof BudgetExhaustedError)) throw error;
+    if (lastDay !== null) writeFileSync(checkpointPath, lastDay);
+    const seconds = Math.round((error.retryAfterMs ?? 0) / 1000);
+    console.error(
+      `  budget spent after ${written} rows at ${lastDay}; rerun in ${seconds}s to continue`,
+    );
+    return EX_TEMPFAIL;
+  }
+
+  rmSync(checkpointPath, { force: true });
+  console.error(`  done: ${written} rows`);
+  return 0;
+}
+
+async function main() {
+  const { values, positionals } = parseArgs({
+    allowPositionals: true,
+    options: {
+      'api-key': { type: 'string' },
+      format: { type: 'string', default: 'ndjson' },
+      out: { type: 'string', default: '.' },
+    },
+  });
+
+  const apiKey = values['api-key'] ?? process.env.THEMEPARKS_API_KEY;
+  if (!apiKey) {
+    console.error('no key: pass --api-key or set THEMEPARKS_API_KEY');
+    return 2;
+  }
+  if (positionals.length === 0) {
+    console.error('usage: node examples/backfill.mjs [--format csv] [--out DIR] PARK_ID...');
+    return 2;
+  }
+  if (values.format !== 'ndjson' && values.format !== 'csv') {
+    console.error(`unknown format ${values.format}; use ndjson or csv`);
+    return 2;
+  }
+
+  mkdirSync(values.out, { recursive: true });
+
+  // One client for every park: the connection pool is worth reusing, and the
+  // budget is per account either way.
+  const tp = new ThemeParks({ apiKey, userAgent: 'themeparks-backfill-example/1' });
+  for (const parkId of positionals) {
+    const status = await backfillPark(tp, parkId, values.out, values.format);
+    // Stop at the first exhausted budget. Carrying on to the next park only
+    // spends the retry-after on 429s.
+    if (status !== 0) return status;
+  }
+  return 0;
+}
+
+process.exitCode = await main();