diff --git a/CHANGELOG.md b/CHANGELOG.md index 11ccfa2..2614faa 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,10 +5,73 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [Unreleased] +## [8.1.0] - 2026-09-23 ### Added +- **`entity(id).history.span()`, `.days()` and `.changeRows()`** — the loop + above the three history calls. + + ```js + const history = tp.entity(DISNEYLAND).history; + const span = await history.span(); + for await (const { entityId, row } of history.days({ + from: span.archiveFrom, + to: span.retrievableThrough, + })) { + // ... + } + ``` + + - `span()` returns `archiveFrom`, `recordedTo` and `retrievableThrough` in + one shape. The underlying coverage documents do not: a park nests them + under `summary`, an entity carries them at the top level under different + names, so without this every caller writes that branch first. + `retrievableThrough` is the end date to bound a backfill by, because it is + what the key may read rather than what the archive holds. + - `days()` pages until the server stops offering a `next`, following that URL + verbatim, and yields `{ entityId, row }` as rows arrive rather than + collecting them. A park's daily call is the one paged call in the family, + so without this a park backfill silently stopped at the first 31 days. + - Both flatten a park envelope and an entity envelope to the same stream, so + a caller writes one loop and does not branch on `'entities' in res`. + - `BudgetExhaustedError` (a `RateLimitError`) is thrown when the history + budget is spent and the server asks for longer than `maxWaitMs` (120000 by + default). It carries `retryAfterMs`, so a backfill can checkpoint and + resume rather than hold a process open for most of an hour. + +- **`examples/backfill.mjs`** — a complete backfill with resume and NDJSON or + CSV output. It pulled Disneyland Resort's whole daily archive, 98,452 rows, + in one run. + +### Fixed + +- **A 429 could park the client for hours.** The transport honoured any + `Retry-After` up to `retry.max` times. That is right for a REST 429, which + asks for seconds, and wrong for a history 429: that budget is hourly, so a + spent one can ask for most of an hour, and three of those is roughly two and + a half hours of a silent process. `RetryConfig` gains `maxRetryAfterMs` + (120000 by default): past it the client does not sleep at all and throws + `RateLimitError` with `retryAfterMs` set. + +- **`EntityHistoryCoverage` was missing the park shape.** + `/entity/{id}/history/coverage` answers a PARK with + `HistoryParkCoverageDocument`, the same way `/history` and `/history/daily` + do, and the type named only `HistoryCoverageDocument`. The two do not + overlap where it counts: a park carries `summary` and `fields`, an entity + carries `firstRecordedAt`, `lastRecordedAt` and `kinds`. A TypeScript user + read `.kinds` off a park's coverage, got `undefined` at runtime, and the + compiler said nothing. The fixture that covered this was hand-written in the + entity shape and named after a park, so it agreed with the code for the same + reason the code was wrong; both coverage fixtures are now captured from + production, and the live smoke test asserts the park shape it actually gets. + +- **The user agent announced the wrong version.** `PACKAGE_VERSION` was still + `7.0.0-alpha.0` in a package at `8.0.0`, so every request announced a version + a major old and nothing failed. A gate test now asserts the `User-Agent` the + server actually receives carries the version `package.json` declares, so + forgetting the bump is a red test rather than a quiet lie in a header. + - **`apiKey` client option.** Sent as the `X-API-Key` header on every request. Every endpoint still answers without one; a key raises the limits, which matters for the history endpoints (30 days of history and 600 diff --git a/README.md b/README.md index 0119a67..6ad1d20 100644 --- a/README.md +++ b/README.md @@ -49,15 +49,15 @@ Jungle Cruise 40 min `new ThemeParks(options)` takes the following keyword options: -| Option | Type | Default | Purpose | -| ----------- | ----------------------------------- | -------------------------------- | ----------------------------------------------------------------------------------------------------- | -| `baseUrl` | `string` | `https://api.themeparks.wiki/v1` | API base URL (point at a mock / staging if you need to). | -| `userAgent` | `string` | `themeparks-sdk-js/` | Sent as the `User-Agent` header. Set this to identify your app. | -| `apiKey` | `string` | none | API key from api.themeparks.wiki, sent as `X-API-Key`. Optional; a key raises the limits. | -| `fetch` | `typeof fetch` | `globalThis.fetch` | Custom fetch implementation. Useful for logging, mocking, or older runtimes. | -| `timeoutMs` | `number` | `10000` | Per-request timeout in milliseconds. | -| `retry` | `Partial` | `{ max: 3, on429: true }` | Retry/backoff behavior. `max` counts retries **beyond** the initial attempt (so `3` = up to 4 total). | -| `cache` | `Cache \| false \| { maxEntries? }` | in-memory LRU | See [Caching](#caching) below. `false` disables caching entirely. | +| Option | Type | Default | Purpose | +| ----------- | ----------------------------------- | -------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `baseUrl` | `string` | `https://api.themeparks.wiki/v1` | API base URL (point at a mock / staging if you need to). | +| `userAgent` | `string` | `themeparks-sdk-js/` | Sent as the `User-Agent` header. Set this to identify your app. | +| `apiKey` | `string` | none | API key from api.themeparks.wiki, sent as `X-API-Key`. Optional; a key raises the limits. | +| `fetch` | `typeof fetch` | `globalThis.fetch` | Custom fetch implementation. Useful for logging, mocking, or older runtimes. | +| `timeoutMs` | `number` | `10000` | Per-request timeout in milliseconds. | +| `retry` | `Partial` | `{ max: 3, on429: true, maxRetryAfterMs: 120000 }` | Retry/backoff behavior. `max` counts retries **beyond** the initial attempt (so `3` = up to 4 total). `maxRetryAfterMs` is the longest `Retry-After` the client will sleep through; past it you get `RateLimitError` instead of a silent wait. | +| `cache` | `Cache \| false \| { maxEntries? }` | in-memory LRU | See [Caching](#caching) below. `false` disables caching entirely. | Example: @@ -189,9 +189,13 @@ if (!('entities' in week)) { for (const d of week.days) console.log(d.date, d.operatingMinutes, d.standby?.p50); } -// Which days, and which live-data fields, are held at all. +// Which days, and which live-data fields, are held at all. A PARK answers the +// park document here too (summary + fields + entities), so narrow, or use +// span() below, which reads both to one shape. const coverage = await barnstormer.history.coverage(); -console.log(coverage.firstRecordedAt, coverage.retrievableThrough, Object.keys(coverage.kinds)); +if (!('summary' in coverage)) { + console.log(coverage.firstRecordedAt, coverage.retrievableThrough, Object.keys(coverage.kinds)); +} ``` Sample output of the first loop: @@ -212,10 +216,11 @@ Three things to know before polling these: days back and 60 history requests an hour; a free key sees 30 days and 600. A day outside the window is a 403 `ApiError` whose `body.error.earliestAllowedDate` names the first day you may ask for. Over - the budget is a 429 whose `Retry-After` can be most of an hour; with the - default `retry.on429` the client sleeps that long before trying again, so a - poller that would rather fail fast passes `retry: { on429: false }` and reads - `err.retryAfterMs`. + the budget is a 429 whose `Retry-After` can be most of an hour. The client + will not sleep that long: past `retry.maxRetryAfterMs` (120000) it stops + retrying and throws `RateLimitError` with `retryAfterMs` set, so a poller + fails fast by default rather than looking hung. A REST 429, which asks for + seconds, is still ridden out. - **Today is not final.** The default cache leaves `changes` and `daily` uncached and keeps `coverage` for an hour. A completed day never changes, so cache it yourself for as long as you like. @@ -223,6 +228,71 @@ Three things to know before polling these: `tp.raw.getEntityHistory(id, query)`, `getEntityHistoryDaily(id, query)` and `getEntityHistoryCoverage(id)` are the underlying calls. +### Backfilling: span, paging, and the budget + +The three calls above are one request each. A backfill is not one request, and +the three things it needs are here rather than in your code. + +```js +import { BudgetExhaustedError, ThemeParks } from 'themeparks'; + +const DISNEYLAND = '7340550b-c14d-4def-80bb-acdb51d49a66'; +const tp = new ThemeParks({ apiKey: process.env.THEMEPARKS_API_KEY }); +const history = tp.entity(DISNEYLAND).history; + +// What exists, and what your key may read. The same three fields whether the +// id is a park or a single ride. +const span = await history.span(); +// -> { archiveFrom: '2021-07-03', recordedTo: '2026-09-22', retrievableThrough: '2026-09-23' } + +// Pages until the server stops offering a `next`, yielding as it goes. +for await (const { entityId, row } of history.days({ + from: span.archiveFrom, + to: span.retrievableThrough, +})) { + console.log(entityId, row.date, row.operatingMinutes, row.standby?.p50); +} +``` + +**Ask the park, not the rides.** Both history endpoints answer every entity in +a park in one request. Pulling the same data ride by ride is around a hundred +times more calls against the same budget. Pass a park id and `days()` takes the +cheap path; every row is tagged with the entity it came from, which is the only +thing you give up. + +**Bound the range with `retrievableThrough`, not `recordedTo`.** The first is +what your key may read, the second is what the archive holds. They differ on +every plan below the top one, and asking past the entitlement is how a long run +ends in 403s. + +**`days()` yields, it does not collect.** Nothing accumulates, so the only thing +that grows is whatever you write the rows to. + +**The budget is hourly.** When it runs out the server asks for a wait the client +will not sit through, and `days()` throws `BudgetExhaustedError` carrying +`retryAfterMs`, so you can write down where you got to: + +```js +let lastDay = null; +try { + for await (const { entityId, row } of history.days({ from, to })) { + write(entityId, row); + lastDay = row.date; + } +} catch (error) { + if (!(error instanceof BudgetExhaustedError)) throw error; + checkpoint(lastDay); + console.error(`resume in ${Math.round(error.retryAfterMs / 1000)}s`); +} +``` + +`history.changeRows(query)` is the same treatment for `changes`: one flattened +stream of `{ entityId, row }` whether you asked a park or a ride. + +A complete backfill with resume and CSV output is in +[`examples/backfill.mjs`](examples/backfill.mjs). It pulled Disneyland Resort's +whole daily archive, 98,452 rows, in one run. + ## Low-level escape hatch Every ergonomic helper is built on top of `tp.raw`, which is a thin, typed 1:1 wrapper over the OpenAPI operations. Use it directly when you want the raw response shape: diff --git a/examples/backfill.mjs b/examples/backfill.mjs new file mode 100644 index 0000000..545a9b5 --- /dev/null +++ b/examples/backfill.mjs @@ -0,0 +1,174 @@ +#!/usr/bin/env node +/** + * Pull a park's whole daily history into a file, and survive the budget. + * + * node examples/backfill.mjs 7340550b-c14d-4def-80bb-acdb51d49a66 + * node examples/backfill.mjs --format csv PARK_ID_A PARK_ID_B + * + * The key comes from --api-key or the THEMEPARKS_API_KEY environment variable. + * + * Three things this shows that are easy to get wrong by hand: + * + * 1. It asks the PARK, not the rides. Both history endpoints answer every + * entity in a park in one request, so a park-level backfill of a large + * resort is around a hundred times fewer calls than the same data pulled + * ride by ride. + * + * 2. It bounds the range with span().retrievableThrough, not with what the + * archive holds. Those are different dates on every plan below the top one, + * and asking past the entitlement is how a long backfill ends in 403s. + * + * 3. It checkpoints. The history budget is hourly, so a spent one can be most + * of an hour from resetting. The SDK raises BudgetExhaustedError rather + * than sleeping through that; this writes down the last day it wrote and + * exits 75 (EX_TEMPFAIL), the code that makes a cron or a systemd timer + * retry rather than alert. + * + * Re-running picks up from the checkpoint. It re-reads the last day on + * purpose: a page can end mid-day, and one duplicate day is cheaper to + * de-duplicate than a missing one is to notice. + */ + +import { + appendFileSync, + existsSync, + mkdirSync, + readFileSync, + rmSync, + statSync, + writeFileSync, +} from 'node:fs'; +import { join } from 'node:path'; +import { parseArgs } from 'node:util'; +import { BudgetExhaustedError, ThemeParks } from 'themeparks'; + +const EX_TEMPFAIL = 75; + +const CSV_COLUMNS = [ + 'entityId', + 'date', + 'firstOperatingAt', + 'lastClosedAt', + 'operatingMinutes', + 'downMinutes', + 'showCount', + 'changes', + 'standbyMin', + 'standbyP50', + 'standbyMean', + 'standbyP90', + 'standbyMax', + 'singleRiderP50', + 'singleRiderMax', +]; + +/** Flatten the nested standby/singleRider statistics into one wide row. */ +function csvRow(entityId, row) { + const s = row.standby; + const sr = row.singleRider; + const cells = [ + entityId, + row.date, + row.firstOperatingAt ?? '', + row.lastClosedAt ?? '', + row.operatingMinutes, + row.downMinutes, + row.showCount ?? '', + row.changes, + s?.min ?? '', + s?.p50 ?? '', + s?.mean ?? '', + s?.p90 ?? '', + s?.max ?? '', + sr?.p50 ?? '', + sr?.max ?? '', + ]; + // No field here can contain a comma or a quote, so this stays a join rather + // than pulling in a CSV writer. + return cells.join(',') + '\n'; +} + +async function backfillPark(tp, parkId, outDir, format) { + const history = tp.entity(parkId).history; + const span = await history.span(); + + const outPath = join(outDir, `${parkId}.${format === 'csv' ? 'csv' : 'ndjson'}`); + const checkpointPath = join(outDir, `${parkId}.checkpoint`); + + const resuming = existsSync(checkpointPath); + const hasRows = existsSync(outPath) && statSync(outPath).size > 0; + const from = resuming ? readFileSync(checkpointPath, 'utf8').trim() : span.archiveFrom; + const to = span.retrievableThrough; + + console.error(`${parkId}: ${from} .. ${to}${resuming ? ' (resumed)' : ''} -> ${outPath}`); + + if (format === 'csv' && !hasRows) { + writeFileSync(outPath, CSV_COLUMNS.join(',') + '\n'); + } + + let written = 0; + let lastDay = null; + try { + for await (const { entityId, row } of history.days({ from, to })) { + appendFileSync( + outPath, + format === 'csv' ? csvRow(entityId, row) : JSON.stringify({ entityId, ...row }) + '\n', + ); + written++; + lastDay = row.date; + if (written % 5000 === 0) console.error(` ${written} rows, at ${lastDay}`); + } + } catch (error) { + if (!(error instanceof BudgetExhaustedError)) throw error; + if (lastDay !== null) writeFileSync(checkpointPath, lastDay); + const seconds = Math.round((error.retryAfterMs ?? 0) / 1000); + console.error( + ` budget spent after ${written} rows at ${lastDay}; rerun in ${seconds}s to continue`, + ); + return EX_TEMPFAIL; + } + + rmSync(checkpointPath, { force: true }); + console.error(` done: ${written} rows`); + return 0; +} + +async function main() { + const { values, positionals } = parseArgs({ + allowPositionals: true, + options: { + 'api-key': { type: 'string' }, + format: { type: 'string', default: 'ndjson' }, + out: { type: 'string', default: '.' }, + }, + }); + + const apiKey = values['api-key'] ?? process.env.THEMEPARKS_API_KEY; + if (!apiKey) { + console.error('no key: pass --api-key or set THEMEPARKS_API_KEY'); + return 2; + } + if (positionals.length === 0) { + console.error('usage: node examples/backfill.mjs [--format csv] [--out DIR] PARK_ID...'); + return 2; + } + if (values.format !== 'ndjson' && values.format !== 'csv') { + console.error(`unknown format ${values.format}; use ndjson or csv`); + return 2; + } + + mkdirSync(values.out, { recursive: true }); + + // One client for every park: the connection pool is worth reusing, and the + // budget is per account either way. + const tp = new ThemeParks({ apiKey, userAgent: 'themeparks-backfill-example/1' }); + for (const parkId of positionals) { + const status = await backfillPark(tp, parkId, values.out, values.format); + // Stop at the first exhausted budget. Carrying on to the next park only + // spends the retry-after on 429s. + if (status !== 0) return status; + } + return 0; +} + +process.exitCode = await main(); diff --git a/package-lock.json b/package-lock.json index 107fa2a..f5540fc 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "themeparks", - "version": "7.1.0", + "version": "8.1.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "themeparks", - "version": "7.1.0", + "version": "8.1.0", "license": "MIT", "devDependencies": { "@types/node": "^26.4.1", diff --git a/package.json b/package.json index f080937..e27822b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "themeparks", - "version": "8.0.0", + "version": "8.1.0", "description": "Official SDK for the ThemeParks.wiki API", "license": "MIT", "repository": "github:ThemeParks/ThemeParks_JavaScript", diff --git a/src/_generated/schema.ts b/src/_generated/schema.ts index 6f3ce82..7e47e6e 100644 --- a/src/_generated/schema.ts +++ b/src/_generated/schema.ts @@ -11,7 +11,10 @@ export interface paths { path?: never; cookie?: never; }; - /** GET /v1/destinations */ + /** + * GET /v1/destinations + * @description Every destination we hold, each with its parks as id and name. This is the entry point: take a destination or park id from here and ask GET /v1/entity/{id}/children for what is inside it, /live for what it is doing, or /schedule for when it is open. Soft-deleted parks are excluded. The list is small and changes when a resort opens or closes a park, which is rarely — cached `public, max-age=300, s-maxage=3600`, and there is no reason to ask for it more than once a run. + */ get: operations["getAllDestinations"]; put?: never; post?: never; @@ -28,7 +31,10 @@ export interface paths { path?: never; cookie?: never; }; - /** GET /v1/entity/{id} */ + /** + * GET /v1/entity/{id} + * @description The entity document: what this thing IS, not what it is doing. Name, entityType, timezone, location, its place in the destination/park hierarchy, and the ride's own attributes. `id` is an entity's UUID, its slug, or — for a destination only — its upstream external id. An id that resolves to nothing is 404, including one malformed enough that the query itself rejects it. Attributes recorded as tags are spread into the document as top-level keys named after the tag — `minimumHeight`, `mayGetWet` and so on — so the set of keys varies by entity and by park; read the ones you need and ignore the rest rather than expecting a fixed shape. For what the entity is doing right now, GET /v1/entity/{id}/live. Cached `public, max-age=300, s-maxage=3600`: this changes when a park changes its data, which is rarely, and polling it faster buys requests rather than freshness. + */ get: operations["getEntityById"]; put?: never; post?: never; @@ -45,7 +51,10 @@ export interface paths { path?: never; cookie?: never; }; - /** GET /v1/entity/{id}/children */ + /** + * GET /v1/entity/{id}/children + * @description The entities beneath this one, as a flat array — id, name, entityType, slug, external id, coordinates and parentId each. `id` is an entity's UUID, its slug, or — for a destination only — its upstream external id. An id that resolves to nothing is 404, including one malformed enough that the query itself rejects it. HOW FAR DOWN depends on what you asked about, and this is the part clients get wrong: a DESTINATION returns every entity in the destination and a PARK returns every entity in the park — the whole subtree, not one level — while every other entityType returns its direct children only. Rebuild the hierarchy from `parentId` rather than assuming one level. Soft-deleted entities are never included. Cached `public, max-age=300, s-maxage=3600`. + */ get: operations["getEntityChildren"]; put?: never; post?: never; @@ -122,7 +131,10 @@ export interface paths { path?: never; cookie?: never; }; - /** GET /v1/entity/{id}/live */ + /** + * GET /v1/entity/{id}/live + * @description Live data for this entity AND everything beneath it, in one call: wait times, ride status, return-time and boarding-group windows, and show times. `id` is an entity's UUID, its slug, or — for a destination only — its upstream external id. An id that resolves to nothing is 404, including one malformed enough that the query itself rejects it. An entity with nothing to report is absent from `liveData` rather than present and empty, so treat a missing entry as no data rather than as a closed ride. `?entityType=ATTRACTION,SHOW` filters the array to those types (comma-separated). Cached `public, max-age=60, s-maxage=60`: the collectors run on a cadence, and a faster poll returns the same body. For what an entity reported in the past rather than now, GET /v1/entity/{id}/history. + */ get: operations["getEntityLiveData"]; put?: never; post?: never; @@ -139,7 +151,10 @@ export interface paths { path?: never; cookie?: never; }; - /** GET /v1/entity/{id}/schedule */ + /** + * GET /v1/entity/{id}/schedule + * @description Opening hours for this entity, for the default window: today through the next 30 days, anchored to the entity's own timezone rather than the caller's or UTC. `id` is an entity's UUID, its slug, or — for a destination only — its upstream external id. An id that resolves to nothing is 404, including one malformed enough that the query itself rejects it. Days the park has not published are absent from the array rather than present as closed. For a specific month, including a past one, use GET /v1/entity/{id}/schedule/{year}/{month}. Cached `public, max-age=300, s-maxage=3600`. + */ get: operations["getEntitySchedule"]; put?: never; post?: never; @@ -156,7 +171,10 @@ export interface paths { path?: never; cookie?: never; }; - /** GET /v1/entity/{id}/schedule/{year}/{month} */ + /** + * GET /v1/entity/{id}/schedule/{year}/{month} + * @description Opening hours for one calendar month in the entity's own timezone. `month` is TWO digits, 01-12 — `/2026/9` is 400, `/2026/09` is right — and `year` is a four-digit year between 1970 and 2150; anything else is 400 before the entity is even looked up. `id` is an entity's UUID, its slug, or — for a destination only — its upstream external id. An id that resolves to nothing is 404, including one malformed enough that the query itself rejects it. Past months are served from what was recorded at the time and are not backfilled, so a month before this entity was collected comes back empty rather than 404. Days the park has not published are absent rather than present as closed. Cached `public, max-age=300, s-maxage=3600`. + */ get: operations["getEntityScheduleYearMonth"]; put?: never; post?: never; @@ -249,6 +267,25 @@ export interface components { } & { [key: string]: unknown; }; + /** @description 400: a path parameter is the wrong shape. Checked before the entity is looked up, so a bad `year` or `month` answers 400 whether or not the id exists. */ + EntityInvalidParameter: { + /** + * @description Always false. Branch on this rather than on the status alone. + * @enum {boolean} + */ + success: false; + error: { + /** @enum {string} */ + type: "Bad request"; + /** @description Says which parameter was rejected and what it must look like, e.g. "Month must be a two-digit number (01-12)". */ + message: string; + /** + * @description Repeats the HTTP status. + * @enum {integer} + */ + code?: 400; + }; + }; EntityLiveData: { /** @description Entity identifier */ id: string; @@ -282,6 +319,25 @@ export interface components { /** @description Longitude coordinate of the entity location */ longitude?: number | null; }; + /** @description 404: nothing resolved from `id`. An id is a UUID, a slug, or — for a destination — its upstream external id; an id malformed enough that the lookup itself rejects it answers 404 as well, rather than 400 or 500. The body is the same whether the entity never existed or has been removed, so it cannot be used to tell those apart. */ + EntityNotFound: { + /** + * @description Always false. Branch on this rather than on the status alone. + * @enum {boolean} + */ + success: false; + error: { + /** @enum {string} */ + type: "Not found"; + /** @description Names the id that did not resolve, e.g. "Entity 00000000-0000-0000-0000-000000000000 not found". */ + message: string; + /** + * @description Repeats the HTTP status. + * @enum {integer} + */ + code?: 404; + }; + }; EntityScheduleResponse: { /** @description Entity identifier */ id?: string; @@ -457,12 +513,12 @@ export interface components { message: string; }; }; - /** @description 400: the range spans more than 31 park-local days. Split it into consecutive calls. */ + /** @description 400: the range is longer than the path allows. The cap is NOT the same on every path, and this one error type is returned by all of them: GET /v1/entity/{id}/history allows 31 park-local days for a single entity and 1 for a PARK; GET /v1/entity/{id}/history/daily allows 3660 (ten years) for a single entity, and for a PARK serves 31 days a page and gives you `next` for the rest. The message names the cap that applied. Split the ask into consecutive calls. */ HistoryErrorRangeTooLong: { error: { /** @enum {string} */ type: "RANGE_TOO_LONG"; - /** @description e.g. "A history call covers at most 31 park-local days (2026-01-01 to 2026-03-01 is 60). Ask for a shorter range." */ + /** @description Names the cap that was exceeded and the span that was asked for, e.g. "A history call covers at most 31 park-local days (2026-01-01 to 2026-03-01 is 60). Ask for a shorter range." The number is the cap for the path that answered, not a constant: read it from the message rather than hard-coding 31. */ message: string; }; }; @@ -497,11 +553,111 @@ export interface components { * @description Start of the range (UTC, whole seconds). The state below is effective from this instant. */ time: string; + /** + * Format: date-time + * @description The UTC instant (whole seconds) at which this state was actually OBSERVED, as opposed to `time`, which is the start of the range you asked for. The two are different questions and only this one tells you whether to trust the state. + * + * A state carried forward from before the range has an `observedAt` BEFORE `time` — sometimes long before, because the lookup is deliberately unbounded and returns the last reading of each kind at any age. So a day we hold nothing for still reports the newest state we ever saw, which is usually right and occasionally very wrong: a ride whose feed simply stopped mid-operation carries its last wait time forward indefinitely. + * + * Compare it against `range.from` to decide. Equal to or after the range start means the state was seen inside the range. Before it means carried forward, and how far back you tolerate is yours to choose — a reading an hour before the day began is ordinary, one from three weeks earlier is not evidence about this day. Absent means nothing survives to carry: we can say nothing at all about the state at the start of this range. + * + * It is the NEWEST instant any kind in this opening was seen. Kinds can be observed at different moments, so an older kind may be staler than this field suggests; treat it as the most generous reading of the opening's age, not a guarantee about every key. Days that genuinely have data are better read from the rows, which carry their own `time`. + */ + observedAt?: string; /** @description Live status at the start of the range; null when unknown. */ status?: string | null; queue?: components["schemas"]["LiveQueue"]; showtimes?: components["schemas"]["LiveShowTime"][] | null; }; + /** @description How far back each entity's record of one field goes, as counts. Calendar years, measured from the day the document was built (`summary.measuredOn`). Counts, not a percentage or an average: a single figure for a park would hide that most of one park's standby entities have two to four years while its oldest reach back to the start of the archive. */ + HistoryParkCoverageDepth: { + /** @description Entities whose record of this field goes back four calendar years or more. */ + fourYearsPlus: number; + /** @description Entities with at least two and under four calendar years. */ + twoToFourYears: number; + /** @description Entities with at least one and under two calendar years. */ + oneToTwoYears: number; + /** @description Entities with under a calendar year, typically something that opened recently rather than a gap. */ + underOneYear: number; + }; + /** @description What history is held across a whole PARK. GET /v1/entity/{id}/history/coverage returns this shape when the entity is a PARK; every other entityType, a DESTINATION included, returns HistoryCoverageDocument for that entity alone. A park records nothing itself, so without this rollup the honest-looking answer for a park would be "nothing". It reports depth and breadth only: it does not detect missing days and does not judge whether a recorded value was correct. */ + HistoryParkCoverageDocument: { + id: string; + name: string; + entityType: string; + parentId: string | null; + destinationId: string | null; + /** @description IANA timezone the park-local days are resolved in. The park's children inherit it. */ + timezone: string; + summary: components["schemas"]["HistoryParkCoverageSummary"]; + /** @description Keyed by live-data field path, in live-data order. */ + fields: { + [key: string]: components["schemas"]["HistoryParkCoverageField"]; + }; + /** @description Every entity of the park history is held for. */ + entities: components["schemas"]["HistoryParkCoverageEntity"][]; + }; + /** @description One entity of the park that history is held for. An entity nothing is held for is ABSENT rather than listed as empty: it is usually a parade, a show or a land, which never had a queue to record, and listing it would read as a gap. */ + HistoryParkCoverageEntity: { + id: string; + name: string; + entityType: string; + /** + * Format: date + * @description The earliest park-local day any field of this entity was recorded. + */ + from: string; + /** + * Format: date + * @description The newest park-local day any field of this entity was recorded. + */ + newest: string; + /** @description The live-data field paths held for this entity, named exactly as the single-entity coverage document names them, so one client type reads both. */ + fields: string[]; + /** @description False when the park no longer lists this entity. Its history is still held and still retrievable, and every count in this document includes it; this says where the entity is now, not what the archive has. */ + stillListed: boolean; + }; + /** @description What one live-data field looks like across the whole park. `entities` counts what is HELD and is never a fraction: entities that have never reported this field are simply absent from the count, because a denominator drawn from entityType would publish parades, shows and lands as missing wait times. */ + HistoryParkCoverageField: { + /** @description How many of the park's entities this field is held for. */ + entities: number; + /** + * Format: date + * @description The earliest park-local day any entity in the park reported this field. + */ + from: string; + /** + * Format: date + * @description The newest park-local day any entity in the park reported it. A day in the past is not staleness: when a park stops publishing a field the ending is recorded, so the span genuinely stops there. + */ + newest: string; + depth: components["schemas"]["HistoryParkCoverageDepth"]; + }; + /** @description The park in four numbers. Every one describes what is held; none is a fraction of a total, and nothing here asserts that anything is missing. */ + HistoryParkCoverageSummary: { + /** @description Entities of this park any live-data history is held for - attractions, restaurants, shows and anything else that has ever reported. Larger than the number with wait times: `fields` breaks it down. Counts entities the park no longer lists as well, since their history is still held; those carry `stillListed: false` in `entities`. */ + entitiesWithData: number; + /** + * Format: date + * @description The earliest park-local day anything in this park was recorded, or null when nothing has been. + */ + archiveFrom: string; + /** + * Format: date + * @description The newest park-local day anything in this park was recorded, or null. + */ + recordedTo: string; + /** + * Format: date + * @description The newest park-local day a caller can actually retrieve. Runs ahead of `recordedTo` by a day or two: the most recent days are served from live data before they are sealed into the archive. Null when nothing is recorded. + */ + retrievableThrough: string; + /** + * Format: date + * @description The park-local day these figures were computed. They move as the archive grows, so a reader comparing two copies of this document needs to know which day each was built. + */ + measuredOn: string; + }; /** @description A day-by-day summary of a whole PARK: every entity of the park that has history, in one call. GET /v1/entity/{id}/history/daily returns THIS shape when the entity is a PARK (entityType: "PARK") and HistoryDailyEnvelope for every other entityType, so a client should branch on the presence of entities[] or on the entity's type. range.from and range.to are always park-local calendar days, and range.to is THIS PAGE's last day rather than the whole range you asked for: a call serves at most 31 park-local days and next carries the rest. */ HistoryParkDailyEnvelope: { id: string; @@ -580,7 +736,7 @@ export interface components { waitTime: number | null; }; RETURN_TIME?: { - state: components["schemas"]["ReturnTimeState"]; + state: components["schemas"]["ReturnTimeState"] | null; /** * Format: date-time * @description Start time of return window @@ -593,15 +749,15 @@ export interface components { returnEnd: string | null; }; PAID_RETURN_TIME?: { - state: components["schemas"]["ReturnTimeState"]; + state: components["schemas"]["ReturnTimeState"] | null; /** Format: date-time */ returnStart: string | null; /** Format: date-time */ returnEnd: string | null; - price: components["schemas"]["PriceData"]; + price: components["schemas"]["PriceData"] | null; }; BOARDING_GROUP?: { - allocationStatus: components["schemas"]["BoardingGroupState"]; + allocationStatus: components["schemas"]["BoardingGroupState"] | null; /** @description Current boarding group start number */ currentGroupStart: number | null; /** @description Current boarding group end number */ @@ -779,6 +935,15 @@ export interface operations { "application/json": components["schemas"]["EntityData"]; }; }; + /** @description Resource not found */ + 404: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["EntityNotFound"]; + }; + }; /** @description Too Many Requests - Rate limit exceeded */ 429: { headers: { @@ -820,6 +985,15 @@ export interface operations { "application/json": components["schemas"]["EntityChildrenResponse"]; }; }; + /** @description Resource not found */ + 404: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["EntityNotFound"]; + }; + }; /** @description Too Many Requests - Rate limit exceeded */ 429: { headers: { @@ -929,7 +1103,7 @@ export interface operations { [name: string]: unknown; }; content: { - "application/json": components["schemas"]["HistoryCoverageDocument"]; + "application/json": components["schemas"]["HistoryCoverageDocument"] | components["schemas"]["HistoryParkCoverageDocument"]; }; }; /** @description Resource not found */ @@ -1034,6 +1208,15 @@ export interface operations { "application/json": components["schemas"]["EntityLiveDataResponse"]; }; }; + /** @description Resource not found */ + 404: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["EntityNotFound"]; + }; + }; /** @description Too Many Requests - Rate limit exceeded */ 429: { headers: { @@ -1075,6 +1258,15 @@ export interface operations { "application/json": components["schemas"]["EntityScheduleResponse"]; }; }; + /** @description Resource not found */ + 404: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["EntityNotFound"]; + }; + }; /** @description Too Many Requests - Rate limit exceeded */ 429: { headers: { @@ -1118,6 +1310,24 @@ export interface operations { "application/json": components["schemas"]["EntityScheduleResponse"]; }; }; + /** @description Bad request */ + 400: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["EntityInvalidParameter"]; + }; + }; + /** @description Resource not found */ + 404: { + headers: { + [name: string]: unknown; + }; + content: { + "application/json": components["schemas"]["EntityNotFound"]; + }; + }; /** @description Too Many Requests - Rate limit exceeded */ 429: { headers: { diff --git a/src/client.ts b/src/client.ts index cc4d866..daef7e4 100644 --- a/src/client.ts +++ b/src/client.ts @@ -2,10 +2,15 @@ import { InMemoryLruCache, ttlForPath, type Cache } from './cache'; import { DestinationsApi } from './ergonomic/destinations'; import { EntityHandle } from './ergonomic/entity'; import { RawClient } from './raw'; -import { Transport, type FetchLike, type RetryConfig } from './transport'; +import { + DEFAULT_MAX_RETRY_AFTER_MS, + Transport, + type FetchLike, + type RetryConfig, +} from './transport'; const DEFAULT_BASE_URL = 'https://api.themeparks.wiki/v1'; -const PACKAGE_VERSION = '7.0.0-alpha.0'; +const PACKAGE_VERSION = '8.1.0'; const DEFAULT_USER_AGENT = `themeparks-sdk-js/${PACKAGE_VERSION}`; export interface ThemeParksOptions { @@ -42,7 +47,11 @@ export class ThemeParks { userAgent: options.userAgent ?? DEFAULT_USER_AGENT, ...(options.apiKey !== undefined ? { apiKey: options.apiKey } : {}), timeoutMs: options.timeoutMs ?? 10_000, - retry: { max: options.retry?.max ?? 3, on429: options.retry?.on429 ?? true }, + retry: { + max: options.retry?.max ?? 3, + on429: options.retry?.on429 ?? true, + maxRetryAfterMs: options.retry?.maxRetryAfterMs ?? DEFAULT_MAX_RETRY_AFTER_MS, + }, fetch: fetchFn, }); this.cache = buildCache(options.cache); diff --git a/src/ergonomic/entity.ts b/src/ergonomic/entity.ts index 4da2d2e..69c532a 100644 --- a/src/ergonomic/entity.ts +++ b/src/ergonomic/entity.ts @@ -1,15 +1,6 @@ import type { components } from '../_generated/schema'; -import type { - Entity, - EntityChildren, - EntityHistory, - EntityHistoryCoverage, - EntityHistoryDaily, - EntityLive, - EntitySchedule, - HistoryQuery, - RawClient, -} from '../raw'; +import type { Entity, EntityChildren, EntityLive, EntitySchedule, RawClient } from '../raw'; +import { HistoryApi } from './history'; export type EntityChild = components['schemas']['EntityChild']; @@ -21,15 +12,6 @@ export interface ScheduleApi { range(start: Date, end: Date): Promise; } -export interface HistoryApi { - /** Every recorded change in the range, one row per change: `GET /entity/{id}/history`. */ - changes(query?: HistoryQuery): Promise; - /** One row per park-local day with operating minutes and wait-time statistics: `GET /entity/{id}/history/daily`. */ - daily(query?: HistoryQuery): Promise; - /** Which days and which live-data fields are held: `GET /entity/{id}/history/coverage`. */ - coverage(): Promise; -} - export class EntityHandle { readonly schedule: ScheduleApi; readonly history: HistoryApi; @@ -43,11 +25,7 @@ export class EntityHandle { month: (year, month) => this.raw.getEntityScheduleMonth(this.id, year, month), range: (start, end) => this.scheduleRange(start, end), }; - this.history = { - changes: (query) => this.raw.getEntityHistory(this.id, query), - daily: (query) => this.raw.getEntityHistoryDaily(this.id, query), - coverage: () => this.raw.getEntityHistoryCoverage(this.id), - }; + this.history = new HistoryApi(raw, id); } get(): Promise { diff --git a/src/ergonomic/history.ts b/src/ergonomic/history.ts new file mode 100644 index 0000000..776012d --- /dev/null +++ b/src/ergonomic/history.ts @@ -0,0 +1,235 @@ +/** + * The loop above the history endpoints. + * + * The raw calls are honest but they hand the caller four jobs: walk the `next` + * links, respect an hourly budget separate from the per-minute one, know that + * asking a PARK returns a different shape from asking a ride, and keep five + * years of rows out of memory. Every customer who buys history writes the same + * loop, and most write it wrong: the first version of our own monitor treated + * a 429 as "no data" and reported all-clear for eight days. + * + * So this does the walking. `days()` pages until the server stops offering a + * `next`, and yields rows rather than returning an array, because a resort's + * five years is not an array. + * + * THE ONE THING THAT MATTERS MOST is that asking about a park uses the PARK + * call. Both history endpoints answer a whole park in one request; asking ride + * by ride costs around a hundred times more for the same data. A caller who + * passes a park id gets the cheap path without having to know the expensive + * one exists. + */ + +import { RateLimitError, type RateLimitErrorInit } from '../errors'; +import type { + EntityHistory, + EntityHistoryCoverage, + EntityHistoryDaily, + HistoryQuery, + RawClient, +} from '../raw'; +import type { components } from '../_generated/schema'; + +type HistoryDailyRow = components['schemas']['HistoryDailyRow']; +type HistoryRow = components['schemas']['HistoryRow']; + +/** + * The hourly history budget is spent and the wait is longer than this client + * will sit through. + * + * Carries `retryAfterMs` from the underlying 429, so a backfill can record + * where it got to and come back after that long rather than holding a process + * open waiting for a budget window to roll. + */ +export class BudgetExhaustedError extends RateLimitError { + constructor(message: string, init: RateLimitErrorInit) { + super(message, init); + this.name = 'BudgetExhaustedError'; + } +} + +/** + * Past this, a 429 is reported rather than waited out. Matches the transport's + * own `retry.maxRetryAfterMs`, so the two agree: the transport stops retrying, + * and this turns what comes back into an error a backfill can act on. + */ +export const DEFAULT_MAX_WAIT_MS = 120_000; + +function asBudgetError(error: unknown, maxWaitMs: number): unknown { + if (!(error instanceof RateLimitError) || error instanceof BudgetExhaustedError) return error; + const wait = error.retryAfterMs; + if (wait === null || wait <= maxWaitMs) return error; + return new BudgetExhaustedError( + `history budget exhausted; retry in ${String(Math.round(wait / 1000))}s ` + + `(longer than maxWaitMs=${String(maxWaitMs)}). Checkpoint and resume.`, + { status: error.status, body: error.body, url: error.url, retryAfterMs: wait }, + ); +} + +/** One daily row, tagged with the entity it belongs to. */ +export interface DailyEntry { + entityId: string; + row: HistoryDailyRow; +} + +/** One recorded change, tagged with the entity it belongs to. */ +export interface ChangeEntry { + entityId: string; + row: HistoryRow; +} + +/** + * The three dates a backfill needs, in one shape for parks and rides. + * + * A park's coverage document nests these under `summary`; an entity's carries + * them at the top level under different names. Without this, every caller + * writes the same branch before they can ask their first question. + */ +export interface HistorySpan { + /** First park-local day the archive holds anything for, or null. */ + archiveFrom: string | null; + /** Newest day in the archive, or null. Runs a few days behind live data. */ + recordedTo: string | null; + /** + * Newest day YOUR key may retrieve, or null. + * + * This, not `recordedTo`, is the end date to bound a backfill by: it is what + * the plan allows rather than what exists, and asking past it is how a long + * run ends in 403s. + */ + retrievableThrough: string | null; +} + +function toSpan(document: EntityHistoryCoverage): HistorySpan { + if ('summary' in document) { + return { + archiveFrom: document.summary.archiveFrom, + recordedTo: document.summary.recordedTo, + retrievableThrough: document.summary.retrievableThrough, + }; + } + return { + archiveFrom: document.firstRecordedAt, + recordedTo: document.lastRecordedAt, + retrievableThrough: document.retrievableThrough, + }; +} + +/** + * A park envelope carries many entities; an entity envelope carries its own + * rows. Both flatten to the same stream, so a caller writes one loop. + */ +function* dailyEntries(envelope: EntityHistoryDaily): Generator { + if ('entities' in envelope) { + for (const entity of envelope.entities) { + for (const row of entity.days) yield { entityId: entity.id, row }; + } + return; + } + for (const row of envelope.days) yield { entityId: envelope.id, row }; +} + +function* changeEntries(envelope: EntityHistory): Generator { + if ('entities' in envelope) { + for (const entity of envelope.entities) { + for (const row of entity.history) yield { entityId: entity.id, row }; + } + return; + } + for (const row of envelope.history) yield { entityId: envelope.id, row }; +} + +export interface BudgetOptions { + /** Past this, a 429 becomes {@link BudgetExhaustedError} instead of a retry. */ + maxWaitMs?: number; +} + +export type DaysOptions = HistoryQuery & BudgetOptions; +export type ChangesOptions = HistoryQuery & BudgetOptions; + +function toQuery(options: HistoryQuery & BudgetOptions): HistoryQuery { + const query: HistoryQuery = {}; + if (options.date !== undefined) query.date = options.date; + if (options.from !== undefined) query.from = options.from; + if (options.to !== undefined) query.to = options.to; + return query; +} + +/** History for one entity id, reached as `tp.entity(id).history`. */ +export class HistoryApi { + constructor( + private readonly raw: RawClient, + private readonly entityId: string, + ) {} + + /** Every recorded change in the range: `GET /entity/{id}/history`. */ + changes(query: HistoryQuery = {}): Promise { + return this.raw.getEntityHistory(this.entityId, query); + } + + /** One row per park-local day: `GET /entity/{id}/history/daily`. */ + daily(query: HistoryQuery = {}): Promise { + return this.raw.getEntityHistoryDaily(this.entityId, query); + } + + /** Which days and which live-data fields are held. */ + coverage(): Promise { + return this.raw.getEntityHistoryCoverage(this.entityId); + } + + /** + * The dates a backfill should run between: one call, and the same three + * fields whether this id is a park or a single ride. + */ + async span(): Promise { + return toSpan(await this.coverage()); + } + + /** + * Every daily row in the range, paged automatically, yielded as it arrives. + * + * Given a park id this uses the park call, which answers every entity in the + * park in one request. Nothing accumulates, so the only thing that grows is + * whatever the caller writes the rows to. + */ + async *days(options: DaysOptions = {}): AsyncGenerator { + const maxWaitMs = options.maxWaitMs ?? DEFAULT_MAX_WAIT_MS; + let envelope: EntityHistoryDaily; + try { + envelope = await this.raw.getEntityHistoryDaily(this.entityId, toQuery(options)); + } catch (error) { + throw asBudgetError(error, maxWaitMs); + } + + for (;;) { + yield* dailyEntries(envelope); + const next = envelope.next; + if (next === null || next === '') return; + try { + // Followed verbatim: the server has already applied every parameter, + // and re-deriving the URL is how a paging loop starts asking for the + // wrong range. + envelope = await this.raw.getUrl(next); + } catch (error) { + throw asBudgetError(error, maxWaitMs); + } + } + } + + /** + * Every recorded change in the range, flattened to one stream. + * + * A park answers one day per call; a single entity answers up to 31 days. + * The caller does not have to know which cap applies: ask for what you want, + * and the API answers or says the range is too long. + */ + async *changeRows(options: ChangesOptions = {}): AsyncGenerator { + const maxWaitMs = options.maxWaitMs ?? DEFAULT_MAX_WAIT_MS; + let envelope: EntityHistory; + try { + envelope = await this.raw.getEntityHistory(this.entityId, toQuery(options)); + } catch (error) { + throw asBudgetError(error, maxWaitMs); + } + yield* changeEntries(envelope); + } +} diff --git a/src/index.ts b/src/index.ts index 698cbd5..71bc6f5 100644 --- a/src/index.ts +++ b/src/index.ts @@ -1,7 +1,13 @@ export { ThemeParks, type ThemeParksOptions } from './client'; export { ApiError, NetworkError, RateLimitError, ThemeParksError, TimeoutError } from './errors'; export { InMemoryLruCache, type Cache } from './cache'; -export type { FetchLike, FetchLikeResponse, RetryConfig, TransportOptions } from './transport'; +export { + DEFAULT_MAX_RETRY_AFTER_MS, + type FetchLike, + type FetchLikeResponse, + type RetryConfig, + type TransportOptions, +} from './transport'; export { RawClient, type Destinations, @@ -15,6 +21,17 @@ export { type HistoryQuery, } from './raw'; export { EntityHandle } from './ergonomic/entity'; +export { + BudgetExhaustedError, + DEFAULT_MAX_WAIT_MS, + HistoryApi, + type BudgetOptions, + type ChangeEntry, + type ChangesOptions, + type DailyEntry, + type DaysOptions, + type HistorySpan, +} from './ergonomic/history'; export { DestinationsApi } from './ergonomic/destinations'; export { currentWaitTime, diff --git a/src/raw.ts b/src/raw.ts index 9f8197b..7b53153 100644 --- a/src/raw.ts +++ b/src/raw.ts @@ -15,7 +15,17 @@ export type EntitySchedule = components['schemas']['EntityScheduleResponse']; export type EntityHistory = | components['schemas']['HistoryEnvelope'] | components['schemas']['HistoryParkRawEnvelope']; -export type EntityHistoryCoverage = components['schemas']['HistoryCoverageDocument']; +/** + * Response of `/entity/{id}/history/coverage`. A PARK answers with + * `HistoryParkCoverageDocument`, exactly as `/history` and `/history/daily` + * do. The two shapes do not overlap where it counts: a park carries `summary` + * and `fields`, an entity carries `firstRecordedAt`, `lastRecordedAt` and + * `kinds`. Narrow on `'summary' in res`, or use `entity(id).history.span()`, + * which reads both to one shape. + */ +export type EntityHistoryCoverage = + | components['schemas']['HistoryCoverageDocument'] + | components['schemas']['HistoryParkCoverageDocument']; /** Response of `/entity/{id}/history/daily`; a PARK answers with `entities[]`, see {@link EntityHistory}. */ export type EntityHistoryDaily = | components['schemas']['HistoryDailyEnvelope'] @@ -84,4 +94,13 @@ export class RawClient { `/entity/${encodeURIComponent(entityId)}/history/daily${queryString(query)}`, ); } + + /** + * GET an absolute URL the API itself handed us, such as a paged response's + * `next`. The server has already applied every parameter; re-deriving the + * URL from its path is how a paging loop starts asking for the wrong range. + */ + getUrl(url: string): Promise { + return this.transport.getUrl(url); + } } diff --git a/src/transport.ts b/src/transport.ts index 49389c3..e4617ae 100644 --- a/src/transport.ts +++ b/src/transport.ts @@ -49,6 +49,18 @@ export interface RetryConfig { max: number; /** If true, HTTP 429 responses are retried (honouring `Retry-After`). */ on429: boolean; + /** + * Longest `Retry-After` this client will sleep through, in milliseconds. + * Defaults to {@link DEFAULT_MAX_RETRY_AFTER_MS}. + * + * A REST 429 asks for seconds and is worth waiting out. A HISTORY 429 is a + * different animal: that budget is hourly, so a spent one can ask for most + * of an hour, and honouring it up to `max` times means a process that sits + * silent for hours and looks hung. Past this cap we do not sleep at all, and + * throw `RateLimitError` carrying `retryAfterMs` so the caller can + * checkpoint and come back. + */ + maxRetryAfterMs?: number; } export interface TransportOptions { @@ -66,6 +78,9 @@ export interface TransportOptions { sleep?: (ms: number) => Promise; } +/** Two minutes: longer than any REST 429 asks for, far short of an hourly budget. */ +export const DEFAULT_MAX_RETRY_AFTER_MS = 120_000; + const defaultSleep = (ms: number): Promise => new Promise((r) => setTimeout(r, ms)); function parseRetryAfter(header: string | null): number | null { @@ -122,11 +137,22 @@ export class Transport { constructor(private readonly opts: TransportOptions) {} async get(path: string): Promise { - return this.request('GET', path); + return this.request('GET', this.opts.baseUrl.replace(/\/$/, '') + path); + } + + /** + * GET an absolute URL the API itself handed us. + * + * Paged history responses carry `next` as an absolute URL with every + * parameter already applied. Re-deriving that from the path would mean + * re-deriving the parameters too, which is how a paging loop quietly starts + * asking for the wrong range. + */ + async getUrl(url: string): Promise { + return this.request('GET', url); } - private async request(method: string, path: string): Promise { - const url = this.opts.baseUrl.replace(/\/$/, '') + path; + private async request(method: string, url: string): Promise { const sleep = this.opts.sleep ?? defaultSleep; let attempt = 0; @@ -171,10 +197,16 @@ export class Transport { const body = await this.safeParseBody(response); const bodyExcerpt = formatBodyExcerpt(body); - if (response.status === 429 && this.opts.retry.on429 && attempt < this.opts.retry.max) { - const retryAfterMs = - parseRetryAfter(response.headers.get('retry-after')) ?? backoff(attempt); - await sleep(retryAfterMs); + const retryAfterMs = parseRetryAfter(response.headers.get('retry-after')); + const cap = this.opts.retry.maxRetryAfterMs ?? DEFAULT_MAX_RETRY_AFTER_MS; + const waitTooLong = retryAfterMs !== null && retryAfterMs > cap; + if ( + response.status === 429 && + this.opts.retry.on429 && + attempt < this.opts.retry.max && + !waitTooLong + ) { + await sleep(retryAfterMs ?? backoff(attempt)); attempt++; continue; } @@ -183,7 +215,7 @@ export class Transport { status: 429, body, url, - retryAfterMs: parseRetryAfter(response.headers.get('retry-after')), + retryAfterMs, ...(bodyExcerpt !== undefined ? { bodyExcerpt } : {}), }); } diff --git a/test/fixtures/mk_attraction_history_coverage.json b/test/fixtures/mk_attraction_history_coverage.json new file mode 100644 index 0000000..a5ca4b6 --- /dev/null +++ b/test/fixtures/mk_attraction_history_coverage.json @@ -0,0 +1,25 @@ +{ + "id": "924a3b2c-6b4b-49e5-99d3-e9dc3f2e8a48", + "name": "The Barnstormer", + "entityType": "ATTRACTION", + "parentId": "75ea578a-adc8-4116-a54d-dccb60765ef9", + "destinationId": "e957da41-3552-4cf6-b636-5babc5cbc4e5", + "timezone": "America/New_York", + "firstRecordedAt": "2021-07-03", + "lastRecordedAt": "2026-09-21", + "retrievableThrough": "2026-09-23", + "kinds": { + "status": { + "first": "2021-07-03", + "last": "2026-09-21" + }, + "queue.STANDBY": { + "first": "2021-07-03", + "last": "2026-09-21" + }, + "queue.RETURN_TIME": { + "first": "2021-10-19", + "last": "2026-09-21" + } + } +} diff --git a/test/fixtures/mk_history_coverage.json b/test/fixtures/mk_history_coverage.json index f202dbb..9478d72 100644 --- a/test/fixtures/mk_history_coverage.json +++ b/test/fixtures/mk_history_coverage.json @@ -5,17 +5,108 @@ "parentId": "e957da41-3552-4cf6-b636-5babc5cbc4e5", "destinationId": "e957da41-3552-4cf6-b636-5babc5cbc4e5", "timezone": "America/New_York", - "firstRecordedAt": "2024-05-14", - "lastRecordedAt": "2024-05-14", - "retrievableThrough": "2024-05-14", - "kinds": { + "summary": { + "entitiesWithData": 109, + "archiveFrom": "2021-07-03", + "recordedTo": "2026-09-21", + "retrievableThrough": "2026-09-23", + "measuredOn": "2026-09-23" + }, + "fields": { "status": { - "first": "2024-05-14", - "last": "2024-05-14" + "entities": 109, + "from": "2021-07-03", + "newest": "2026-09-21", + "depth": { + "fourYearsPlus": 85, + "twoToFourYears": 10, + "oneToTwoYears": 8, + "underOneYear": 6 + } + }, + "showtimes": { + "entities": 58, + "from": "2021-07-04", + "newest": "2026-09-21", + "depth": { + "fourYearsPlus": 34, + "twoToFourYears": 9, + "oneToTwoYears": 9, + "underOneYear": 6 + } }, "queue.STANDBY": { - "first": "2024-05-14", - "last": "2024-05-14" + "entities": 56, + "from": "2021-07-03", + "newest": "2026-09-21", + "depth": { + "fourYearsPlus": 41, + "twoToFourYears": 9, + "oneToTwoYears": 2, + "underOneYear": 4 + } + }, + "queue.RETURN_TIME": { + "entities": 26, + "from": "2021-10-19", + "newest": "2026-09-21", + "depth": { + "fourYearsPlus": 23, + "twoToFourYears": 3, + "oneToTwoYears": 0, + "underOneYear": 0 + } + }, + "queue.PAID_RETURN_TIME": { + "entities": 3, + "from": "2021-10-19", + "newest": "2026-09-21", + "depth": { + "fourYearsPlus": 2, + "twoToFourYears": 1, + "oneToTwoYears": 0, + "underOneYear": 0 + } + }, + "queue.BOARDING_GROUP": { + "entities": 2, + "from": "2023-03-18", + "newest": "2025-11-14", + "depth": { + "fourYearsPlus": 0, + "twoToFourYears": 2, + "oneToTwoYears": 0, + "underOneYear": 0 + } + } + }, + "entities": [ + { + "id": "290942fd-89c9-4680-98df-b86b49752f6a", + "name": "Cinderella Castle: A Beacon of Magic", + "entityType": "SHOW", + "from": "2021-10-03", + "newest": "2023-03-31", + "fields": ["showtimes", "status"], + "stillListed": false + }, + { + "id": "92aad875-9134-4a2f-bad0-8d8c7cc59cf1", + "name": "Mickey & Minnie\u2019s Very Merry Memories", + "entityType": "SHOW", + "from": "2021-12-23", + "newest": "2021-12-31", + "fields": ["showtimes", "status"], + "stillListed": false + }, + { + "id": "a5241f3b-4ab5-4902-b5ba-435132ef553d", + "name": "Splash Mountain", + "entityType": "ATTRACTION", + "from": "2021-07-03", + "newest": "2023-01-22", + "fields": ["queue.RETURN_TIME", "queue.STANDBY", "status"], + "stillListed": false } - } + ] } diff --git a/test/live/smoke.test.ts b/test/live/smoke.test.ts index 267c5ee..b2061fa 100644 --- a/test/live/smoke.test.ts +++ b/test/live/smoke.test.ts @@ -28,10 +28,21 @@ describe('live smoke tests against api.themeparks.wiki', () => { expect(Array.isArray(s.schedule)).toBe(true); }); - it('Magic Kingdom history coverage parses', async () => { + it('Magic Kingdom history coverage parses as a PARK document', async () => { const c = await tp.entity(MK_ID).history.coverage(); expect(c.timezone).toBe('America/New_York'); - expect(typeof c.kinds).toBe('object'); + // MK is a PARK, so this is the park shape. Asserting `kinds` here read + // undefined against production and only ever ran in the drift workflow, + // so CI never saw it. + if (!('summary' in c)) throw new Error('expected a park coverage document'); + expect(typeof c.summary.archiveFrom).toBe('string'); + expect(Object.keys(c.fields).length).toBeGreaterThan(0); + }); + + it('span() reads a park and a ride to the same three dates', async () => { + const park = await tp.entity(MK_ID).history.span(); + expect(typeof park.archiveFrom).toBe('string'); + expect(typeof park.retrievableThrough).toBe('string'); }); it("Magic Kingdom today's history answers for the whole park", async () => { diff --git a/test/unit/history-paging.test.ts b/test/unit/history-paging.test.ts new file mode 100644 index 0000000..0b1b151 --- /dev/null +++ b/test/unit/history-paging.test.ts @@ -0,0 +1,261 @@ +/** + * The loop above the history endpoints: paging, park flattening, span, and the + * hourly budget. + * + * The raw calls are covered in history.test.ts. These are about what a caller + * would otherwise have to write themselves, and get wrong in the same three + * ways every time: stopping at page one, branching on the park shape by hand, + * and treating a 429 as no data. + */ + +import { describe, it, expect, vi } from 'vitest'; +import { readFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { ThemeParks } from '../../src/client'; +import { BudgetExhaustedError } from '../../src/ergonomic/history'; +import { RateLimitError } from '../../src/errors'; +import type { FetchLike } from '../../src/transport'; + +async function loadFixture(name: string): Promise> { + return JSON.parse(await readFile(resolve(__dirname, '../fixtures', name), 'utf8')) as Record< + string, + unknown + >; +} + +function json(body: unknown, init: { status?: number; headers?: Record } = {}) { + return new Response(JSON.stringify(body), { + status: init.status ?? 200, + headers: { 'content-type': 'application/json', ...(init.headers ?? {}) }, + }); +} + +function client(fetchFn: unknown, options: Record = {}) { + return new ThemeParks({ fetch: fetchFn as FetchLike, cache: false, ...options }); +} + +async function collect(iterable: AsyncIterable): Promise { + const out: T[] = []; + for await (const item of iterable) out.push(item); + return out; +} + +describe('days() paging', () => { + it('follows the server next URL verbatim and stops at null', async () => { + const page = await loadFixture('mk_history_daily.json'); + const urls: string[] = []; + const fetchFn = vi.fn((url: string | URL) => { + urls.push(String(url)); + const last = urls.length > 1; + return Promise.resolve( + json({ + ...page, + next: last ? null : 'https://api.themeparks.wiki/v1/entity/mk/history/daily?cursor=abc', + }), + ); + }); + + const rows = await collect(client(fetchFn).entity('mk').history.days()); + + expect(urls).toHaveLength(2); + // Not rebuilt from the path. The server has already applied every + // parameter, and re-deriving the URL is how a paging loop quietly starts + // asking for the wrong range. + expect(urls[1]).toBe('https://api.themeparks.wiki/v1/entity/mk/history/daily?cursor=abc'); + // Both pages' rows arrived, none twice. + const perPage = rows.length / 2; + expect(Number.isInteger(perPage)).toBe(true); + expect(perPage).toBeGreaterThan(0); + }); + + it('a single unpaged response costs exactly one call', async () => { + const page = await loadFixture('mk_history_daily.json'); + page.next = null; + const fetchFn = vi.fn(() => Promise.resolve(json(page))); + await collect(client(fetchFn).entity('mk').history.days()); + expect(fetchFn).toHaveBeenCalledOnce(); + }); + + it('flattens a park envelope to the same stream as an entity one', async () => { + const park = await loadFixture('mk_history_daily.json'); + const fetchFn = vi.fn(() => Promise.resolve(json(park))); + const rows = await collect(client(fetchFn).entity('mk').history.days()); + + const entities = park.entities as Array<{ id: string; days: unknown[] }>; + const expected = entities.flatMap((e) => e.days.map(() => e.id)); + expect(rows.map((r) => r.entityId)).toEqual(expected); + // Every row is tagged, which is the only thing the caller loses by asking + // the park instead of the rides one at a time. + expect(rows.every((r) => typeof r.entityId === 'string' && r.entityId !== '')).toBe(true); + }); + + it('does not accumulate: rows arrive before the last page does', async () => { + const page = await loadFixture('mk_history_daily.json'); + let served = 0; + const fetchFn = vi.fn(() => { + served++; + return Promise.resolve( + json({ ...page, next: served >= 3 ? null : 'https://api.themeparks.wiki/v1/p' }), + ); + }); + + const seen: number[] = []; + for await (const row of client(fetchFn).entity('mk').history.days()) { + expect(row.entityId).toBeTruthy(); + seen.push(served); + break; + } + // The first row was yielded while only one page had been fetched. A + // version that collected everything first would report 3 here. + expect(seen[0]).toBe(1); + }); +}); + +describe('span()', () => { + it('reads a park summary and an entity top level to the same shape', async () => { + const park = await loadFixture('mk_history_coverage.json'); + const entity = await loadFixture('mk_attraction_history_coverage.json'); + + const parkSpan = await client(vi.fn(() => Promise.resolve(json(park)))) + .entity('mk') + .history.span(); + const entitySpan = await client(vi.fn(() => Promise.resolve(json(entity)))) + .entity('ride') + .history.span(); + + expect(Object.keys(parkSpan).sort()).toEqual(Object.keys(entitySpan).sort()); + expect(parkSpan.archiveFrom).toBe((park.summary as { archiveFrom: string }).archiveFrom); + expect(entitySpan.archiveFrom).toBe(entity.firstRecordedAt); + expect(entitySpan.retrievableThrough).toBe(entity.retrievableThrough); + }); + + it('keeps retrievableThrough distinct from recordedTo', async () => { + // A backfill bounded by recordedTo asks past the entitlement and ends in + // 403s. These are different dates on every plan below the top one. + const park = await loadFixture('mk_history_coverage.json'); + const span = await client(vi.fn(() => Promise.resolve(json(park)))) + .entity('mk') + .history.span(); + const summary = park.summary as { recordedTo: string; retrievableThrough: string }; + expect(span.recordedTo).toBe(summary.recordedTo); + expect(span.retrievableThrough).toBe(summary.retrievableThrough); + }); +}); + +describe('the hourly history budget', () => { + function limited(retryAfter: string) { + return vi.fn(() => + Promise.resolve( + json( + { error: 'HISTORY_RATE_LIMITED' }, + { status: 429, headers: { 'retry-after': retryAfter } }, + ), + ), + ); + } + + it('raises BudgetExhaustedError instead of sleeping through a long wait', async () => { + const fetchFn = limited('2700'); + const tp = client(fetchFn); + + const error = await collect(tp.entity('mk').history.days()).catch((e: unknown) => e); + + expect(error).toBeInstanceOf(BudgetExhaustedError); + expect((error as BudgetExhaustedError).retryAfterMs).toBe(2_700_000); + // The shipped retry policy is in force here on purpose: turning retries + // off in the test is exactly how this hid in the Python sibling, where the + // transport rode out 45 minutes three times before the error could fire. + expect(fetchFn).toHaveBeenCalledOnce(); + }); + + it('still rides out an ordinary REST 429', async () => { + const slept: number[] = []; + const fetchFn = limited('2'); + const tp = new ThemeParks({ fetch: fetchFn as unknown as FetchLike, cache: false }); + // Only the sleep is substituted; the retry policy under test is the real one. + ( + tp as unknown as { transport: { opts: { sleep: (ms: number) => Promise } } } + ).transport.opts.sleep = (ms: number) => { + slept.push(ms); + return Promise.resolve(); + }; + + const error = await collect(tp.entity('ride').history.days()).catch((e: unknown) => e); + + expect(error).toBeInstanceOf(RateLimitError); + expect(error).not.toBeInstanceOf(BudgetExhaustedError); + expect(slept).toEqual([2000, 2000, 2000]); + expect(fetchFn).toHaveBeenCalledTimes(4); + }); + + it('applies on a later page too, not just the first call', async () => { + const page = await loadFixture('mk_history_daily.json'); + let call = 0; + const fetchFn = vi.fn(() => { + call++; + if (call === 1) { + return Promise.resolve(json({ ...page, next: 'https://api.themeparks.wiki/v1/p2' })); + } + return Promise.resolve( + json( + { error: 'HISTORY_RATE_LIMITED' }, + { status: 429, headers: { 'retry-after': '2700' } }, + ), + ); + }); + + const error = await collect(client(fetchFn).entity('mk').history.days()).catch( + (e: unknown) => e, + ); + // A backfill runs out of budget mid-walk far more often than on its first + // call, so the page loop has to raise the same error the first call does. + expect(error).toBeInstanceOf(BudgetExhaustedError); + }); + + it('a raised maxWaitMs lets a longer wait through as a plain RateLimitError', async () => { + const tp = new ThemeParks({ + fetch: limited('2700') as unknown as FetchLike, + cache: false, + retry: { max: 0 }, + }); + const error = await collect(tp.entity('mk').history.days({ maxWaitMs: 3_600_000 })).catch( + (e: unknown) => e, + ); + expect(error).toBeInstanceOf(RateLimitError); + expect(error).not.toBeInstanceOf(BudgetExhaustedError); + }); +}); + +describe('the API key', () => { + it('treats an empty key as no key', async () => { + // An unset environment variable arrives as '' far more often than as + // undefined, and sending an empty key is a 401 rather than an anonymous + // request. + const seen: Array> = []; + const fetchFn = vi.fn((_url: string | URL, init?: { headers?: Record }) => { + seen.push(init?.headers ?? {}); + return Promise.resolve(json({ destinations: [] })); + }); + await client(fetchFn, { apiKey: '' }).destinations.list(); + expect(seen[0]).not.toHaveProperty('x-api-key'); + }); + + it('reaches a page fetched by absolute URL, not just the first call', async () => { + const page = await loadFixture('mk_history_daily.json'); + const seen: Array> = []; + let call = 0; + const fetchFn = vi.fn((_url: string | URL, init?: { headers?: Record }) => { + seen.push(init?.headers ?? {}); + call++; + return Promise.resolve( + json({ ...page, next: call >= 2 ? null : 'https://api.themeparks.wiki/v1/p2' }), + ); + }); + await collect(client(fetchFn, { apiKey: 'tpw_example' }).entity('mk').history.days()); + // getUrl takes a different path into the transport from get. A key that + // reached only the first call would drop the caller to the anonymous + // window partway through their own backfill. + expect(seen).toHaveLength(2); + expect(seen[1]!['x-api-key']).toBe('tpw_example'); + }); +}); diff --git a/test/unit/history.test.ts b/test/unit/history.test.ts index 2fbe38d..b39951e 100644 --- a/test/unit/history.test.ts +++ b/test/unit/history.test.ts @@ -144,12 +144,37 @@ describe('history responses', () => { }); }); - it('coverage names the recorded days per field', async () => { + // Magic Kingdom is a PARK, and /history/coverage answers a park with the + // park document: summary + fields + entities, no `kinds`. The fixture this + // test used to read was hand-written in the entity shape and named after a + // park, so it agreed with the code for the same reason the code was wrong. + // Both fixtures below are captured from production. + it('coverage of a PARK summarises the park and names the fields it holds', async () => { const fixture = await loadFixture('mk_history_coverage.json'); const raw = new RawClient(transportReturning(fixture)); const res = await raw.getEntityHistoryCoverage('75ea578a-adc8-4116-a54d-dccb60765ef9'); expect(res.timezone).toBe('America/New_York'); - expect(res.kinds['status']).toEqual({ first: '2024-05-14', last: '2024-05-14' }); + if (!('summary' in res)) throw new Error('expected a park coverage document'); + expect(res.summary.archiveFrom <= res.summary.recordedTo).toBe(true); + // retrievableThrough is what YOUR key may read, not what the archive + // holds. Bounding a backfill by the wrong one ends it in 403s. + expect(typeof res.summary.retrievableThrough).toBe('string'); + expect(Object.keys(res.fields)).toContain('queue.STANDBY'); + // entities[] is truncated in the fixture; the shape is what matters. + expect(res.entities.length).toBeGreaterThan(0); + }); + + it('coverage of a single entity names the recorded days per field', async () => { + const fixture = await loadFixture('mk_attraction_history_coverage.json'); + const raw = new RawClient(transportReturning(fixture)); + const res = await raw.getEntityHistoryCoverage('some-attraction'); + expect(res.timezone).toBe('America/New_York'); + if ('summary' in res) throw new Error('expected an entity coverage document'); + expect(Object.keys(res.kinds)).toContain('queue.STANDBY'); + expect(res.kinds['queue.STANDBY']).toMatchObject({ + first: expect.any(String), + last: expect.any(String), + }); }); it('a day outside the window is an ApiError carrying the earliest allowed date', async () => { diff --git a/test/unit/version.test.ts b/test/unit/version.test.ts new file mode 100644 index 0000000..0201394 --- /dev/null +++ b/test/unit/version.test.ts @@ -0,0 +1,63 @@ +/** + * The version the SDK announces must be the version it is. + * + * `PACKAGE_VERSION` is a literal in client.ts. It said `7.0.0-alpha.0` in a + * package at `8.0.0`, so every request announced a version a major old and + * nothing anywhere failed. The Python sibling had the same bug and was two + * majors out. + * + * A literal only stays right while someone remembers to change it, and across + * two releases nobody did. This makes forgetting a red test instead of a quiet + * lie in a header. + */ + +import { describe, it, expect, vi } from 'vitest'; +import { readFile } from 'node:fs/promises'; +import { resolve } from 'node:path'; +import { ThemeParks } from '../../src/client'; +import type { FetchLike } from '../../src/transport'; + +async function declaredVersion(): Promise { + const pkg = JSON.parse( + await readFile(resolve(__dirname, '../../package.json'), 'utf8'), + ) as Record; + return pkg.version as string; +} + +async function sentUserAgent(): Promise { + const seen: Array> = []; + const fetchFn = vi.fn((_url: string | URL, init?: { headers?: Record }) => { + seen.push(init?.headers ?? {}); + return Promise.resolve( + new Response(JSON.stringify({ destinations: [] }), { + headers: { 'content-type': 'application/json' }, + }), + ); + }); + await new ThemeParks({ + fetch: fetchFn as unknown as FetchLike, + cache: false, + }).destinations.list(); + return seen[0]!['user-agent']!; +} + +describe('the announced version', () => { + it('package.json declares a plain semver version', async () => { + // Guard the helper: a lookup that silently found nothing would make the + // test below pass for the wrong reason. + expect(await declaredVersion()).toMatch(/^\d+\.\d+\.\d+(-[0-9A-Za-z.-]+)?$/); + }); + + it('is the one the package declares', async () => { + // This is the assertion the drift would have failed. It is checked through + // the header the server actually receives, not through the constant, so a + // correct constant wired up wrongly fails here too. + expect(await sentUserAgent()).toBe(`themeparks-sdk-js/${await declaredVersion()}`); + }); + + it('is not a stale prerelease of an older major', async () => { + const [sdkMajor] = (await sentUserAgent()).split('/')[1]!.split('.'); + const [pkgMajor] = (await declaredVersion()).split('.'); + expect(sdkMajor).toBe(pkgMajor); + }); +});