diff --git a/use-crystallize/skills/mass-operations/SKILL.md b/use-crystallize/skills/mass-operations/SKILL.md new file mode 100644 index 0000000..4f40f71 --- /dev/null +++ b/use-crystallize/skills/mass-operations/SKILL.md @@ -0,0 +1,232 @@ +--- +name: mass-operations +description: Author, validate, run, and monitor Crystallize mass operation files — the JSON batch format executed by the mass-operations runner as a BulkTask. Use this skill whenever the user wants to bulk create, update, or upsert Crystallize data (products, folders, documents, variants, prices, stock, customers, orders, subscription contracts, shapes, pieces, topics, flows, price lists, paths, images), seed or migrate a tenant, import a catalog or customer/order backlog, fix or replay data outside API rate limits, publish or unpublish items at scale, or debug a failed or stuck bulk task. Trigger on mentions of "mass operation", "mass-operation", "bulk task", "bulkTask", "operations.json", "intent", "_ref", "createMassOperationBulkTask", "operationLogs", or on any request phrased as "import X into Crystallize", "update all products", "migrate this tenant" — even when the user does not name the mass operations feature itself. +metadata: + author: Crystallize + version: "1.0" +--- + +# Crystallize Mass Operations + +A mass operation file is a strict JSON document listing every mutation you want Crystallize to run. +Each entry is an **operation**. The **mass-operations runner** consumes them as a **BulkTask**, applying +the same validation and side effects as the equivalent GraphQL mutation. + +Use this for: tenant seeding/migration, catalog and customer/order imports, replaying or fixing data +outside API rate limits, coordinated multi-step changes, and large-scale content updates. + +**The runner acts as a tenant admin.** These files are infrastructure artefacts — review them like code. + +> Verified against `@crystallize/schema` v6.13.0 and the behaviour of the mass-operations +> runner as of 2026-08-18. These are observed runtime behaviours, not contractual API +> guarantees — re-check anything load-bearing before depending on it. +> Several examples in the public docs do not validate, and several documented behaviours differ +> from what the runner does — see `references/intents.md` § "Documentation discrepancies". + +## Decide the execution path first + +| Situation | Path | +| --- | --- | +| User has the Crystallize CLI, or is doing a production migration | **CLI** (default — recommended) | +| Browser/app context, CI without CLI, or building tooling around it | **Raw API** (see `references/lifecycle.md`) | + +## Workflow + +### 1. Establish tenant context before writing anything + +Do not invent IDs, shape identifiers, component IDs, or languages. Confirm with the user, or pull them: + +``` +crystallize mass-operation dump-content-model +``` + +Component IDs that don't exist on the target shape are the single most common cause of a failed task. + +### 2. Write the file + +```json +{ + "version": "1.0.0", + "operations": [ + { + "intent": "piece/upsert", + "identifier": "rating-system", + "name": "Rating System", + "components": [ + { + "id": "name", + "name": "Name", + "type": "singleLine", + "config": { "singleLine": { "required": true } } + } + ] + } + ] +} +``` + +Top-level shape (`OperationsSchema`): + +- `version` — **must be exactly `"1.0.0"`.** This is a behaviour switch, not a label: the runner picks + the highest converter registered at a version `<=` this string. See "The `version` field" below. +- `operations` — ordered array; executed **sequentially, in order**. An empty array fails the task. + +Every operation: + +- has an `intent` — a literal string in a discriminated union. An unrecognised intent fails validation. +- carries the matching GraphQL input's fields **inlined at the top level** (not nested under `input`). +- accepts an optional `_ref` (non-empty string) for downstream references. + +**Required fields per intent are listed in `references/intents.md` — check them before writing.** +For `_ref` chaining, handlebars, and helpers, read `references/templating.md`. + +### 3. The `version` field selects the converter set + +The runner resolves each intent to the newest handler whose version is equal to or lower than the +file's `version`. Two sets exist: `0.0.1` (legacy) and `1.0.0` (current). + +- `"1.0.0"` — current converters. **Always use this.** +- `"0.0.1"` — legacy converters with no result normalisation. This is where the "upsert returns a + different shape" folklore comes from; see `references/templating.md`. +- `"1"`, `"1.0"`, `"*"` — **pass schema validation, then crash the whole task at runtime.** The version + string is fed to `semver.eq`/`semver.lt`, which throw `TypeError: Invalid Version: 1` on any + non-full semver. Verified empirically against semver 7.8.5. + +### 4. Identify items by `resourceIdentifier` where it is actually implemented + +`resourceIdentifier` is your own stable key for an entity. It makes files re-runnable and portable +across tenants without hardcoding Crystallize IDs. The schema's `checkResourceIdentifierOrId` +refinement accepts it in place of `itemId`/`id`/`topicId`: + +``` +Expected at least a resourceIdentifier or an id/itemId/topicID. +``` + +**But passing schema validation does not mean the runner honours it.** Only these intents actually +resolve it: `product|document|folder /update` and `/upsert`, `item/delete`, `topic/update|upsert|delete`, +`image/register`, and all `item/paths/*`. Everywhere else it is either rejected at runtime or silently +ignored — the full table is in `references/intents.md` § "Where `resourceIdentifier` actually works". + +### 5. Order operations as a dependency tree + +**A thing must exist before anything references it.** Execution is strictly sequential, so every +dependency is positional — the referencing operation has to come later in the array than the operation +that creates its target. This is a universal rule: it applies to content exactly as it applies to the +content model. There is no deferred resolution and no second pass. + +| Reference | Must already exist | +| --- | --- | +| `shape/upsert` choice → piece | the `piece/upsert` | +| item's `shapeIdentifier` | the `shape/upsert` | +| `tree.parentId` | the parent `folder/create` (or `{{ defaults.rootItemId }}`) | +| `topicIds` on an item | the `topic/*` operations | +| `itemRelations` content | every item it points at | +| `item/updateComponent/*` | the item, and the component on its shape | +| `product/variant/*`, `…/price/modify`, `…/stock/modify` | the product, price variant, stock location | +| `item/flow/stage/addItems` | the `flow/*` defining that stage | +| `order`/`subscription-contract` customer link | the `customer/upsert` | +| any `{{ myRef.… }}` | the operation carrying that `_ref` | + +Chain with `_ref` where there is no natural identifier, and with a shared `identifier` / +`resourceIdentifier` where there is. + +**Nothing checks this for you.** The schema validates structure, not existence, so a file referencing +something that isn't there yet uploads and starts happily — it fails partway through, after earlier +operations have already been applied, with no rollback. Worse, `item/updateComponent/item` runs with +content validation disabled (see `references/limits.md`), so a dangling reference written that way may +not raise at all. Order the +array correctly rather than relying on an error. + +### 6. Validate and run + +``` +crystallize mass-operation run +``` + +Validates, requests presigned upload, pushes the file, creates the bulk task with `autoStart`, and waits +while tailing logs. Add `--no-interactive` for CI. + +To validate without the CLI, parse the file with `OperationsSchema` from `@crystallize/schema/mass-operation`. +**Do this locally every time** — the server discards per-field detail (see step 7). + +Raw API sequence: `references/lifecycle.md`. + +### 7. Monitor and verify + +Lifecycle: `pending → started → complete | error`. There is **no `running` status** — the enum is +exactly `{pending, started, complete, error}`. + +**Always check `operationLogs`, not just task status.** A task reaches `complete` if the loop finishes, +even when individual operations failed — each failure is logged and the runner moves to the next +operation. There is no rollback. Per-operation `status` is `success`, `partial`, or `failure`, with +`statusCode` `200`, `206`, or `500` respectively. + +Three ways an operation can leave **no log entry at all**: + +- its converter returned no command (see `references/intents.md` § "Silent skips"), +- the whole task died first, +- schema validation rejected the file, so nothing ran. + +A schema violation or unparseable JSON marks the task `error` before any operation runs, and +`bulkTask.info.error` reads only `Invalid Operation File` — the per-field issues go to the worker's own +logs, not to you. Validate locally to see them. + +## Rules that prevent most failures + +- **An item `upsert` with neither `itemId` nor `resourceIdentifier` always CREATES** — on every run. + `externalReference` is *not* a lookup key. This is the number-one duplicate-data trap: give every + `product|document|folder /upsert` a `resourceIdentifier`. Identifier-keyed upserts (`piece`, `shape`, + `customer`, `customer/group`, `pricelist`, `flow`, `product/variant`) are genuinely idempotent; + `order/upsert` and `subscription-contract/upsert` are not unless you pass a real `id`. +- **`item/unpublish` is accepted by the schema but not supported by the runner.** It validates and + uploads fine, then fails the whole task at execution and abandons every remaining operation. It is + the only one of the 59 intents in this position. Unpublish via the Core API `unpublishItem` + mutation instead. +- **Prefer `upsert` for re-runnability — but upserts derive from the *create* input schema**, so they + need the *full* create payload, not a sparse patch. `product/upsert` requires `tree`, `vatTypeId`, and + `variants` exactly as `product/create` does. (`product/variant/update` is the exception — it's + `.partial()`, so it genuinely accepts a sparse patch.) +- **At `version: "1.0.0"` there is no upsert return-shape trap.** The v1.0.0 converters normalise every + upsert result to a flat `{id}` or `{identifier}`. The `{{#if x.id.id}}…{{else}}…{{/if}}` guard seen in + the public docs is legacy `0.0.1` behaviour — harmless but unnecessary. See `references/templating.md`. +- **`richText.html` is an array of strings**, not a string. +- **Shape and piece component settings go under `config`**, keyed by type: + `{"id": "name", "name": "Name", "type": "singleLine", "config": {"singleLine": {"required": true}}}`. + The inline form the public docs use (`"singleLine": {"required": true}` as a sibling of `type`) + **validates and is then silently discarded** — the component is created with default settings. +- **`componentChoice` / `componentMultipleChoice` need ≥2 choices, and each choice needs a `type`.** + When a choice carries more than one field, make it a `piece/upsert` and reference it with + `{"type": "piece", "config": {"piece": {"identifier": "…"}}}` — an inline `components` array on a + choice is not a valid API structure. Structural components (`contentChunk`, `componentChoice`, + `componentMultipleChoice`) can never be direct children of each other; put a piece in between. +- **ID fields accept handlebars.** `IdSchema` is `/^(?:[0-9a-f]{24}|{{.*}})$/` — a 24-char hex ID or a + `{{ ... }}` expression, nothing else. This is why `{{ defaults.vatTypeIds.[0] }}` and + `{{ defaults.rootItemId }}` work where a literal placeholder like `"TODO"` fails validation. +- **A handlebars expression that fails to render is written through literally.** The renderer catches + the error, logs it, and returns the *raw template string* — so a bad reference silently stores + `{{ myRef.id }}` as data rather than failing the operation. +- **`item/paths/set*` needs two operations to converge.** If anything must be removed, that run *only* + removes; the add is expected as a separate operation. Not atomic, despite the name. +- **Respect the domain limits — the schema does not.** A file that exceeds one validates, uploads, and + fails at execution, or worse succeeds with altered data. **One invalid operation rejects the entire + file** (all-or-nothing parse), so chunk large jobs. The caps that bite most: **30 topics** per + `topic/create` (counting the whole `children` subtree), **50 items** per `item/flow/stage/addItems`, + **500 cart items** per order, **250 `topicIds`** per item, **75** item relations, and component + nesting depth **5**. Default string fields cap at **256** chars. +- **A large class of limits fails silently** — no error, no log, wrong data stored. Notably: only the + first component-type key in a content object is used and the rest are discarded; `null` content is + coerced to empty and wipes the component; `propertiesTable` is rebuilt from the shape config; + `numeric` `decimalPlaces` floors the stored value; choice/selection values not in the config are + filtered out; strings are never trimmed. Read `references/limits.md` before writing content at scale. +- **Chunk large jobs** so a failure doesn't force replaying everything. +- **Never guess an intent name.** The union in `references/intents.md` is exhaustive as of v6.13.0. +- **Presigned URLs are short-lived.** Upload immediately after requesting one. + +## Reference files + +- `references/intents.md` — all 59 intents, required fields, verified `_ref` outputs, silent skips, + limits, where `resourceIdentifier` works, corrected examples, docs bugs +- `references/templating.md` — `_ref`, positional refs, handlebars, `defaults.*`, `upload`, `fetch*` +- `references/limits.md` — every enforced limit: file-level, batch caps, silent data loss, config + traps, string/number bounds, structural rules +- `references/lifecycle.md` — raw API upload/run/monitor, CLI commands, troubleshooting diff --git a/use-crystallize/skills/mass-operations/references/intents.md b/use-crystallize/skills/mass-operations/references/intents.md new file mode 100644 index 0000000..8ad4445 --- /dev/null +++ b/use-crystallize/skills/mass-operations/references/intents.md @@ -0,0 +1,362 @@ +# Intents + +`intent` is a `z.literal` in a `z.discriminatedUnion`. The list below is **exhaustive** for +`@crystallize/schema` v6.13.0 — 59 intents. An intent not listed here will fail validation. + +The rest of the operation object is the matching GraphQL input's fields, **inlined at the top level**. + +Runtime facts on this page describe what the mass-operations runner actually does, verified as of +2026-08-18. They are observed behaviours, not contractual guarantees. + +## Contents + +- [Not supported by the runner](#not-supported-by-the-runner) +- [Required fields per intent](#required-fields-per-intent) +- [Where `resourceIdentifier` actually works](#where-resourceidentifier-actually-works) +- [Which upserts are idempotent](#which-upserts-are-idempotent) +- [Properties exposed to `_ref`](#properties-exposed-to-_ref) +- [Silent skips](#silent-skips) +- [Limits](#limits) +- [Schema quirks worth knowing](#schema-quirks-worth-knowing) +- [Corrected examples](#corrected-examples) +- [Documentation discrepancies](#documentation-discrepancies) + +## Not supported by the runner + +**`item/unpublish` is in the schema but the runner has no handler for it.** It is the only one of the +59 intents in this position. + +The failure mode is unusually harsh. The file validates locally, uploads, and the task starts — then +fails when it reaches the operation, and unlike an ordinary per-operation error it is **not** caught +and logged. The task is marked `error` and **every remaining operation is abandoned**. Operations +already applied are not rolled back. + +Unpublish via the Core API `unpublishItem` mutation instead. + +## Required fields per intent + +Derived by parsing empty payloads against the real schema. `_ref` is optional on every intent. +"+ id" means `checkResourceIdentifierOrId` applies at the *schema* level: supply `itemId`/`id`/`topicId` +**or** `resourceIdentifier`. Whether the runner then honours `resourceIdentifier` is a separate +question — see the next section. + +| Intent | Required fields | +| --- | --- | +| `folder/create` | `name`, `shapeIdentifier`, `tree`, `language` | +| `folder/update` | as create, + id | +| `folder/upsert` | as create | +| `document/create` | `name`, `shapeIdentifier`, `tree`, `language` | +| `document/update` | as create, + id | +| `document/upsert` | as create | +| `product/create` | `name`, `shapeIdentifier`, `tree`, `vatTypeId`, `variants`, `language` | +| `product/update` | as create, + id | +| `product/upsert` | as create | +| `item/updateComponent/item` | `language`, `component`, + id — **runner requires `itemId` specifically** | +| `item/updateComponent/sku` | `language`, `component`, `sku` | +| `item/publish` | `language`, + id — **runner requires `itemId` specifically** | +| `item/unpublish` | schema-valid, **not implemented** — see above | +| `item/delete` | + id. Passing *both* `itemId` and `resourceIdentifier` throws | +| `item/flow/stage/addItems` | `items` (min 1), `stageIdentifier`. Optional `moveFromFlowIdentifier`, `actionConfig` | +| `shape/create` | `identifier`, `name` | +| `shape/update`, `shape/upsert` | `identifier`, `name` | +| `piece/create`, `piece/update`, `piece/upsert` | `identifier`, `name` | +| `product/variant/create` | `sku`, `name`, `language`, `productId` | +| `product/variant/update` | `language`, `sku` — **everything else optional** (`.partial()`) | +| `product/variant/upsert` | `sku`, `name`, `language`, `productId` | +| `product/variant/delete` | `sku` | +| `product/variant/stock/modify` | `sku`, `quantity`, `stockLocationIdentifier` | +| `product/variant/price/modify` | `sku`, `priceVariantIdentifier`, `price` | +| `customer/create`, `customer/upsert` | `identifier` | +| `customer/update` | per `UpdateCustomerInputSchema` | +| `customer/group/create`, `customer/group/update`, `customer/group/upsert` | per group input schema | +| `order/register` | `cart`, `customer` | +| `order/update` | `cart`, `customer`, + id — **runner requires `id` specifically** | +| `order/upsert` | `cart`, `customer` — **`pipelines` is omitted from this intent** | +| `subscription-contract/create` | `customerIdentifier`, `subscriptionPlan`, `status`, `item`, `recurring` | +| `subscription-contract/update` | as create, + id — **runner requires `id` specifically** | +| `subscription-contract/upsert` | as create | +| `pricelist/create`, `pricelist/upsert` | `identifier`, `name`, `priceVariants`, `selectedProductVariants`, `targetAudience` | +| `pricelist/update` | per `UpdatePriceListInputSchema` | +| `topic/create` | `name`, `language` | +| `topic/update`, `topic/upsert` | `name`, `language`, + id (`topicId` or `resourceIdentifier`) | +| `topic/delete` | + id | +| `flow/create`, `flow/upsert` | `name`, `stages`, `identifier` (`type` drives restriction mapping) | +| `flow/update` | `identifier` + per `UpdateFlowInputSchema` | +| `image/register` | `key` | +| `item/paths/addAliases`, `setAliases`, `removeAliases` | `language`, `paths`, + `itemId`/`resourceIdentifier` | +| `item/paths/addHistory`, `setHistory`, `removeHistory` | `language`, `paths`, + `itemId`/`resourceIdentifier` | +| `item/paths/addShortcuts`, `setShortcuts` | **`shortcuts`** (`[{parentId, position?}]`), + `itemId`/`resourceIdentifier` | +| `item/paths/removeShortcuts` | **`parentIds`**, + `itemId`/`resourceIdentifier` | + +The three shortcut intents take `shortcuts`/`parentIds`, **not `paths`**, and they ignore any `language` +you pass — the converter hardcodes the tenant default language for shortcut operations. + +## Where `resourceIdentifier` actually works + +The schema accepts `resourceIdentifier` far more widely than the runner implements it. Several +converters carry a literal `ResourceIdentifier is not implemented yet` error. + +| Intent | Behaviour when you supply only `resourceIdentifier` | +| --- | --- | +| `product\|document\|folder /update` | ✅ resolved via `loadByResourceIdentifier` (scoped by `language`) | +| `product\|document\|folder /upsert` | ✅ resolved; used to decide create-vs-update | +| `product\|document\|folder /create` | ✅ stored on the new item, so later ops can find it | +| `item/delete` | ✅ resolved. Supplying both it and `itemId` **throws** | +| `topic/update`, `topic/upsert` | ✅ resolved; mismatch against a supplied `topicId` throws | +| `topic/delete` | ✅ resolved (default language) | +| `topic/create` | ✅ stored on the new topic | +| `image/register` | ✅ used as a dedupe key — if already registered, the operation is skipped | +| `item/paths/*` | ✅ resolved; throws `Item with resourceIdentifier … not found` if missing | +| `item/updateComponent/item` | ❌ **throws** `itemId is required … ResourceIdentifier is not implemented yet` | +| `item/publish` | ❌ **throws** `Operation is missing itemId. ResourceIdentifier is not implemented yet` | +| `order/update` | ❌ **throws** `Operation is missing id. ResourceIdentifier is not implemented yet` | +| `order/upsert` | ❌ **silently registers a brand-new order** | +| `subscription-contract/update` | ❌ **throws** the same "not implemented" error | +| `subscription-contract/upsert` | ❌ **silently creates a new contract** | + +For the ❌ rows, resolve the ID yourself first — either with a `_ref` to the operation that created the +entity, or with a `fetch*` helper (`references/templating.md`). + +## Which upserts are idempotent + +An `upsert` is only idempotent if the converter has something to look the entity up by. + +| Intent | Lookup key | Re-run safe? | +| --- | --- | --- | +| `piece/upsert` | `identifier` | ✅ | +| `shape/upsert` | `identifier` | ✅ | +| `customer/upsert` | `identifier` | ✅ | +| `customer/group/upsert` | `identifier` | ✅ | +| `pricelist/upsert` | `identifier` | ✅ | +| `flow/upsert` | `identifier` | ✅ | +| `product/variant/upsert` | `productId` + `sku` | ✅ | +| `topic/upsert` | `topicId` or `resourceIdentifier` (one is required) | ✅ | +| `product\|document\|folder /upsert` | `itemId` or `resourceIdentifier` — **only if you supply one** | ⚠️ | +| `order/upsert` | `id` only | ⚠️ | +| `subscription-contract/upsert` | `id` only | ⚠️ | + +The ⚠️ rows are the trap. With neither key present the runner short-circuits straight to a create, every run. +`externalReference` is *not* consulted. A `product/upsert` keyed only on `externalReference` creates a +duplicate product on every run. The item lookup is also **language-scoped**, so upserting the same item +under a second language without an `itemId` will create a second item. + +## Properties exposed to `_ref` + +An operation's `_ref` output is the normalised result of the command it ran. The same value is stored +in `OperationLog.output`, which is the authoritative view. + +**Verified** (at `version: "1.0.0"`): + +| Intent | `_ref` output | +| --- | --- | +| `product/upsert`, `folder/upsert`, `document/upsert` | `{ id: string }` — flat, in all branches | +| `shape/upsert` | `{ identifier: string }` | +| `piece/create`, `piece/upsert` | `{ identifier: string }` | +| `customer/create`, `customer/update`, `customer/upsert` | `{ identifier: string }` | +| `product/variant/create`, `/update`, `/upsert` | the **full variant DTO** minus `tenantId`/`language` (id, sku, name, isDefault, priceVariants, stockLocations, components, …); falls back to `{ sku }` if the variant can't be resolved | +| `product/variant/delete` | `{ sku: string }` | +| `item/publish` | `{ language, success: string[], failure: [{ itemId, error }] }` | +| `item/updateComponent/item`, `item/updateComponent/sku` | a **bare ID string**, not `{ id }` — reference it as `{{ myRef }}` | +| `product/variant/stock/modify` | a **bare ID string**, same as above | + +**Not verified.** Every other intent returns whatever its underlying operation produces, and the shape +varies — several return nothing at all, which normalises to `null`. The public docs' table for these +is unreliable and is not reproduced here. Read the `output` field of a real `OperationLog` before +depending on one, or key off `resourceIdentifier` + a `fetch*` helper instead +(`references/templating.md`). + +Note that `item/publish` is special-cased by the runner and stores its raw result. + +## Silent skips + +If a converter returns `null`/`undefined`, the runner's `if (command)` guard is false: **no command +runs, no `OperationLog` row is written, and no `_ref` output is saved.** The operation vanishes. A +`_ref` pointing at it resolves to nothing, and positional `_ref` numbering shifts (see +`references/templating.md`). + +Known cases: + +- `image/register` whose `resourceIdentifier` is already registered — intentional dedupe. +- `item/delete` where neither `itemId` nor a resolvable `resourceIdentifier` yields an id. +- `topic/delete` where the `resourceIdentifier` doesn't resolve. +- `item/paths/setAliases | setShortcuts | setHistory` targeting the tree **root** node. + +If an operation you expected has no log entry, this is why — not a monitoring gap. + +## Limits + +Moved to its own file: **`references/limits.md`** — every bound that can break a mass operation, or +silently change what it writes, from an exhaustive source audit. + +The headline ones: **one invalid operation rejects the entire file**; **30 topics** per `topic/create` +counting the whole subtree; **50 items** per `item/flow/stage/addItems`; **500 cart items** per order; +**250 `topicIds`** per item; **75** item relations; component nesting depth **5**; and a large class of +**silent** truncations and drops that raise no error at all. + +## Schema quirks worth knowing + +- **Upsert = create schema.** `UpsertProductOperationSchema` extends `CreateProductInputSchema`, not the + update one. So an upsert needs the full create payload — it is *not* a sparse patch. Same for + document, folder, piece, shape, customer, customer group, pricelist, topic, flow, variant, and + subscription contract. +- **`piece/update` also uses `CreatePieceInputSchema`**, so it too requires the full payload — and the + converter sends the whole input, so omitted components are dropped, not preserved. +- **`order/upsert` omits `pipelines`** — present on `order/register`, rejected here. +- **`product/variant/update` omits `sku` and `isDefault` from the variant input, then `.partial()`s the + rest**, re-adding `sku` as required. It's the only genuinely patch-style intent. +- **`flow/*` requires `identifier`** on top of the flow input. +- **`product/variant/create` and `/upsert` require `productId`**; `/update` and `/delete` key off `sku`. +- **`version` regex is `/^(\d+\.)?(\d+\.)?(\*|\d+)$/`** — looser than the runner. Only `1.0.0` is safe; + `1`, `1.0` and `*` validate and then crash the task. See `SKILL.md` § "The `version` field". +- **`priceVariants[].tierType` and `.tiers` are stripped by the schema but re-attached by the runner.** + `enrichPriceVariantTiers` walks the parsed result against the raw JSON and puts them back for + `product/create|update|upsert` and `product/variant/create|update|upsert`. So tiered pricing works + even though the fields don't appear in the schema. Matching is by `identifier`, falling back to array + position only when no raw entry has one. +- **`variant.topicIds` is silently dropped.** Also stripped by the schema, and *not* re-attached — + there is an explicit `blocked on @crystallize/schema bump` comment in both variant converters. + Assign variant topics another way. +- **`item/updateComponent/item` runs with `disableContentValidation: true`.** Component content is + written without validation, so malformed content lands silently. `item/updateComponent/sku` does + not disable it. + +## Corrected examples + +### Customer → order → component (docs example, valid as published) + +```json +{ + "version": "1.0.0", + "operations": [ + { + "intent": "customer/upsert", + "identifier": "customer-for-order-123", + "firstName": "John", + "lastName": "Doe", + "type": "individual" + }, + { + "intent": "order/register", + "customer": { "identifier": "customer-for-order-123", "type": "individual" }, + "additionalInformation": "Please deliver between 9am-5pm", + "cart": [ + { + "sku": "SP-RED-001", + "name": "Sample Product", + "productId": "67e5d2d12d31ee752710a74b", + "quantity": 2, + "price": { + "currency": "USD", + "gross": 1000, + "net": 800, + "tax": { "name": "VAT", "percent": 20 } + } + } + ] + }, + { + "intent": "item/updateComponent/item", + "itemId": "632958a35dfc2c90cbbad20d", + "language": "en", + "component": { + "componentId": "title", + "singleLine": { "text": "Mass operation updated title" } + } + } + ] +} +``` + +The customer link uses a **shared identifier**, not `_ref`. Prefer that where a natural identifier +exists. Note `order/register` is not idempotent — re-running this file registers a second order. + +### Product upsert → component update + +The published docs version omits `tree`, `vatTypeId` and `variants`, passes `richText.html` as a +string, and — the part that actually costs you data — keys the upsert on `externalReference` alone, so +it duplicates the product on every run. Corrected: + +```json +{ + "version": "1.0.0", + "operations": [ + { + "_ref": "rootProducts", + "intent": "product/upsert", + "resourceIdentifier": "product-sku-12345", + "externalReference": "SKU-12345", + "name": "My Product", + "language": "en", + "shapeIdentifier": "product", + "tree": { "parentId": "{{ defaults.rootItemId }}" }, + "vatTypeId": "{{ defaults.vatTypeIds.[0] }}", + "variants": [{ "sku": "SKU-12345", "name": "My Product", "isDefault": true }] + }, + { + "intent": "item/updateComponent/item", + "itemId": "{{ rootProducts.id }}", + "language": "en", + "component": { + "componentId": "description", + "richText": { "html": ["

Updated description

"] } + } + } + ] +} +``` + +`resourceIdentifier` is what makes the upsert an upsert. `{{ rootProducts.id }}` is a plain string at +`version: "1.0.0"` — the `{{#if rootProducts.id.id}}` dance in the docs is legacy `0.0.1` behaviour. + +### Piece upsert (docs version is invalid) + +The published version omits `name` on the component. Every component definition needs both `id` and +`name`: + +```json +{ + "version": "1.0.0", + "operations": [ + { + "intent": "piece/upsert", + "identifier": "rating-system", + "name": "Rating System", + "components": [ + { "id": "name", "name": "Name", "type": "singleLine", "singleLine": { "required": true } } + ] + } + ] +} +``` + +## Documentation discrepancies + +Points where differs from what the runner +does. The skill follows the runner. + +### Examples that fail schema validation + +1. `piece/upsert` example omits `name` on the component definition. +2. `product/upsert` example omits `tree`, `vatTypeId` and `variants` (upserts extend the *create* + schema). +3. `richText.html` is passed as a string; the schema requires an array of strings. + +### Statements contradicted by the runner + +4. Task lifecycle is documented as `pending → started → running → complete | error`. There is no + `running` state; the enum is `{pending, started, complete, error}`. +5. The `_ref` property table is largely wrong for v1.0.0 — upserts are normalised to flat `{id}` / + `{identifier}`, `item/updateComponent/*` and `product/variant/stock/modify` return a bare string, + and `product/variant/*` returns a full DTO rather than `{sku}`. +6. The "upsert returns `{id}` on create and `{id:{id}}` on update" guidance describes `version: "0.0.1"` + only, but the docs pair it with `version: "1.0.0"` examples. +7. `resourceIdentifier` is presented as generally available. It is unimplemented on + `item/updateComponent/item`, `item/publish`, `order/update|upsert` and + `subscription-contract/update|upsert`. +8. `item/unpublish` is documented as a usable intent; it has no converter and aborts the whole task. +9. `item/paths/*Shortcuts` are documented as taking `paths`; they take `shortcuts` / `parentIds`. +10. `item/paths/set*` is presented as a set operation; it removes *or* adds in a single run, never both. +11. `version` is presented as a version label. It selects the converter set, and three of the four + formats the regex accepts crash the runner. +12. `externalReference` is used as the de-duplication key in the flagship `product/upsert` example, + which duplicates data on re-run. diff --git a/use-crystallize/skills/mass-operations/references/lifecycle.md b/use-crystallize/skills/mass-operations/references/lifecycle.md new file mode 100644 index 0000000..533330c --- /dev/null +++ b/use-crystallize/skills/mass-operations/references/lifecycle.md @@ -0,0 +1,306 @@ +# Lifecycle: upload, run, monitor + +Verified against runner and Core API behaviour as of 2026-08-18. + +## Contents + +- [CLI path](#cli-path) +- [Raw API path](#raw-api-path) +- [What the runner does with your file](#what-the-runner-does-with-your-file) +- [Monitoring](#monitoring) +- [Troubleshooting](#troubleshooting) +- [Browser CORS](#browser-cors) + +## CLI path + +Recommended for anything production-facing. The CLI validates locally against the same schema the +server enforces, so schema problems surface **with per-field detail** before upload — the server throws +that detail away (see Troubleshooting). + +``` +crystallize mass-operation dump-content-model +``` +Generates a starter file containing the tenant's current shapes and pieces. Use this to ground shape +identifiers and component IDs instead of guessing them. + +``` +crystallize mass-operation run +``` +Validates → requests presigned upload → pushes the file → creates the bulk task with `autoStart` → +waits for completion while tailing logs. + +``` +crystallize mass-operation execute-mutations [image-mapping-file] +``` +Runs client-side GraphQL mutations alongside mass operations, for steps the runner doesn't cover. + +All commands support `--no-interactive` for CI and reuse stored credentials. `--legacy-spec` converts +an old Spec File into a mass operation file. + +The CLI lives in `CrystallizeAPI/tools` — a different repo, so these command signatures are **not** +verified against the runner source and may drift. `crystallize mass-operation --help` is authoritative. + +## Raw API path + +### Endpoints and auth + +| Endpoint | URL | Auth | +| --- | --- | --- | +| Core API | `https://api.crystallize.com/@{tenant}` | Headers below | +| File upload | Returned in `generatePresignedUploadRequest.url` | Presigned — no auth needed | + +``` +Content-Type: application/json +X-Crystallize-Access-Token-Id: +X-Crystallize-Access-Token-Secret: +``` + +### 1. Request a presigned upload + +```graphql +mutation GeneratePresignedUpload($filename: String!, $contentType: String!) { + generatePresignedUploadRequest( + input: { + type: MASS_OPERATIONS + filename: $filename + contentType: $contentType + } + ) { + ... on PresignedUploadRequest { + url + fields { name value } + } + ... on BasicError { error errorName } + } +} +``` + +`MASS_OPERATIONS` routes to a dedicated bucket, separate from `MEDIA` and `STATIC`. Presigned URLs are +short-lived — upload immediately. + +### 2. Upload the file + +Multipart form POST. **Order matters: all presigned fields first, the file last.** + +```typescript +async function uploadToPresignedUrl( + presignedUrl: string, + fields: Array<{ name: string; value: string }>, + fileContent: string +): Promise { + const formData = new FormData(); + + // Add all presigned fields first (order matters!) + for (const field of fields) { + formData.append(field.name, field.value); + } + + // Add the file last + const blob = new Blob([fileContent], { type: 'application/json' }); + formData.append('file', blob); + + const response = await fetch(presignedUrl, { method: 'POST', body: formData }); + if (!response.ok) throw new Error(`Upload failed: ${response.status}`); +} + +// The storage key is inside `fields` — there is no top-level `key` +const storageKey = fields.find(f => f.name === 'key')?.value; +``` + +Equivalent with curl: + +```bash +curl -X POST "$PRESIGNED_URL" \ + -F "key=$KEY" \ + -F "bucket=$BUCKET" \ + -F "X-Amz-Algorithm=$ALGORITHM" \ + -F "X-Amz-Credential=$CREDENTIAL" \ + -F "X-Amz-Date=$DATE" \ + -F "X-Amz-Security-Token=$TOKEN" \ + -F "Policy=$POLICY" \ + -F "X-Amz-Signature=$SIGNATURE" \ + -F "file=@operations.json" +``` + +### 3. Register the bulk task + +```graphql +mutation CreateMassOperationBulkTask($key: String!, $autoStart: Boolean) { + createMassOperationBulkTask(input: { key: $key, autoStart: $autoStart }) { + ... on BulkTaskMassOperation { id status } + ... on BasicError { error errorName } + } +} +``` + +`autoStart: true` dispatches the runner immediately. Omit it, or set false, to start later. + +### 4. Start it (only if `autoStart` wasn't true) + +```graphql +mutation StartMassOperationBulkTask($id: ID!) { + startMassOperationBulkTask(id: $id) { + ... on BulkTaskMassOperation { id status } + ... on BasicError { error errorName } + } +} +``` + +Note: this resolver reads the task **before** dispatching it, so the `status` in the response is the +pre-start value (`pending`). Poll `bulkTask` for the real state. + +Both mutations are flagged `EXPERIMENTAL: the full feature set is not yet complete.` in the schema. + +## What the runner does with your file + +Worth knowing when a task behaves oddly: + +1. A task is only picked up while its status is `pending`. Anything else is ignored outright. +2. The file is fetched from S3 and parsed with `OperationsSchema.safeParse`. On failure the worker logs + each issue to **its own** logs and throws `Invalid Operation File`. +3. `enrichPriceVariantTiers` re-attaches `tierType`/`tiers` that the schema stripped from + `priceVariants` (see `intents.md`). +4. An empty `operations` array stops the task as `error` with cause + `No operations retrieved from spec file`. +5. Files above a configured size threshold are not run in the worker at all — the worker broadcasts a + spawn request and a standalone task runs them. Behaviour is identical; only the execution host + differs. +6. Operations then run **strictly sequentially** in array order. Queued image uploads for each operation + run concurrently (up to 10) and are awaited before the next operation starts. + +## Monitoring + +### Task status + +```graphql +query GetBulkTaskStatus($id: ID!) { + bulkTask(id: $id) { + ... on BulkTaskMassOperation { + id + status + info { error errorName stack } + } + ... on BasicError { error errorName } + } +} +``` + +Also available: `bulkTasks(filter: { type: massOperation })`. + +Lifecycle: **`pending → started → complete | error`**. The status enum is exactly +`{pending, started, complete, error}` — there is **no `running` state**, despite what the public docs +say. + +| Status | Meaning | +| --- | --- | +| `pending` | Created but not started. Only a `pending` task will be picked up | +| `started` | The operation loop is running | +| `complete` | The loop finished. **Individual operations may still have failed** | +| `error` | The task aborted — check `info` | + +`complete` means "the runner reached the end of the array", not "everything worked". + +### Per-operation logs + +```graphql +query OperationLogs($id: ID!, $first: Int) { + operationLogs(filter: { operationId: $id }, first: $first) { + edges { + node { status statusCode message input output } + } + pageInfo { hasNextPage endCursor } + } +} +``` + +`operationId` is the **bulk task id**. It's a paginated connection, so page through it — a large import +will not return every log in one call. The filter also accepts two fields the docs don't mention: +`status` (`success` | `partial` | `failure`) and `operationTypeStartsWith`. Filtering on +`status: failure` is the fastest triage. + +Each entry stores the original input payload, the command executed, the result, a `status` +(`success` / `partial` / `failure`), and a `statusCode`. + +| status | statusCode | When | +| --- | --- | --- | +| `success` | `200` | Command executed without throwing | +| `partial` | `206` | `item/publish` only — some items failed, or zero succeeded | +| `failure` | `500` | The command threw, or the converter failed to build it | + +Those are the only three the runner emits. The underlying DTO permits `201/400/401/403/404/502/503/504` +too, but nothing writes them. + +**Check these even when the task says `complete`.** Operation-level failures are logged and the runner +moves on to the next operation — there is no implicit rollback. + +**An operation with no log entry was skipped, not lost.** Some converters return no command (an already +registered image, an unresolvable delete, a root-node path set) and the runner writes nothing at all +for those. See `intents.md` § "Silent skips". + +Image uploads appear as their own entries with `input.intent = "image/upload"`, separate from the +operation whose `{{ upload ... }}` queued them. + +### Error example + +```json +{ + "bulkTask": { + "id": "abc123", + "status": "error", + "info": { + "error": "Invalid Operation File", + "errorName": "Error", + "stack": "Error: Invalid Operation File\n at OperationsFetcherService.fetch..." + } + } +} +``` + +Schema violations and unparseable JSON mark the whole task `error` and no operation runs — unlike a +runtime failure on a single operation, which is logged and stepped over. + +## Troubleshooting + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `Invalid Operation File`, no detail | Server logs per-field zod issues to its own logger, not to `info` | Parse locally with `OperationsSchema`, or run via the CLI | +| Task `error`: `No converter found for intent item/unpublish` | `item/unpublish` is unimplemented in the runner | Remove it; use the Core API `unpublishItem` mutation | +| Task `error`: `TypeError: Invalid Version: 1` | `version` is `1`, `1.0` or `*` — valid per the regex, invalid per semver | Use `"1.0.0"` | +| Task `error`: `No operations retrieved from spec file` | Empty `operations` array, or an unreadable/empty upload | Check the file actually uploaded | +| Task stuck in `pending` | `autoStart: false` and never started | Call `startMassOperationBulkTask` | +| Re-run created duplicate products/folders | Item `upsert` without `itemId` or `resourceIdentifier` always creates | Add a `resourceIdentifier` to every item upsert | +| Re-run created duplicate orders/contracts | `order/upsert` and `subscription-contract/upsert` only dedupe on a real `id` | Track ids, or accept non-idempotence | +| `ResourceIdentifier is not implemented yet` | That intent doesn't support it | See `intents.md` § "Where `resourceIdentifier` actually works" | +| Operation missing entirely from `operationLogs` | Converter returned no command | See `intents.md` § "Silent skips" | +| A field literally contains `{{ myRef.id }}` | Template render threw; the renderer returns the raw string | Fix the reference; check the intent's real `_ref` output | +| Reference resolves to an object, not an ID | Running at `version: "0.0.1"` | Use `"1.0.0"`, where upserts are normalised flat | +| `fetch*` returned pre-update data | Fetch cache isn't invalidated by that intent | Use an uncached helper (`*ByResourceIdentifier`, `fetchProductVariantBySku`) | +| Variant `topicIds` didn't apply | Stripped by the schema, not re-attached by the runner | Assign variant topics via another API | +| Component content saved but wrong | `item/updateComponent/item` runs with content validation disabled | Verify component IDs against `dump-content-model` | +| `400 Bad Request` on mutations | Missing union type fragments | Add `... on BulkTaskMassOperation` and `... on BasicError` | +| `Failed to fetch` on upload | Browser CORS restriction | Upload server-side or proxy the S3 request | +| `key` is undefined | Looking for a top-level `key` | Read it from the fields array: `fields.find(f => f.name === 'key').value` | +| Task `error` with no message | Not querying `info` | Add `info { error errorName stack }` to the `bulkTask` query | +| `Invalid component ID …` | Component doesn't exist on that shape | Re-check against `dump-content-model` output | + +## Browser CORS + +The S3 bucket may block cross-origin requests from a browser. Options, in order of preference: + +1. **Use the CLI** — recommended for production migrations +2. **Server-side proxy** — route uploads through your backend +3. **Dev proxy** — Vite/webpack dev server + +```javascript +export default defineConfig({ + server: { + proxy: { + '/api/s3-upload': { + target: 'https://crystallize-mass-operations-production.s3.eu-central-1.amazonaws.com', + changeOrigin: true, + rewrite: (path) => path.replace(/^\/api\/s3-upload/, ''), + }, + }, + }, +}); +``` diff --git a/use-crystallize/skills/mass-operations/references/limits.md b/use-crystallize/skills/mass-operations/references/limits.md new file mode 100644 index 0000000..3b06de8 --- /dev/null +++ b/use-crystallize/skills/mass-operations/references/limits.md @@ -0,0 +1,226 @@ +# Limits + +Every bound that can break a mass operation, or silently change what it writes. + +Compiled from an exhaustive audit of runner behaviour as of 2026-08-18, covering every component +type, primitive and value object. These are **observed limits, not documented API guarantees** — +re-check anything load-bearing before depending on it. + +## Contents + +- [How limits fail](#how-limits-fail) +- [File-level](#file-level) +- [Batch caps per operation](#batch-caps-per-operation) +- [Silent data loss](#silent-data-loss) +- [Config traps](#config-traps) +- [Component capacity](#component-capacity) +- [String and number bounds](#string-and-number-bounds) +- [Structural rules](#structural-rules) + +## How limits fail + +Three distinct modes. Knowing which one applies tells you whether to expect an error at all. + +| Mode | What you see | Blast radius | +| --- | --- | --- | +| **schema-validation** | task `error`, `Invalid Operation File`, nothing runs | the whole file | +| **runtime-throw** | that operation logs `failure` / 500, run continues | one operation | +| **silent** | **nothing** — no error, no log, wrong data stored | however many rows carry it | + +The silent class is the dangerous one and it is large. Local validation cannot catch any of it, +because the value is legal — it is the *behaviour* that differs from what you wrote. + +## File-level + +| Limit | Value | Mode | +| --- | --- | --- | +| **One invalid operation rejects the entire file** | all-or-nothing `safeParse` | schema-validation | +| Spec file size (presigned POST content-length-range) | **50 MiB** max, 1 byte min (`UPLOAD_MAX_SIZE`) | upload rejected | +| Offload to standalone task | **1 MiB** (`MASS_OPERATIONS_STANDALONE_TASK_FILE_SIZE_THRESHOLD`) | none — runs elsewhere | +| Operations per file | **no cap** | — | +| `version` format | `/^(\d+\.)?(\d+\.)?(\*|\d+)$/` — but only `1.0.0` works, see `SKILL.md` | runtime-throw | + +**All-or-nothing validation is the one to internalise.** There is no partial acceptance: operation +500 of 500 being malformed means operations 1–499 never run. Chunk large files so one bad row costs +you one chunk, and validate locally before every upload. + +## Batch caps per operation + +| Intent | Cap | Counting subtlety | +| --- | --- | --- | +| `topic/create`, `topic/upsert` | **30 topics** | counts the **whole subtree recursively** — 1 root + 29 descendants | +| `item/flow/stage/addItems` | **50 items** | raw array length; throws during conversion, so the op is skipped with a 500 | +| `order/*` | **500 cart items** | | +| `order/*` | **100 applied promotions**, **50 related orders** | | +| Any item or variant | **250 `topicIds`** | product variants share one **deduplicated union** across all variants | +| `flow/*` | **30 stages**, **8 actions per stage** | nested `onFailure` actions are **not** counted | +| `customer/*`, `customer/group/*` | **5 parents**, **20 addresses**, hierarchy depth **5** | | + +`item/publish` has a 50-id batch cap that you **cannot reach** — the converter always sends exactly +one `itemId`. + +## Silent data loss + +Nothing below raises. Ordered by how many intents carry the risk. + +### The component-content mapper + +All component content passes through a single input mapper, and it is the largest source of silent +loss in the whole format. + +- **Only the first present type key is used.** `{componentId: "media", images: [...], videos: [...]}` + writes one and **silently discards the other**. Same rule in the config mapper. +- **`null` is coerced to empty, wiping the component.** `{"componentId":"gallery","images":null}` + stores an empty list rather than erroring. Same for `numeric: null` (writes a contentless + component), `chunks` missing/null (empty chunk list), `paragraphs` omitted (**wipes the whole + collection**), per-paragraph `images` omitted (wipes that paragraph's images). +- **Unrecognised component types are dropped** by `filterComponents()` via the mapper's `unknownType` + branch — no error, the component simply does not appear. +- **Stored content is dropped when its type no longer matches the definition**, e.g. after a shape + change. + +### `propertiesTable` is rebuilt from the shape config + +If the shape config declares `sections`, the content constructor **ignores your structure entirely** +and regenerates it: + +- sections are matched **by array index only**; sections beyond `config.sections.length` are dropped +- property keys not declared in the config are dropped +- your section title is **discarded and replaced** by the config's +- property order is forced to the config's key order +- every configured key is materialised as a valueless row, so omitting a key does not omit the row +- mentioning the component with an empty content object writes a **full empty skeleton** +- a property `value: null` is silently dropped, storing a bare key + +Treat `propertiesTable` as config-driven: the shape decides the shape of the data, not your payload. + +### `numeric` decimal places + +- **`decimalPlaces` FLOORS the stored number.** It is not a display setting — the value is truncated + on write, and it does not round. +- **`decimalPlaces: 0` takes a `parseInt(String(...))` branch that destroys small-magnitude numbers.** +- An empty-string `unit` is silently dropped in content, but **throws** in shape config. + +### Choice and selection filter to configured options + +- A `componentChoice` selection that is not one of the configured choices is **silently discarded**. +- `componentMultipleChoice` entries not matching a configured choice are **filtered out**. +- `selection` keys not present in the shape config's options are **filtered out**. +- `itemRelations`: falsy entries in `itemIds`/`skus` are filtered; an all-falsy array becomes empty. + +In every case the write "succeeds" with fewer values than you sent. If you are seeding a shape and its +content in one file, an ordering mistake (step 5 in `SKILL.md`) shows up here as quiet data loss +rather than an error. + +### Strings, dates, identifiers + +- **`ValidatedString` never trims.** Leading/trailing whitespace is stored *and counts toward every + length cap*. +- **`ValidatedString` coerces via `toString()`** — a number or boolean becomes its string form rather + than being rejected. An explicit `null` raises a raw `TypeError`. +- **Datetime is re-parsed by `new Date()` and re-emitted as UTC ISO.** Offsets are normalised away. +- **Topic `pathIdentifier` is sliced to 64 chars *before* slugification**, so a long topic name + silently produces a different stored path. +- **`KeyValuePair` coerces an empty-string value to `null`.** +- **Tree-path collisions are silently rewritten** with a random suffix — one 5-char attempt, one + 10-char, then a forced 15-char. Your requested path is not necessarily the stored one. +- **Colour entries are whitelisted to exactly seven fields**; anything else is dropped. + +### Retention + +- **Path history keeps 100 entries** per item/language, enforced by a Mongo `$slice: -100`. No error + type exists. +- **Archived published versions keep 50** per item/language; older ones are removed on publish. + +### Images + +- Remote download cap **50 MiB**, per-image timeout **30 s**. Both surface as separate `image/upload` + log entries with status 500 — **the operation that queued the image still reports success.** + +## Config traps + +Cases where two components behave oppositely from the same JSON. + +- **`max: 0` on a `files` component silently means 512.** A `max` of `0` is treated as "unset" for files and + falls back to the ceiling. The same `max: 0` on `images` or `videos` rejects **every** entry. + Identical JSON, opposite outcome. +- **`required: false` disables the configured `min` entirely.** `{min: 3, required: false}` accepts + zero items. **Omitting `required` is stricter than setting it to `false`.** +- **`richText` `min`/`max` of 0 is a silent no-op** — the opposite of `singleLine`. +- **`item/updateComponent/item` skips ALL component content validation**: min/max counts, file size, MIME type and `required` are simply not executed. The identical payload + via `item/updateComponent/sku`, `product/upsert`, `document/upsert` **is** validated and throws. + Only reference existence and type are still checked. Enforcement depends on which intent you chose. +- **Order line items and payment objects are `.strict()`** — they reject *any* unknown key, while + nearly every other schema silently strips them. The one place a typo errors instead of vanishing. + +## Component capacity + +Ceilings on the `max` you may configure, and therefore on content. + +| Component | Cap | +| --- | --- | +| `itemRelations` | **75** (quick-select folders: 100) | +| `images`, `files`, `videos`, `gridRelations` | **512** | +| `colors` | **100** | +| `numeric` `decimalPlaces` | 0–**64** | +| Configurable `min`/`max` bounds | max **1048576**, min **256** | +| Component nesting depth | **5**, following piece expansion | + +## String and number bounds + +| Field | Bound | +| --- | --- | +| **Default for every unqualified string** | min 1, **max 256** | +| Shape / piece / flow identifier | **2–64**, charset `[A-Za-z0-9]` plus `; : + @ (` and space | +| `resourceIdentifier` | 1–**256**, charset `[A-Za-z0-9]` plus `. - _ / @` | +| `externalReference` | 1–256 | +| Item / catalogue item name | **512** | +| Product variant name | **1024** | +| SKU | **512** | +| Variant attribute key / value | **128** / **2048** | +| Component name / description | 256 / 1024 | +| `singleLine` text, meta value, tree path | **1048576** | +| Meta key | 256 | +| Image alt text / URL | 1024 / 10240 | +| Order `additionalInformation` | 10240 | +| Language code | 2–20 | +| Currency code | 10 | +| Topic display colour | 4–9, `/^#([A-Fa-f0-9]{3}|[A-Fa-f0-9]{6})$/` | +| Uploaded filename (basename of the `{{upload}}` URL) | 3–512 | +| Every `id` field | exactly 24 lowercase hex, or `{{ ... }}` | +| Price | −1e9 … 1e9 | +| Percent (order tax & discount) | −1000 … 1000 | +| Order cart item quantity | 0 … 1e6 | +| Item tree position | 1 … 100000 | +| Latitude / longitude | ±90 / ±180 | +| Focal point | x, y each 0 … 1 | +| **`product/variant/price/modify` price** | **minimum 1** — you cannot set 0 this way | +| **`product/variant/stock/modify` quantity** | **positive integer** | +| Stock (elsewhere) | `ValidatedInteger`, **fractional input silently truncated** | + +## Structural rules + +Minimums and shape rules, not maximums. + +- `componentChoice` requires **≥ 2** choices; `componentMultipleChoice` **≥ 1**; `contentChunk` **≥ 1** + component. +- `contentChunk` with `repeatable: false` accepts exactly **1** chunk. +- `selection` needs ≥ 1 option, keys unique. +- `product/*` requires a **non-empty `variants`** array; `pricelist/*` a non-empty `priceVariants`; + `item/flow/stage/addItems` a non-empty `items`. +- Component identifiers must be **unique within a component list**. +- `min` must not exceed `max` in any component config. +- **Structural components cannot nest** — `contentChunk`, `componentChoice`, `componentMultipleChoice` + can never be direct children of each other. Put a piece in between. +- `variantComponents` are allowed **only on product shapes**. +- Handlebars enrichment substitutes into **string values only** — a `{{ ... }}` inside a number or + boolean is never resolved. +- At least one of `resourceIdentifier` / `id` / `itemId` / `topicId` is required where + `checkResourceIdentifierOrId` applies. + +## Operational note + +A worker that dies mid-run leaves the task stuck in `started`. SQS redelivery after the 90 s +visibility timeout is **refused**, because `getTask` accepts only `pending` tasks and `markStarted` +fires before the run — so there is no double execution, but also no automatic retry. A task sitting in +`started` with no progress needs manual intervention. diff --git a/use-crystallize/skills/mass-operations/references/templating.md b/use-crystallize/skills/mass-operations/references/templating.md new file mode 100644 index 0000000..5305a86 --- /dev/null +++ b/use-crystallize/skills/mass-operations/references/templating.md @@ -0,0 +1,244 @@ +# Templating: references, helpers, and variables + +Operations run sequentially, so a later operation can consume an earlier one's output. The templating +layer is Handlebars (async, via `handlebars-async-helpers`) — `{{ ... }}` expressions are resolved by +the runner immediately before each operation is converted to a command. + +**The schema is templating-aware.** `IdSchema` is `/^(?:[0-9a-f]{24}|{{.*}})$/`: an ID field accepts a +24-character hex ID *or* a handlebars expression, and nothing else. A placeholder like `"TODO"` or +`""` fails validation immediately — which is useful, because it means malformed references +are caught locally rather than at runtime. + +Verified against runner behaviour as of 2026-08-18. + +## Contents + +- [How enrichment works](#how-enrichment-works) +- [`_ref` chaining](#_ref-chaining) +- [Positional references](#positional-references) +- [The upsert return-shape trap (legacy only)](#the-upsert-return-shape-trap-legacy-only) +- [Default context variables](#default-context-variables) +- [Uploading images](#uploading-images) +- [Fetch helpers](#fetch-helpers) +- [`valueOf`](#valueof) + +## How enrichment works + +`MassOperationEnricherService.enrich` walks the operation object recursively and renders **every string +value** as a Handlebars template. Two consequences: + +- Any string field can contain `{{ ... }}`, not just ID fields. +- The `_ref` key itself is skipped, so a `_ref` name is never templated. + +**A template that throws is swallowed.** `MassOperationTemplateRenderer.renderTemplate` catches the +error, logs it, and returns the **original template string**. So a broken reference does not fail the +operation — it writes the literal `{{ myRef.id }}` into your data, and you find out later. On an ID +field the downstream command then fails with a confusing cast error; on a text field it just persists. +Grep your `operationLogs` output for `{{` after a run. + +## `_ref` chaining + +Tag an operation with `_ref` (a non-empty string), then reference its output by that name. + +```json +[ + { + "_ref": "rooms", + "intent": "folder/create", + "name": "My Rooms", + "language": "en", + "shapeIdentifier": "folder", + "tree": { "parentId": "6902a7a78f7d45cf23343fab" } + }, + { + "intent": "folder/create", + "name": "My Kitchen", + "language": "en", + "shapeIdentifier": "folder", + "tree": { "parentId": "{{ rooms.id }}" } + } +] +``` + +"My Kitchen" is created inside "My Rooms". + +Which properties a `_ref` exposes depends on the intent — see `intents.md` § "Properties exposed to +`_ref`". Some outputs are a bare string rather than an object; for those, use `{{ myRef }}` directly. +An operation whose converter produced no command saves nothing at all, so a `_ref` to it resolves to +empty (see `intents.md` § "Silent skips"). + +## Positional references + +Undocumented, but implemented and unit-tested: a reference whose first segment parses as a number is +looked up by **operation number** rather than by `_ref`. + +```handlebars +{{ 0.firstName }} +``` + +`MassOperationTemplateRenderer.getOperationId` does `Number.isNaN(+id)` and routes to +`findByOperationNumber(taskId, n)` when it's numeric. + +The counter is **0-based and assigned at save time**, not from your file's array index: +`MassOperationRepository.save` sets `operationNumber = count(existing rows for this task)`. Only +operations that actually produced a command are saved — a silently-skipped operation writes no row, +so everything after it shifts down by one relative to the file. + +**Prefer named `_ref`.** Positional references are brittle against exactly the failure mode that is +hardest to notice. + +## The upsert return-shape trap (legacy only) + +The public docs warn that an upsert returns `{ id }` when it creates and `{ id: { id } }` when it +updates, and prescribe a Handlebars conditional to cope. + +**That applies to `version: "0.0.1"` only.** At `version: "1.0.0"` the runner flattens every upsert +result before storing it, whichever path the backend took — create or update. `product/upsert`, +`folder/upsert` and `document/upsert` all yield a flat `{ id: string }`. `shape/upsert`, `piece/create|upsert` and +`customer/create|update|upsert` normalise to `{ identifier: string }`. + +So at `1.0.0`, this is correct and sufficient: + +```handlebars +{{ rootProducts.id }} +``` + +The defensive form still works — `id.id` is undefined on a flat `{id}`, so the `else` branch fires — +but it is noise: + +```handlebars +{{#if rootProducts.id.id}}{{ rootProducts.id.id }}{{else}}{{ rootProducts.id }}{{/if}} +``` + +Keep it only if you genuinely target `version: "0.0.1"`. You should not. + +## Default context variables + +Injected by the runner without any `_ref`, assembled once per (tenant, task) by `DefaultsService`: + +| Variable | Source | +| --- | --- | +| `{{ defaults.taskId }}` | the bulk task id | +| `{{ defaults.tenantId }}` | tenant ObjectId | +| `{{ defaults.tenantIdentifier }}` | tenant identifier, falling back to the id | +| `{{ defaults.rootItemId }}` | the catalogue tree root | +| `{{ defaults.languages.[0] }}` | **tenant default language**; `[1..]` are the custom languages | +| `{{ defaults.vatTypeIds.[0] }}` | vat type ids, **first 100 only** (the repository query is capped) | + +These are the idiomatic way to satisfy required ID fields portably. `product/create` and +`product/upsert` both require `vatTypeId`, which must match `IdSchema` — so +`"vatTypeId": "{{ defaults.vatTypeIds.[0] }}"` is how you write a product operation that runs against +any tenant. Likewise `{{ defaults.rootItemId }}` for a top-level `tree.parentId`. + +Note the `.[0]` bracket syntax for array indexing — Handlebars requires it for numeric segments. +`defaults.vatTypeIds` has no guaranteed ordering, so `[0]` means "some vat type", not "the default +one" — name the vat type explicitly if it matters. + +## Uploading images + +```handlebars +{{ upload "https://my-image.com/image.jpg" }} +{{ upload "https://my-image.com/image.jpg" "hero-image-1" }} +``` + +The optional **second argument is a `resourceIdentifier`** (undocumented). If an image is already +registered under it in this tenant, `upload` returns that image's existing key and nothing is +re-fetched — this is what makes image imports re-runnable. + +```json +{ + "intent": "image/register", + "key": "{{ upload \"https://my-image.com/image.jpg\" \"hero-image-1\" }}", + "resourceIdentifier": "hero-image-1" +} +``` + +Mechanics worth knowing: + +- `upload` **does not upload during the expression.** It reserves a storage key and queues the job. + The actual download-and-upload runs after the operation completes, on a `p-queue` with concurrency + **10**, and the runner awaits `queue.onIdle()` before the next operation. +- Within one task, repeated calls with the same URL return the same key (deduped in the runner's + SQLite scratch DB) — so calling `upload` inline in several operations is safe. +- The filename is `path.basename(sourceUrl)`, so query strings and redirect URLs can produce odd names. +- Image uploads are logged as their own entries with `"intent": "image/upload"` and statusCode 200/500, + **separate** from the operation that called `upload`. A failed image does not fail its operation. +- You can call `upload` inline anywhere a key is expected in another operation. + +The file is strict JSON, so inner double quotes must be escaped as `\"` — or use single quotes, as the +fetch helpers below do. + +## Fetch helpers + +For pulling in entities that already exist in the tenant, rather than ones created earlier in the file. +Ten helpers, all registered via `@HandlebarsHelper`: + +| Helper | Arguments | +| --- | --- | +| `fetchItemById` | `itemId`, `language`, `version` | +| `fetchItemByResourceIdentifier` | `resourceIdentifier`, `language` | +| `fetchOrderById` | `orderId` | +| `fetchSubscriptionContractById` | `id` | +| `fetchCustomerByIdentifier` | `identifier` | +| `fetchCustomerGroupByIdentifier` | `identifier` | +| `fetchPriceListByIdentifier` | `identifier` | +| `fetchProductVariantBySku` | `sku`, `language` | +| `fetchTopicByResourceIdentifier` | `resourceIdentifier`, `language` | +| `fetchImageByResourceIdentifier` | `resourceIdentifier`, `language` | + +**Always pass every argument explicitly.** Handlebars appends its own options hash as the final +argument, so omitting `language` puts that object into the `language` slot rather than falling back to +a default. The runner's own tests call the helper as +`fetchItemById('item-1', 'en', 'published', {})` — the trailing `{}` is the options hash. + +```handlebars +{{ valueOf (fetchItemById '626af86d6f9c909e39ecb8a6' 'en' 'published') 'name' }} +{{ valueOf (fetchOrderById '626af86d6f9c909e39ecb8a6') 'reference' }} +``` + +`fetchItemById`'s `version` argument is coerced: **only the literal `'published'` is honoured, anything +else silently becomes `'draft'`** (`version === 'published' ? 'published' : 'draft'`). So `'latest'` or +`'live'` quietly reads the draft. + +### Caching and staleness + +Six helpers read through a per-task cache (`MassOperationResourceCacheService`): `fetchItemById`, +`fetchOrderById`, `fetchCustomerByIdentifier`, `fetchCustomerGroupByIdentifier`, +`fetchPriceListByIdentifier`, `fetchSubscriptionContractById`. The cache is emptied at task start and +end. + +After each successful operation the runner calls `refreshFetchedResourceCache`, which invalidates only +these intents: + +`product|document|folder /update|upsert` · `order/update|upsert` · `customer/update|upsert` · +`customer/group/update|upsert` · `pricelist/update|upsert` · `subscription-contract/update|upsert` + +**Anything else leaves the cache stale.** In particular `item/updateComponent/item|sku`, +`product/variant/*`, `topic/*` and `image/register` do *not* invalidate — so a `fetchItemById` after a +component update in the same task can return pre-update data. The item refresh is also keyed on +`operation.itemId`, so an update resolved via `resourceIdentifier` alone may not invalidate either. + +The four uncached helpers — `fetchItemByResourceIdentifier`, `fetchProductVariantBySku`, +`fetchTopicByResourceIdentifier`, `fetchImageByResourceIdentifier` — hit the repository every time and +are always fresh. Prefer them when you need to read something you just wrote. + +`fetchOrderById` and `fetchSubscriptionContractById` load by raw id and then verify tenant ownership, +returning `null` for a foreign-tenant record — indistinguishable from "not found". + +The `*ByResourceIdentifier` / `*ByIdentifier` variants pair naturally with writing `resourceIdentifier` +on your create/upsert operations: you set a stable key on the way in, then look entities up by that key +later, without ever hardcoding a Crystallize ID. This is the main workaround for intents whose `_ref` +output is unhelpful, and for the intents where the runner ignores `resourceIdentifier` (see +`intents.md`). + +## `valueOf` + +Reads a property off a helper result. It **splits on `.`**, so dotted paths work: + +```handlebars +{{ valueOf (fetchItemById '626af86d…' 'en' 'draft') 'name' }} +{{ valueOf (fetchItemById '626af86d…' 'en' 'draft') 'components.description.content.plainText' }} +``` + +Implementation is `prop.split('.').reduce((acc, curr) => acc?.[curr], await obj)`, so a missing segment +yields `undefined` (rendered as an empty string) rather than throwing.