fix(init): scaffolded CI key guidance names policy:read

CHANGELOG.md

@@ -24,6 +24,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- The scaffolded CI workflow's key guidance (`evalshift init --ci`) named only
+  `run:create` + `run:read`. That workflow defaults to `fail-on: policy`, which
+  reads the hosted policy-check endpoint (`policy:read`); a key scoped to only
+  `run:create` + `run:read` got a 403 there and the action silently fell back
+  to `fail-on: regression`. The scaffold, `DOCS.md`, `llms-full.txt`,
+  `docs/github-action.md`, and `docs/hosted.md` now name `run:create` +
+  `run:read` + `policy:read` for the CI key.
+
 - Under SQLAlchemy 2.1, which fresh installs resolve (`sqlalchemy>=2.0`), a
   `CacheStore` opened on an in-memory SQLite database could silently lose
   concurrent writes. The default on-disk cache used by CLI runs was not

DOCS.md

@@ -795,7 +795,7 @@ The CLI checks for this wherever it writes or validates config — `capture sync
 
 Equal pins, `${{ }}` expressions, unparseable versions, and an editable install without metadata (`0.0.0+unknown`) are silent. The check is advisory: it never edits a workflow and never changes an exit code, and in CI it is a no-op by construction (the running CLI *is* the pin). Config `version: 1` is not bumped for additive fields, nor for a removal that fails the load with a message naming the key — see [Configuration](docs/configuration.md#config-version-policy).
 
-Secrets needed: a provider API key matching your config's models, and `EVALSHIFT_TOKEN` — a service account key from Settings → API tokens → Service accounts, scoped to `run:create` + `run:read`, stored as an encrypted repository or environment secret. Not a personal token, never a literal in the workflow YAML, and never reachable from `pull_request_target`. Rotate by minting the successor first (24h grace), updating the secret, confirming a green run, then letting the old key expire. One thing a scoped key can't do, by design: auto-create the project (`project:create` is owner-only — pre-create it and set `create-project: false`). Full guidance: the action's [README](https://github.com/babaliauskas/evalshift-action#readme).
+Secrets needed: a provider API key matching your config's models, and `EVALSHIFT_TOKEN` — a service account key from Settings → API tokens → Service accounts, scoped to `run:create` + `run:read` + `policy:read`, stored as an encrypted repository or environment secret. Not a personal token, never a literal in the workflow YAML, and never reachable from `pull_request_target`. Rotate by minting the successor first (24h grace), updating the secret, confirming a green run, then letting the old key expire. One thing a scoped key can't do, by design: auto-create the project (`project:create` is owner-only — pre-create it and set `create-project: false`). Full guidance: the action's [README](https://github.com/babaliauskas/evalshift-action#readme).
 
 ---
 

docs/github-action.md

@@ -64,9 +64,11 @@ your project.
 ## Required secrets
 
 - `EVALSHIFT_TOKEN`: a **service account key** from the web app (Settings → API
-  tokens → Service accounts), scoped to `run:create` + `run:read`. Not a personal
-  token — that one dies with its owner's membership and takes the pipeline with
-  it. A scoped key cannot auto-create the hosted project (`project:create` is
+  tokens → Service accounts), scoped to `run:create` + `run:read` + `policy:read`.
+  Not a personal token — that one dies with its owner's membership and takes the
+  pipeline with it. `policy:read` is what lets the default `fail-on: policy` gate
+  read the hosted verdict; without it the check falls back to `fail-on: regression`
+  silently. A scoped key cannot auto-create the hosted project (`project:create` is
   owner-only), so create the project once in the web app and set
   `create-project: false`.
 - Provider API keys used by the source, target, judge, or embedding models.

docs/hosted.md

@@ -44,7 +44,8 @@ For CI, mint a **service account key** instead. In the hosted web app: Settings
 tokens → Service accounts. A service account is an org-owned machine identity that never
 consumes a seat and can only ever hold the `member` or `viewer` role, so no CI credential is
 owner-equivalent. Scope the key to the permission keys the job actually needs (`run:create`
-plus `run:read` is enough to push a run and read a diff), store it as an encrypted CI
+plus `run:read` plus `policy:read` covers pushing a run, reading a diff, and the default
+`fail-on: policy` gate), store it as an encrypted CI
 secret, and pass it as `EVALSHIFT_TOKEN` — do not run `evalshift login` on a runner.
 
 Keys rotate with an overlap: mint the successor, update the secret, confirm a green run,

llms-full.txt

@@ -205,8 +205,9 @@ evalshift login [--token es_...] [--host URL] [--no-browser] [--timeout SECS=900
   wrong for CI. For CI mint a SERVICE ACCOUNT KEY in the hosted web app (Settings -> API
   tokens -> Service accounts): org-owned machine identity, never owner-equivalent (role is
   `member` or `viewer` only), consumes no seat, survives the employee who created it. Scope it
-  to the permission keys the job needs (`run:create` + `run:read` covers push + diff), store it
-  as an encrypted CI secret, pass it as EVALSHIFT_TOKEN -- do not run `login` on a runner.
+  to the permission keys the job needs (`run:create` + `run:read` + `policy:read` covers push,
+  diff, and the default policy gate), store it as an encrypted CI secret, pass it as
+  EVALSHIFT_TOKEN -- do not run `login` on a runner.
   Rotation is overlapping keys: mint successor, update secret, confirm a green run, let the
   predecessor expire (24h default grace).
 evalshift logout | evalshift whoami [--host] [--token]

src/evalshift_cli/cli/commands/_scaffold.py

@@ -132,7 +132,8 @@
 #        EVALSHIFT_TOKEN       hosted EvalShift service-account key (es_...).
 #                              Mint it in the web app (org Settings -> API
 #                              tokens -> Service accounts) scoped to
-#                              run:create + run:read. Not a personal token.
+#                              run:create + run:read + policy:read. Not a
+#                              personal token.
 #        __PROVIDER_API_KEY__      key for the provider your evalshift.yaml models
 #                              use. Add further keys here (and under `env:`
 #                              below) if your judge/embedding models live in

tests/unit/test_init.py

@@ -418,6 +418,15 @@ def test_gates_on_hosted_policy_verdict(self, in_tmp: Path) -> None:
         assert "fail-on: policy" in body
         assert "fail-on: regression" not in body
 
+    def test_token_scope_names_policy_read(self, in_tmp: Path) -> None:
+        # The scaffolded workflow defaults to `fail-on: policy`, which reads
+        # the hosted policy-check endpoint (`policy:read`). A key scoped to
+        # only `run:create` + `run:read` gets a 403 on that check and the
+        # action silently falls back to `fail-on: regression` -- so the
+        # scaffolded key guidance must name all three scopes together.
+        body, _ = self._workflow(in_tmp)
+        assert "run:create + run:read + policy:read" in body
+
     def test_pins_the_scaffolding_cli_version(self, in_tmp: Path) -> None:
         body, _ = self._workflow(in_tmp)
         assert f'evalshift-version: "{evalshift_cli.__version__}"' in body