diff --git a/CHANGELOG.md b/CHANGELOG.md index 82bd8e6..6edcbc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,6 +24,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- The scaffolded CI workflow's key guidance (`evalshift init --ci`) named only + `run:create` + `run:read`. That workflow defaults to `fail-on: policy`, which + reads the hosted policy-check endpoint (`policy:read`); a key scoped to only + `run:create` + `run:read` got a 403 there and the action silently fell back + to `fail-on: regression`. The scaffold, `DOCS.md`, `llms-full.txt`, + `docs/github-action.md`, and `docs/hosted.md` now name `run:create` + + `run:read` + `policy:read` for the CI key. + - Under SQLAlchemy 2.1, which fresh installs resolve (`sqlalchemy>=2.0`), a `CacheStore` opened on an in-memory SQLite database could silently lose concurrent writes. The default on-disk cache used by CLI runs was not diff --git a/DOCS.md b/DOCS.md index 60baea7..504fa8e 100644 --- a/DOCS.md +++ b/DOCS.md @@ -795,7 +795,7 @@ The CLI checks for this wherever it writes or validates config — `capture sync Equal pins, `${{ }}` expressions, unparseable versions, and an editable install without metadata (`0.0.0+unknown`) are silent. The check is advisory: it never edits a workflow and never changes an exit code, and in CI it is a no-op by construction (the running CLI *is* the pin). Config `version: 1` is not bumped for additive fields, nor for a removal that fails the load with a message naming the key — see [Configuration](docs/configuration.md#config-version-policy). -Secrets needed: a provider API key matching your config's models, and `EVALSHIFT_TOKEN` — a service account key from Settings → API tokens → Service accounts, scoped to `run:create` + `run:read`, stored as an encrypted repository or environment secret. Not a personal token, never a literal in the workflow YAML, and never reachable from `pull_request_target`. Rotate by minting the successor first (24h grace), updating the secret, confirming a green run, then letting the old key expire. One thing a scoped key can't do, by design: auto-create the project (`project:create` is owner-only — pre-create it and set `create-project: false`). Full guidance: the action's [README](https://github.com/babaliauskas/evalshift-action#readme). +Secrets needed: a provider API key matching your config's models, and `EVALSHIFT_TOKEN` — a service account key from Settings → API tokens → Service accounts, scoped to `run:create` + `run:read` + `policy:read`, stored as an encrypted repository or environment secret. Not a personal token, never a literal in the workflow YAML, and never reachable from `pull_request_target`. Rotate by minting the successor first (24h grace), updating the secret, confirming a green run, then letting the old key expire. One thing a scoped key can't do, by design: auto-create the project (`project:create` is owner-only — pre-create it and set `create-project: false`). Full guidance: the action's [README](https://github.com/babaliauskas/evalshift-action#readme). --- diff --git a/docs/github-action.md b/docs/github-action.md index 70949cc..5b34ad5 100644 --- a/docs/github-action.md +++ b/docs/github-action.md @@ -64,9 +64,11 @@ your project. ## Required secrets - `EVALSHIFT_TOKEN`: a **service account key** from the web app (Settings → API - tokens → Service accounts), scoped to `run:create` + `run:read`. Not a personal - token — that one dies with its owner's membership and takes the pipeline with - it. A scoped key cannot auto-create the hosted project (`project:create` is + tokens → Service accounts), scoped to `run:create` + `run:read` + `policy:read`. + Not a personal token — that one dies with its owner's membership and takes the + pipeline with it. `policy:read` is what lets the default `fail-on: policy` gate + read the hosted verdict; without it the check falls back to `fail-on: regression` + silently. A scoped key cannot auto-create the hosted project (`project:create` is owner-only), so create the project once in the web app and set `create-project: false`. - Provider API keys used by the source, target, judge, or embedding models. diff --git a/docs/hosted.md b/docs/hosted.md index f6831be..5dbfd63 100644 --- a/docs/hosted.md +++ b/docs/hosted.md @@ -44,7 +44,8 @@ For CI, mint a **service account key** instead. In the hosted web app: Settings tokens → Service accounts. A service account is an org-owned machine identity that never consumes a seat and can only ever hold the `member` or `viewer` role, so no CI credential is owner-equivalent. Scope the key to the permission keys the job actually needs (`run:create` -plus `run:read` is enough to push a run and read a diff), store it as an encrypted CI +plus `run:read` plus `policy:read` covers pushing a run, reading a diff, and the default +`fail-on: policy` gate), store it as an encrypted CI secret, and pass it as `EVALSHIFT_TOKEN` — do not run `evalshift login` on a runner. Keys rotate with an overlap: mint the successor, update the secret, confirm a green run, diff --git a/llms-full.txt b/llms-full.txt index 4c6bfae..9295028 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -205,8 +205,9 @@ evalshift login [--token es_...] [--host URL] [--no-browser] [--timeout SECS=900 wrong for CI. For CI mint a SERVICE ACCOUNT KEY in the hosted web app (Settings -> API tokens -> Service accounts): org-owned machine identity, never owner-equivalent (role is `member` or `viewer` only), consumes no seat, survives the employee who created it. Scope it - to the permission keys the job needs (`run:create` + `run:read` covers push + diff), store it - as an encrypted CI secret, pass it as EVALSHIFT_TOKEN -- do not run `login` on a runner. + to the permission keys the job needs (`run:create` + `run:read` + `policy:read` covers push, + diff, and the default policy gate), store it as an encrypted CI secret, pass it as + EVALSHIFT_TOKEN -- do not run `login` on a runner. Rotation is overlapping keys: mint successor, update secret, confirm a green run, let the predecessor expire (24h default grace). evalshift logout | evalshift whoami [--host] [--token] diff --git a/src/evalshift_cli/cli/commands/_scaffold.py b/src/evalshift_cli/cli/commands/_scaffold.py index b3bfd14..408a340 100644 --- a/src/evalshift_cli/cli/commands/_scaffold.py +++ b/src/evalshift_cli/cli/commands/_scaffold.py @@ -132,7 +132,8 @@ # EVALSHIFT_TOKEN hosted EvalShift service-account key (es_...). # Mint it in the web app (org Settings -> API # tokens -> Service accounts) scoped to -# run:create + run:read. Not a personal token. +# run:create + run:read + policy:read. Not a +# personal token. # __PROVIDER_API_KEY__ key for the provider your evalshift.yaml models # use. Add further keys here (and under `env:` # below) if your judge/embedding models live in diff --git a/tests/unit/test_init.py b/tests/unit/test_init.py index 263d21b..32bd98c 100644 --- a/tests/unit/test_init.py +++ b/tests/unit/test_init.py @@ -418,6 +418,15 @@ def test_gates_on_hosted_policy_verdict(self, in_tmp: Path) -> None: assert "fail-on: policy" in body assert "fail-on: regression" not in body + def test_token_scope_names_policy_read(self, in_tmp: Path) -> None: + # The scaffolded workflow defaults to `fail-on: policy`, which reads + # the hosted policy-check endpoint (`policy:read`). A key scoped to + # only `run:create` + `run:read` gets a 403 on that check and the + # action silently falls back to `fail-on: regression` -- so the + # scaffolded key guidance must name all three scopes together. + body, _ = self._workflow(in_tmp) + assert "run:create + run:read + policy:read" in body + def test_pins_the_scaffolding_cli_version(self, in_tmp: Path) -> None: body, _ = self._workflow(in_tmp) assert f'evalshift-version: "{evalshift_cli.__version__}"' in body