From fa469ac8fea1ed8ac2820aac6978dc07216edc4e Mon Sep 17 00:00:00 2001 From: Evan Hearne Date: Thu, 20 Aug 2026 09:27:46 +0100 Subject: [PATCH] add AI contextification files. This commit adds AGENTS.md to give agents context on the repo from a glance, and also modifies README.md so that the purpose of the repo is clearer to newcomers. CLAUDE.md adds a symlink to AGENTS.md as Claude does not read AGENTS.md out of the box. README.md adds examples of actual use, explains clearly the purpose of the repo, and dives into how to work with the project. --- AGENTS.md | 116 +++++++++++++++++++++++++++ CLAUDE.md | 1 + README.md | 234 ++++++++++++++++++++++++++++++++++++++++++++++++++---- 3 files changed, 337 insertions(+), 14 deletions(-) create mode 100644 AGENTS.md create mode 100644 CLAUDE.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..6464f64c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,116 @@ +# AGENTS.md + +Guidance for AI agents working in this repository. Read this before making changes. +See `README.md` for the user-facing explanation of what the project does. + +## What this repo is + +`multi-operator-manager` (MOM) is an OpenShift Go module that is **both a CLI tool and a +library**: + +- The **tool** (`multi-operator-manager` binary) tests, debugs, and analyzes OpenShift + operators *offline* (no live cluster). +- The **library** (`pkg/library/...`) is imported by real operators so they become + "compatible" — i.e. expose the three MOM verbs. + +A compatible operator turns its reconcile logic into a function of its inputs: given input +resources, it computes the mutations it *would* make, without touching a cluster. It does +this via three subcommands: `input-resources`, `apply-configuration`, `output-resources`. + +## The core: `pkg/library` + +`pkg/library` is the heart of the project — the shared implementation of the operator +contract, and the part other repos import. It does two jobs: + +- **Defines the contract.** It provides the ready-made cobra commands for the three verbs and + the types they exchange, so an operator doesn't implement them from scratch. An operator + becomes compatible by importing these packages and supplying three things: its input list, + its output list, and its decision logic (`pkg/sampleoperator` is the worked example). +- **Runs and checks the results.** It contains the machinery that drives an operator once + (`SimpleOperatorStarter` and the run-once helpers), writes the resulting mutations to disk, + and compares two result sets for equivalence — the same logic the test harness relies on. + +Its three subpackages map one-to-one to the verbs: + +- `libraryinputresources/` — the `input-resources` verb and input types. +- `libraryapplyconfiguration/` — the `apply-configuration` verb, the run-once machinery, and + mutation output/comparison. +- `libraryoutputresources/` — the `output-resources` verb and output types. + +Because this is the public surface other operators depend on, treat changes to its exported +types and function signatures as **breaking changes** and weigh them accordingly. + +For a real-world consumer of this contract, see the +[cluster-authentication-operator example in the README](README.md#real-world-example-cluster-authentication-operator) +(`pkg/sampleoperator` is the in-repo example). + +## Commands + +``` +make build # build both binaries (multi-operator-manager, sample-operator) into repo root +make check # verify + unit tests (run before considering a change done) +make test-unit # go test ./pkg/... ./cmd/... +make verify # gofmt/vet/etc. via build-machinery-go +make test-operator-integration # build + run sample-operator against test-data, compare vs expected output +make update-test-operator-integration # same, but REGENERATE the expected output (use after intended changes) +make test-e2e # requires openshift-tests in PATH; rarely needed locally +make clean # remove built binary + test-output +``` + +Run a single package's tests directly with `go test ./pkg/library/libraryinputresources/...`. + +## Repo layout + +- `cmd/` — two `main` packages: `multi-operator-manager` and `sample-operator`. +- `pkg/cmd/multi-operator-manager/` — the cobra CLI tree (`test`, `sample-operator`, + `create-input-resources`). +- `pkg/library/` — the reusable contract; **this is the public API other repos import**: + - `libraryinputresources/` — `input-resources` verb + input types. + - `libraryapplyconfiguration/` — `apply-configuration` verb, the operator launch/run-once + machinery, and mutation output/comparison. + - `libraryoutputresources/` — `output-resources` verb + output types. +- `pkg/sampleoperator/` — reference compatible operator (the example to copy from). +- `pkg/test/testapplyconfiguration/` — the test harness behind `test apply-configuration`. +- `test-data/apply-configuration/*/` — integration fixtures: each dir with a `test.yaml` has + an `input-dir/` and an `expected-output/` (the known-good result to compare against). +- `vendor/` — dependencies are **vendored**; dep changes go through `go mod` + build-machinery + vendoring, not hand edits. + +## Conventions and gotchas + +- **Output must be reproducible in content, and order-independent.** `apply-configuration` + should compute the same *set* of desired mutations for the same inputs, regardless of the + order controllers happen to run in. To support this, time is injected via `Clock` / `--now` + and reads go through the injected `MutationTrackingClient` (in `ApplyConfigurationInput`) + rather than a live cluster or disk; and `SimpleOperatorStarter` shuffles controllers before + running them and rejects duplicate controller names. + + Practical consequence: do **not** introduce `time.Now()`, `math/rand`, env/file reads, or + network/cluster calls into operator logic — use the injected clock and client instead, and + don't rely on controller run order. +- **A test passes when the *meaningful* output matches — not when the files are identical.** + The harness does not require the run's output to be byte-for-byte equal to + `expected-output/`. Instead it compares the actual resource mutations field-by-field, and it + ignores events entirely (`EquivalentApplyConfigurationResultIgnoringEvents` in + `equivalence.go`). So differences that are only in events, formatting, or ordering will + still pass; only a difference in the actual mutations an operator wants to make will fail a + test. +- **Do not hand-edit `expected-output/` or `test-output/`.** These are generated. To update + the expected output after an intentional behavior change, run + `make update-test-operator-integration` (or the harness with `--replace-expected-output`), + then review the diff. +- **Output is structured by cluster type + verb + GVR.** Mutations land under + `Configuration/`, `Management/`, or `UserWorkload/`, then `Create`/`ApplyStatus`/etc. When + reasoning about a change, look at which of these buckets shifts. +- **`output-resources` must be complete.** Any GVR an operator mutates must be declared + there; mutations to undeclared resources are filtered out (a bug signal). + +## When adding or changing behavior + +1. Make the code change (keep output reproducible/order-independent — see above). +2. `make check` for unit tests + verify. +3. If integration output changes intentionally, regenerate the expected output with + `make update-test-operator-integration` and inspect the diff to confirm only the intended + mutations changed. +4. If you add a new operator capability, mirror the pattern in `pkg/sampleoperator/` and add + a fixture under `test-data/apply-configuration/`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..eef4bd20 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md \ No newline at end of file diff --git a/README.md b/README.md index a3d50f0b..2b4f97e1 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,229 @@ -Multi-Operator-Manager +# Multi-Operator-Manager (MOM) -# Building -* `make build` +MOM is a command-line tool (binary) that lets you **test, debug, and compare OpenShift +operators offline — without a live cluster**. +Normally an operator only runs against a live cluster: it watches resources and writes +changes back, deciding and acting in the same step. That makes it hard to test and +impossible to simply ask *"what would you do?"* without it actually doing it. MOM changes +that: you hand an operator a folder of resources describing the cluster's current state, and +it prints the exact changes it *would* make — without applying any of them. -# Testing your compatible operator -`./multi-operator-manager test apply-configuration --test-dir=./test-data/apply-configuration/ --output-dir=./test-output --preserve-policy=KeepAlways` +To work with MOM, an operator exposes three commands ("verbs"): -The `./test-output` directory will be created and a `junit.xml` inside will summarize the results. +| Verb | In plain terms | +| --------------------- | -------------------------------------------------------------------------- | +| `input-resources` | "Everything I need to read before I can decide anything." | +| `apply-configuration` | "Given those resources, the changes I'd make (create/update/delete/status)." | +| `output-resources` | "The complete list of things I'm ever allowed to change." | -## Defining a test -An example is contained in `test-data`. -You can organize your tests however you wish, but every directory with a `test.yaml` is considered a test and must have -an `input-dir` and an `expected-output` dir. +You don't write these from scratch: the packages under `pkg/library/` provide the commands, +and the operator just supplies three functions — its input list, its output list, and its +decision logic (see `pkg/sampleoperator` for a working example). Once an operator exposes +these three verbs, it is **compatible** and MOM can drive it. -TODO probably allow missing to mean no output. It's painful otherwise. +Because every compatible operator answers in the same standard format, MOM can **test** an +operator (compare its output to a known-good result), **debug** a real cluster (replay its +state through the operator on your laptop), and **compare** operators to each other (e.g. +spot two that would fight over the same resource) — all without touching a running cluster. + +The reference implementation lives under `pkg/sampleoperator` and is exposed through the +`sample-operator` subcommand. + +## Why MOM — example scenarios + +### 1. CI regression testing, no cluster required + +You change a controller in your operator and want to prove it only affected what you +intended. `apply-configuration` is a pure function — given a directory of input resources +it prints the exact mutations it *would* make — so CI can diff that output against the +checked-in expected output: + +``` +./multi-operator-manager test apply-configuration \ + --test-dir=./test-data/apply-configuration/ \ + --output-dir=./test-output \ + --preserve-policy=KeepAlways +``` + +**What you learn:** whether your change altered the operator's decisions. The run is +deterministic (time is pinned via `now` in each `test.yaml`) and takes milliseconds — no +cluster, no flakiness. `test-output/junit.xml` summarizes pass/fail per test. + +### 2. Debug a real cluster from a must-gather + +A customer cluster is misbehaving and all you have is a must-gather. Build an input +directory from it, then run `apply-configuration` to see what the operator *would* do +against that exact state — on your laptop, with no cluster access: + +``` +# 1. capture the resources the operator declares it reads + input-resources > pertinent-resources.yaml + +# 2. extract just those resources from the must-gather into an input dir +./multi-operator-manager create-input-resources from-must-gather \ + --must-gather-dir= \ + --input-resources=pertinent-resources.yaml \ + --output-dir= + +# 3. run the operator against that input dir + apply-configuration \ + --input-dir= \ + --output-dir= +``` + +**What you learn:** the operator's intended creates/updates/deletes against real cluster +state, reproducibly, offline. + +### 3. Multi-operator conflict and dependency analysis + +Every compatible operator declares what it reads (`input-resources`) and the complete set +it may mutate (`output-resources`). Line those lists up across operators: + +``` + output-resources > a-out.yaml + output-resources > b-out.yaml + input-resources > a-in.yaml +``` + +**What you learn:** +- A resource (GVR) appearing in two operators' `output-resources` is a latent **write + conflict** (they will fight over it on a live cluster). +- Matching one operator's `input-resources` against another's `output-resources` reveals a + **dependency / ordering** edge. +- A mutation emitted by `apply-configuration` for a GVR that is *not* in `output-resources` + is an out-of-bounds bug. + +This is the "multi" the project is named for: reducing each operator to comparable, +declarative pieces so a whole cluster's worth of them can be reasoned about together. + +## Building + +``` +make build +``` + +This produces two binaries in the repo root: + +| Binary | What it is | +| ------------------------- | --------------------------------------------------------------------------------- | +| `multi-operator-manager` | The MOM tool itself — runs the tests, builds inputs from must-gather, etc. | +| `sample-operator` | A reference **compatible** operator, used to exercise MOM and as a copyable example. | + +`multi-operator-manager` is what you run against *your* operators; `sample-operator` is a +minimal operator that already speaks the three verbs, so you can see the whole flow end to +end without wiring up a real operator. + +## Command overview + +``` +multi-operator-manager +├── test +│ └── apply-configuration # run compatible operators against fixtures and diff output +├── sample-operator # reference compatible operator +│ ├── input-resources +│ ├── apply-configuration +│ └── output-resources +└── create-input-resources + └── from-must-gather # build input resources from a must-gather dump +``` + +## Testing your compatible operator + +``` +./multi-operator-manager test apply-configuration \ + --test-dir=./test-data/apply-configuration/ \ + --output-dir=./test-output \ + --preserve-policy=KeepAlways +``` + +The `./test-output` directory is created and a `junit.xml` inside summarizes the results. + +Useful flags for `test apply-configuration`: + +- `--test-dir` (required) — directory of tests, searched recursively. +- `--output-dir` (required) — where results (and `junit.xml`) are written. +- `--preserve-policy` — how much of the run output to keep. +- `--replace-expected-output` — delete each test's `expected-output` and replace it with + the current run's values (use to regenerate the expected output). + +### Defining a test + +Examples live in `test-data`. You can organize tests however you like: **every directory +containing a `test.yaml` is a test**, and must have an `input-dir` and an `expected-output` +directory. `test.yaml` must name the operator binary under test (`binaryName`). + +> TODO: allow a missing `expected-output` to mean "no output". It's painful otherwise. + +## The operator contract (verb flags) + +Each compatible operator exposes the three verbs. As implemented by the shared libraries: + +- `input-resources` / `output-resources` — print the declared resource lists. +- `apply-configuration`: + - `--input-dir` — directory holding the input resources. + - `--output-dir` — directory where computed mutations are written. + - `--controllers` — controllers to enable: `*` (all), `foo` (enable `foo`), + `-foo` (disable `foo`). Default: `*`. + - `--now` — the value to use for `time.Now` during the run (for deterministic output). + +## Real-world example: cluster-authentication-operator + +[cluster-authentication-operator](https://github.com/openshift/cluster-authentication-operator) +is a production operator that adopts the MOM contract. It's a useful reference for how to +make your own operator compatible, because MOM support there is almost entirely **additive** — +the normal operator keeps working unchanged. + +It adds a small `mom` command group and reuses its existing controller graph: + +- **Wiring** (`cmd/authentication-operator/main.go`) — registers the three verbs as + subcommands (`apply-configuration`, `input-resources`, `output-resources`). +- **The two declarations** (`pkg/cmd/mom/input_resources_command.go`, + `output_resources_command.go`) — thin wrappers around the MOM libraries that declare what + the operator reads and what it is allowed to write (the latter partitioned into + configuration / management / user-workload resources). +- **The run** (`pkg/cmd/mom/apply_configuration_command.go`) — builds the operator from the + MOM-supplied input and calls `RunOnce`, returning desired mutations instead of applying + them. +- **The shared core** (`pkg/operator/replacement_starter.go`) — the key design point: one + controller graph serves both modes. A production constructor uses real clients; a MOM + constructor swaps in the mutation-tracking client and a static feature-gate accessor (no + live cluster to observe). Both register each controller in two forms — a long-running + `Run` for production and a single-shot `Sync` for MOM's `RunOnce`. + +The pattern to copy for your own operator: + +1. Refactor operator construction so one starter exposes both `Run` (continuous) and `Sync` + (run-once) forms of each controller. +2. Add a MOM input constructor that swaps real clients for the mutation-tracking client. +3. Add three thin commands that declare inputs, declare outputs, and call `RunOnce`. + +## Building input resources from a must-gather + +Replay what an operator *would* do against a real cluster snapshot. Supply the operator's +declared inputs as a "pertinent resources" file (its `input-resources` output), and the +command extracts exactly those objects from the must-gather: + +``` + input-resources > pertinent-resources.yaml + +./multi-operator-manager create-input-resources from-must-gather \ + --must-gather-dir= \ + --input-resources=pertinent-resources.yaml \ + --output-dir= +``` + +- `--must-gather-dir` — location of the must-gather output. +- `--input-resources` — file listing the pertinent resources to extract (the operator's + `input-resources` output). +- `--output-dir` — where the minimal output is written. +- `--operator-binary` — intended to derive the pertinent resources by calling the operator + directly, but **not yet implemented** (currently errors); use `--input-resources` instead. ## Testing this repo -`make test-operator-integration` will run the `sample-operator` against the local test data here. -### test.yaml -This repo contains examples, but to test your operator the operator binary name must be present. +``` +make test-operator-integration +``` + +runs the `sample-operator` against the local test data in `test-data`.