diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 00000000..6464f64c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,116 @@ +# AGENTS.md + +Guidance for AI agents working in this repository. Read this before making changes. +See `README.md` for the user-facing explanation of what the project does. + +## What this repo is + +`multi-operator-manager` (MOM) is an OpenShift Go module that is **both a CLI tool and a +library**: + +- The **tool** (`multi-operator-manager` binary) tests, debugs, and analyzes OpenShift + operators *offline* (no live cluster). +- The **library** (`pkg/library/...`) is imported by real operators so they become + "compatible" — i.e. expose the three MOM verbs. + +A compatible operator turns its reconcile logic into a function of its inputs: given input +resources, it computes the mutations it *would* make, without touching a cluster. It does +this via three subcommands: `input-resources`, `apply-configuration`, `output-resources`. + +## The core: `pkg/library` + +`pkg/library` is the heart of the project — the shared implementation of the operator +contract, and the part other repos import. It does two jobs: + +- **Defines the contract.** It provides the ready-made cobra commands for the three verbs and + the types they exchange, so an operator doesn't implement them from scratch. An operator + becomes compatible by importing these packages and supplying three things: its input list, + its output list, and its decision logic (`pkg/sampleoperator` is the worked example). +- **Runs and checks the results.** It contains the machinery that drives an operator once + (`SimpleOperatorStarter` and the run-once helpers), writes the resulting mutations to disk, + and compares two result sets for equivalence — the same logic the test harness relies on. + +Its three subpackages map one-to-one to the verbs: + +- `libraryinputresources/` — the `input-resources` verb and input types. +- `libraryapplyconfiguration/` — the `apply-configuration` verb, the run-once machinery, and + mutation output/comparison. +- `libraryoutputresources/` — the `output-resources` verb and output types. + +Because this is the public surface other operators depend on, treat changes to its exported +types and function signatures as **breaking changes** and weigh them accordingly. + +For a real-world consumer of this contract, see the +[cluster-authentication-operator example in the README](README.md#real-world-example-cluster-authentication-operator) +(`pkg/sampleoperator` is the in-repo example). + +## Commands + +``` +make build # build both binaries (multi-operator-manager, sample-operator) into repo root +make check # verify + unit tests (run before considering a change done) +make test-unit # go test ./pkg/... ./cmd/... +make verify # gofmt/vet/etc. via build-machinery-go +make test-operator-integration # build + run sample-operator against test-data, compare vs expected output +make update-test-operator-integration # same, but REGENERATE the expected output (use after intended changes) +make test-e2e # requires openshift-tests in PATH; rarely needed locally +make clean # remove built binary + test-output +``` + +Run a single package's tests directly with `go test ./pkg/library/libraryinputresources/...`. + +## Repo layout + +- `cmd/` — two `main` packages: `multi-operator-manager` and `sample-operator`. +- `pkg/cmd/multi-operator-manager/` — the cobra CLI tree (`test`, `sample-operator`, + `create-input-resources`). +- `pkg/library/` — the reusable contract; **this is the public API other repos import**: + - `libraryinputresources/` — `input-resources` verb + input types. + - `libraryapplyconfiguration/` — `apply-configuration` verb, the operator launch/run-once + machinery, and mutation output/comparison. + - `libraryoutputresources/` — `output-resources` verb + output types. +- `pkg/sampleoperator/` — reference compatible operator (the example to copy from). +- `pkg/test/testapplyconfiguration/` — the test harness behind `test apply-configuration`. +- `test-data/apply-configuration/*/` — integration fixtures: each dir with a `test.yaml` has + an `input-dir/` and an `expected-output/` (the known-good result to compare against). +- `vendor/` — dependencies are **vendored**; dep changes go through `go mod` + build-machinery + vendoring, not hand edits. + +## Conventions and gotchas + +- **Output must be reproducible in content, and order-independent.** `apply-configuration` + should compute the same *set* of desired mutations for the same inputs, regardless of the + order controllers happen to run in. To support this, time is injected via `Clock` / `--now` + and reads go through the injected `MutationTrackingClient` (in `ApplyConfigurationInput`) + rather than a live cluster or disk; and `SimpleOperatorStarter` shuffles controllers before + running them and rejects duplicate controller names. + + Practical consequence: do **not** introduce `time.Now()`, `math/rand`, env/file reads, or + network/cluster calls into operator logic — use the injected clock and client instead, and + don't rely on controller run order. +- **A test passes when the *meaningful* output matches — not when the files are identical.** + The harness does not require the run's output to be byte-for-byte equal to + `expected-output/`. Instead it compares the actual resource mutations field-by-field, and it + ignores events entirely (`EquivalentApplyConfigurationResultIgnoringEvents` in + `equivalence.go`). So differences that are only in events, formatting, or ordering will + still pass; only a difference in the actual mutations an operator wants to make will fail a + test. +- **Do not hand-edit `expected-output/` or `test-output/`.** These are generated. To update + the expected output after an intentional behavior change, run + `make update-test-operator-integration` (or the harness with `--replace-expected-output`), + then review the diff. +- **Output is structured by cluster type + verb + GVR.** Mutations land under + `Configuration/`, `Management/`, or `UserWorkload/`, then `Create`/`ApplyStatus`/etc. When + reasoning about a change, look at which of these buckets shifts. +- **`output-resources` must be complete.** Any GVR an operator mutates must be declared + there; mutations to undeclared resources are filtered out (a bug signal). + +## When adding or changing behavior + +1. Make the code change (keep output reproducible/order-independent — see above). +2. `make check` for unit tests + verify. +3. If integration output changes intentionally, regenerate the expected output with + `make update-test-operator-integration` and inspect the diff to confirm only the intended + mutations changed. +4. If you add a new operator capability, mirror the pattern in `pkg/sampleoperator/` and add + a fixture under `test-data/apply-configuration/`. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 00000000..eef4bd20 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1 @@ +@AGENTS.md \ No newline at end of file diff --git a/README.md b/README.md index a3d50f0b..2b4f97e1 100644 --- a/README.md +++ b/README.md @@ -1,23 +1,229 @@ -Multi-Operator-Manager +# Multi-Operator-Manager (MOM) -# Building -* `make build` +MOM is a command-line tool (binary) that lets you **test, debug, and compare OpenShift +operators offline — without a live cluster**. +Normally an operator only runs against a live cluster: it watches resources and writes +changes back, deciding and acting in the same step. That makes it hard to test and +impossible to simply ask *"what would you do?"* without it actually doing it. MOM changes +that: you hand an operator a folder of resources describing the cluster's current state, and +it prints the exact changes it *would* make — without applying any of them. -# Testing your compatible operator -`./multi-operator-manager test apply-configuration --test-dir=./test-data/apply-configuration/ --output-dir=./test-output --preserve-policy=KeepAlways` +To work with MOM, an operator exposes three commands ("verbs"): -The `./test-output` directory will be created and a `junit.xml` inside will summarize the results. +| Verb | In plain terms | +| --------------------- | -------------------------------------------------------------------------- | +| `input-resources` | "Everything I need to read before I can decide anything." | +| `apply-configuration` | "Given those resources, the changes I'd make (create/update/delete/status)." | +| `output-resources` | "The complete list of things I'm ever allowed to change." | -## Defining a test -An example is contained in `test-data`. -You can organize your tests however you wish, but every directory with a `test.yaml` is considered a test and must have -an `input-dir` and an `expected-output` dir. +You don't write these from scratch: the packages under `pkg/library/` provide the commands, +and the operator just supplies three functions — its input list, its output list, and its +decision logic (see `pkg/sampleoperator` for a working example). Once an operator exposes +these three verbs, it is **compatible** and MOM can drive it. -TODO probably allow missing to mean no output. It's painful otherwise. +Because every compatible operator answers in the same standard format, MOM can **test** an +operator (compare its output to a known-good result), **debug** a real cluster (replay its +state through the operator on your laptop), and **compare** operators to each other (e.g. +spot two that would fight over the same resource) — all without touching a running cluster. + +The reference implementation lives under `pkg/sampleoperator` and is exposed through the +`sample-operator` subcommand. + +## Why MOM — example scenarios + +### 1. CI regression testing, no cluster required + +You change a controller in your operator and want to prove it only affected what you +intended. `apply-configuration` is a pure function — given a directory of input resources +it prints the exact mutations it *would* make — so CI can diff that output against the +checked-in expected output: + +``` +./multi-operator-manager test apply-configuration \ + --test-dir=./test-data/apply-configuration/ \ + --output-dir=./test-output \ + --preserve-policy=KeepAlways +``` + +**What you learn:** whether your change altered the operator's decisions. The run is +deterministic (time is pinned via `now` in each `test.yaml`) and takes milliseconds — no +cluster, no flakiness. `test-output/junit.xml` summarizes pass/fail per test. + +### 2. Debug a real cluster from a must-gather + +A customer cluster is misbehaving and all you have is a must-gather. Build an input +directory from it, then run `apply-configuration` to see what the operator *would* do +against that exact state — on your laptop, with no cluster access: + +``` +# 1. capture the resources the operator declares it reads + input-resources > pertinent-resources.yaml + +# 2. extract just those resources from the must-gather into an input dir +./multi-operator-manager create-input-resources from-must-gather \ + --must-gather-dir= \ + --input-resources=pertinent-resources.yaml \ + --output-dir= + +# 3. run the operator against that input dir + apply-configuration \ + --input-dir= \ + --output-dir= +``` + +**What you learn:** the operator's intended creates/updates/deletes against real cluster +state, reproducibly, offline. + +### 3. Multi-operator conflict and dependency analysis + +Every compatible operator declares what it reads (`input-resources`) and the complete set +it may mutate (`output-resources`). Line those lists up across operators: + +``` + output-resources > a-out.yaml + output-resources > b-out.yaml + input-resources > a-in.yaml +``` + +**What you learn:** +- A resource (GVR) appearing in two operators' `output-resources` is a latent **write + conflict** (they will fight over it on a live cluster). +- Matching one operator's `input-resources` against another's `output-resources` reveals a + **dependency / ordering** edge. +- A mutation emitted by `apply-configuration` for a GVR that is *not* in `output-resources` + is an out-of-bounds bug. + +This is the "multi" the project is named for: reducing each operator to comparable, +declarative pieces so a whole cluster's worth of them can be reasoned about together. + +## Building + +``` +make build +``` + +This produces two binaries in the repo root: + +| Binary | What it is | +| ------------------------- | --------------------------------------------------------------------------------- | +| `multi-operator-manager` | The MOM tool itself — runs the tests, builds inputs from must-gather, etc. | +| `sample-operator` | A reference **compatible** operator, used to exercise MOM and as a copyable example. | + +`multi-operator-manager` is what you run against *your* operators; `sample-operator` is a +minimal operator that already speaks the three verbs, so you can see the whole flow end to +end without wiring up a real operator. + +## Command overview + +``` +multi-operator-manager +├── test +│ └── apply-configuration # run compatible operators against fixtures and diff output +├── sample-operator # reference compatible operator +│ ├── input-resources +│ ├── apply-configuration +│ └── output-resources +└── create-input-resources + └── from-must-gather # build input resources from a must-gather dump +``` + +## Testing your compatible operator + +``` +./multi-operator-manager test apply-configuration \ + --test-dir=./test-data/apply-configuration/ \ + --output-dir=./test-output \ + --preserve-policy=KeepAlways +``` + +The `./test-output` directory is created and a `junit.xml` inside summarizes the results. + +Useful flags for `test apply-configuration`: + +- `--test-dir` (required) — directory of tests, searched recursively. +- `--output-dir` (required) — where results (and `junit.xml`) are written. +- `--preserve-policy` — how much of the run output to keep. +- `--replace-expected-output` — delete each test's `expected-output` and replace it with + the current run's values (use to regenerate the expected output). + +### Defining a test + +Examples live in `test-data`. You can organize tests however you like: **every directory +containing a `test.yaml` is a test**, and must have an `input-dir` and an `expected-output` +directory. `test.yaml` must name the operator binary under test (`binaryName`). + +> TODO: allow a missing `expected-output` to mean "no output". It's painful otherwise. + +## The operator contract (verb flags) + +Each compatible operator exposes the three verbs. As implemented by the shared libraries: + +- `input-resources` / `output-resources` — print the declared resource lists. +- `apply-configuration`: + - `--input-dir` — directory holding the input resources. + - `--output-dir` — directory where computed mutations are written. + - `--controllers` — controllers to enable: `*` (all), `foo` (enable `foo`), + `-foo` (disable `foo`). Default: `*`. + - `--now` — the value to use for `time.Now` during the run (for deterministic output). + +## Real-world example: cluster-authentication-operator + +[cluster-authentication-operator](https://github.com/openshift/cluster-authentication-operator) +is a production operator that adopts the MOM contract. It's a useful reference for how to +make your own operator compatible, because MOM support there is almost entirely **additive** — +the normal operator keeps working unchanged. + +It adds a small `mom` command group and reuses its existing controller graph: + +- **Wiring** (`cmd/authentication-operator/main.go`) — registers the three verbs as + subcommands (`apply-configuration`, `input-resources`, `output-resources`). +- **The two declarations** (`pkg/cmd/mom/input_resources_command.go`, + `output_resources_command.go`) — thin wrappers around the MOM libraries that declare what + the operator reads and what it is allowed to write (the latter partitioned into + configuration / management / user-workload resources). +- **The run** (`pkg/cmd/mom/apply_configuration_command.go`) — builds the operator from the + MOM-supplied input and calls `RunOnce`, returning desired mutations instead of applying + them. +- **The shared core** (`pkg/operator/replacement_starter.go`) — the key design point: one + controller graph serves both modes. A production constructor uses real clients; a MOM + constructor swaps in the mutation-tracking client and a static feature-gate accessor (no + live cluster to observe). Both register each controller in two forms — a long-running + `Run` for production and a single-shot `Sync` for MOM's `RunOnce`. + +The pattern to copy for your own operator: + +1. Refactor operator construction so one starter exposes both `Run` (continuous) and `Sync` + (run-once) forms of each controller. +2. Add a MOM input constructor that swaps real clients for the mutation-tracking client. +3. Add three thin commands that declare inputs, declare outputs, and call `RunOnce`. + +## Building input resources from a must-gather + +Replay what an operator *would* do against a real cluster snapshot. Supply the operator's +declared inputs as a "pertinent resources" file (its `input-resources` output), and the +command extracts exactly those objects from the must-gather: + +``` + input-resources > pertinent-resources.yaml + +./multi-operator-manager create-input-resources from-must-gather \ + --must-gather-dir= \ + --input-resources=pertinent-resources.yaml \ + --output-dir= +``` + +- `--must-gather-dir` — location of the must-gather output. +- `--input-resources` — file listing the pertinent resources to extract (the operator's + `input-resources` output). +- `--output-dir` — where the minimal output is written. +- `--operator-binary` — intended to derive the pertinent resources by calling the operator + directly, but **not yet implemented** (currently errors); use `--input-resources` instead. ## Testing this repo -`make test-operator-integration` will run the `sample-operator` against the local test data here. -### test.yaml -This repo contains examples, but to test your operator the operator binary name must be present. +``` +make test-operator-integration +``` + +runs the `sample-operator` against the local test data in `test-data`.