From 9f67ee46d52d0d4a57bd6fe3c1fadea4aa5db39f Mon Sep 17 00:00:00 2001 From: Thomas Schmelzer Date: Wed, 2 Sep 2026 19:36:31 +0400 Subject: [PATCH] feat: one repos.yml entry can be a folder of repos `- folder: ~/repos/jebel-quant` puts every checkout sitting directly inside that directory on the board. Sixteen lines become one, and the one cannot drift out of step with the disk the way sixteen can: clone a repo into the folder and it joins at the next restart, delete it and it leaves. This is not the whole-root directory walk that the explicit list replaced. That walk took any checkout anywhere under one mount whose origin happened to look plausible, and nobody had said where to look. A folder is a directory you wrote down, read one level deep and no further - `folder: ~` finding your entire disk is exactly the failure the one-level rule exists to prevent, and it is also why `folder: ~/repos` correctly finds nothing when the checkouts live at `~/repos//`. The refusals follow what a `path` entry already does, for the same reason: a board that is quietly short a folder's worth of repos is worse than one that did not come up. A folder that is not there, or that holds no checkouts at all, stops the collector. What is *not* fatal is a clone inside the folder with no usable origin remote - somebody's scratch clone, warned about and skipped, because a folder entry is not the deliberate statement that naming a path is. A `path` to that same clone is still refused. An entry of its own wins, and the dedupe that makes that work is on the checkout path rather than on `namespace/name`. A folder leaves out any checkout another entry names by `path`, so one repo inside a listed folder keeps a `repo:` override for its upstream - and the whole point of such an override is that the name comes out different, which is why matching on the name cannot do it. Where the collision really is on the name (`- repo: org/x` beside the folder org/x is checked out in), the two merge into one repo either way round: the entry decides the name and the forge, the folder still supplies the path it found. Verified against the real fleet on this machine: two folders plus a `path` override plus a bare `repo:` resolve to 11 repos, with the override renamed and the bare entry picking up a checkout path it could not previously express. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 15 ++- collector/jq_collector/repos.py | 183 ++++++++++++++++++++++++++++--- collector/tests/test_fleet.py | 189 ++++++++++++++++++++++++++++++++ docs/configuration.md | 79 +++++++++++-- docs/index.md | 15 ++- repos.example.yml | 29 ++++- 6 files changed, 472 insertions(+), 38 deletions(-) diff --git a/README.md b/README.md index 876d221..5bd278c 100644 --- a/README.md +++ b/README.md @@ -10,8 +10,9 @@ A Grafana board for the state of your repo fleet — template drift, CI on the default branch, open pull requests, and the working copies on this machine — with Prometheus keeping the history and six alert rules on top. -One container, one file. Nothing is discovered: a repo is on the board because -you listed it in `repos.yml`, and for no other reason. +One container, one file. Nothing is discovered behind your back: a repo is on +the board because `repos.yml` names it — or names the folder you keep it in — +and for no other reason. ![The fleet board: tiles counting repos monitored, open PRs and issues, CI red on main, PRs with red checks, local clones out of sync, dirty working copies, branch protection and Dependabot, above a seven-day trend of open problems by kind.](dashboard.png) @@ -22,12 +23,13 @@ table of exactly which repos are behind its number — see ## Recipe -Write a `repos.yml` — one entry per repo you want on the board: +Write a `repos.yml` — one entry per repo you want on the board, or one +entry for a folder of them: ```yaml repos: - path: ~/repos/jebel-quant/rhiza # a checkout on this machine - - path: ~/repos/cvxgrp/cvxsimulator + - folder: ~/repos/cvxgrp # every checkout in a folder - repo: Jebel-Quant/actions # monitored, but not cloned here - repo: acme/platform/infra/web # GitLab, on the same board forge: gitlab @@ -35,7 +37,10 @@ repos: `namespace/name` comes from each checkout's `origin`, so the path is all you write, and the path is used as written — a checkout does not have to live at -`//`. **GitHub and GitLab repos share one board**; the forge +`//`. **A `folder:` puts every checkout inside it on the +board**, one level deep, which is how an org you keep whole stays one line +instead of twenty — see +[A folder of repos](docs/configuration.md#a-folder-of-repos). **GitHub and GitLab repos share one board**; the forge is read off the origin's host where there is a checkout, and stated with `forge: gitlab` where there is not — see [Configuration](docs/configuration.md#github-and-gitlab-in-the-same-fleet). diff --git a/collector/jq_collector/repos.py b/collector/jq_collector/repos.py index 089d465..ab286a6 100644 --- a/collector/jq_collector/repos.py +++ b/collector/jq_collector/repos.py @@ -16,6 +16,15 @@ mounted, or that repo is not checked out here: the GitHub panels still report on it and the working-copy panels have nothing to say. That is the same shape as an entry with no ``path`` at all. + +One entry may also name a ``folder:``, and then every checkout sitting directly +inside it is on the board. That is still the list deciding membership - the +folder is named here, one level is scanned and no deeper, and a repo joins +because a folder you wrote down holds it. It is not the old whole-root walk, +which took any checkout anywhere under one mount whose origin looked plausible. +Cloning a repo into a listed folder does add it to the board at the next +restart, which is the point: an org you keep whole is one line instead of +twenty, and the twenty cannot drift out of step with the disk. """ from __future__ import annotations @@ -83,6 +92,100 @@ def _declared_forge(item: dict, index: int) -> str | None: return declared +def _folder(item: dict, index: int, host_root: str, claimed: frozenset[str]) -> list[dict]: + """A ``folder:`` entry -> one ``path:`` entry per checkout inside it. + + Only the folder's own children are looked at, never their children in turn. + A folder is a statement about where you keep a set of repos, and one level + is what that means; recursing would make ``folder: ~`` a whole-disk walk, + which is the discovery this file exists to have got rid of. + + A checkout some other entry names by ``path`` is left to that entry - + ``claimed`` is every such path in the file. That is how one repo inside a + listed folder gets an override: the folder covers the rest, and the entry + written out for that one checkout is the only entry for it. Matching on the + path rather than on ``namespace/name`` is what makes a ``repo:`` override + work, since the whole point of one is that the name comes out different. + + A folder that is not there is refused, exactly as an unreachable ``path`` + is, and so is one holding no checkouts at all. Both look identical to a + fleet that is quietly short a folder's worth of repos, and the point of + naming the folder was to get those repos on the board. + """ + raw = str(item.get("folder") or "").strip() + for key in ("path", "repo"): + if item.get(key): + raise FleetError( + f"entry {index}: `folder: {raw}` cannot also carry a `{key}` - " + "a folder stands for however many repos are in it, so there is " + "no one path or name to give it. List the repo on its own entry." + ) + # Validated here so a bad `forge:` on the folder is refused once, against + # the line that was actually written, rather than per checkout found. + forge = _declared_forge(item, index) + + folder = resolve_path(raw, host_root) + try: + children = sorted(entry.path for entry in os.scandir(folder) if entry.is_dir()) + except OSError as exc: + raise FleetError( + f"entry {index}: cannot read folder {raw} (looked in {folder}): {exc}. " + "Mount the home directory it lives under, or list the repos in it " + "one by one." + ) from exc + + checkouts = [child for child in children if _is_checkout(child)] + if not checkouts: + raise FleetError( + f"entry {index}: folder {raw} holds no git checkouts (looked in {folder}). " + "A folder is scanned one level deep, so name the folder the " + "checkouts are directly in." + ) + # The emptiness check is on what is in the folder, not on what is left + # after the claimed ones go: a folder whose every checkout has an entry of + # its own is a redundant line, not a mistake worth refusing to start over. + taken = [child for child in checkouts if child not in claimed] + if len(taken) == len(checkouts): + log.info("folder %s: %d checkouts", raw, len(taken)) + else: + log.info( + "folder %s: %d checkouts, %d left to an entry of their own", + raw, + len(taken), + len(checkouts) - len(taken), + ) + return [{"path": child, "forge": forge} for child in taken] + + +def _expand( + item: Any, index: int, host_root: str, claimed: frozenset[str] +) -> list[tuple[Any, bool]]: + """One config entry -> the entries to read, and whether each was swept up. + + Everything is one entry, written outright, except a ``folder:`` - which is + however many checkouts are in it. The flag is what lets an entry named + outright override one a folder found, instead of colliding with it. + """ + if isinstance(item, dict) and item.get("folder"): + return [(found, True) for found in _folder(item, index, host_root, claimed)] + return [(item, False)] + + +def _claimed_paths(entries: list, host_root: str) -> frozenset[str]: + """Every checkout the file names by ``path``, resolved for this filesystem. + + A folder skips these, so an entry written out for one checkout inside a + listed folder is that checkout's only entry. + """ + claimed = set() + for item in entries: + if isinstance(item, str): + claimed.add(resolve_path(item, host_root)) + elif isinstance(item, dict) and item.get("path"): + claimed.add(resolve_path(str(item["path"]), host_root)) + return frozenset(claimed) + + def _entry(item: Any, index: int, host_root: str) -> tuple[str, str | None, str]: """One entry -> ``(namespace/name, checkout path or None, forge)``.""" # `- ~/repos/foo` is accepted as shorthand for `- path: ~/repos/foo`. @@ -168,20 +271,72 @@ def load( fleet: list[str] = [] paths: dict[str, str] = {} forges: dict[str, str] = {} + # Named outright rather than swept up by a folder, which is what decides + # who wins when both describe the same repo. + named: set[str] = set() + claimed = _claimed_paths(entries, host_root) for index, item in enumerate(entries, start=1): - full_name, path, forge = _entry(item, index, host_root) - if full_name in fleet: - # Two forges can host the same `namespace/name`, and the board's - # whole label scheme is that one `repo` value is one repo. Refusing - # is the same choice the duplicate case has always made: a merged - # pair would report one repo's CI under the other's name, and read - # as a working board while doing it. - clash = forges[full_name] - detail = f" - on {clash} and on {forge}" if clash != forge else "" - raise FleetError(f"{full_name} is listed twice{detail}") - fleet.append(full_name) - forges[full_name] = forge - if path is not None: - paths[full_name] = path + for entry, swept in _expand(item, index, host_root, claimed): + try: + full_name, path, forge = _entry(entry, index, host_root) + except FleetError as exc: + if not swept: + raise + # A checkout with no usable origin is somebody's scratch clone + # sitting in the folder. Refusing would take the whole board + # down over a repo nobody asked to monitor; a folder entry is + # not the deliberate statement that a listed path is. + log.warning("skipping %s: %s", entry["path"], exc) + continue + + # Exactly one of the two entries names the repo outright: a folder + # overlapping an entry, which is not the duplicate case below. + overlap = (not swept) != (full_name in named) + if full_name in forges and overlap: + # One entry names this repo outright and the other is a folder + # that swept it up - `- repo: org/x` next to the folder org/x + # is checked out in. The entry written for the repo itself is + # the deliberate statement, so it decides the name and the + # forge whichever order the two were written in; the folder can + # still supply the checkout path, since that is where the repo + # is on disk and the other entry may not have said. + # + # (A folder never reaches here for a checkout some entry names + # by `path` - it skips those outright, which is what lets a + # `repo:` override rename one repo inside a listed folder.) + log.info("%s: named outright, so the folder does not list it too", full_name) + if not swept: + named.add(full_name) + forges[full_name] = forge + if path is not None: + paths[full_name] = path if not swept else paths.get(full_name, path) + continue + + if full_name in forges: + # Two forges can host the same `namespace/name`, and the board's + # whole label scheme is that one `repo` value is one repo. Refusing + # is the same choice the duplicate case has always made: a merged + # pair would report one repo's CI under the other's name, and read + # as a working board while doing it. + clash, first = forges[full_name], paths.get(full_name) + if clash != forge: + detail = f" - on {clash} and on {forge}" + elif first and path and first != path: + detail = f" - checked out at {first} and at {path}" + else: + detail = "" + raise FleetError(f"{full_name} is listed twice{detail}") + + fleet.append(full_name) + forges[full_name] = forge + if not swept: + named.add(full_name) + if path is not None: + paths[full_name] = path + + if not fleet: + # Reachable only when every checkout a folder turned up was skipped for + # want of an origin: the file said something, and none of it survived. + raise FleetError(f"{source} named no repo the collector could identify") return tuple(fleet), paths, forges diff --git a/collector/tests/test_fleet.py b/collector/tests/test_fleet.py index 2896f7a..16135d2 100644 --- a/collector/tests/test_fleet.py +++ b/collector/tests/test_fleet.py @@ -714,3 +714,192 @@ def test_the_duplicate_message_stays_plain_when_the_forge_matches(tmp_path): with pytest.raises(repos.FleetError, match="acme/web is listed twice$"): repos.load(write_fleet(tmp_path, body)) + + +# -- a folder of repos ------------------------------------------------------- +# +# One entry, however many checkouts are in it. Membership is still decided by +# the file - the folder is named there, and only its own children are looked at +# - which is what keeps this from being the whole-root walk it replaced. + + +def test_every_checkout_in_a_folder_joins_the_fleet(tmp_path): + make_checkout(tmp_path, "org", "alpha") + make_checkout(tmp_path, "org", "beta") + + fleet, paths, forges = repos.load(write_fleet(tmp_path, f" - folder: {tmp_path / 'org'}\n")) + + assert fleet == ("org/alpha", "org/beta") + assert paths == { + "org/alpha": str(tmp_path / "org" / "alpha"), + "org/beta": str(tmp_path / "org" / "beta"), + } + assert forges == {"org/alpha": "github", "org/beta": "github"} + + +def test_a_folder_is_scanned_one_level_and_no_deeper(tmp_path): + """`folder: ~` would otherwise be a whole-disk walk. + + A folder says where you keep a set of repos, and one level is what that + means. Anything more is the discovery this file exists to have got rid of. + """ + make_checkout(tmp_path, "org", "alpha") + make_checkout(tmp_path / "org" / "deeper", "org", "buried") + (tmp_path / "org" / "not-a-repo").mkdir() + (tmp_path / "org" / "notes.md").write_text("x\n") + + fleet, _paths, _forges = repos.load(write_fleet(tmp_path, f" - folder: {tmp_path / 'org'}\n")) + + assert fleet == ("org/alpha",) + + +def test_a_folder_that_is_not_there_is_refused(tmp_path): + """The same call as an unreachable `path`: better a refusal than a board + quietly short a folder's worth of repos.""" + with pytest.raises(repos.FleetError, match="cannot read folder"): + repos.load(write_fleet(tmp_path, f" - folder: {tmp_path / 'absent'}\n")) + + +def test_a_folder_with_no_checkouts_in_it_is_refused(tmp_path): + """Naming the folder was a request for the repos in it; there are none.""" + (tmp_path / "org").mkdir() + (tmp_path / "org" / "not-a-repo").mkdir() + + with pytest.raises(repos.FleetError, match="holds no git checkouts"): + repos.load(write_fleet(tmp_path, f" - folder: {tmp_path / 'org'}\n")) + + +@pytest.mark.parametrize("key", ["path", "repo"]) +def test_a_folder_cannot_also_be_a_path_or_a_repo(tmp_path, key): + """A folder stands for many repos, so there is no one path or name for it.""" + make_checkout(tmp_path, "org", "alpha") + body = f" - folder: {tmp_path / 'org'}\n {key}: whatever/it-is\n" + + with pytest.raises(repos.FleetError, match="cannot also carry"): + repos.load(write_fleet(tmp_path, body)) + + +@pytest.mark.parametrize("explicit_first", [True, False]) +def test_an_entry_naming_a_repo_outright_wins_over_the_folder(tmp_path, explicit_first): + """How one repo in a listed folder gets a `repo:` override. + + The checkout is a fork and the board should follow upstream. Colliding + would mean the folder could not be used at all; the deliberate entry wins, + whichever order the two are written in. + """ + make_checkout(tmp_path, "org", "alpha") + fork = make_checkout(tmp_path, "org", "beta") + folder = f" - folder: {tmp_path / 'org'}\n" + override = f" - path: {fork}\n repo: upstream/beta\n" + + fleet, paths, _forges = repos.load( + write_fleet(tmp_path, override + folder if explicit_first else folder + override) + ) + + assert set(fleet) == {"org/alpha", "upstream/beta"} + assert "org/beta" not in paths + assert paths["upstream/beta"] == str(fork) + + +def test_a_repo_listed_outright_and_swept_up_by_a_folder_is_one_repo(tmp_path): + """Belt and braces in repos.yml is not a duplicate to refuse over.""" + path = make_checkout(tmp_path, "org", "alpha") + body = f" - folder: {tmp_path / 'org'}\n - path: {path}\n" + + fleet, paths, _forges = repos.load(write_fleet(tmp_path, body)) + + assert fleet == ("org/alpha",) + assert paths == {"org/alpha": str(path)} + + +def test_two_folders_holding_the_same_repo_are_refused(tmp_path): + """Two checkouts, one row: there is no saying which one the board means.""" + make_checkout(tmp_path / "one", "org", "alpha") + make_checkout(tmp_path / "two", "org", "alpha") + body = f" - folder: {tmp_path / 'one' / 'org'}\n - folder: {tmp_path / 'two' / 'org'}\n" + + with pytest.raises(repos.FleetError, match="checked out at .* and at "): + repos.load(write_fleet(tmp_path, body)) + + +def test_a_declared_forge_covers_every_checkout_in_the_folder(tmp_path): + """A whole folder on the other forge is one line, not one line per repo.""" + make_checkout(tmp_path, "acme", "web", origin="git@gitlab.com:acme/web.git") + make_checkout(tmp_path, "acme", "api", origin="git@gitlab.com:acme/api.git") + body = f" - folder: {tmp_path / 'acme'}\n forge: gitlab\n" + + assert repos.load(write_fleet(tmp_path, body))[2] == { + "acme/api": "gitlab", + "acme/web": "gitlab", + } + + +def test_a_bad_forge_on_a_folder_is_refused_once(tmp_path): + make_checkout(tmp_path, "org", "alpha") + body = f" - folder: {tmp_path / 'org'}\n forge: gitbucket\n" + + with pytest.raises(repos.FleetError, match="forge 'gitbucket'"): + repos.load(write_fleet(tmp_path, body)) + + +def test_a_scratch_clone_in_the_folder_is_skipped_not_fatal(tmp_path, caplog): + """A clone with no origin cannot be named, and nobody asked to monitor it. + + A listed `path` to the same clone is still refused - that path was a + deliberate statement. A folder is not, so one stray clone must not take the + whole board down. + """ + make_checkout(tmp_path, "org", "alpha") + scratch = make_checkout(tmp_path, "org", "scratch") + subprocess.run(["git", "-C", str(scratch), "remote", "remove", "origin"], check=True) + + with caplog.at_level("WARNING"): + fleet, paths, _forges = repos.load( + write_fleet(tmp_path, f" - folder: {tmp_path / 'org'}\n") + ) + + assert fleet == ("org/alpha",) + assert str(scratch) not in paths.values() + assert "skipping" in caplog.text + + +def test_a_folder_of_nothing_but_scratch_clones_refuses_to_start(tmp_path): + """The file said something and none of it survived - that is not a fleet.""" + scratch = make_checkout(tmp_path, "org", "scratch") + subprocess.run(["git", "-C", str(scratch), "remote", "remove", "origin"], check=True) + + with pytest.raises(repos.FleetError, match="could identify"): + repos.load(write_fleet(tmp_path, f" - folder: {tmp_path / 'org'}\n")) + + +def test_a_folder_is_read_through_the_host_mount(tmp_path): + """`~/repos/org` in repos.yml is /host/repos/org inside the container.""" + make_checkout(tmp_path / "repos", "org", "alpha") + + fleet, paths, _forges = repos.load( + write_fleet(tmp_path, " - folder: ~/repos/org\n"), host_root=str(tmp_path) + ) + + assert fleet == ("org/alpha",) + assert paths == {"org/alpha": str(tmp_path / "repos" / "org" / "alpha")} + + +@pytest.mark.parametrize("outright_first", [True, False]) +def test_naming_a_repo_the_folder_also_holds_is_one_entry(tmp_path, outright_first): + """`- repo: org/alpha` beside the folder org/alpha is checked out in. + + The entry written for the repo itself is the deliberate one, so it decides + the name and the forge either way round - but the folder still says where + the checkout is, which the other entry never claimed to know. + """ + path = make_checkout(tmp_path, "org", "alpha") + outright = " - repo: org/alpha\n" + folder = f" - folder: {tmp_path / 'org'}\n" + + fleet, paths, forges = repos.load( + write_fleet(tmp_path, outright + folder if outright_first else folder + outright) + ) + + assert fleet == ("org/alpha",) + assert paths == {"org/alpha": str(path)} + assert forges == {"org/alpha": "github"} diff --git a/docs/configuration.md b/docs/configuration.md index 6d8d1d2..878e0a0 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -11,27 +11,82 @@ environment variables on the `docker run`. ## The fleet is an explicit list -`repos.yml` names every monitored repo, one -entry per checkout on this machine: +`repos.yml` names every monitored repo — one entry per checkout on this +machine, or one entry for a folder full of them: ```yaml repos: - path: ~/repos/jebel-quant/rhiza - - path: ~/repos/cvxgrp/cvxsimulator - - repo: Jebel-Quant/actions # monitored, but not checked out here + - folder: ~/repos/cvxgrp # every checkout directly inside it + - repo: Jebel-Quant/actions # monitored, but not checked out here ``` `owner/name` is read from each checkout's `origin` remote, so the path is all -you write. Nothing is discovered: a repo is on the board because it is in this -file, and for no other reason. The collector reads the file itself, at startup, -from `/config/repos.yml` — nothing is generated from it, so there is no second -file to fall out of step. +you write. Nothing is discovered behind your back: a repo is on the board +because this file names it, or names the folder it sits in, and for no other +reason. The collector reads the file itself, at startup, from +`/config/repos.yml` — nothing is generated from it, so there is no second file +to fall out of step. This replaced a whole-org GitHub sweep plus a directory walk under one mounted root. Both decided membership on their own: a new repo in the org arrived unasked, a shared org like cvxgrp dragged in 100+ repos that were not yours, and any checkout that happened to sit under the root joined the board because its -origin looked right. +origin looked right. A `folder:` is not that walk back: it is one directory you +wrote down, read one level deep, and the repos it holds are the repos you keep +there. + +## A folder of repos + +Where you keep a whole org checked out, name the folder and every checkout in +it is on the board: + +```yaml +repos: + - folder: ~/repos/jebel-quant +``` + +Sixteen repos become one line, and the line cannot drift out of step with the +disk the way sixteen can — clone a repo into the folder and it joins at the +next `docker restart jq-fleet`, `rm -rf` one and it leaves. That is the trade: +a folder is convenient exactly because you are no longer deciding repo by repo, +so keep folders for the directories you want whole and list the repos +one by one where you want only some of them. + +The rules, all of which exist so the board cannot go quietly short: + +- **One level, never deeper.** Only the folder's own children are looked at, so + `folder: ~/repos` finds nothing when your repos live in `~/repos//` + — name the folders the checkouts are directly in. Recursing would make + `folder: ~` the whole-disk walk this file exists to have got rid of. +- **A missing folder, or one with no checkouts in it, refuses to start.** You + asked for the repos in it and there are none; an unreachable `path` is refused + for the same reason. +- **A directory that is not a checkout is passed over**, and so is a clone with + no `origin` remote to name it by — that is somebody's scratch clone, logged as + a warning and skipped rather than taken as fatal. A `path` naming the same + clone is still refused, because that path was a deliberate statement. +- **An entry of its own wins.** A folder leaves out any checkout another entry + names by `path`, so one repo inside a listed folder can carry a `repo:` + override for its upstream: + + ```yaml + repos: + - folder: ~/repos/forks + - path: ~/repos/forks/cvxpy # a fork; the board follows upstream + repo: cvxpy/cvxpy + ``` + + Matching on the path rather than on `owner/name` is what makes that work, + since the point of the override is that the name comes out different. An + entry that names a repo the folder also holds (`- repo: org/x`) is likewise + one repo, not a duplicate: the entry decides the name and the forge, and the + folder still supplies the checkout path. +- **`forge: gitlab` on a folder covers every checkout in it**, though with + checkouts to read the origin off it is not needed at all. + +`JQ_IGNORE` is the way to drop one repo a folder sweeps up without listing the +rest by hand. Edit `repos.yml`, `docker restart jq-fleet`, and the fleet is whatever you just wrote. Both halves of the collector read the same list, so the GitHub panels @@ -109,10 +164,12 @@ alarming. |---|---|---| | `path` | | A checkout on this machine, written as you would write it yourself. `~` is your home directory — which the container sees as the single `-v "$HOME:/host:ro"` mount — and a relative path is relative to it too. | | `repo` | | `owner/name`. Optional next to a `path` — it overrides the origin, which is what you want for a fork whose board should follow upstream. On its own it monitors a repo you have not cloned: GitHub panels are gathered, the working-copy panels stay empty for that row. | +| `folder` | | A directory full of checkouts. Every checkout directly inside it joins the fleet, one level deep and no further — see [A folder of repos](#a-folder-of-repos). Cannot be combined with `path` or `repo`, which describe one repo each. | A bare string is shorthand for `path`. Duplicate entries, a path that is not a -checkout, and an entry with neither key all stop the collector at startup — -better a refusal than a board that is quietly one repo short. +checkout, a folder that is missing or empty of checkouts, and an entry with +neither key all stop the collector at startup — better a refusal than a board +that is quietly one repo short. The one thing that is *not* fatal is a `path` that cannot be reached alongside an explicit `repo:`. That is what running without the `$HOME` mount looks like, diff --git a/docs/index.md b/docs/index.md index 6feaeda..a76a632 100644 --- a/docs/index.md +++ b/docs/index.md @@ -10,8 +10,9 @@ A Grafana board for the state of your repo fleet — template drift, CI on the default branch, open pull requests, and the working copies on your machine — with Prometheus keeping the history and six alert rules on top. -One container, one file. **Nothing is discovered:** a repo is on the board -because you listed it in `repos.yml`, and for no other reason. +One container, one file. **Nothing is discovered behind your back:** a repo is +on the board because `repos.yml` names it — or names the folder you keep it in — +and for no other reason. ## What you need @@ -20,17 +21,19 @@ token that can read the repos you list. ## The recipe -Write a `repos.yml` — one entry per repo you want on the board: +Write a `repos.yml` — one entry per repo you want on the board, or one +entry for a folder of them: ```yaml repos: - path: ~/repos/jebel-quant/rhiza # a checkout on this machine - - path: ~/repos/cvxgrp/cvxsimulator + - folder: ~/repos/cvxgrp # every checkout in a folder - repo: Jebel-Quant/actions # monitored, but not cloned here ``` -`owner/name` comes from each checkout's `origin`, so the path is all you write. -Then: +`owner/name` comes from each checkout's `origin`, so the path is all you write, +and a [`folder:`](configuration.md#a-folder-of-repos) is every checkout inside +it — one line for an org you keep whole. Then: ```bash docker run -d --name jq-fleet \ diff --git a/repos.example.yml b/repos.example.yml index 555adb1..f1abf4a 100644 --- a/repos.example.yml +++ b/repos.example.yml @@ -5,8 +5,8 @@ # /config/repos.yml and read at startup; nothing is generated from it, so there # is no second file to fall out of step. # -# Nothing is discovered. A repo appears on the board because it is listed here, -# and for no other reason. +# A repo appears on the board because it is listed here - by its own path, or +# by a folder you keep it in - and for no other reason. repos: # The usual entry: a path to a checkout, written the way you would write it @@ -16,6 +16,18 @@ repos: - path: ~/repos/jebel-quant/monitoring - path: ~/repos/jebel-quant/rhiza + # A whole folder: every checkout sitting directly inside it is on the board, + # so an org you keep complete is one line instead of twenty, and the twenty + # cannot drift out of step with what is actually on disk. Clone a repo into + # the folder and it joins at the next restart. + # + # Only the folder's own children are looked at, never their children in turn + # - name the folder the checkouts are directly in. A directory that is not a + # checkout is passed over, and so is a clone with no origin remote to name it + # by; a folder that is missing, or that holds no checkouts at all, refuses to + # start, because you asked for the repos in it and there are none. + # - folder: ~/repos/cvxgrp + # Only some of a large shared org is yours, so name those repos one by one. - path: ~/repos/cvxgrp/cvxsimulator @@ -24,6 +36,13 @@ repos: # - path: ~/repos/forks/cvxpy # repo: cvxpy/cvxpy + # This works inside a listed folder too: a folder leaves out any checkout + # another entry names by `path`, so the two lines below put every repo in + # ~/repos/forks on the board with cvxpy following upstream. + # - folder: ~/repos/forks + # - path: ~/repos/forks/cvxpy + # repo: cvxpy/cvxpy + # A repo with no checkout on this machine. GitHub panels (CI, pull requests, # template drift, coverage) are gathered; the working-copy panels stay empty # for it. This is also the form to use for every entry if you would rather @@ -48,6 +67,12 @@ repos: # A GitLab checkout, where the forge follows from the origin remote's host. # - path: ~/repos/acme/web + # A whole folder on GitLab. Stating the forge once covers every checkout the + # folder turns up, though with a checkout to read the origin off it is not + # needed at all. + # - folder: ~/repos/acme + # forge: gitlab + # Two notes on a mixed fleet: # # - Set GITLAB_TOKEN as well as GITHUB_TOKEN. Public projects are readable