feat: one repos.yml entry can be a folder of repos

README.md

@@ -10,8 +10,9 @@ A Grafana board for the state of your repo fleet — template drift, CI on the
 default branch, open pull requests, and the working copies on this machine —
 with Prometheus keeping the history and six alert rules on top.
 
-One container, one file. Nothing is discovered: a repo is on the board because
-you listed it in `repos.yml`, and for no other reason.
+One container, one file. Nothing is discovered behind your back: a repo is on
+the board because `repos.yml` names it — or names the folder you keep it in —
+and for no other reason.
 
 ![The fleet board: tiles counting repos monitored, open PRs and issues, CI red on main, PRs with red checks, local clones out of sync, dirty working copies, branch protection and Dependabot, above a seven-day trend of open problems by kind.](dashboard.png)
 
@@ -22,20 +23,24 @@ table of exactly which repos are behind its number — see
 
 ## Recipe
 
-Write a `repos.yml` — one entry per repo you want on the board:
+Write a `repos.yml` — one entry per repo you want on the board, or one
+entry for a folder of them:
 
 ```yaml
 repos:
   - path: ~/repos/jebel-quant/rhiza     # a checkout on this machine
-  - path: ~/repos/cvxgrp/cvxsimulator
+  - folder: ~/repos/cvxgrp              # every checkout in a folder
   - repo: Jebel-Quant/actions           # monitored, but not cloned here
   - repo: acme/platform/infra/web       # GitLab, on the same board
     forge: gitlab
 ```
 
 `namespace/name` comes from each checkout's `origin`, so the path is all you
 write, and the path is used as written — a checkout does not have to live at
-`<root>/<owner>/<name>`. **GitHub and GitLab repos share one board**; the forge
+`<root>/<owner>/<name>`. **A `folder:` puts every checkout inside it on the
+board**, one level deep, which is how an org you keep whole stays one line
+instead of twenty — see
+[A folder of repos](docs/configuration.md#a-folder-of-repos). **GitHub and GitLab repos share one board**; the forge
 is read off the origin's host where there is a checkout, and stated with
 `forge: gitlab` where there is not — see
 [Configuration](docs/configuration.md#github-and-gitlab-in-the-same-fleet).

collector/jq_collector/repos.py

@@ -16,6 +16,15 @@
 mounted, or that repo is not checked out here: the GitHub panels still report
 on it and the working-copy panels have nothing to say. That is the same shape
 as an entry with no ``path`` at all.
+
+One entry may also name a ``folder:``, and then every checkout sitting directly
+inside it is on the board. That is still the list deciding membership - the
+folder is named here, one level is scanned and no deeper, and a repo joins
+because a folder you wrote down holds it. It is not the old whole-root walk,
+which took any checkout anywhere under one mount whose origin looked plausible.
+Cloning a repo into a listed folder does add it to the board at the next
+restart, which is the point: an org you keep whole is one line instead of
+twenty, and the twenty cannot drift out of step with the disk.
 """
 
 from __future__ import annotations
@@ -83,6 +92,100 @@ def _declared_forge(item: dict, index: int) -> str | None:
     return declared
 
 
+def _folder(item: dict, index: int, host_root: str, claimed: frozenset[str]) -> list[dict]:
+    """A ``folder:`` entry -> one ``path:`` entry per checkout inside it.
+
+    Only the folder's own children are looked at, never their children in turn.
+    A folder is a statement about where you keep a set of repos, and one level
+    is what that means; recursing would make ``folder: ~`` a whole-disk walk,
+    which is the discovery this file exists to have got rid of.
+
+    A checkout some other entry names by ``path`` is left to that entry -
+    ``claimed`` is every such path in the file. That is how one repo inside a
+    listed folder gets an override: the folder covers the rest, and the entry
+    written out for that one checkout is the only entry for it. Matching on the
+    path rather than on ``namespace/name`` is what makes a ``repo:`` override
+    work, since the whole point of one is that the name comes out different.
+
+    A folder that is not there is refused, exactly as an unreachable ``path``
+    is, and so is one holding no checkouts at all. Both look identical to a
+    fleet that is quietly short a folder's worth of repos, and the point of
+    naming the folder was to get those repos on the board.
+    """
+    raw = str(item.get("folder") or "").strip()
+    for key in ("path", "repo"):
+        if item.get(key):
+            raise FleetError(
+                f"entry {index}: `folder: {raw}` cannot also carry a `{key}` - "
+                "a folder stands for however many repos are in it, so there is "
+                "no one path or name to give it. List the repo on its own entry."
+            )
+    # Validated here so a bad `forge:` on the folder is refused once, against
+    # the line that was actually written, rather than per checkout found.
+    forge = _declared_forge(item, index)
+
+    folder = resolve_path(raw, host_root)
+    try:
+        children = sorted(entry.path for entry in os.scandir(folder) if entry.is_dir())
+    except OSError as exc:
+        raise FleetError(
+            f"entry {index}: cannot read folder {raw} (looked in {folder}): {exc}. "
+            "Mount the home directory it lives under, or list the repos in it "
+            "one by one."
+        ) from exc
+
+    checkouts = [child for child in children if _is_checkout(child)]
+    if not checkouts:
+        raise FleetError(
+            f"entry {index}: folder {raw} holds no git checkouts (looked in {folder}). "
+            "A folder is scanned one level deep, so name the folder the "
+            "checkouts are directly in."
+        )
+    # The emptiness check is on what is in the folder, not on what is left
+    # after the claimed ones go: a folder whose every checkout has an entry of
+    # its own is a redundant line, not a mistake worth refusing to start over.
+    taken = [child for child in checkouts if child not in claimed]
+    if len(taken) == len(checkouts):
+        log.info("folder %s: %d checkouts", raw, len(taken))
+    else:
+        log.info(
+            "folder %s: %d checkouts, %d left to an entry of their own",
+            raw,
+            len(taken),
+            len(checkouts) - len(taken),
+        )
+    return [{"path": child, "forge": forge} for child in taken]
+
+
+def _expand(
+    item: Any, index: int, host_root: str, claimed: frozenset[str]
+) -> list[tuple[Any, bool]]:
+    """One config entry -> the entries to read, and whether each was swept up.
+
+    Everything is one entry, written outright, except a ``folder:`` - which is
+    however many checkouts are in it. The flag is what lets an entry named
+    outright override one a folder found, instead of colliding with it.
+    """
+    if isinstance(item, dict) and item.get("folder"):
+        return [(found, True) for found in _folder(item, index, host_root, claimed)]
+    return [(item, False)]
+
+
+def _claimed_paths(entries: list, host_root: str) -> frozenset[str]:
+    """Every checkout the file names by ``path``, resolved for this filesystem.
+
+    A folder skips these, so an entry written out for one checkout inside a
+    listed folder is that checkout's only entry.
+    """
+    claimed = set()
+    for item in entries:
+        if isinstance(item, str):
+            claimed.add(resolve_path(item, host_root))
+        elif isinstance(item, dict) and item.get("path"):
+            claimed.add(resolve_path(str(item["path"]), host_root))
+    return frozenset(claimed)
+
+
 def _entry(item: Any, index: int, host_root: str) -> tuple[str, str | None, str]:
     """One entry -> ``(namespace/name, checkout path or None, forge)``."""
     # `- ~/repos/foo` is accepted as shorthand for `- path: ~/repos/foo`.
@@ -168,20 +271,72 @@ def load(
     fleet: list[str] = []
     paths: dict[str, str] = {}
     forges: dict[str, str] = {}
+    # Named outright rather than swept up by a folder, which is what decides
+    # who wins when both describe the same repo.
+    named: set[str] = set()
+    claimed = _claimed_paths(entries, host_root)
     for index, item in enumerate(entries, start=1):
-        full_name, path, forge = _entry(item, index, host_root)
-        if full_name in fleet:
-            # Two forges can host the same `namespace/name`, and the board's
-            # whole label scheme is that one `repo` value is one repo. Refusing
-            # is the same choice the duplicate case has always made: a merged
-            # pair would report one repo's CI under the other's name, and read
-            # as a working board while doing it.
-            clash = forges[full_name]
-            detail = f" - on {clash} and on {forge}" if clash != forge else ""
-            raise FleetError(f"{full_name} is listed twice{detail}")
-        fleet.append(full_name)
-        forges[full_name] = forge
-        if path is not None:
-            paths[full_name] = path
+        for entry, swept in _expand(item, index, host_root, claimed):
+            try:
+                full_name, path, forge = _entry(entry, index, host_root)
+            except FleetError as exc:
+                if not swept:
+                    raise
+                # A checkout with no usable origin is somebody's scratch clone
+                # sitting in the folder. Refusing would take the whole board
+                # down over a repo nobody asked to monitor; a folder entry is
+                # not the deliberate statement that a listed path is.
+                log.warning("skipping %s: %s", entry["path"], exc)
+                continue
+
+            # Exactly one of the two entries names the repo outright: a folder
+            # overlapping an entry, which is not the duplicate case below.
+            overlap = (not swept) != (full_name in named)
+            if full_name in forges and overlap:
+                # One entry names this repo outright and the other is a folder
+                # that swept it up - `- repo: org/x` next to the folder org/x
+                # is checked out in. The entry written for the repo itself is
+                # the deliberate statement, so it decides the name and the
+                # forge whichever order the two were written in; the folder can
+                # still supply the checkout path, since that is where the repo
+                # is on disk and the other entry may not have said.
+                #
+                # (A folder never reaches here for a checkout some entry names
+                # by `path` - it skips those outright, which is what lets a
+                # `repo:` override rename one repo inside a listed folder.)
+                log.info("%s: named outright, so the folder does not list it too", full_name)
+                if not swept:
+                    named.add(full_name)
+                    forges[full_name] = forge
+                if path is not None:
+                    paths[full_name] = path if not swept else paths.get(full_name, path)
+                continue
+
+            if full_name in forges:
+                # Two forges can host the same `namespace/name`, and the board's
+                # whole label scheme is that one `repo` value is one repo. Refusing
+                # is the same choice the duplicate case has always made: a merged
+                # pair would report one repo's CI under the other's name, and read
+                # as a working board while doing it.
+                clash, first = forges[full_name], paths.get(full_name)
+                if clash != forge:
+                    detail = f" - on {clash} and on {forge}"
+                elif first and path and first != path:
+                    detail = f" - checked out at {first} and at {path}"
+                else:
+                    detail = ""
+                raise FleetError(f"{full_name} is listed twice{detail}")
+
+            fleet.append(full_name)
+            forges[full_name] = forge
+            if not swept:
+                named.add(full_name)
+            if path is not None:
+                paths[full_name] = path
+
+    if not fleet:
+        # Reachable only when every checkout a folder turned up was skipped for
+        # want of an origin: the file said something, and none of it survived.
+        raise FleetError(f"{source} named no repo the collector could identify")
 
     return tuple(fleet), paths, forges