Reuse deployed CVMFS releases

CHANGELOG.md

@@ -27,6 +27,8 @@ Covers `bits`, `lcg.bits` (recipes), and `bits-recipe-tools`. Entries tagged **[
 - **[Improvement]** `621dded` warn on provider version conflict; `4d92e95` point the "package not found" error at the provider mechanism; `ee78182` docs.
 
 ## CVMFS layout, merged views & relaxed reuse
+- **[Feature]** `--reuse-from <modules-path>|cvmfs` reuses deployed components via their published modulefiles/`init.sh` (sourced in place from `/cvmfs`); `--reuse-policy strict|relaxed`, `<src>::policy` sugar, per-dependency reuse, corrected pkg-config prefixes, and `cvmfs`-resolution from the `cvmfs_modules_template`.
+- **[Change]** Removed the legacy `cvmfs://` `--remote-store` reuse: the `build_id` graft (`--reuse-base`/`--reuse-cvmfs`, `CVMFSRemoteSync`, `select_build_id`/`graftable_match`) is retired in favour of `--reuse-from`; a `cvmfs://` `--remote-store` now errors and points at `--reuse-from` (see ADR-0001, superseded).
 - **[Feature]** `0632f28` / `b70abac` / `b1dfc1d` / `e8d1222` / `ea04075` merged symlink-farm view: one-entry-per-var env, opt-in `enter/setenv --view`, view-aware `load`/`printenv` + age-based GC, path remap (fixes PyROOT).
 - **[Feature]** `58f13b6` / `c2d99be` / `f59a547` / `53e91d4` / `80b6a08` published per-`build_id` views on CVMFS, `bits publish --view`, per-tree pre-publish primitive, CVMFS layout recorded in `.meta.json`.
 - **[Feature]** `8f193fa`→`c52f5d7`, `cf19966` ADR-0001 import pipeline: modulefile harvest → classify → closure/`build_id` → overlay → `bits import` (build-sufficient from modulefiles).

bits

@@ -5,6 +5,19 @@
 
 BITSDIR="$(dirname "$0")"
 
+# `.bitscmd` saved-arg profile (see `bits use`): inject the [common] + [<action>]
+# tokens right after the action, BEFORE anything is parsed, so both the build
+# path and the module commands (q/enter/…, which read ARGV below) see them.
+# Runs python only when a profile exists; `use` opts out; explicit args stay last
+# (they win on single-value options). Safe no-op if the action isn't first.
+if [[ -f .bitscmd ]]; then
+  _bits_prof=()
+  while IFS= read -r -d '' _bits_t; do _bits_prof+=("$_bits_t"); done \
+    < <(BITS_SELF="${BITSDIR}" python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.bits_use import main; sys.exit(main())' --rewrite0 -- "$@")
+  [[ ${#_bits_prof[@]} -gt 0 ]] && set -- "${_bits_prof[@]}"
+  unset _bits_prof _bits_t
+fi
+
 ARGV=("$@"); ARGC=$#   # ARGC must be a plain integer, not an array
 
 # Optional phase timing for diagnosing slow load/enter: set BITS_TIMING=1 to
@@ -375,6 +388,23 @@ if [[ "${1:-}" == "store" ]]; then
   exec "$BITSDIR/bitsStore" "$@"
 fi
 
+# `bits use …` → save/show/clear a per-command arg profile in ./.bitscmd, so
+# repeated commands stay short (the injection at the top of this script consumes
+# it). Handled here, before work-dir/module setup: it only touches ./.bitscmd.
+if [[ "${1:-}" == "use" ]]; then
+  shift
+  BITS_SELF="${BITSDIR}" exec python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.bits_use import main; sys.exit(main())' "$@"
+fi
+
+# `bits cvmfs {platforms,show,summary} …` → read-only inspection of a deployed
+# CVMFS bits tree: platforms + host compatibility, a package's build/provenance
+# from .meta.json (no jq), and a per-build_id summary. Reads the tree only, so it
+# needs no work dir; distinct from cvmfs-stage/cvmfs-publish (producer side).
+if [[ "${1:-}" == "cvmfs" ]]; then
+  shift
+  BITS_SELF="${BITSDIR}" exec python3 -c 'import os,sys; sys.path.insert(0, os.environ.get("BITS_SELF","")); from bits_helpers.cvmfs_inspect import main; sys.exit(main())' "$@"
+fi
+
 # `bits cvmfs-stage …` → producer-side CVMFS staging (ADR-0011).
 # Prepares a package into a staging S3 prefix with the canonical publisher and
 # names the catalog prepub must graft. Handled here, before work-dir/module
@@ -400,7 +430,7 @@ fi
 for arg in "$@"
 do
   case $arg in
-    architecture|brew|build|certify|clean|cleanup|compliance|cvmfs-path|deps|doctor|gc|init|publish|stats|status|store-stats|verify|version|-debug|-d)
+    architecture|brew|build|certify|clean|cleanup|compliance|cvmfs-path|deps|doctor|gc|import|init|publish|stats|status|store-stats|verify|version|-debug|-d)
          mkdir -p "$BITS_WORK_DIR" || echo "Cannot create directory: $BITS_WORK_DIR"
         "$BITSDIR/bitsBuild" "$@"
 	    exit $?

bits_helpers/args.py

@@ -380,8 +380,8 @@ def doParseArgs():
           "manifest), closure-check the set, stamp it with one deterministic "
           "build_id, and generate a per-build_id overlay (build-sufficient bits "
           "modulefiles + module-side .meta.json + .cvmfscatalog) that "
-          "'bits build --reuse-policy relaxed --reuse-base <build_id>' can graft "
-          "without recompiling. See ADR-0001."
+          "'bits build --reuse-from <modules-path>|cvmfs' can reuse without "
+          "recompiling."
       ),
   )
   import_parser.add_argument("-w", "--work-dir", dest="workDir",
@@ -397,6 +397,15 @@ def doParseArgs():
                              metavar="FILE", default=None,
                              help="JSON manifest to import instead of harvesting (fallback when "
                                   "no modulefiles exist).")
+  import_parser.add_argument("--trusted", dest="importTrusted", action="store_true",
+                             help="Trusted mode: harvest a bits-built deployment directly, reading "
+                                  "its own modulefiles (--modulepath) and re-anchoring them to "
+                                  "--install-base, capturing package hashes from the install tree. "
+                                  "Deterministic (no modulecmd); publishable strict reuse.")
+  import_parser.add_argument("--install-base", dest="importInstallBase",
+                             metavar="DIR", default=None,
+                             help="With --trusted, the absolute Packages root the modulefiles' "
+                                  "BASEDIR resolves to (and where each package's .meta.json lives).")
   import_parser.add_argument("--aliases", dest="importAliases",
                              metavar="FILE", default=None,
                              help="JSON name-alias map (foreign -> bits names).")
@@ -673,10 +682,6 @@ def doParseArgs():
                             is set to the same value). May be set to a default store on some
                             architectures; use --no-remote-store to disable it in that case.
                             """)
-  build_remote.add_argument("--reuse-cvmfs", dest="reuseCvmfs", action="store_true",
-                            help=("Reuse already-deployed components from the CVMFS area declared by the "
-                                  "defaults `cvmfs_dir:` field. Sets --remote-store to cvmfs://<cvmfs_dir> "
-                                  "when no remote store is given."))
   build_remote.add_argument("--sign-manifest", dest="signManifest", default=None, metavar="KEY.pem",
                             help=("After the build, sign the build manifest with this Ed25519 private key "
                                   "(PEM) so consumers can verify it for trusted reuse. Public verification "
@@ -727,13 +732,14 @@ def doParseArgs():
                                   "build_id) for fast local dev; the result is loose-provenance and is "
                                   "refused by the publish path. Falls back to the defaults `reuse_policy:` "
                                   "value, else 'strict'."))
-  build_remote.add_argument("--reuse-base", dest="reuseBase", metavar="BUILD_ID", default=None,
-                            help=("With --reuse-policy relaxed, the build_id of the deployed release to "
-                                  "graft packages from. Falls back to the defaults `reuse_base:` value. "
-                                  "Special values auto-select from the cvmfs:// store: 'latest' (newest "
-                                  "build_id for the build target) and 'latest-common' (newest build_id "
-                                  "shared by all requested packages); the same happens when left empty "
-                                  "under relaxed. The chosen build_id is announced."))
+  build_remote.add_argument("--reuse-from", dest="reuseFrom", metavar="PATH|cvmfs", default=None,
+                            help=("Reuse deployed components via their published modulefiles at this "
+                                  "absolute modules-tree path (distinct from --remote-store, which is the "
+                                  "tarball store). The literal 'cvmfs' resolves the exact location from the "
+                                  "defaults `system:` layout (module_dir under cvmfs_dir); fails if that is "
+                                  "not configured. A trailing '::relaxed' or '::strict' also sets the reuse "
+                                  "policy (e.g. 'cvmfs::relaxed'); it must agree with --reuse-policy if both "
+                                  "are given."))
   build_remote.add_argument("--build-local", dest="buildLocal", metavar="PKG[,PKG...]", default="",
                             help=("Comma-separated packages to always build locally even under "
                                   "--reuse-policy relaxed (e.g. a package you need patched), rather than "

bits_helpers/bits_use.py

@@ -0,0 +1,222 @@
+# SPDX-FileCopyrightText: 2015-2026 CERN
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+"""`bits use` — save reusable command-line args in ``./.bitscmd`` so repeated
+commands stay short. Distinct from ``.bitsrc`` (typed key=value settings):
+``.bitscmd`` holds raw CLI tokens, structured by the command they apply to.
+
+A ``[common]`` section is injected into arch-aware commands (put only broadly
+accepted args here, i.e. ``--architecture``, which ``build`` AND ``q``/``enter``/
+``clean`` all take); a per-command section (``[build]``, ``[q]``, …) adds args
+that command accepts — e.g. ``--defaults`` belongs in ``[build]``, NOT
+``[common]`` (module commands and ``clean``/``import`` don't accept it). Injected
+BEFORE the user's own args, so those override single-value options. Example::
+
+    [common]
+    --architecture x86_64-el9-gcc14-opt
+
+    [build]
+    --defaults lcg::release::gcc14::opt --docker --sandbox off --reuse-from cvmfs::relaxed
+
+Prototype — runs standalone::
+
+    python3 -m bits_helpers.bits_use common --architecture x86_64-el9-gcc14-opt
+    python3 -m bits_helpers.bits_use build  --docker --sandbox off
+    python3 -m bits_helpers.bits_use            # show all sections
+    python3 -m bits_helpers.bits_use --clear [SECTION]
+"""
+
+import os
+import shlex
+import sys
+
+PROFILE = ".bitscmd"
+COMMON = "common"          # section injected into every command ('global' alias)
+
+
+def _join(tokens):
+    try:
+        return shlex.join(tokens)          # Python 3.8+
+    except AttributeError:                 # pragma: no cover
+        return " ".join(shlex.quote(t) for t in tokens)
+
+
+def read_all(path=PROFILE):
+    """Parse the profile into an ordered ``{section: [tokens]}`` dict.
+
+    ``[name]`` opens a section; lines before any header belong to ``common``;
+    ``#`` comments and blank lines are ignored. Each section's lines are joined
+    and shlex-split into tokens.
+    """
+    sections, cur, buf = {}, COMMON, []
+    if not os.path.exists(path):
+        return sections
+
+    def _flush():
+        if buf:
+            sections.setdefault(cur, [])
+            sections[cur] += shlex.split(" ".join(buf))
+    try:
+        with open(path) as fh:
+            for raw in fh:
+                line = raw.strip()
+                if not line or line.startswith("#"):
+                    continue
+                if line.startswith("[") and line.endswith("]"):
+                    _flush(); buf = []
+                    cur = line[1:-1].strip().lower()
+                    cur = COMMON if cur in ("global", COMMON) else cur
+                    sections.setdefault(cur, [])
+                    continue
+                buf.append(line)
+        _flush()
+    except (OSError, ValueError):
+        # ValueError: a malformed token (e.g. an unbalanced quote) in the
+        # profile. Fail safe — ignore the profile rather than crash a build.
+        return {}
+    return sections
+
+
+def write_section(section, tokens, path=PROFILE):
+    """Replace *section* with *tokens*, preserving the other sections."""
+    section = COMMON if section in ("global", COMMON) else section.lower()
+    sections = read_all(path)
+    sections[section] = list(tokens)
+    _write_all(sections, path)
+    return path
+
+
+def clear_section(section=None, path=PROFILE):
+    """Clear one section, or the whole profile when *section* is None."""
+    if section is None:
+        try:
+            os.unlink(path); return True
+        except OSError:
+            return False
+    sections = read_all(path)
+    section = COMMON if section in ("global", COMMON) else section.lower()
+    if section in sections:
+        del sections[section]
+        _write_all(sections, path)
+        return True
+    return False
+
+
+def _write_all(sections, path=PROFILE):
+    order = [COMMON] + [s for s in sections if s != COMMON]
+    with open(path, "w") as fh:
+        for s in order:
+            toks = sections.get(s)
+            if not toks:
+                continue
+            fh.write("[%s]\n%s\n\n" % (s, _join(toks)))
+
+
+def merged_argv(command, user_args, path=PROFILE):
+    """Args to run for *command*: ``[common]`` then ``[command]`` then the
+    user's own args (which come last and win on single-value options)."""
+    sec = read_all(path)
+    return sec.get(COMMON, []) + sec.get((command or "").lower(), []) + list(user_args)
+
+
+# Top-level flags that may precede the action (from the root argparse parser);
+# skipped when locating the action token.
+TOP_FLAGS = {"-d", "--debug", "-n", "--dry-run"}
+
+# The profile is injected ONLY into arch-aware commands (they all accept
+# `--architecture`, the intended `[common]` content). Meta commands (use, cvmfs,
+# store, version, help, cvmfs-stage/publish) and `verify` (accepts neither
+# --architecture nor --defaults) take a different option set and are excluded.
+# Note: several of these accept --architecture but NOT --defaults, so --defaults
+# belongs in per-command sections ([build], …), never in [common].
+INJECT_ACTIONS = {
+    "build", "deps", "doctor", "status", "clean", "cleanup", "gc",
+    "import", "publish", "certify", "compliance",
+    "q", "query", "enter", "setenv", "printenv", "load", "unload",
+}
+
+
+def _find_action(argv):
+    """Index of the action token in *argv* (first non-top-flag word), or None."""
+    for i, tok in enumerate(argv):
+        if tok in TOP_FLAGS:
+            continue
+        if tok.startswith("-"):
+            return None          # an option before any action → leave as-is
+        return i
+    return None
+
+
+def rewrite_argv(argv, path=PROFILE):
+    """Return *argv* with the ``.bitscmd`` profile injected right after the action
+    token: ``[common]`` plus ``[<action>]``, for arch-aware actions only
+    (``INJECT_ACTIONS``). A no-op when there is no profile, no action, or the
+    action is a meta command (``use``/``cvmfs``/``store``/…). This is the single
+    entry point the wrapper calls at startup.
+    """
+    argv = list(argv)
+    sec = read_all(path)
+    if not sec:
+        return argv
+    ai = _find_action(argv)
+    if ai is None:
+        return argv
+    action = argv[ai].lower()
+    if action not in INJECT_ACTIONS:
+        return argv
+    inject = sec.get(COMMON, []) + sec.get(action, [])
+    if not inject:
+        return argv
+    return argv[:ai + 1] + inject + argv[ai + 1:]
+
+
+# ── CLI ──────────────────────────────────────────────────────────────────────
+
+def _show(path=PROFILE):
+    sec = read_all(path)
+    if not sec:
+        print("no .bitscmd in %s" % os.getcwd()); return 0
+    for s, toks in sec.items():
+        if toks:
+            print("[%s] %s" % (s, _join(toks)))
+    return 0
+
+
+def main(argv=None):
+    argv = list(sys.argv[1:] if argv is None else argv)
+    # Wrapper hook: emit the profile-injected argv, NUL-separated, for the args
+    # after '--'. `mapfile -t -d '' arr < <(bits_use --rewrite0 -- "$@")`.
+    if argv and argv[0] == "--rewrite0":
+        rest = argv[2:] if len(argv) > 1 and argv[1] == "--" else argv[1:]
+        out = rewrite_argv(rest)
+        if out != rest:                        # injection happened → tell the user
+            n = len(out) - len(rest)
+            sys.stderr.write("using .bitscmd (+%d arg%s)\n" % (n, "" if n == 1 else "s"))
+        # NUL-terminate EVERY token (trailing NUL included) so the wrapper's
+        # `while read -d ''` loop captures the final token — an unterminated last
+        # field is assigned but the loop exits before appending it (dropping it).
+        sys.stdout.write("".join(t + "\0" for t in out))
+        return 0
+    if not argv:
+        return _show()
+    if argv[0] in ("--clear", "clear"):
+        sect = argv[1] if len(argv) > 1 else None
+        ok = clear_section(sect)
+        print(("cleared %s" % (sect or "all")) if ok else "nothing to clear")
+        return 0
+    # Optional leading section name (a bare word); otherwise default to common.
+    if argv[0] and not argv[0].startswith("-"):
+        section, rest = argv[0], argv[1:]
+    else:
+        section, rest = COMMON, argv
+    if not rest:
+        print("no args given for [%s]" % section); return 1
+    write_section(section, rest)
+    print("saved to [%s] in %s: %s" % (
+        COMMON if section in ("global", COMMON) else section.lower(),
+        PROFILE, _join(rest)))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())