Reuse deployed CVMFS releases - #110
Merged
Merged
Conversation
Introduce --reuse-from <abs-path>|cvmfs, the module-based reuse source, kept separate from --remote-store (the tarball store). 'cvmfs' resolves module_path from the defaults system: layout, failing if unset. Pure resolver resolve_reuse_from; nothing consumes the value yet.
Add rewrite_module_anchor: replace a deployed modulefile's shared $::env(BASEDIR) with an absolute install base so a reused copy is self-anchoring (the first BASE on MODULEPATH no longer pins every package's location). write_overlay now writes a pre-rendered modulefile when an entry carries one; foreign entries still regenerate from ops. Mechanism for trusted harvest; no producer wired yet.
bits import --trusted reads a deployed release's own modulefiles (--modulepath) and re-anchors each to --install-base, strips the now unneeded BASE dep, and captures package hash + build_id from the install tree's .meta.json. Deterministic (no modulecmd); feeds the same overlay writer. Produces the self-anchoring modules a strict reuse consumes.
generate_initdotsh gains a per-dependency branch: a dep marked reuse_module_id (satisfied from a reused CVMFS release) is set up via 'module use <overlay>' + 'module load <id>' instead of sourcing its init.sh, while locally-built deps are unchanged. Per-dependency, so a legacy package can consume a module-reused dep. Dormant until the marker and reuse_modulepath are wired (next step).
The import subcommand is dispatched in bitsBuild but was missing from the wrapper's command allowlist, so 'bits import' failed with 'Unknown command'. Add it to the route list.
The deployed BASE dependency is a multi-line 'if ![ is-loaded BASE ] {
module load BASE }' block; the old per-line strip dropped the first two
lines but left the orphan '}', so the reused modulefile failed to load
(invalid command name }). Make strip_base_dep brace-aware: consume
through the matching '}'. Found by the testbed module load.
When --reuse-from is set, doBuild runs import_trusted_release to build the local re-anchored overlay (install base from the layout) and threads its path into generate_initdotsh. import_trusted_release now returns the overlay path. Dormant: nothing is marked reused yet, so builds are unchanged; a --reuse-from build just logs the imported overlay.
After each package's hash is finalized, if the reuse overlay satisfies it (strict: same remote hash, publishable; relaxed: any version in the one-release overlay) set reuse_module_id so consumers 'module load' it instead of sourcing its init.sh. defaults-*/--build-local never grafted. The package still builds here; skipping its build is the next step.
With --reuse-from cvmfs the Packages root comes from the layout; with an explicit .../Modules/modulefiles path and no layout, derive it by the deployment convention (Modules/modulefiles<->Packages, the map BASE uses) so the explicit form works without configuring the layout.
A package satisfied by the reuse overlay is now set up from modules only: it adopts a consistent identity, records already_installed, and continues before fetch_symlinks — so the legacy CVMFS tarball synthesis + relocate path is skipped entirely (that was the relocate-me.sh failure). Devel packages excluded. Initialise mainBuildFamily before the loop so reusing the target package doesn't leave the final banner unset.
args.buildLocal is normalized to a list (args.py:1965), but the 4b reuse guard treated it as a comma string and called .split() -> AttributeError when --build-local was passed. Accept both list and string forms.
generate_initdotsh was given the host overlay abspath, so 'module use' inside the build container pointed at a nonexistent /home/... path. Pass the overlay under init_workDir (the container workdir under --docker), so it resolves at $WORK_DIR/MODULES/<build_id>/<arch> in the container.
Replace module use/load (which needs modulecmd in the image) with the standard mechanism: source the dep's deployed init.sh from CVMFS. Point its $WORK_DIR/$BITS_ARCH_PREFIX at the Packages base while sourcing; BITS_ARCH_PREFIX="." (non-null) survives the deployed init.sh's `:= <arch>` default that "" would not. Mount /cvmfs into the build container when reuse is active. No image change, no modulecmd.
A reused dep's deployed init.sh transitively re-sources its own deps (guarded on _REVISION) from CVMFS. A local-built prerequisite must be set up first so that guard skips the CVMFS re-source, which misses a local-only build (e.g. bits-recipe-tools before a reused CMake).
The relaxed build_id graft path warns 'packages will be built / no packages will be grafted' when it has no cvmfs:// store or --reuse-base. Under --reuse-from overlay reuse those warnings are misleading — the overlay grafts independently. Gate them on args.reuseOverlay. Non-reuse-from output unchanged.
resolve_tag ran only under `if "source"` (git), so a tarball-only recipe with `tag: "v%(version)s"` kept the raw tag; it leaked into commit_hash and the SOURCES/<pkg>/<version>/<tag> path, breaking the source copy. Resolve it on the sources path too — no-op for literal tags, and matches the git path.
A deployed .pc can bake a prefix= that doesn't match its CVMFS location (publish-time relocation), so pkg-config resolves a missing libdir and a consumer's find_package fails (xrootd -> Davix). Stage corrected .pc copies under $WORK_DIR with prefix set to the reused dep's real root, prepend to PKG_CONFIG_PATH.
The revision scan reports a TARS symlink whose store tarball is gone (leftover from a failed build or a cleanup) then skips it and rebuilds — self-healing, not actionable. It fired on every build as WARNING noise; make it debug. The unparseable-symlink case stays a warning.
A trailing ::relaxed/::strict on --reuse-from sets the reuse policy alongside the source (e.g. cvmfs::relaxed). --reuse-policy stays canonical: it wins, and if both are given they must agree (error otherwise). Absolute paths never contain ::, so the split is unambiguous.
When the defaults declare no module layout (module_dir/cvmfs_dir), --reuse-from cvmfs now derives the modulefiles base from the group's cvmfs_modules_template (one declaration drives publish and reuse). Expanded with the deployed (raw) arch so it matches where packages live, not the build-qualified family.
Retire the ADR-0001 graft path now that --reuse-from module overlay is the proven mechanism. Drops the build_id auto-select, the graftable_match performCvmfsMatch callback, the reuseBase default, and the reuseCvmfs -> remote-store=cvmfs:// wiring. Overlay reuse and the relaxed publish guard are untouched. getPackageList's now-unused callback param goes next.
Drop the performCvmfsMatch graft branch from getPackageList, the from_cvmfs hash-adoption in storeHashes, and the from_cvmfs closure-loose provenance check (nothing sets from_cvmfs after 5/1). Legacy graft tests removed (test_reuse_resolver stubbed pending git rm); provenance loose is now driven solely by untracked_requires.
…(5/3) The cvmfs:// remote-store reader (synthesized-tarball reuse) is retired. remote_from_url now errors on a cvmfs:// --remote-store, pointing at --reuse-from; the http+write dual path and all other backends are unchanged. Legacy CVMFSRemoteSync tests removed (fetch_symlinks stub pending git rm).
The build_id-graft flags and the select_build_id/graftable_match helpers they drove have no consumers after 5/1-5/3. Remove the flags, empty cvmfs_reuse.py (stub pending git rm), retarget the bits-import help at --reuse-from, and prune the legacy arg tests.
bits use [section] <args> records reusable CLI args (INI-style: [common] plus per-command sections) so repeated commands stay short. The wrapper injects [common]+[<action>] right after the action token before parsing, so build and the module commands (q/enter) both see it and explicit args still win. New bits_use.py (rewrite_argv/--rewrite0) + tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes it possible to re-use bits packages deployed on CVMFS in subsequent builds.