diff --git a/.agents/skills/create-module/SKILL.md b/.agents/skills/create-module/SKILL.md new file mode 100644 index 000000000..0874fdd34 --- /dev/null +++ b/.agents/skills/create-module/SKILL.md @@ -0,0 +1,134 @@ +--- +name: create-module +description: Create a new AirStack module repository by hand — author the thin module.yaml manifest, lay out colcon packages with canonical-default launch args (never remaps), add test_stack/ and CI, and validate with tools/validate_module.py. Use when packaging a capability (planner, estimator, sim extension, vehicle data) as a standalone module repo per RFC #379. +license: Apache-2.0 +metadata: + author: AirLab CMU + repository: AirStack +--- + +# Skill: Create an AirStack Module Repo + +## When to Use + +Creating a **new AirStack module repo** — a thin, standalone repository that +distributes an optional capability (a planner, state estimator, world model, Isaac +Sim extension, vehicle definition, …) without forking AirStack. Defined by +[RFC #379](https://github.com/castacks/AirStack/discussions/379) §2 (manifest) and +[RFC #385](https://github.com/castacks/AirStack/discussions/385) §2 (repo anatomy). + +Not this skill: adding a trunk-resident package (use `add-ros2-package`), or wiring +an existing module into a bringup (use `integrate-module-into-layer`). + +> **Status:** `airstack module create` scaffolding arrives in a later phase. Until +> then this skill documents the **by-hand procedure**; only the manifest schema and +> validator (`tools/validate_module.py`) exist today. + +## Module Repo Anatomy (RFC #385 §2) + +A module is a *thin* repo: ordinary colcon packages, a small manifest carrying deps +and identity only, and a `test_stack/` that is both its CI target and its living +install documentation. + +``` +my-module/ +├── module.yaml # the thin manifest — deps, identity, tests; NO wiring +├── my_module/ # ordinary colcon package(s) +│ ├── package.xml # rosdep keys live here (dep tier 1) +│ ├── src/ config/ test/ # co-located unit tests, standard colcon convention +│ └── launch/ +│ └── my_module.launch.xml # topic args DEFAULT to canonical names; never remaps +├── test_stack/ # reference-stack copy with this module wired in +│ ├── modules.repos # PINNED to tags/commits — never branches +│ ├── launch/stack.launch.xml # the ONE place this module is wired +│ ├── docker-compose.yaml +│ ├── wiring.md # generated in CI from the running graph +│ └── README.md +├── Dockerfile.module # optional dep tier 2 — against ARG BASE_IMAGE only +├── .github/workflows/ +│ └── ci.yml # ~10 lines: uses castacks/AirStack/.github/workflows/ +│ # module-system-tests.yml@vX.Y.Z (later phase) +└── README.md +``` + +## The Manifest (`module.yaml`) + +Schema + full field reference: [`common/module_schema/`](../../../common/module_schema/README.md). +Summary: + +- **Required:** `name` (snake_case), `description`, `maintainer` (email), + `license`, `type` (`isaac_extension` | `ros_package` | `data` | `platform`), + `airstack_compat` (semver range vs trunk `.env` `VERSION`, e.g. + `">=0.19.0 <0.21.0"` — never a branch name), `targets` + (non-empty: `robot` | `gcs` | `isaac-sim` | `ms-airsim`). +- **Optional:** `deps` `{apt, pip}`, `dockerfile` (path ending + `Dockerfile.module`), `overlay_image`, `compose`, `assets` + (`[{url (https), sha256, dest}]` — no Git LFS), `docs`, `foxglove`, `hooks` + (`host_setup`: idempotent, no sudo, writes only inside the module checkout), + `tests` (`packages` + `marks` from the known mark set). +- **Deliberately absent: wiring.** No slot, role, or topic metadata — unknown keys + are rejected. A module's interface is its launch file's declared args + (`ros2 launch --show-args`). + +Minimal working example: [`tests/fixtures/modules/hello_module/module.yaml`](../../../tests/fixtures/modules/hello_module/module.yaml). + +## The Canonical-Defaults Launch Rule + +The one interface convention that does the plug-and-play work (RFC #379 §2, §4): + +- Expose **every topic endpoint as a launch arg**, and **default it to the + canonical name** from the interface conventions spec (today: + `docs/robot/autonomy/integration_checklist.md`). +- **NEVER put `` in a module launch file**, and never hardcode a topic in + node code. All cross-module remaps live in the *stack's* entry launch file — + the single-locus wiring rule. + +```xml + + + + + + + + +``` + +(The `` *inside* the node block binds the node's internal name to the +declared arg — that is the mechanism, not a cross-module rewire. What is forbidden +is remapping other modules' topics or overriding canonical names in module launch +files: in a conventional stack, including the module must require **zero** remaps, +so only deviations appear in stack files.) + +## Steps (by hand, until `airstack module create` lands) + +1. Create the repo with the anatomy above; write `module.yaml` first. +2. Write the package(s) following `add-ros2-package` conventions (package.xml + format 3, co-located `test/`). +3. Author the module launch file under the canonical-defaults rule. +4. Validate: + + ```bash + python3 tools/validate_module.py path/to/my-module + ``` + + Exit 0 and `{"valid": true, "errors": []}` on stdout is the gate. Dir mode also + checks that declared `dockerfile`/`compose`/`hooks`/`docs` paths exist and warns + when `tests.packages` entries match no directory. +5. Copy a trunk reference stack into `test_stack/` and wire the module in its + `stack.launch.xml` (reference stacks land in a later phase; until then model it + on the bringup you tested against). +6. Add `ci.yml` calling the reusable `module-system-tests.yml` workflow (later + phase) with pinned `airstack_ref` and the marks for your module category + (RFC #379 §5: global planner → `waypoint_flight`,`autonomy`; state estimator → + `liveliness`,`sensors`,`takeoff_hover_land`; world model/perception → + `liveliness`,`sensors`; sim extension → `liveliness`). + +## References + +- Manifest schema + validator: [`common/module_schema/`](../../../common/module_schema/README.md) +- Fixture module: [`tests/fixtures/modules/hello_module/`](../../../tests/fixtures/modules/hello_module/) +- Contract tests: [`tests/meta/test_module_manifest_contract.py`](../../../tests/meta/test_module_manifest_contract.py) +- RFC #379 (design), RFC #385 (directory atlas) +- Related skills: [add-ros2-package](../add-ros2-package), [write-launch-file](../write-launch-file), [run-system-tests](../run-system-tests) diff --git a/.env b/.env index 836b4ce69..fa8836b2c 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0" +VERSION="0.20.0-alpha.1" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/.github/orchestrator/README.md b/.github/orchestrator/README.md index f2df1ff22..ed1bb39e5 100644 --- a/.github/orchestrator/README.md +++ b/.github/orchestrator/README.md @@ -150,6 +150,31 @@ osmo workflow list --name gha-runner- --pool airstack-ci osmo workflow list --name gha-runner- --pool airstack-ci --status RUNNING PENDING WAITING ``` +## Module repos + +Module repos (`asm_*`) that call trunk's reusable +[`module-system-tests.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/module-system-tests.yml) with the +default `runs-on: [self-hosted, airstack-ephemeral]` queue jobs **in their own +repo**, and the orchestrator polls exactly one `repo:` per instance. To add an +`asm_` repo to the poll list, run a second orchestrator instance against it: + +1. **Extend the PAT.** The fine-grained GitHub PAT must also cover the module + repo with `Actions: read/write` + `Administration: read/write` (JIT runner + registration is per-repo). Reuse the existing PAT file if it covers the + repo, else stage a second one. +2. **Copy the config.** `/etc/airstack-orchestrator/config.yaml` → + `config-asm-.yaml` with `repo: "castacks/asm_"` and a + **distinct `workflow_name_prefix`** (e.g. `gha-runner-asm-`) so the + two instances' orphan sweeps don't cancel each other's OSMO workflows. +3. **Run a second service instance** pointing at the new config and its own + state file (copy `airstack-orchestrator.service`, adjust `ExecStart`'s + `--config` and `--state`, e.g. `--state /var/lib/airstack-orchestrator/state-asm-.json`). + +First-party only: the reusable workflow refuses callers outside the castacks +org, mirroring the fork-PR block. Org-level polling across registered repos +(one instance, many repos) is the RFC #379 Phase 4 replacement for this +per-repo setup. + ## Operational notes - **State file**: `/var/lib/airstack-orchestrator/state.json` is the in-flight job tracker (`job_id → workflow_id`). Wiping it triggers an orphan sweep on the next reap iteration — active `gha-runner-*` workflows will be cancelled. Don't wipe it while jobs are mid-flight unless that's what you want. diff --git a/.github/workflows/module-system-tests.yml b/.github/workflows/module-system-tests.yml new file mode 100644 index 000000000..cc743b19e --- /dev/null +++ b/.github/workflows/module-system-tests.yml @@ -0,0 +1,548 @@ +name: Module System Tests + +# Reusable system-test workflow for AirStack module repos (RFC #379 §5). +# +# A module repo (asm_*) calls this workflow to run trunk's existing system-test +# suite — unchanged — against a pinned AirStack ref with the module added to +# the checkout. The module repo never copies a test. +# +# jobs: +# system-tests: +# uses: castacks/AirStack/.github/workflows/module-system-tests.yml@v0.19.0 +# with: +# airstack_ref: v0.19.0 # pin and checkout ref move together +# marks: "build_packages or liveliness" +# sim: isaacsim +# +# First-party only: the job refuses callers outside the castacks org. External +# modules go through the dispatch-triggered test bench (RFC #379 §5, Phase 4). +# +# Registry secrets are optional. When the caller passes them (castacks org +# secrets provide DOCKER_REGISTRY_* to org repos — the internal registry is +# airlab-docker.andrew.cmu.edu/airstack, NOT ghcr), image prep pulls versioned +# and floating cache tags instead of cold-building. Without them the workflow +# still works; it just image-builds from scratch when pulls miss. +# +# The workflow_dispatch trigger exists so trunk maintainers can smoke-test this +# workflow against any module repo without a caller in place: +# +# gh workflow run module-system-tests.yml --repo castacks/AirStack \ +# --ref \ +# -f module_repo=castacks/asm_dfm2_disturbances -f module_ref=main \ +# -f airstack_ref=develop -f marks="build_packages or liveliness" +# +# GPU runner note: with the default runs_on, the job queues for the ephemeral +# OSMO-backed runners (.github/orchestrator/). The orchestrator polls one repo +# per instance, so a module repo must be added to the poll list before its +# calls can be picked up — see .github/orchestrator/README.md "Module repos". +# +# Docs: docs/development/module_ci.md + +on: + workflow_call: + inputs: + airstack_ref: + description: "Tag/branch/SHA of castacks/AirStack to test the module against. Pin it together with the workflow's own @. Must include the module CLI (airstack module add/sync)." + type: string + required: true + marks: + description: "pytest marks expression (see tests/pytest.ini). build_packages is auto-prepended when absent so code is built before launch tests run." + type: string + default: "build_packages or liveliness" + sim: + description: "Sim targets, comma-separated: isaacsim,msairsim." + type: string + default: "isaacsim" + num_robots: + description: "Robot counts, comma-separated (e.g. 1,3)." + type: string + default: "1" + stress_iterations: + description: "Iterations per (sim, num_robots) config." + type: string + default: "1" + stable_duration: + description: "Seconds for the test_stable polling window." + type: string + default: "120" + module_tests_dir: + description: "Module-relative directory of extra pytest tests, appended to the pytest paths only if it exists in the module checkout." + type: string + default: "tests" + runs_on: + description: "JSON array string parsed into runs-on." + type: string + default: '["self-hosted","airstack-ephemeral"]' + timeout_minutes: + description: "Job timeout in minutes." + type: number + default: 120 + secrets: + DOCKER_REGISTRY_URL: + description: "Internal Docker registry host (castacks org secret; airlab-docker.andrew.cmu.edu). Optional — enables registry-cache image prep." + required: false + DOCKER_REGISTRY_USERNAME: + description: "Registry username (castacks org secret). Optional." + required: false + DOCKER_REGISTRY_PASSWORD: + description: "Registry password (castacks org secret). Optional." + required: false + + # Trunk-maintainer smoke path: same inputs as workflow_call, plus the module + # repo/ref to test (workflow_call derives those from the caller instead). + # timeout_minutes is omitted here (workflow_dispatch caps at 10 inputs); the + # job falls back to 120 when it is unset. + workflow_dispatch: + inputs: + module_repo: + description: "Module repo to test (owner/name, e.g. castacks/asm_dfm2_disturbances)." + type: string + required: true + module_ref: + description: "Module ref (tag/branch/SHA)." + type: string + default: "main" + airstack_ref: + description: "Tag/branch/SHA of castacks/AirStack to test the module against." + type: string + default: "develop" + marks: + description: "pytest marks expression. build_packages is auto-prepended when absent." + type: string + default: "build_packages or liveliness" + sim: + description: "Sim targets, comma-separated: isaacsim,msairsim." + type: string + default: "isaacsim" + num_robots: + description: "Robot counts, comma-separated (e.g. 1,3)." + type: string + default: "1" + stress_iterations: + description: "Iterations per (sim, num_robots) config." + type: string + default: "1" + stable_duration: + description: "Seconds for the test_stable polling window." + type: string + default: "120" + module_tests_dir: + description: "Module-relative extra pytest dir (appended only if it exists)." + type: string + default: "tests" + runs_on: + description: "JSON array string parsed into runs-on." + type: string + default: '["self-hosted","airstack-ephemeral"]' + +permissions: + contents: read + +jobs: + module-tests: + name: Module System Tests + runs-on: ${{ fromJSON(inputs.runs_on) }} + timeout-minutes: ${{ inputs.timeout_minutes || 120 }} + # Mirror the registry secrets into env so step-level `if:` expressions can + # check whether registry-cache mode is available — `secrets.*` itself is + # not addressable from `if:` expressions. + env: + DOCKER_REGISTRY_URL: ${{ secrets.DOCKER_REGISTRY_URL }} + DOCKER_REGISTRY_PASSWORD: ${{ secrets.DOCKER_REGISTRY_PASSWORD }} + steps: + # First-party policy (RFC #379 §5): org GPU runners and registry secrets + # never serve repos outside castacks. In a reusable workflow the github + # context is the CALLER's, so repository_owner is the module repo's org. + # External modules use the dispatch-triggered test bench when it lands + # (Phase 4); until then this is a hard refusal, not a degraded mode. + - name: Enforce first-party module policy + if: github.repository_owner != 'castacks' + run: | + echo "::error::module-system-tests.yml is first-party only: the calling repository must live in the castacks org (caller is '${{ github.repository }}'). External modules will be served by the dispatch-triggered test bench (RFC #379 §5); it does not exist yet." + exit 1 + + - name: Checkout AirStack (trunk under test) + uses: actions/checkout@v4 + with: + repository: castacks/AirStack + ref: ${{ inputs.airstack_ref }} + submodules: recursive + + # workflow_call: the module is the calling repo at the triggering SHA. + # workflow_dispatch: the module is named explicitly by inputs. + - name: Resolve module source + id: module_src + env: + EVENT_NAME: ${{ github.event_name }} + DISPATCH_REPO: ${{ inputs.module_repo }} + DISPATCH_REF: ${{ inputs.module_ref }} + CALLER_REPO: ${{ github.repository }} + CALLER_SHA: ${{ github.sha }} + run: | + if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then + repo="$DISPATCH_REPO" + ref="${DISPATCH_REF:-main}" + else + repo="$CALLER_REPO" + ref="$CALLER_SHA" + fi + if [[ -z "$repo" ]]; then + echo "::error::No module repository resolved (module_repo input is required for workflow_dispatch)." + exit 1 + fi + echo "repo=$repo" >> "$GITHUB_OUTPUT" + echo "ref=$ref" >> "$GITHUB_OUTPUT" + echo "Module under test: $repo @ $ref" + + - name: Checkout module under test + uses: actions/checkout@v4 + with: + repository: ${{ steps.module_src.outputs.repo }} + ref: ${{ steps.module_src.outputs.ref }} + path: module-under-test + submodules: recursive + + - name: Install test dependencies + # Ubuntu 24.04 marks the system Python as externally-managed (PEP 668), + # so `pip install` outside a venv is rejected. Use a venv and prepend + # its bin/ to $GITHUB_PATH so subsequent steps pick up `pytest` + # automatically. + run: | + sudo apt-get update -qq + sudo apt-get install -y --no-install-recommends python3-venv + python3 -m venv .venv + echo "$GITHUB_WORKSPACE/.venv/bin" >> "$GITHUB_PATH" + .venv/bin/pip install --upgrade pip + .venv/bin/pip install -r tests/requirements.txt + + - name: Validate module manifest + run: python3 tools/validate_module.py module-under-test + + # Artifact/summary identity: the manifest name, falling back to the repo + # basename. Sanitized because artifact names reject several characters. + - name: Resolve module name + id: module + env: + MODULE_REPO: ${{ steps.module_src.outputs.repo }} + run: | + python3 - <<'PYEOF' + import os, re + name = "" + try: + import yaml + with open("module-under-test/module.yaml") as f: + name = str(yaml.safe_load(f).get("name") or "") + except Exception: + pass + if not name: + name = os.environ["MODULE_REPO"].rsplit("/", 1)[-1] + name = re.sub(r'[^A-Za-z0-9._-]+', "-", name) + with open(os.environ["GITHUB_OUTPUT"], "a") as f: + f.write(f"name={name}\n") + print(f"Module name: {name}") + PYEOF + + - name: Ensure airstack.sh is executable + run: chmod +x airstack.sh + + # Register the module with the trunk checkout via the module CLI + # (contract: `airstack module add ` places the module and + # runs its declared hooks.host_setup; `airstack module sync` materializes + # the overlay — ROS packages into robot/ros_ws/src/modules/, Isaac + # extensions onto the extension path, compose fragments into + # COMPOSE_FILE). `--no-hooks` is attempted first so a hook failure can't + # masquerade as an add failure; the fallback covers a CLI that doesn't + # know the flag. hooks.host_setup is then run explicitly below — the + # manifest schema requires hooks to be idempotent, so running it after an + # add that already ran it is safe. + - name: Add module to the AirStack checkout + run: | + ./airstack.sh module add ./module-under-test --no-hooks \ + || ./airstack.sh module add ./module-under-test + ./airstack.sh module sync + + - name: Run module host_setup hook (if declared) + run: | + hook="$(python3 -c " + import yaml + with open('module-under-test/module.yaml') as f: + m = yaml.safe_load(f) or {} + print((m.get('hooks') or {}).get('host_setup') or '') + ")" + if [[ -n "$hook" ]]; then + echo "Running hooks.host_setup: $hook (idempotent by manifest contract)" + (cd module-under-test && bash "$hook") + else + echo "No hooks.host_setup declared." + fi + + - name: Create Isaac Sim omni_pass.env + if: contains(inputs.sim, 'isaacsim') + run: | + mkdir -p simulation/isaac-sim/docker + cat > simulation/isaac-sim/docker/omni_pass.env <<'EOF' + OMNI_USER=guest + OMNI_PASS=guest + OMNI_SERVER="omniverse://airlab-nucleus.andrew.cmu.edu/NVIDIA/Assets/Isaac/5.1" + ACCEPT_EULA=Y + OMNI_ENV_PRIVACY_CONSENT=Y + EOF + + # Compose the pytest argv from the explicit inputs. Modeled on trunk's + # system-tests.yml parse block, minus the issue_comment parsing (inputs + # arrive structured here). Rules kept from trunk: + # - build_packages is auto-prepended when marks lack it, so code is + # built before launch tests run; + # - marks that never launch a sim (pure build_packages) switch to + # pull-only image prep so we don't bake isaac-sim for a colcon test; + # - build_docker marks skip image prep entirely (those tests rebuild). + # Added here: the module's own pytest dir is appended when it exists, + # and -c tests/pytest.ini pins the config when a second path would move + # pytest's rootdir/inifile discovery off tests/. + - name: Compose pytest args + id: parse + env: + INPUT_MARKS: ${{ inputs.marks }} + INPUT_SIM: ${{ inputs.sim }} + INPUT_NUM_ROBOTS: ${{ inputs.num_robots }} + INPUT_ITERATIONS: ${{ inputs.stress_iterations }} + INPUT_STABLE: ${{ inputs.stable_duration }} + INPUT_MODULE_TESTS_DIR: ${{ inputs.module_tests_dir }} + run: | + python3 <<'PYEOF' + import os, shlex + + marks = os.environ.get('INPUT_MARKS', '').strip() or 'build_packages or liveliness' + sim = os.environ.get('INPUT_SIM', '').strip() or 'isaacsim' + + # When the caller specified marks without build_packages, prepend it + # so code is built before launch tests try to use it (same rule as + # trunk system-tests.yml). + if 'build_packages' not in marks: + marks = f'build_packages or {marks}' + + args = ['-m', marks, '--sim', sim] + if (n := os.environ.get('INPUT_NUM_ROBOTS', '').strip()): + args.extend(['--num-robots', n]) + if (it := os.environ.get('INPUT_ITERATIONS', '').strip()): + args.extend(['--stress-iterations', it]) + if (st := os.environ.get('INPUT_STABLE', '').strip()): + args.extend(['--stable-duration', st]) + + # Marks that bring up sim/robot containers need compose images; a + # pure build_packages selection does not, so pull-only there (trunk's + # `only_packages` rule). waypoint_flight is included in the heavy set + # for correctness even though trunk's list predates the mark. + heavy = any(m in marks for m in ( + 'liveliness', 'sensors', 'takeoff_hover_land', 'autonomy', + 'waypoint_flight', 'build_docker', 'optitrack', + )) + no_image_build = not heavy + skip_prep = 'build_docker' in marks + + # Append the module's own pytest dir when it exists. Adding a second + # collection path moves pytest's rootdir to the common ancestor, so + # pin the config file explicitly in that case. + module_tests = '' + tests_dir = os.environ.get('INPUT_MODULE_TESTS_DIR', '').strip() + if tests_dir: + candidate = os.path.join('module-under-test', tests_dir) + if os.path.isdir(candidate): + module_tests = candidate + args.extend(['-c', 'tests/pytest.ini']) + + quoted = ' '.join(shlex.quote(a) for a in args) + with open(os.environ['GITHUB_OUTPUT'], 'a') as f: + f.write(f'pytest_args={quoted}\n') + f.write(f'resolved_marks={marks}\n') + f.write(f'sim={sim}\n') + f.write(f'skip_image_prep={"true" if skip_prep else "false"}\n') + f.write(f'no_image_build={"true" if no_image_build else "false"}\n') + f.write(f'module_tests_dir={module_tests}\n') + + print(f'Resolved pytest args: {quoted}') + print(f'Module tests dir: {module_tests or "(none)"}') + print(f'Skip image prep: {skip_prep}') + print(f'No image build (pull/retag only): {no_image_build}') + PYEOF + + # Optional registry-cache mode. When the caller passes the registry + # secrets we log in to the internal Docker registry; the next step then + # sets AIRSTACK_REGISTRY_CACHE=1 so airstack.sh pre-pulls + uses BuildKit + # inline cache. When secrets are absent both steps are skipped and image + # prep falls back to cold builds. + # + # Read-only on purpose: AIRSTACK_REGISTRY_CACHE_PUSH stays unset here so + # a module run can consume the floating cache tag but never republish + # it. Only trunk's docker-build.yml (main/develop) writes it. + - name: Log in to internal Docker registry + id: docker_login + if: ${{ env.DOCKER_REGISTRY_URL != '' && env.DOCKER_REGISTRY_PASSWORD != '' }} + uses: docker/login-action@v3 + with: + registry: ${{ secrets.DOCKER_REGISTRY_URL }} + username: ${{ secrets.DOCKER_REGISTRY_USERNAME }} + password: ${{ secrets.DOCKER_REGISTRY_PASSWORD }} + + - name: Enable registry-cache mode + if: ${{ steps.docker_login.outcome == 'success' }} + run: echo "AIRSTACK_REGISTRY_CACHE=1" >> "$GITHUB_ENV" + + - name: Disable compose image builds + if: ${{ steps.parse.outputs.no_image_build == 'true' }} + run: echo "AIRSTACK_NO_IMAGE_BUILD=1" >> "$GITHUB_ENV" + + # The ephemeral runner starts with no local images. `airstack_env` in + # tests/conftest.py fails fast if compose images are missing, so prep + # them here. Profile-gated services (ms-airsim, isaac-sim) are skipped + # by compose unless their profile is active, so we mirror the fixture's + # profile selection from the sim input. Pull versioned tags, then retag + # floating cache_* tags onto the VERSION name (unreleased trunk refs + # have no versioned tag). Fall back to image-build only when pull-only + # mode is off. Skipped when marks contain build_docker — those tests + # build themselves. + - name: Ensure Docker images present + if: ${{ steps.parse.outputs.skip_image_prep != 'true' }} + env: + AIRSTACK_ROOT: ${{ github.workspace }} + SIM_INPUT: ${{ steps.parse.outputs.sim }} + NO_IMAGE_BUILD: ${{ steps.parse.outputs.no_image_build }} + run: | + profiles=desktop + [[ ",$SIM_INPUT," == *,msairsim,* ]] && profiles="$profiles,ms-airsim" + [[ ",$SIM_INPUT," == *,isaacsim,* ]] && profiles="$profiles,isaac-sim" + export COMPOSE_PROFILES="$profiles" + echo "Pulling images for COMPOSE_PROFILES=$COMPOSE_PROFILES (no_image_build=$NO_IMAGE_BUILD)" + + # Pull from registry; tolerate per-image failures so we can detect + # what's still missing afterwards instead of aborting on the first + # gap. `--progress=quiet` suppresses per-layer progress; errors + # still surface on stderr. + ./airstack.sh --progress=quiet image-pull --ignore-pull-failures || true + + # Versioned tags can miss (unreleased trunk refs). Seed from floating + # cache_* tags. + cache_tag="$(grep -E '^CACHE_TAG=' .env 2>/dev/null | cut -d= -f2 | tr -d '"' || true)" + cache_tag="${cache_tag:-cache}" + while IFS= read -r img; do + [[ -z "$img" ]] && continue + if docker image inspect "$img" --format '{{.Id}}' >/dev/null 2>&1; then + continue + fi + # Replace :v_ with :_ (unpublished versioned tags) + cache_img="$(python3 -c "import re,sys; print(re.sub(r':v[^_]+_', f':{sys.argv[2]}_', sys.argv[1], count=1))" "$img" "$cache_tag")" + echo "Versioned tag missing; trying cache tag $cache_img" + if docker pull --quiet "$cache_img"; then + docker tag "$cache_img" "$img" + echo "Retagged $cache_img -> $img" + else + echo "Cache tag pull failed for $cache_img" + fi + done < <(docker compose -f docker-compose.yaml config --images) + + missing=() + while IFS= read -r img; do + [[ -z "$img" ]] && continue + if ! docker image inspect "$img" --format '{{.Id}}' >/dev/null 2>&1; then + missing+=("$img") + fi + done < <(docker compose -f docker-compose.yaml config --images) + + if (( ${#missing[@]} > 0 )); then + echo "Images still missing after pull/retag:" + printf ' - %s\n' "${missing[@]}" + if [[ "$NO_IMAGE_BUILD" == "true" ]]; then + echo "::error::Pull-only mode (pure build_packages marks) will not image-build. Include a mark that launches the sim, or run trunk's /pytest -m build_docker once to publish cache tags." + exit 1 + fi + echo "Falling back to image-build" + ./airstack.sh --progress=quiet image-build + else + echo "All required images present after pull/retag — skipping build." + fi + + - name: Run tests + env: + AIRSTACK_ROOT: ${{ github.workspace }} + DISPLAY: "" + PYTEST_ARGS: ${{ steps.parse.outputs.pytest_args }} + MODULE_TESTS_DIR: ${{ steps.parse.outputs.module_tests_dir }} + run: | + # Re-split the shell-quoted args from the parse step so we forward + # them to pytest as a proper argv list (preserving values like + # `-m 'a or b'`). sys.stdout.write is intentional: print('') emits + # one blank line, which mapfile turns into an empty positional path + # and makes pytest recurse from the repository root. + mapfile -t ARGS < <(python3 -c "import os, shlex, sys; sys.stdout.write(''.join(f'{arg}\\n' for arg in shlex.split(os.environ['PYTEST_ARGS'])))") + for arg in "${ARGS[@]}"; do + if [[ -z "$arg" ]]; then + echo "::error::Refusing an empty pytest argument because it expands collection to the repository root." + exit 2 + fi + done + PATHS=(tests/) + if [[ -n "$MODULE_TESTS_DIR" ]]; then + PATHS+=("$MODULE_TESTS_DIR") + fi + set +e + pytest "${PATHS[@]}" \ + "${ARGS[@]}" \ + -v -s \ + --log-cli-level=INFO \ + --log-cli-format='%(asctime)s [%(levelname)s] %(name)s: %(message)s' \ + --log-cli-date-format='%H:%M:%S' + pytest_status=$? + set -e + if (( pytest_status != 0 )); then + exit "$pytest_status" + fi + + # A successful collect-only/non-executed campaign is not a passing + # system test. Make that distinction visible in the job conclusion. + python3 <<'PYEOF' + import json + from pathlib import Path + + candidates = list(Path("tests/results").glob("*/run_meta.json")) + if not candidates: + raise SystemExit("::error::pytest succeeded without run_meta.json") + latest = max(candidates, key=lambda path: path.stat().st_mtime) + outcome = json.loads(latest.read_text()).get("outcome") + if outcome not in {"simulation", "non_simulation"}: + raise SystemExit( + f"::error::pytest did not execute a complete campaign ({outcome})" + ) + PYEOF + + - name: Upload test results + uses: actions/upload-artifact@v4 + if: always() + with: + name: module-test-results-${{ steps.module.outputs.name }}-${{ github.run_id }} + path: tests/results/ + retention-days: 90 + + - name: Write job summary + if: always() + env: + MODULE_NAME: ${{ steps.module.outputs.name }} + MODULE_REPO: ${{ steps.module_src.outputs.repo }} + MODULE_REF: ${{ steps.module_src.outputs.ref }} + AIRSTACK_REF: ${{ inputs.airstack_ref }} + RESOLVED_MARKS: ${{ steps.parse.outputs.resolved_marks }} + SIM: ${{ steps.parse.outputs.sim }} + NUM_ROBOTS: ${{ inputs.num_robots }} + MODULE_TESTS_DIR: ${{ steps.parse.outputs.module_tests_dir }} + run: | + { + echo "## Module System Tests" + echo "" + echo "| Field | Value |" + echo "|---|---|" + echo "| Module | \`$MODULE_NAME\` (\`$MODULE_REPO\` @ \`$MODULE_REF\`) |" + echo "| AirStack ref | \`${AIRSTACK_REF:-develop}\` |" + echo "| Marks | \`${RESOLVED_MARKS:-—}\` |" + echo "| Sim | \`${SIM:-—}\` |" + echo "| Robots | \`${NUM_ROBOTS:-1}\` |" + echo "| Module pytest dir | \`${MODULE_TESTS_DIR:-—}\` |" + } >> "$GITHUB_STEP_SUMMARY" diff --git a/airstack.sh b/airstack.sh index eba64b112..fae7a6178 100755 --- a/airstack.sh +++ b/airstack.sh @@ -1051,7 +1051,8 @@ function print_launch_config { # Dump for reproducibility (precursor of RFC #380's effective_config.yaml). # Best-effort: a read-only checkout (e.g. the tests container) skips it. - local run_dir="$PROJECT_ROOT/.airstack/runs/$(date +%Y-%m-%d_%H-%M-%S)" + # PID suffix keeps back-to-back runs within the same second in distinct dirs. + local run_dir="$PROJECT_ROOT/.airstack/runs/$(date +%Y-%m-%d_%H-%M-%S)_$$" if mkdir -p "$run_dir" 2>/dev/null && printf '%s\n' "${lines[@]}" > "$run_dir/effective_config.env" 2>/dev/null; then log_info " effective config saved to ${run_dir#$PROJECT_ROOT/}/effective_config.env" fi diff --git a/common/module_schema/README.md b/common/module_schema/README.md new file mode 100644 index 000000000..533829f8b --- /dev/null +++ b/common/module_schema/README.md @@ -0,0 +1,96 @@ +# AirStack Module Manifest Schema + +The schema for `module.yaml` — the **thin module manifest** from +[RFC #379 §2](https://github.com/castacks/AirStack/discussions/379): dependencies, +identity, and test metadata only. **Wiring deliberately does not live here.** A +module's interface is its launch file's declared args (with canonical-name defaults) +plus the interface conventions spec; its wiring lives in whatever stack includes it. + +- Schema: [`module.schema.json`](module.schema.json) +- Validator: [`tools/validate_module.py`](../../tools/validate_module.py) +- Authoring workflow: [`.agents/skills/create-module`](../../.agents/skills/create-module/SKILL.md) + +## Validate a module + +```bash +# Module directory (schema + cross-file checks): +python3 tools/validate_module.py path/to/module_repo + +# Bare manifest (schema only): +python3 tools/validate_module.py path/to/module.yaml +``` + +Human-readable errors go to stderr; a JSON verdict goes to stdout for scripts: + +```json +{"valid": false, "errors": [{"path": "maintainer", "message": "required property is missing"}]} +``` + +Exit code 0 when valid, 1 otherwise. Contract tests: +[`tests/meta/test_module_manifest_contract.py`](../../tests/meta/test_module_manifest_contract.py). + +## Fields + +### Required + +| Field | Shape | Notes | +|---|---|---| +| `name` | `^[a-z][a-z0-9_]*$` | module identity, snake_case | +| `description` | string, ≥ 8 chars | one-liner for the registry catalog | +| `maintainer` | email | required by governance (RFC #379 §8) | +| `license` | non-empty string | checked at registration | +| `type` | `isaac_extension` \| `ros_package` \| `data` \| `platform` | `platform` is future (RFC #380 §6) | +| `airstack_compat` | semver range | vs trunk `.env` `VERSION`, e.g. `">=0.19.0 <0.21.0"`, `">=0.19.0-alpha.18 <0.20.0"`. Branch names (`main`) and partial versions (`0.19`) are invalid. | +| `targets` | non-empty array of `robot` \| `gcs` \| `isaac-sim` \| `ms-airsim` | host containers touched | + +### Optional (with defaults) + +| Field | Default | Notes | +|---|---|---| +| `deps` | `{apt: [], pip: []}` | dependency tier 1 (RFC #379 §6); rosdep keys stay in `package.xml` | +| `dockerfile` | `null` | tier 2: repo-relative path ending `Dockerfile.module`, written against `ARG BASE_IMAGE` | +| `overlay_image` | `null` | tier 3: prebuilt overlay image reference | +| `compose` | `null` | repo-relative compose fragment (mounts, env) | +| `assets` | `[]` | `[{url, sha256, dest}]`: `url` must be `https://`, `sha256` is 64 lowercase hex chars, `dest` is a relative path (no `..`) | +| `docs` | `{readme: README.md}` | or `{dir: , nav: }` | +| `foxglove` | `null` | layout/panel fragment so operators can see the module | +| `hooks` | — | `{host_setup: