From cce3977cae9701f73d2eb3f8e0fd76ce6d1a2045 Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" <41898282+github-actions[bot]@users.noreply.github.com> Date: Wed, 20 May 2026 22:27:10 +0000 Subject: [PATCH 01/21] Bump VERSION to after sync from main --- .env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.env b/.env index 079796374..d16b8c649 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.18.0" +VERSION="0.19.0-alpha.0" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. From 25d52da5759eb731a34f90e9b5e731bb80926418 Mon Sep 17 00:00:00 2001 From: Sebastian Scherer Date: Fri, 22 May 2026 19:54:24 -0400 Subject: [PATCH 02/21] feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO (#352) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO Adds a privileged Docker-in-Docker workspace task that lets a developer run the full AirStack docker-compose stack on OSMO and attach an IDE over SSH, with Isaac Sim WebRTC livestream + Foxglove websocket exposed via osmo port-forward. Components: - osmo/workspace/{Dockerfile,entrypoint.sh,sshd_config}: airstack-osmo-workspace image. Ubuntu 24.04 + sshd (pubkey-only) + Docker CE + Docker Compose + nvidia-container-toolkit + fuse-overlayfs (DinD-on-overlayfs needs it, otherwise dockerd falls back to vfs which bloats AirStack images ~10x). - osmo/workflows/airstack-dev.yaml: single privileged GPU task. Materializes Nucleus + airlab-docker secrets from OSMO credentials, clones AirStack, starts inner dockerd, runs `airstack up` with desktop + isaac-sim-livestream Compose profiles. - simulation/isaac-sim: isaac-sim-livestream Compose service that runs Pegasus standalone with --/app/livestream/enabled=true and exposes WebRTC port ranges 47995-48012 / 49000-49007 / 49100; launch script gates headless+livestream extension on ISAAC_SIM_LIVESTREAM env var. - .airstack/modules/osmo.sh: airstack osmo:{up,ide,foxglove,webrtc,logs,down} CLI wrappers around `osmo workflow submit` / `port-forward` / `cancel`. Persists the active workflow id and validates it's still running before each command (prevents the stale-state 410 error). - airstack.sh: bash 4+ re-exec bootstrap (macOS ships 3.2; the CLI uses `declare -A`). - osmo/README.md + docs/tutorials/airstack_on_osmo.md: admin pool setup (privileged_allowed) + per-user credentials (airlab-docker-login, airlab-nucleus) + student-facing IDE attach + WebRTC/Foxglove flow. Pool requirements: privileged_allowed: true, GPU pool with nvidia-container-toolkit on the host, ample node ephemeral storage (AirStack images extracted are ~50-100Gi via fuse-overlayfs; vfs needs ~500Gi+). Co-authored-by: Cursor * fix(osmo): harden CLI + workspace image against stale-state, port-forward race, and cursor-server install hangs Four bugs that bit the first end-to-end runs (airstack-dev-10 → -13): - _osmo_wf_id: validate saved workflow id against `osmo workflow query` before returning. Without this, the state file at ~/.airstack/osmo-state outlives the workflow it points at and every subsequent osmo:webrtc / osmo:foxglove / osmo:ide call surfaces the same confusing "Workflow airstack-dev-N is not running! (status 410)" instead of the obvious "run airstack osmo:up to launch a fresh workflow". - cmd_osmo_up: `osmo workflow submit --set-env` is variadic. Passing two separate `--set-env A=1 --set-env B=2` silently drops the first one — this is what made airstack-dev-11 fail with "ERROR: SSH_PUB_KEY not set" when --branch was passed alongside the pubkey. Collapse the K=V pairs into a single --set-env. - cmd_osmo_ide: previously launched the IDE before starting the port-forward, so Cursor/VS Code would try to SSH localhost:2200 a few hundred ms before the tunnel listener existed and fail with "connect to host localhost port 2200: Connection refused". Now: detect an existing forward and reuse it (also avoids the "Address already in use" if osmo:foxglove was started in parallel), otherwise spawn the forward in the background, wait up to 30s for it to bind, then launch the IDE. Ctrl+C tears down the spawned forward cleanly via a trap. - workspace image / entrypoint: Cursor Remote-SSH hung indefinitely on airstack-dev-13 because (a) cursor-server's installer fell back to wget when curl timed out and wget was not in the image, and (b) a /tmp/cursor-remote-lock.* file left behind by the first crashed install blocked every silent retry. Add wget to the apt install list and rm -f the stale Cursor / VS Code remote lock files at the very top of entrypoint.sh so each fresh pod starts from a clean slate. Co-authored-by: Cursor * fix(osmo): correct osmo:logs CLI invocation; install Foxglove extensions locally on osmo:foxglove osmo:logs was invoking `osmo workflow logs workspace --follow`, but the real CLI takes the task via `-t TASK` (not positionally) and has no `--follow` flag at all — so the command failed immediately with "unrecognized arguments: workspace --follow". Replace with a polling loop that uses `-t workspace -n ` on a short interval, prints only the suffix that appeared since the previous fetch (find-the-last-seen-line trick; degrades to "reprint tail" with a warning if the cursor outruns -n), and exits cleanly once the workflow reaches a terminal state. Tunables: OSMO_LOGS_TASK / OSMO_LOGS_TAIL / OSMO_LOGS_INTERVAL. osmo:foxglove now installs the AirStack Foxglove extensions (robot-commands / waypoint-editor / polygon-editor) into the laptop's local Foxglove user-extensions directory before opening the port-forward. Without this, custom panels show up as "Unknown panel type: robot-commands.Robot Tasks" in the laptop's Foxglove Desktop because it has no way to discover the extension folders that live inside the GCS container. To avoid duplicating the install logic, the existing gcs/foxglove_extensions/install.py is refactored to read FOXGLOVE_EXT_SRC / FOXGLOVE_EXT_DST env vars (the in-container call already in gcs/docker/gcs-base-docker-compose.yaml keeps working unchanged via defaults). The wrapper sets those vars to ${PROJECT_ROOT}/gcs/foxglove_extensions and ~/.foxglove-studio/extensions respectively, overridable with OSMO_FOXGLOVE_EXT_DIR / skippable with OSMO_FOXGLOVE_SKIP_EXTENSIONS=1. Co-authored-by: Cursor * fix(osmo): pin Kit livestream UDP media port to 49099 so osmo:webrtc actually shows pixels Kit 107's WebRTC livestream picks a UDP media port dynamically. The documented `omni.services.livestream.nvcf` defaults (minHostPort=47998 maxHostPort=48020 fixedHostPort=0) are ignored by the stock standalone Kit binary — on airstack-dev-13 it bound to UDP 49042, outside both the Compose-published range AND the default `osmo:webrtc --udp` forward of `47995-48012,49000-49007`. Result: TCP signaling on 49100 worked, the WebRTC Streaming Client window opened, but every SRTP media packet was dropped → black viewport plus the recurring `NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1` underflow in Kit's log. Pin the media port via three `app.livestream.*` settings set on `SimulationApp` before `omni.kit.livestream.webrtc` is enabled, so whichever code path the carb.livestream-rtc.plugin consults lands on the same port: app.livestream.fixedHostPort = 49099 app.livestream.minHostPort = 49099 app.livestream.maxHostPort = 49099 49099 is a deliberate one-off from the 49100 TCP signaling port — same neighborhood, easy to remember. Verified live on airstack-dev-13 after `docker compose up -d --force-recreate isaac-sim-livestream`: Kit binds UDP 49099 (`/proc/net/udp` hex BFCB on 0.0.0.0) and docker-proxy publishes it from the pod host network. Knock-on cleanups: - `simulation/isaac-sim/docker/docker-compose.yaml` shrinks the isaac-sim-livestream `ports:` from 27 forwarded ports (`47995-48012, 49000-49007 TCP+UDP, 49100 TCP`) to just two: `49100/tcp` + `49099/udp`. - `.airstack/modules/osmo.sh` shrinks `OSMO_WEBRTC_TCP` to `49100` and `OSMO_WEBRTC_UDP` to `49099`, so `airstack osmo:webrtc` spawns two port-forwards instead of thirty. - `.gitignore` ignores `.DS_Store` so working from a Mac doesn't leak Finder metadata. After pulling this commit into a running pod: `docker compose up -d --force-recreate isaac-sim-livestream` to apply the new port mapping; then re-run `airstack osmo:webrtc` on the laptop to pick up the new forward ranges. The standalone WebRTC Streaming Client connects to `localhost` (same address as before) and now actually receives frames. Co-authored-by: Cursor * fix(osmo): render Kit GUI in WebRTC stream; document SSH agent forward for in-pod git push Two paper-cuts that bit airstack-dev-13 after the WebRTC media port pin landed (commit 2d9b1611): (1) The WebRTC stream showed only the bare 3D viewport — no menu bar, no toolbar, no panels, no console. Cause: SimulationApp's default when `headless=True` is to also hide the UI (`hide_ui=True`). The NVIDIA reference at `simulation/isaac-sim/standalone_examples/api/isaacsim.simulation_app/livestream.py` explicitly opts back into UI rendering plus picks explicit window sizing and `display_options=3286` to keep the default grid/axes visible. Mirror that config in `example_one_px4_pegasus_launch_script.py` when `ISAAC_SIM_LIVESTREAM=true` (local desktop dev keeps the minimal `headless=False` path unchanged). (2) The pod has no SSH private key, only an `authorized_keys` for inbound connections from the user's laptop. As a result, `git push` from inside the Cursor / VS Code Remote-SSH session inside the pod fails with "Permission denied (publickey)". sshd inside the workspace image already has `AllowAgentForwarding yes` baked in via `osmo/workspace/sshd_config`; the missing piece is purely on the Mac side. Update the `~/.ssh/config` block in the tutorial to include `ForwardAgent yes` (so the local agent's keys are exposed in the pod), `AddKeysToAgent yes` (auto-load on first push), and `UseKeychain yes` (macOS-only Keychain unlock without passphrase prompts; ignored on Linux). Adds an `ssh-add -l` smoke-test note. Co-authored-by: Cursor * fix(osmo): make osmo:setup idempotent + paste-safe; document Nucleus auth-debug path osmo:setup hit two failure modes that wasted a debug session each: - `osmo credential set` is not an upsert for GENERIC creds — re-running setup (e.g. to rotate a Nucleus API token) failed with `400 duplicate key value violates unique constraint "credential_pkey"` and bailed before reaching the airlab-nucleus credential. Delete-then-set each credential so re-running is idempotent. - Bracket-paste mode and cross-OS clipboards routinely smuggle invisible bytes around long pastes. Nucleus's auth endpoint silently DENIES a token with one extra trailing byte, with no actionable error from the client side. _osmo_prompt now strips leading/trailing whitespace and CR/NUL bytes via a new _osmo_trim helper, and warns when bytes were stripped. cmd_osmo_setup additionally JWT-shape-checks the Nucleus token (must be eyJ...) before submitting it, so a wrong paste fails at setup time instead of silently DENIED at pod boot. Also documents how to debug the "Login Required: Unable to connect server omniverse://airlab-nucleus..." popup: SSH the Nucleus host and tail base_stack-nucleus-auth-1 for InternalCredentials.auth status: DENIED. Adds a "Nucleus connectivity from OSMO" section to the admin README clarifying that Nucleus over HTTPS uses a single 443 (no need to open the native 3009-3180 range from the OSMO cluster), per NVIDIA's TLS docs. Co-authored-by: Cursor * fix(osmo): use Nucleus API-token auth, with double-dollar to survive compose parser The OSMO entrypoint was writing OMNI_USER= alongside an API token JWT in OMNI_PASS, which routes the JWT through the password- verification path. Nucleus silently DENIES — visible only in base_stack-nucleus-auth-1 as `InternalCredentials.auth … 'username': '' … status: DENIED` (no Tokens.auth_with_api_token call). Kit then pops "Login Required: Unable to connect server omniverse://...". omniclient expects the literal sentinel username `$omni-api-token` paired with the JWT as the password. The entrypoint now detects a JWT-shaped OMNI_PASS (header starts with `eyJ`) and emits OMNI_USER=$$omni-api-token into omni_pass.env. The `$$` is intentional: docker-compose v2 interpolates env_file values, and a single `$` would be eaten by the parser (`OMNI_USER=$omni-api-token` becomes `OMNI_USER=-api-token` after ${omni}- expansion to empty). The container ultimately sees OMNI_USER=$omni-api-token, which is the correct sentinel. Also note for the next debugger: `docker compose restart` does NOT re-read env_file. Use `docker compose up -d ` to recreate the container after editing omni_pass.env. Updates omni_pass_TEMPLATE.env header to document the API-token pattern explicitly (with the $$ caveat), and adds a troubleshooting row that distinguishes "wrong auth path" (DENIED with no Tokens.auth_with_api_token call) from "bad/expired token" (Tokens.auth_with_api_token: DENIED). Co-authored-by: Cursor * docs(osmo): make OSMO the recommended dev path, single clone-the-repo flow Reposition the OSMO tutorial as AirStack's recommended day-to-day development path (not just a fallback for laptops without GPUs) and collapse it onto a single recipe: clone the repo, then drive everything through the airstack osmo:* wrappers in .airstack/modules/osmo.sh. - docs/tutorials/airstack_on_osmo.md - Retitle + rewrite the intro to lead with five concrete advantages (pooled GPUs, no local CUDA/Docker/driver maintenance, same image as CI + field robots, one-command onboarding, hardware bigger than your laptop). Demote the Linux+GPU-desktop path to an escape hatch. - Drop the Mac/Windows/no-GPU framing in 'Who is this for?' and the mermaid laptop subgraph label. - Add 'a local clone of AirStack' to Prerequisites; remove it from the 'do not need' list. - Replace Option A/B credential split with a single ./airstack.sh osmo:setup recipe; move the three raw osmo credential set calls into a collapsible 'Under the hood' footnote. - Replace each step's raw osmo workflow ... command with the corresponding airstack osmo:up/logs/ide/webrtc/foxglove/down wrapper; preserve the raw form in 'Under the hood' footnotes that cross-link cmd_osmo_* in .airstack/modules/osmo.sh. - Drop the export WF=... paragraph — the wrappers read the id from ~/.airstack/osmo-state automatically; AIRSTACK_OSMO_WF overrides per-invocation. \$WF now only appears inside the raw-form footnotes. - Sweep Troubleshooting + What-survives tables: redirect raw port-forward fixes to the airstack osmo:* equivalents and rename the section to 'What survives airstack osmo:down?'. - Fix WebRTC edge label (49100/tcp + 49099/udp) to match the pinned ports the workflow actually uses today. Companion cleanups now that the privileged_allowed flip is automatic on the OSMO autosync side (synchronize_osmo_team_pools.py forces privileged_allowed: true on every platform of every pool, so students never see the 'platform does not have privileged flag enabled' error): - osmo/README.md: drop the 'Most common blocker' privileged warning, the privileged_allowed row from the pool-requirements table, and the 'privileged GPU pod' / '(privileged, GPU)' descriptors in the architecture summary. Simplify the validation-stage SSH-failure hint. - osmo/workflows/airstack-dev.yaml: trim the long DinD-requires-privileged comment to a one-liner (the privileged: true directive itself stays). - .airstack/modules/osmo.sh: remove the special-case 'privileged flag enabled' error branch in cmd_osmo_up — it should never fire now. Co-authored-by: Cursor * fix(osmo): make osmo:logs actually stream + survive pod host-key churn osmo:logs was silent because cmd_osmo_logs wrapped osmo workflow logs in $( ... ) on the assumption that -n LAST_N_LINES exits after dumping the tail. Empirically the CLI keeps the stream open as new lines arrive (it already behaves like tail -f, despite --help advertising only -n), so command substitution waited forever and printed nothing. Drop the polling loop and just exec the command directly. Each fresh OSMO pod also ships a new sshd host key, so every osmo:up trips StrictHostKeyChecking against the previous workflow's fingerprint and SSH/Cursor abort with "Host key for [localhost]:2200 has changed". Switch the recommended ~/.ssh/config block (and osmo/README.md) to the ephemeral-host pattern (StrictHostKeyChecking no + UserKnownHostsFile /dev/null + LogLevel ERROR), and have cmd_osmo_ide ssh-keygen -R the stale loopback entry on every run so users on the old config get unblocked automatically. Co-authored-by: Cursor * fix(osmo): auto-pin --branch to local checkout + clean error UX when workflow dies The pod's entrypoint clones AirStack fresh from GitHub on every workflow start (the pod fs is ephemeral). It defaulted to `main`, so any developer testing branch-only OSMO changes silently ran their pod against stale `main` code — most visibly: COMPOSE_PROFILES=desktop,isaac-sim-livestream resolved to "desktop" alone on `main` because the isaac-sim-livestream service only exists on the feature branch, so isaac-sim never came up and `airstack osmo:webrtc` showed a blank stream. - cmd_osmo_up now defaults --branch to the local repo's current branch (git rev-parse --abbrev-ref HEAD). Detached HEAD or non-git checkouts fall back to `main` cleanly. Pass --branch explicitly to override. - New _osmo_check_branch_pushed warns up-front when the about-to- submit branch has no upstream, is ahead of origin, or has an uncommitted working tree. The pod doesn't see your laptop's edits. Separately, when an OSMO workflow gets canceled mid-flight (osmo:down in another shell, or OSMO timing it out), the in-flight port-forward and logs streams raise OSMOUserError("Workflow X is not running!") from inside an asyncio Task. The CLI prints "Task exception was never retrieved" + a multi-line Traceback that buries the actual one-line cause. New _osmo_pf_filter awk script collapses that into a single [ERROR] line pointing at `airstack osmo:up`. Wired into webrtc, foxglove, and logs. webrtc also gains a cleanup trap that kills the backgrounded UDP port-forward on EXIT/INT/TERM so we don't leak it against a dead workflow. Tutorial Step 2 documents the new --branch default and the "pod-clones-from-GitHub-not-your-laptop" gotcha. Co-authored-by: Cursor * perf(osmo): bump inner dockerd concurrency to saturate 10 GbE pulls dockerd's defaults of --max-concurrent-downloads=3 / --max-concurrent -uploads=5 cap a fresh airstack-dev pod's image-pull at ~300 MiB/s against the airlab-backup-10g registry — single-stream TLS tops out around 300-500 MiB/s per core, and three parallel streams of unevenly sized blobs serialize down to that ceiling. Ceph (1014 TiB, 92 OSDs, SSD pools) and 10 GbE both have far more headroom than that. Bump to 10/10 to overlap enough blob downloads to saturate the pipe. Threaded through the DOCKERD_MAX_DOWNLOADS / DOCKERD_MAX_UPLOADS env vars so a pool can be tuned at submit time without rebuilding the workspace image. Workspace image needs a rebuild + push for this to take effect: cd osmo/workspace docker build -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest . docker push airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest Co-authored-by: Cursor * docs(osmo): require buildx --platform linux/amd64 for workspace image A plain `docker build && docker push` on an Apple Silicon Mac silently produces a linux/arm64-only `latest` manifest. OSMO workers are amd64, so every subsequent workflow fails at the outer pod-image pull with "no match for platform in manifest" before the entrypoint even runs — a confusing failure mode whose root cause lives entirely in the push, not in the workflow yaml or the entrypoint. Switch the README and the Dockerfile docstring to the buildx form, explain the why, and document the post-push manifest check. Co-authored-by: Cursor * perf(osmo): move dockerd data-root to /osmo/run for native overlay2 The OSMO pod's `/` is itself a containerd overlay snapshot, and Linux refuses to stack a second overlayfs on top of an overlay rootfs — which is why the inner dockerd was falling through to fuse-overlayfs. That costs a kernel↔userspace FUSE round-trip on every `creat()` during layer extraction, which murders throughput on apt/pip/ROS layers (measured: 32-50 MB/s for small-file-heavy layers vs 480 MB/s for big-file layers in the same pull). Pointing dockerd at /osmo/run/docker (the kubelet emptyDir backed by ext4 on /dev/vda3) lets the existing overlay2-first fallback chain actually succeed on its first try, restoring kernel-overlay extraction performance. emptyDir lifetime matches the workflow lifetime, so the docker layer cache gets the right scope automatically. Falls back to /var/lib/docker if /osmo/run isn't present so the image still works in non-OSMO test contexts. Co-authored-by: Cursor * updated version * added virtual display for GL context * added virtual display for droan_gl * droan_gl patch * run Xvfb in its own tmux session * updated dockerfile + version * typo in docs Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in osmo logs, renamed airstack-isaac-sim to just isaac-sim Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in container name for isaac-sim-livestream Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * airstack-dev version overwrite removed Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Cursor Co-authored-by: krrishj18 Co-authored-by: Andrew Jong Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- .airstack/modules/osmo.sh | 719 ++++++++++++++++++ .env | 4 +- .gitignore | 2 + airstack.sh | 41 + docs/getting_started/index.md | 7 + docs/tutorials/airstack_on_osmo.md | 591 ++++++++++++++ docs/tutorials/index.md | 1 + gcs/foxglove_extensions/install.py | 36 +- mkdocs.yml | 1 + osmo/README.md | 301 ++++++++ osmo/workflows/airstack-dev.yaml | 99 +++ osmo/workspace/Dockerfile | 112 +++ osmo/workspace/entrypoint.sh | 281 +++++++ osmo/workspace/sshd_config | 41 + robot/docker/Dockerfile.robot | 2 + robot/docker/docker-compose.yaml | 5 + .../isaac-sim/docker/docker-compose.yaml | 58 ++ .../isaac-sim/docker/omni_pass_TEMPLATE.env | 31 +- .../example_one_px4_pegasus_launch_script.py | 72 +- 19 files changed, 2383 insertions(+), 21 deletions(-) create mode 100755 .airstack/modules/osmo.sh create mode 100644 docs/tutorials/airstack_on_osmo.md create mode 100644 osmo/README.md create mode 100644 osmo/workflows/airstack-dev.yaml create mode 100644 osmo/workspace/Dockerfile create mode 100755 osmo/workspace/entrypoint.sh create mode 100644 osmo/workspace/sshd_config diff --git a/.airstack/modules/osmo.sh b/.airstack/modules/osmo.sh new file mode 100755 index 000000000..053decbee --- /dev/null +++ b/.airstack/modules/osmo.sh @@ -0,0 +1,719 @@ +#!/usr/bin/env bash + +# osmo.sh — AirStack-on-OSMO convenience commands. +# +# Wraps `osmo workflow submit/port-forward/logs/cancel` for the +# osmo/workflows/airstack-dev.yaml workflow so a Mac/Windows student doesn't +# have to memorize the WebRTC port range or the entry-script path. +# +# This module is pure bash + the cross-platform `osmo` CLI — no Docker +# dependency. Safe to run on a laptop with no AirStack runtime. +# +# Most commands need a workflow id. `osmo:up` saves the id to +# $OSMO_STATE_FILE; the other commands read it from there. You can also +# override it for a single invocation by exporting AIRSTACK_OSMO_WF. + +# State directory and file: ~/.airstack/osmo-state stores the most recent +# workflow id submitted with `airstack osmo:up`. +OSMO_STATE_DIR="${HOME}/.airstack" +OSMO_STATE_FILE="${OSMO_STATE_DIR}/osmo-state" + +# WebRTC livestream ports — must match the ports published by the +# isaac-sim-livestream service in +# simulation/isaac-sim/docker/docker-compose.yaml AND the +# app.livestream.fixedHostPort setting pinned in the Pegasus launch script +# (simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_launch_script.py). +# +# Two ports total: +# TCP 49100 — omni.kit.livestream.webrtc WebSocket signaling +# UDP 49099 — SRTP media (pinned; Kit 107 otherwise picks dynamically and +# escapes both the compose-published and CLI-forwarded ranges) +OSMO_WEBRTC_TCP="49100" +OSMO_WEBRTC_UDP="49099" + +# GCS Foxglove websocket: container 8765 → host 8766 (per +# gcs/docker/docker-compose.yaml). +OSMO_FOXGLOVE_PORT="8766:8766" + +# SSH port-forward: local 2200 → pod 22. +OSMO_SSH_PORT="2200:22" + +# Default `osmo workflow port-forward` connect-timeout (24h). +OSMO_PF_TIMEOUT="${OSMO_PF_TIMEOUT:-86400}" + +# Helper: ensure the osmo CLI is on PATH. +function _osmo_check_cli { + if ! command -v osmo >/dev/null 2>&1; then + log_error "osmo CLI not found on PATH. Install from https://github.com/NVIDIA/OSMO and run 'osmo login'." + return 1 + fi +} + +# Helper: strip leading/trailing whitespace + CR/NUL bytes from the +# variable named in $1. +# +# Why this exists: bracket-paste mode and cross-OS clipboards (RDP, VNC, +# Windows-side note apps) routinely smuggle invisible bytes around long +# pastes — Nucleus API tokens (JWT, ~1 KB) and SSH keys are the usual +# victims. Nucleus's auth endpoint silently `DENIES` a token that has +# one extra trailing byte, with no actionable error from the client side. +# Stripping defensively at prompt time saves an entire round-trip of +# "regenerate token → still denied → check auth-service log" debugging. +function _osmo_trim { + local var_name="$1" + local val="${!var_name}" + local original_len="${#val}" + val="${val//$'\r'/}" + val="${val//$'\0'/}" + val="${val#"${val%%[![:space:]]*}"}" + val="${val%"${val##*[![:space:]]}"}" + if [ "${#val}" -ne "$original_len" ]; then + log_warn "Stripped $((original_len - ${#val})) whitespace/control byte(s) from ${var_name}." + fi + printf -v "$var_name" '%s' "$val" +} + +# Helper: read a value with prompt; supports -s for silent (passwords). +# +# Visible prompts switch the TTY out of canonical mode for the duration of +# the read. Without this, macOS caps each input line at MAX_CANON = 1024 +# bytes (per ) and rings the terminal bell on Enter when +# the buffer overflows. Nucleus API tokens are JWTs ~950 bytes long, so +# `Nucleus API token: ` lands right at the cap. `stty -icanon` makes +# the kernel deliver bytes to bash as they're typed, with no line-buffer +# limit; bash's `read` still terminates on newline normally. +# +# We use a trap to guarantee the saved stty is restored if the user Ctrl-Cs +# mid-paste — otherwise the shell would be left in raw mode. +# +# After reading we always run _osmo_trim — see comment there. +function _osmo_prompt { + local var_name="$1" + local prompt_text="$2" + local silent="${3:-false}" + local saved_stty="" + + if [ "$silent" = "true" ]; then + # Passwords are short — canonical-mode cap is fine here. + read -r -s -p "${prompt_text}: " "$var_name" + printf "\n" >&2 + else + if [ -t 0 ]; then + saved_stty="$(stty -g 2>/dev/null || true)" + if [ -n "$saved_stty" ]; then + trap 'stty "$saved_stty" 2>/dev/null; trap - INT' INT + stty -icanon 2>/dev/null + fi + fi + read -r -p "${prompt_text}: " "$var_name" + if [ -n "$saved_stty" ]; then + stty "$saved_stty" 2>/dev/null + trap - INT + fi + fi + + _osmo_trim "$var_name" + + if [ -z "${!var_name}" ]; then + log_error "Empty input for ${var_name}; aborting." + return 1 + fi +} + +# osmo:setup — interactively register the three OSMO credentials AirStack +# needs (airlab-docker-registry, airlab-docker-login, airlab-nucleus). +# Idempotent — re-running rotates the credentials. +function cmd_osmo_setup { + _osmo_check_cli || return 1 + + cat >&2 <<'EOF' + +This sets up the three per-user OSMO credentials AirStack-on-OSMO needs: + + 1. airlab-docker-registry (REGISTRY) — for OSMO to pull the workspace image + 2. airlab-docker-login (GENERIC) — for the inner dockerd to pull AirStack images + 3. airlab-nucleus (GENERIC) — for Isaac Sim Nucleus access + +You'll be asked for: + + - your Andrew ID (no @andrew.cmu.edu suffix) + - your AirLab Docker password (same as your Andrew password) + - your Nucleus API token (https://airlab-nucleus.andrew.cmu.edu/omni/web3/ + → right-click cloud → API Tokens). NOT your Andrew password. + +Values go directly to OSMO; nothing is written to disk locally. + +EOF + + local andrew_id andrew_password nucleus_token + _osmo_prompt andrew_id "Andrew ID" false || return 1 + _osmo_prompt andrew_password "AirLab Docker password (hidden)" true || return 1 + _osmo_prompt nucleus_token "Nucleus API token" false || return 1 + + # Sanity-check the Nucleus token shape. Nucleus issues RS256 JWTs: + # base64url(header).base64url(payload).base64url(signature), with the + # header always starting `eyJ` (base64url of `{"`). Catching a wrong + # paste here (e.g. Andrew password, or token without the trailing + # signature segment) saves the user from a silent `InternalCredentials + # .auth: DENIED` round-trip later on. We do not validate the signature. + case "$nucleus_token" in + eyJ*.*.*) ;; # looks like a 3-segment JWT + *) + log_error "That doesn't look like a Nucleus API token." + log_error " - Expected: a JWT of the form eyJ……… (~1 KB long)" + log_error " - Got: ${#nucleus_token} chars, prefix '$(printf '%s' "$nucleus_token" | head -c 8)…'" + log_error " Generate one at https://airlab-nucleus.andrew.cmu.edu/omni/web3/" + log_error " → right-click cloud icon → API Tokens → Create." + return 1 + ;; + esac + + local omni_server="${OMNI_SERVER:-omniverse://airlab-nucleus.andrew.cmu.edu/NVIDIA/Assets/Isaac/5.1}" + local airlab_registry="${AIRLAB_REGISTRY:-airlab-docker.andrew.cmu.edu}" + + # `osmo credential set` is NOT an upsert for GENERIC credentials — re-setting + # one that already exists fails with `400 duplicate key value violates unique + # constraint "credential_pkey"`. Delete first so re-running osmo:setup + # (e.g. to rotate a Nucleus token) is idempotent. The `|| true` swallows the + # "credential not found" case on a first-time run. + log_info "Refreshing airlab-docker-registry (REGISTRY)..." + osmo credential delete airlab-docker-registry >/dev/null 2>&1 || true + osmo credential set airlab-docker-registry \ + --type REGISTRY \ + --payload "registry=${airlab_registry}" \ + "username=${andrew_id}" \ + "auth=${andrew_password}" \ + || { log_error "osmo credential set airlab-docker-registry failed"; return 1; } + + log_info "Refreshing airlab-docker-login (GENERIC)..." + osmo credential delete airlab-docker-login >/dev/null 2>&1 || true + osmo credential set airlab-docker-login \ + --type GENERIC \ + --payload "username=${andrew_id}" \ + "password=${andrew_password}" \ + || { log_error "osmo credential set airlab-docker-login failed"; return 1; } + + log_info "Refreshing airlab-nucleus (GENERIC)..." + osmo credential delete airlab-nucleus >/dev/null 2>&1 || true + osmo credential set airlab-nucleus \ + --type GENERIC \ + --payload "omni_user=${andrew_id}" \ + "omni_pass=${nucleus_token}" \ + "omni_server=${omni_server}" \ + || { log_error "osmo credential set airlab-nucleus failed"; return 1; } + + log_info "All three credentials registered. List them with: osmo credential list" + log_info "Next: airstack osmo:up [--pool POOL]" +} + +# Helper: pick the first existing SSH public key on the host. +function _osmo_pick_pubkey { + local candidates=( + "${HOME}/.ssh/id_ed25519.pub" + "${HOME}/.ssh/id_ecdsa.pub" + "${HOME}/.ssh/id_rsa.pub" + ) + for k in "${candidates[@]}"; do + if [ -f "$k" ]; then + echo "$k" + return 0 + fi + done + return 1 +} + +# Helper: get the active workflow id (env override first, then state file). +# +# The state file persists across shell sessions, so it can easily go stale +# (e.g. a previous airstack-dev-N is now FAILED/CANCELED). To avoid the +# confusing "Workflow airstack-dev-10 is not running!" 410 error from the +# downstream osmo command, this helper verifies the saved id is still in a +# live state (PENDING / RUNNING) before returning it. +function _osmo_wf_id { + local wf + if [ -n "${AIRSTACK_OSMO_WF:-}" ]; then + wf="${AIRSTACK_OSMO_WF}" + elif [ -f "${OSMO_STATE_FILE}" ]; then + wf="$(cat "${OSMO_STATE_FILE}")" + else + log_error "No workflow id found. Run 'airstack osmo:up' first, or export AIRSTACK_OSMO_WF=." + return 1 + fi + + # Validate the workflow is still alive (only when osmo CLI is available). + if command -v osmo >/dev/null 2>&1; then + local status + status="$(osmo workflow query "${wf}" 2>/dev/null | awk -F': +' '/^Status/ {print $2; exit}' | tr -d ' \r\n')" + case "${status}" in + PENDING|RUNNING|"") + # "" means we couldn't reach osmo; let the downstream + # command surface the real error rather than failing here. + ;; + *) + log_error "Saved workflow '${wf}' is ${status}, not running." + log_warn "Run 'airstack osmo:up' to launch a fresh one, or:" + log_warn " rm ${OSMO_STATE_FILE}" + log_warn " export AIRSTACK_OSMO_WF=" + return 1 + ;; + esac + fi + + echo "${wf}" + return 0 +} + +# Helper: persist the workflow id. +function _osmo_save_wf_id { + mkdir -p "${OSMO_STATE_DIR}" + echo "$1" > "${OSMO_STATE_FILE}" + log_info "Saved workflow id '$1' to ${OSMO_STATE_FILE}" +} + +# Helper: best-effort detection of the user's current AirStack branch so +# `airstack osmo:up` can default --branch to whatever the user is editing +# locally. Returns the branch name on stdout, or empty if we shouldn't +# auto-pin (detached HEAD, not a git repo, etc.). +# +# Why default to the local branch: the pod's entrypoint clones AirStack +# fresh from GitHub on every workflow start (the pod fs is ephemeral, so +# nothing else makes sense). If we don't tell it which branch, it +# defaults to `main` — and any developer testing branch-only OSMO +# changes (compose services, entrypoint tweaks, workflow yaml edits) +# silently runs against stale `main` code instead of their work. +# Defaulting to the local branch makes "edit on laptop, push, osmo:up" +# the natural workflow. +function _osmo_local_branch { + if ! command -v git >/dev/null 2>&1; then + return 0 + fi + local b + b="$(git -C "${PROJECT_ROOT}" rev-parse --abbrev-ref HEAD 2>/dev/null)" || return 0 + case "$b" in + ""|HEAD) return 0 ;; # detached HEAD or empty + esac + echo "$b" +} + +# Helper: warn if the about-to-submit branch isn't safely pushed. The +# pod clones from GitHub, so unpushed commits / dirty working tree don't +# make it into the pod even if the user thinks they did. Catching this +# before submit avoids a 60-90s "wait for pod, then realize" round trip. +function _osmo_check_branch_pushed { + local branch="$1" + command -v git >/dev/null 2>&1 || return 0 + local repo="${PROJECT_ROOT}" + [ -d "${repo}/.git" ] || return 0 + + local local_sha upstream_sha + local_sha="$(git -C "$repo" rev-parse "${branch}" 2>/dev/null)" || return 0 + + # Look for a remote-tracking branch first (the explicit upstream + # set by `git push -u`); fall back to origin/. + upstream_sha="$(git -C "$repo" rev-parse "${branch}@{upstream}" 2>/dev/null)" + if [ -z "$upstream_sha" ]; then + upstream_sha="$(git -C "$repo" rev-parse "origin/${branch}" 2>/dev/null)" + fi + + if [ -z "$upstream_sha" ]; then + log_warn "Branch '${branch}' has no upstream on origin — the pod's clone will fail. Run: git push -u origin ${branch}" + return 0 + fi + + if [ "$local_sha" != "$upstream_sha" ]; then + local ahead behind + ahead="$(git -C "$repo" rev-list --count "${upstream_sha}..${local_sha}" 2>/dev/null)" + behind="$(git -C "$repo" rev-list --count "${local_sha}..${upstream_sha}" 2>/dev/null)" + if [ "${ahead:-0}" -gt 0 ]; then + log_warn "Local '${branch}' is ${ahead} commit(s) ahead of origin/${branch} — the pod will clone the older origin tip. Run: git push" + fi + if [ "${behind:-0}" -gt 0 ]; then + log_info "Local '${branch}' is ${behind} commit(s) behind origin/${branch} (pod will clone the newer origin tip)." + fi + fi + + if [ -n "$(git -C "$repo" status --porcelain 2>/dev/null)" ]; then + log_warn "Working tree has uncommitted changes — the pod will not see them. Commit + push first if you want the pod to pick them up." + fi +} + +# osmo:up — submit airstack-dev.yaml with the local pubkey injected. +# +# Usage: airstack osmo:up [--pool POOL] [--key PATH] [--branch BRANCH] +# +# --branch defaults to the local repo's current branch (or `main` if we +# can't detect one), and is passed through as AIRSTACK_BRANCH so the +# pod's entrypoint clones the matching code. Pass `--branch main` +# explicitly to override. +function cmd_osmo_up { + _osmo_check_cli || return 1 + + local pool="${OSMO_POOL:-}" + local pubkey_file="" + local branch="" + local branch_explicit=false + local extra_args=() + + while [ $# -gt 0 ]; do + case "$1" in + --pool) pool="$2"; shift 2 ;; + --key) pubkey_file="$2"; shift 2 ;; + --branch) branch="$2"; branch_explicit=true; shift 2 ;; + *) extra_args+=("$1"); shift ;; + esac + done + + if [ -z "$pubkey_file" ]; then + if ! pubkey_file="$(_osmo_pick_pubkey)"; then + log_error "No SSH public key found in ~/.ssh. Generate one with: ssh-keygen -t ed25519" + return 1 + fi + fi + log_info "Using SSH public key: ${pubkey_file}" + + local workflow_yaml="${PROJECT_ROOT}/osmo/workflows/airstack-dev.yaml" + if [ ! -f "$workflow_yaml" ]; then + log_error "Workflow file not found: ${workflow_yaml}" + return 1 + fi + + # Auto-pin --branch to the local checkout if the user didn't pass one. + if [ "$branch_explicit" = false ] && [ -z "$branch" ]; then + branch="$(_osmo_local_branch)" + if [ -n "$branch" ]; then + log_info "Auto-detected local branch '${branch}'; pod will clone from origin/${branch} (override with --branch main)." + else + log_info "Could not detect local branch (detached HEAD?); pod will clone from origin/main." + fi + fi + if [ -n "$branch" ]; then + _osmo_check_branch_pushed "$branch" + fi + + local cmd=(osmo workflow submit "$workflow_yaml") + if [ -n "$pool" ]; then + cmd+=(--pool "$pool") + else + log_warn "No --pool provided and OSMO_POOL is unset; using your osmo profile's default pool." + fi + # IMPORTANT: `osmo workflow submit --set-env` is variadic. Passing two + # separate `--set-env A=1 --set-env B=2` silently drops the first one + # (only the last `--set-env` flag's values are kept). We collect all + # K=V pairs and pass them under a single `--set-env`. + local env_kvs=("SSH_PUB_KEY=$(cat "$pubkey_file")") + if [ -n "$branch" ]; then + env_kvs+=("AIRSTACK_BRANCH=${branch}") + fi + cmd+=(--set-env "${env_kvs[@]}") + if [ ${#extra_args[@]} -gt 0 ]; then + cmd+=("${extra_args[@]}") + fi + + log_info "Submitting: ${cmd[*]}" + local output + if ! output="$("${cmd[@]}" 2>&1)"; then + echo "$output" >&2 + log_error "osmo workflow submit failed." + return 1 + fi + echo "$output" + + # Parse the workflow id out of the submit output. The cookbook examples + # show "Workflow ID - " formatted output (see OSMO + # submission.rst). Match that line. + local wf_id + wf_id="$(echo "$output" | awk -F'- ' '/^Workflow ID/ {print $2; exit}' | tr -d ' \r\n')" + if [ -z "$wf_id" ]; then + log_warn "Could not parse workflow id from submit output. Set it manually:" + log_warn " echo > ${OSMO_STATE_FILE}" + return 0 + fi + _osmo_save_wf_id "$wf_id" + + log_info "Next steps:" + log_info " airstack osmo:logs # follow startup until 'sshd listening'" + log_info " airstack osmo:ide # port-forward sshd + open VS Code" + log_info " airstack osmo:webrtc # forward Isaac Sim WebRTC ports" + log_info " airstack osmo:foxglove # forward GCS Foxglove websocket" + log_info " airstack osmo:down # cancel the workflow" +} + +# osmo:logs — follow the workspace task logs. +# +# Despite the `osmo workflow logs --help` output advertising only `-n +# LAST_N_LINES` (no `--follow`), the CLI in fact streams the tail and keeps +# the connection open as new lines arrive — i.e. it already behaves like +# `tail -f`. We just exec it in the foreground so the user sees output +# immediately and can Ctrl+C to stop. (An earlier implementation wrapped +# this in `out=$(osmo workflow logs ...)`; command substitution waits for +# the process to exit, which never happened, so nothing was ever printed.) +function cmd_osmo_logs { + _osmo_check_cli || return 1 + local wf; wf="$(_osmo_wf_id)" || return 1 + + local task="${OSMO_LOGS_TASK:-workspace}" + local lines="${OSMO_LOGS_TAIL:-500}" + + log_info "Following ${task} logs for ${wf} (last ${lines} lines, then live; Ctrl+C to stop)" + + # Filter stderr for the same OSMOUserError-when-workflow-dies case + # the port-forward path hits — same noisy asyncio Traceback + + # "Task exception was never retrieved" header. _osmo_pf_filter + # collapses it into one clean log line. + osmo workflow logs "${wf}" -t "${task}" -n "${lines}" \ + 2> >(_osmo_pf_filter "${wf}") +} + +# osmo:ide — port-forward sshd + (optionally) launch VS Code/Cursor on the +# `airstack-osmo` host. Runs the port-forward in the foreground so closing +# the terminal closes the tunnel. +# +# Usage: airstack osmo:ide [--no-open] [code|cursor] +function cmd_osmo_ide { + _osmo_check_cli || return 1 + local wf; wf="$(_osmo_wf_id)" || return 1 + + local open_ide=true + local ide_cmd="" + while [ $# -gt 0 ]; do + case "$1" in + --no-open) open_ide=false; shift ;; + code|cursor) ide_cmd="$1"; shift ;; + *) log_warn "Ignoring unknown osmo:ide arg: $1"; shift ;; + esac + done + + if [ -z "$ide_cmd" ]; then + if command -v cursor >/dev/null 2>&1; then + ide_cmd="cursor" + elif command -v code >/dev/null 2>&1; then + ide_cmd="code" + else + log_warn "Neither 'cursor' nor 'code' found on PATH; will only port-forward (open the IDE manually and Connect to Host airstack-osmo)." + open_ide=false + fi + fi + + log_info "Make sure ~/.ssh/config has a 'Host airstack-osmo' entry pointing at localhost:2200, User root." + + # Local TCP port the user's IDE will connect to (the local side of the + # `--port LOCAL:REMOTE` mapping). + local local_port="${OSMO_SSH_PORT%%:*}" + + # Every fresh OSMO pod ships a new sshd host key. If the user's + # ~/.ssh/known_hosts still has an entry for [localhost]:${local_port} + # from a previous workflow, ssh aborts with "Host key for [localhost] + # :${local_port} has changed and you have requested strict checking", + # which the IDE surfaces as a generic "could not connect" error. + # + # The recommended ~/.ssh/config block for `airstack-osmo` uses + # `UserKnownHostsFile /dev/null`, which sidesteps this entirely — but + # users who set up before that change still have a stale entry on + # disk. Scrub it defensively on every osmo:ide invocation. ssh-keygen + # -R is idempotent: a no-op if the entry doesn't exist. + if command -v ssh-keygen >/dev/null 2>&1; then + ssh-keygen -R "[localhost]:${local_port}" >/dev/null 2>&1 || true + fi + + # Reuse an existing forward if one is already listening (the user might + # have run this from a second terminal, or osmo:foxglove already opened + # a multi-port forward). Otherwise spawn one in the background and wait + # for it to bind before launching the IDE — this avoids the race where + # Cursor/VS Code tries to SSH before the tunnel exists and dies with + # "connect to host localhost port 2200: Connection refused". + local pf_pid="" + if nc -z localhost "$local_port" 2>/dev/null; then + log_info "Port ${local_port} is already listening; reusing existing port-forward." + else + log_info "osmo workflow port-forward ${wf} workspace --port ${OSMO_SSH_PORT} --connect-timeout ${OSMO_PF_TIMEOUT}" + osmo workflow port-forward "$wf" workspace --port "$OSMO_SSH_PORT" --connect-timeout "$OSMO_PF_TIMEOUT" \ + > "${OSMO_STATE_DIR}/ssh-pf.log" 2>&1 & + pf_pid=$! + # Wait up to 30s for the tunnel to start accepting connections. + local waited=0 + until nc -z localhost "$local_port" 2>/dev/null; do + sleep 1; waited=$((waited+1)) + if [ "$waited" -ge 30 ]; then + log_error "Timed out waiting for port-forward on :${local_port} after ${waited}s." + log_error " port-forward log: ${OSMO_STATE_DIR}/ssh-pf.log" + kill "$pf_pid" 2>/dev/null + return 1 + fi + if ! kill -0 "$pf_pid" 2>/dev/null; then + log_error "port-forward exited early. Tail:" + tail -10 "${OSMO_STATE_DIR}/ssh-pf.log" >&2 + return 1 + fi + done + log_info "Port-forward established on localhost:${local_port} (pid ${pf_pid})." + fi + + if [ "$open_ide" = true ]; then + # vscode-remote URI launches the IDE pre-attached to the remote host. + local uri="vscode-remote://ssh-remote+airstack-osmo/root/AirStack" + log_info "Launching ${ide_cmd} → ${uri}" + ( "$ide_cmd" --folder-uri "$uri" >/dev/null 2>&1 || \ + "$ide_cmd" "$uri" >/dev/null 2>&1 || \ + log_warn "Could not launch ${ide_cmd} automatically; open it and pick airstack-osmo from Remote-SSH manually." ) & + fi + + if [ -n "$pf_pid" ]; then + log_info "Leave this terminal running for the length of your session (Ctrl+C to disconnect)." + # Forward Ctrl+C to the port-forward and clean up. + trap 'kill "$pf_pid" 2>/dev/null; exit 0' INT TERM + wait "$pf_pid" + else + log_info "Existing port-forward owns the tunnel; this command will exit immediately." + log_info "Stop the tunnel with: pkill -f 'osmo workflow port-forward' or airstack osmo:down" + fi +} + +# Helper: filter `osmo workflow port-forward` stderr through awk to +# suppress the asyncio traceback that erupts whenever the workflow gets +# canceled mid-flight (e.g. via osmo:down in another shell, or because +# OSMO timed it out). The CLI raises OSMOUserError("Workflow X is not +# running!") from inside an asyncio Task, which then prints "Task +# exception was never retrieved" + a multi-line Traceback that obscures +# the actual one-line cause. We translate that into a single clean log +# line and drop everything else. +function _osmo_pf_filter { + local wf="$1" + awk -v WF="$wf" ' + /^Task exception was never retrieved/ { skipping=1; next } + /^future:/ { skipping=1; next } + /^Traceback \(most recent call last\):/ { skipping=1; next } + /^ File "/ { next } + /^src\.lib\.utils\.osmo_errors\.OSMOUserError/ { + sub(/^src\.lib\.utils\.osmo_errors\.OSMOUserError: */, "") + printf "\033[0;31m[ERROR]\033[0m %s (run `airstack osmo:up` to start a new workflow)\n", $0 + next + } + /OSMOUserError: Workflow .* is not running!/ { + printf "\033[0;31m[ERROR]\033[0m Workflow %s is no longer running (run `airstack osmo:up` to start a new one).\n", WF + next + } + skipping && /^$/ { skipping=0; next } + skipping { next } + { print } + ' >&2 +} + +# Helper: run `osmo workflow port-forward` with the noise filter +# attached. Returns the underlying exit code so callers can decide +# whether to retry / fail. Args after the helper name are passed to +# `osmo workflow port-forward` verbatim. +function _osmo_run_port_forward { + osmo workflow port-forward "$@" 2> >(_osmo_pf_filter "$1") +} + +# osmo:webrtc — forward both Isaac Sim WebRTC port ranges (TCP in this +# terminal, spawn UDP in the background). Cleans up the UDP child on +# exit (Ctrl+C, foreground TCP failure, or the workflow disappearing +# mid-stream) so we don't leak a port-forward into the user's process +# table. +function cmd_osmo_webrtc { + _osmo_check_cli || return 1 + local wf; wf="$(_osmo_wf_id)" || return 1 + + log_info "Spawning UDP port-forward in background: ${OSMO_WEBRTC_UDP}" + nohup osmo workflow port-forward "$wf" workspace \ + --port "$OSMO_WEBRTC_UDP" --udp \ + --connect-timeout "$OSMO_PF_TIMEOUT" \ + > "${OSMO_STATE_DIR}/webrtc-udp.log" 2>&1 & + local udp_pid=$! + log_info " UDP log: ${OSMO_STATE_DIR}/webrtc-udp.log (pid ${udp_pid})" + + # Tear the UDP fork down when this function exits, by any path. + # Without this, hitting Ctrl+C on the TCP foreground (or the + # workflow being canceled, which surfaces as the foreground exiting + # non-zero) leaves the UDP `osmo workflow port-forward` running + # against a dead workflow until the user notices and pkill's it. + trap ' + if kill -0 "'"${udp_pid}"'" 2>/dev/null; then + kill "'"${udp_pid}"'" 2>/dev/null + wait "'"${udp_pid}"'" 2>/dev/null + fi + trap - EXIT INT TERM + ' EXIT INT TERM + + log_info "Foreground TCP port-forward: ${OSMO_WEBRTC_TCP}" + log_info "Open the Omniverse Streaming Client / WebRTC client at http://localhost" + _osmo_run_port_forward "$wf" workspace \ + --port "$OSMO_WEBRTC_TCP" \ + --connect-timeout "$OSMO_PF_TIMEOUT" +} + +# osmo:foxglove — install the AirStack Foxglove extensions into the local +# Foxglove Desktop user-extensions dir, then forward the GCS Foxglove +# websocket. +# +# The extension install is the same script the GCS container runs on +# startup — gcs/foxglove_extensions/install.py — invoked with env-var +# overrides that point at the local laptop dirs. Default destination on +# Linux/macOS is ~/.foxglove-studio/extensions (Foxglove's canonical user +# extensions path; the macOS rebrand still reads from here). Override +# with OSMO_FOXGLOVE_EXT_DIR, or skip the install entirely with +# OSMO_FOXGLOVE_SKIP_EXTENSIONS=1 (e.g. when using app.foxglove.dev +# which doesn't load local extensions anyway). +function cmd_osmo_foxglove { + _osmo_check_cli || return 1 + local wf; wf="$(_osmo_wf_id)" || return 1 + + local ext_src="${PROJECT_ROOT}/gcs/foxglove_extensions" + local ext_dst="${OSMO_FOXGLOVE_EXT_DIR:-${HOME}/.foxglove-studio/extensions}" + + if [ "${OSMO_FOXGLOVE_SKIP_EXTENSIONS:-0}" != "1" ] && [ -d "${ext_src}" ]; then + if command -v python3 >/dev/null 2>&1; then + log_info "Installing Foxglove extensions to ${ext_dst}" + FOXGLOVE_EXT_SRC="${ext_src}" FOXGLOVE_EXT_DST="${ext_dst}" \ + python3 "${ext_src}/install.py" \ + || log_warn "Foxglove extension install failed; panels like 'Robot Tasks' may show as 'Unknown panel type' in Foxglove" + else + log_warn "python3 not found on PATH — skipping Foxglove extension install." + log_warn " Custom panels (Robot Tasks, Waypoint Editor, Polygon Editor) will show as 'Unknown panel type'." + log_warn " Install python3 (e.g. 'brew install python') or copy ${ext_src}/* manually to ${ext_dst}." + fi + elif [ "${OSMO_FOXGLOVE_SKIP_EXTENSIONS:-0}" = "1" ]; then + log_info "Skipping Foxglove extension install (OSMO_FOXGLOVE_SKIP_EXTENSIONS=1)." + fi + + log_info "osmo workflow port-forward ${wf} workspace --port ${OSMO_FOXGLOVE_PORT} --connect-timeout ${OSMO_PF_TIMEOUT}" + log_info "Then in Foxglove Desktop: Open connection → ws://localhost:8766" + log_info " Layouts → Import from file → ${ext_src}/airstack_default.json" + log_info " (Restart Foxglove Desktop once if newly-installed panels still show as 'Unknown panel type'.)" + _osmo_run_port_forward "$wf" workspace \ + --port "$OSMO_FOXGLOVE_PORT" \ + --connect-timeout "$OSMO_PF_TIMEOUT" +} + +# osmo:down — cancel the active workflow. Reminds you to push first. +function cmd_osmo_down { + _osmo_check_cli || return 1 + local wf; wf="$(_osmo_wf_id)" || return 1 + + log_warn "About to cancel workflow '${wf}'." + log_warn "Anything not pushed to git in /root/AirStack inside the pod will be LOST." + log_warn "Hit Ctrl-C in the next 5 seconds to abort." + sleep 5 + osmo workflow cancel "$wf" + rm -f "${OSMO_STATE_FILE}" +} + +# Register commands from this module. +function register_osmo_commands { + COMMANDS["osmo:setup"]="cmd_osmo_setup" + COMMANDS["osmo:up"]="cmd_osmo_up" + COMMANDS["osmo:logs"]="cmd_osmo_logs" + COMMANDS["osmo:ide"]="cmd_osmo_ide" + COMMANDS["osmo:webrtc"]="cmd_osmo_webrtc" + COMMANDS["osmo:foxglove"]="cmd_osmo_foxglove" + COMMANDS["osmo:down"]="cmd_osmo_down" + + COMMAND_HELP["osmo:setup"]="One-time per-user OSMO credential setup (airlab-docker-registry, airlab-docker-login, airlab-nucleus)" + COMMAND_HELP["osmo:up"]="Submit osmo/workflows/airstack-dev.yaml with your SSH pubkey injected (--pool POOL, --key PATH, --branch BRANCH)" + COMMAND_HELP["osmo:logs"]="Follow the workspace task logs (osmo workflow logs -t workspace -n 500; OSMO_LOGS_TASK / OSMO_LOGS_TAIL override)" + COMMAND_HELP["osmo:ide"]="Port-forward sshd (2200:22) and open VS Code/Cursor on Host airstack-osmo" + COMMAND_HELP["osmo:webrtc"]="Port-forward Isaac Sim WebRTC ranges (TCP foreground + UDP background)" + COMMAND_HELP["osmo:foxglove"]="Install AirStack Foxglove extensions locally, then port-forward GCS Foxglove websocket (8766:8766). Override target dir with OSMO_FOXGLOVE_EXT_DIR; skip install with OSMO_FOXGLOVE_SKIP_EXTENSIONS=1." + COMMAND_HELP["osmo:down"]="Cancel the active workflow (push to git before running this)" +} diff --git a/.env b/.env index d16b8c649..aab86845e 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.0" +VERSION="0.19.0-alpha.1" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. @@ -53,4 +53,4 @@ DEBUG_RVIZ="false" # "true" or "false". If true, launches RViz alongside the ro # offboard API streaming out. this is so that ports don't conflict for multi-agent FCU communication. OFFBOARD_BASE_PORT=14540 -ONBOARD_BASE_PORT=14580 +ONBOARD_BASE_PORT=14580 \ No newline at end of file diff --git a/.gitignore b/.gitignore index a5776557c..4868b5c74 100644 --- a/.gitignore +++ b/.gitignore @@ -99,3 +99,5 @@ common/rayfronts/ # Docker build cache (root-owned subdirs cause permission warnings on `git add`) robot/docker/cache/ +.DS_Store +gcs/.DS_Store diff --git a/airstack.sh b/airstack.sh index 3d1e955e3..78b475c07 100755 --- a/airstack.sh +++ b/airstack.sh @@ -5,6 +5,47 @@ # This script provides a unified interface for common development tasks # in the AirStack project, including setup, installation, and container management. +# Re-exec under bash 4+ if necessary. macOS ships bash 3.2 which can't handle +# `declare -A` (associative arrays) used throughout this script. Searches for +# a newer bash via $AIRSTACK_BASH, then common Homebrew install paths, then +# any `bash` on PATH that reports version >= 4. Sets AIRSTACK_REEXEC_BASH=1 +# to guard against infinite re-exec loops. +if [ -z "${AIRSTACK_REEXEC_BASH:-}" ] && [ "${BASH_VERSINFO[0]:-0}" -lt 4 ]; then + _airstack_candidates=( + "${AIRSTACK_BASH:-}" + /opt/homebrew/bin/bash # Apple Silicon Homebrew + /usr/local/bin/bash # Intel Homebrew + /opt/local/bin/bash # MacPorts + ) + if command -v bash5 >/dev/null 2>&1; then + _airstack_candidates+=("$(command -v bash5)") + fi + # Add any `bash` on PATH whose version is >= 4 (other than the one we just + # got here from, which is < 4 by the if-check above). + for _alt in $(command -v -a bash 2>/dev/null); do + _airstack_candidates+=("$_alt") + done + + for _airstack_alt_bash in "${_airstack_candidates[@]}"; do + [ -z "$_airstack_alt_bash" ] && continue + [ -x "$_airstack_alt_bash" ] || continue + # Probe BASH_VERSINFO[0] without sourcing the script. + if "$_airstack_alt_bash" -c '[ "${BASH_VERSINFO[0]:-0}" -ge 4 ]' 2>/dev/null; then + export AIRSTACK_REEXEC_BASH=1 + exec "$_airstack_alt_bash" "$0" "$@" + fi + done + + cat >&2 <<'EOF' +[ERROR] airstack.sh requires bash 4 or newer (your bash is 3.x). + macOS ships bash 3.2 by default; install a modern bash with: + brew install bash + Or set AIRSTACK_BASH=/path/to/bash >= 4 before invoking this script. +EOF + exit 1 +fi +unset AIRSTACK_REEXEC_BASH + set -e # Script directory diff --git a/docs/getting_started/index.md b/docs/getting_started/index.md index e7bb64235..3b319ec00 100644 --- a/docs/getting_started/index.md +++ b/docs/getting_started/index.md @@ -1,5 +1,12 @@ # Getting Started +!!! tip "On Mac, Windows, or no GPU?" + + This page assumes a Linux desktop with an NVIDIA GPU. If that's not you, + use [AirStack on OSMO](../tutorials/airstack_on_osmo.md) instead — you + only need an SSH key, the `osmo` CLI, and VS Code or Cursor. No local + Docker, no NVIDIA drivers, no `airstack install`. + !!! warning "" AirStack is currently in ALPHA and only meant for internal usage. diff --git a/docs/tutorials/airstack_on_osmo.md b/docs/tutorials/airstack_on_osmo.md new file mode 100644 index 000000000..e9cfa8974 --- /dev/null +++ b/docs/tutorials/airstack_on_osmo.md @@ -0,0 +1,591 @@ +# AirStack on OSMO — Recommended Remote Development Workflow + +This is AirStack's recommended day-to-day development path going forward. +You submit one OSMO workflow that spins up a GPU pod running the full +three-container AirStack stack (Isaac Sim, robot-desktop, GCS), attach VS +Code or Cursor to it over Remote-SSH, and stream Isaac Sim and the GCS +Foxglove dashboard back to your browser. + +Why this is the recommended path: + +- **Pooled GPUs.** A lab's GPUs are shared on-demand across the whole team + instead of pinned one-per-desktop. Onboarding doesn't require buying + hardware. +- **No local CUDA / Docker / driver maintenance.** Your laptop just needs + `git`, an SSH key, and an IDE. macOS, Windows, and Linux all work + identically. +- **Same image as CI and field robots.** The OSMO pod runs the exact + Docker images that the system tests and deployed robots run, so your + dev environment can't drift away from production. +- **One-command onboarding.** A new student goes from zero to "Isaac Sim + streaming into my browser" with `airstack osmo:setup` followed by + `airstack osmo:up` — no install marathon. +- **Hardware bigger than your laptop.** The pod has more CPU/RAM/GPU than + most dev laptops, even if you have a GPU laptop. + +> **Still want local development on a Linux+GPU desktop?** It works and +> can be faster for tight inner loops — see +> [Getting Started](../getting_started/index.md). It just isn't the +> recommended default anymore. + +## Who is this for? + +Anyone developing AirStack — Mac, Windows, or Linux, with or without a +local GPU. + +You're comfortable using `git` from a terminal, you have an SSH key +(`~/.ssh/id_ed25519` or similar), and you have either VS Code or Cursor +installed. That's the entire local-machine bar. + +## Architecture in a sentence + +`airstack osmo:up` (which wraps `osmo workflow submit`) spins up a GPU pod +that runs sshd plus a Docker-in-Docker daemon. Inside that pod, `airstack +up` brings up the familiar three AirStack containers (Isaac Sim, +robot-desktop, GCS). Your IDE attaches over Remote-SSH; Isaac Sim and +Foxglove are reached via separate port-forwards. + +```mermaid +flowchart LR + subgraph laptop [Your laptop] + ide[VS Code or Cursor + Remote-SSH] + osmo[osmo CLI] + fox[app.foxglove.dev] + webrtc[Isaac Sim WebRTC client] + end + subgraph pod [OSMO workspace pod - GPU] + sshd[sshd] + inner[Inner dockerd] + isaac[isaac-sim container] + robot[robot-desktop container] + gcs[gcs container] + end + osmo -- submit and port-forward --> pod + ide -- ssh on 2200 --> sshd + fox -- ws on 8766 --> gcs + webrtc -- "WebRTC on 49100/tcp, 49099/udp" --> isaac + inner --> isaac + inner --> robot + inner --> gcs +``` + +## Prerequisites + +| You need | Why | +|---|---| +| A local clone of AirStack (`git clone https://github.com/castacks/AirStack.git`) | The `airstack osmo:*` wrappers, the workflow YAML, and the Foxglove extensions all live in the repo | +| The [`osmo` CLI](https://github.com/NVIDIA/OSMO) on your `PATH` | Submitting workflows and port-forwarding | +| `osmo login` done once | Stores your auth token in `~/.config/osmo` | +| An SSH keypair (e.g. `~/.ssh/id_ed25519`) | The pod authorises your pubkey at submit time. Generate one with `ssh-keygen -t ed25519` if you don't already have one. | +| **VS Code with the Remote-SSH extension** *or* **Cursor with its Remote-SSH equivalent** | Where you'll actually edit AirStack code | +| Optional: Foxglove desktop app, or just `app.foxglove.dev` | View ROS topics | +| Optional: an Omniverse Streaming Client / WebRTC browser client | View the streamed Isaac Sim render | + +You **do not** need: Docker, NVIDIA drivers, `airstack install`, `airstack +setup`, sudo, or Linux. + +> **Lab admin prerequisites (someone else's job, once).** A lab admin +> pushes the `airstack-osmo-workspace` image to +> `airlab-docker.andrew.cmu.edu`. Details in +> [`osmo/README.md`](https://github.com/castacks/AirStack/blob/main/osmo/README.md). +> +> **Your job, once:** the next step. + +## Step 0 — Register your OSMO credentials (one time) + +OSMO credentials are **per-user** (each Andrew ID has its own Nucleus token, +its own AirLab Docker password, its own OSMO profile). You register them +once with the `osmo` CLI on your laptop and OSMO injects them into every +workflow you submit afterwards. They never leave your OSMO profile and your +laptop never sees the values again. + +You need three credentials. The exact names matter — the workflow YAML +references them by these exact names. + +From your AirStack clone, run: + +```bash +git clone https://github.com/castacks/AirStack.git +cd AirStack +./airstack.sh osmo:setup +``` + +This prompts for your Andrew ID, AirLab Docker password, and Nucleus API +token (get one at → +right-click cloud icon → **API Tokens** → Create), then registers the +three credentials with OSMO. The values go directly to your OSMO profile +— nothing is written to local disk. + +> **macOS prereq: bash 4+.** macOS ships bash 3.2 by default and the +> `airstack` CLI needs bash 4+. If you see +> `airstack.sh requires bash 4 or newer`, install a modern bash with: +> +> ```bash +> brew install bash +> ``` +> +> No further config needed — `airstack.sh` auto-detects the Homebrew bash +> at `/opt/homebrew/bin/bash` (Apple Silicon) or `/usr/local/bin/bash` +> (Intel) and re-execs under it. You don't need to change your login shell. + +### Verify + +List your credentials: + +```bash +osmo credential list +``` + +You should see all three (`airlab-docker-registry`, `airlab-docker-login`, +`airlab-nucleus`). To rotate any of them later, just re-run +`./airstack.sh osmo:setup`. + +
+Under the hood — the three raw `osmo credential set` calls + +`airstack osmo:setup` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_setup`) is equivalent to running these three commands by hand +— useful for debugging or rotating one credential at a time: + +```bash +# 1. AirLab Docker registry (REGISTRY) — for OSMO's outer image-pull of +# airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace +osmo credential set airlab-docker-registry \ + --type REGISTRY \ + --payload registry=airlab-docker.andrew.cmu.edu \ + username= \ + auth='' + +# 2. AirLab Docker login (GENERIC) — for the *inner* dockerd inside the +# pod to `docker login` and pull the AirStack image set +osmo credential set airlab-docker-login \ + --type GENERIC \ + --payload username= \ + password='' + +# 3. AirLab Nucleus (GENERIC) — for Isaac Sim to authenticate against +# omniverse://airlab-nucleus.andrew.cmu.edu (API token, NOT password) +osmo credential set airlab-nucleus \ + --type GENERIC \ + --payload omni_user= \ + omni_pass='' \ + omni_server=omniverse://airlab-nucleus.andrew.cmu.edu/NVIDIA/Assets/Isaac/5.1 +``` + +
+ +> **Why three credentials?** It's tempting to consolidate. The reason for +> the split: OSMO REGISTRY credentials drive Kubernetes `imagePullSecrets` +> (auto-attached, never exposed as env vars), while GENERIC credentials are +> what get injected as env vars inside the running container. The pod +> needs **both** kinds of access — outer pull of the workspace image, plus +> inner login from the inner dockerd to pull AirStack images. + +## Step 1 — Add an SSH config entry (one time) + +VS Code and Cursor's Remote-SSH "Connect to Host…" picker reads +`~/.ssh/config`. Add this block once and the host shows up by name forever: + +```bash +cat >> ~/.ssh/config <<'EOF' + +Host airstack-osmo + HostName localhost + Port 2200 + User root + # Every OSMO workflow boots a fresh pod with a fresh sshd host key, so + # any saved fingerprint for [localhost]:2200 will be wrong on the next + # `airstack osmo:up`. Skip the host-key check here: this alias only + # connects via the local port-forward, so the security boundary is + # OSMO's authenticated control-plane tunnel — not the SSH fingerprint. + # /dev/null keeps known_hosts clean (no stale entries pile up); LogLevel + # ERROR silences the "Permanently added [localhost]:2200" banner. + StrictHostKeyChecking no + UserKnownHostsFile /dev/null + LogLevel ERROR + # SSH agent forwarding so `git push` from inside the pod uses your + # local laptop's SSH key (the pod's sshd has AllowAgentForwarding yes + # baked in by osmo/workspace/sshd_config). Without this, the pod has + # no key to push to github.com with — its ~/.ssh/ only holds the + # authorized_keys file for inbound connections. + ForwardAgent yes + # macOS Keychain integration — first push from the pod auto-loads + # your key into the local ssh-agent and unlocks it via the system + # keychain (no passphrase prompts). Harmless on Linux: those clients + # ignore the option. AddKeysToAgent works on both OSes. + AddKeysToAgent yes + UseKeychain yes +EOF +``` + +The `localhost:2200` is what we'll port-forward to in step 4. + +> **Already added the old block?** If your `~/.ssh/config` still has +> `StrictHostKeyChecking accept-new` for `airstack-osmo` from an earlier +> setup, replace it with the three lines above. As a one-time cleanup of +> the stale fingerprint left behind by previous pods, also run: +> +> ```bash +> ssh-keygen -R "[localhost]:2200" +> ``` +> +> `airstack osmo:ide` does this scrub for you on every run, so you only +> need it once when migrating. + +> **Smoke-test the agent forward** once the pod is up: SSH in and run +> `ssh-add -l` — you should see your local key listed. If you see "The +> agent has no identities", run `ssh-add ~/.ssh/id_ed25519` on your +> laptop and reconnect. + +## Step 2 — Submit the workflow + +From the AirStack clone: + +```bash +./airstack.sh osmo:up --pool airstack +``` + +This submits +[`osmo/workflows/airstack-dev.yaml`](https://github.com/castacks/AirStack/blob/main/osmo/workflows/airstack-dev.yaml) +with two things injected: + +- your local SSH pubkey as `SSH_PUB_KEY` — that's what authorises + **your** key on **this** workflow (each student passes their own at + submit time; the lab admin doesn't manage a global `authorized_keys` + file). +- `AIRSTACK_BRANCH` set to your local repo's current branch — the pod + ignores your laptop's working tree (it's ephemeral and runs in a + different machine room) and clones AirStack fresh from GitHub on + every workflow start, so this is how it knows which branch to use. + Override with `--branch main` if you want the pod to track main even + while you're on a feature branch. + +> **The pod clones from GitHub, not your laptop.** Local edits (and +> commits you haven't pushed) won't make it into the pod. `airstack +> osmo:up` warns you up-front if your branch is ahead of origin or has +> uncommitted changes — `git push` first if you want the pod to pick +> them up. + +`airstack osmo:up` prints a workflow id like `airstack-dev-1` and stores +it in `~/.airstack/osmo-state`, so the rest of the `airstack osmo:*` +commands in this tutorial pick it up automatically — no `export WF=...` +needed. To target a specific workflow for a single invocation, export +`AIRSTACK_OSMO_WF=`. + +
+Under the hood — raw `osmo workflow submit` + +`airstack osmo:up` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_up`) is equivalent to: + +```bash +osmo workflow submit osmo/workflows/airstack-dev.yaml \ + --pool airstack \ + --set-env "SSH_PUB_KEY=$(cat ~/.ssh/id_ed25519.pub)" +``` + +Save the printed workflow id as `$WF` if you're using the raw form, and +substitute it for `airstack osmo:*` in the rest of the tutorial. + +
+ +## Step 3 — Wait for the stack to come up + +Tail the lead task's logs and watch for milestones: + +```bash +./airstack.sh osmo:logs +``` + +Expected milestones, in order (each is one line in the log): + +1. `[entrypoint] sshd listening on :22` — VS Code/Cursor can attach. +2. `[entrypoint] dockerd ready` — the inner Docker daemon is up. +3. `Successfully built airstack_isaac-sim` *(or `Pulled` if pre-built)* — + the image set is in place. +4. `isaac-sim-livestream ... started` +5. `airstack-robot-desktop-1 ... started` +6. `airstack-gcs-1 ... started` + +If step (1) appears, you can attach the IDE while the rest is still +spinning up — the bring-up will continue in the background. + +
+Under the hood — raw `osmo workflow logs` + +`airstack osmo:logs` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_logs`) just exec's: + +```bash +osmo workflow logs $WF -t workspace -n 500 +``` + +The `osmo` CLI's `workflow logs` command prints the last N lines and then +keeps the stream open as new lines arrive (it already behaves like `tail +-f`, even though `--help` only documents `-n LAST_N_LINES`). Ctrl+C to +stop. Override the task / tail length with `OSMO_LOGS_TASK` / +`OSMO_LOGS_TAIL` env vars. + +
+ +## Step 4 — Forward sshd and attach the IDE + +In one terminal, run: + +```bash +./airstack.sh osmo:ide +``` + +This (a) starts the `localhost:2200 → pod:22` port-forward with a 24h +connect-timeout (matching the workflow's `exec_timeout`), waits for the +tunnel to come up, then (b) launches Cursor or VS Code (whichever it +finds on `PATH`) pre-attached to +`vscode-remote://ssh-remote+airstack-osmo/root/AirStack`. **Leave the +terminal running** for the length of your session — closing it tears the +tunnel down. + +The IDE installs its remote server in the pod on first connect (~50 MB, +slower on a fresh pod, cached on subsequent connects). Then: + +1. The IDE should open `/root/AirStack` automatically. (If not: + **Open Folder…** → `/root/AirStack`.) +2. Open the integrated terminal — you're root in `/root/AirStack`. +3. Edit code in the IDE; the changes land directly on the pod's disk. + +Verify everything is wired up by running: + +```bash +docker ps +``` + +You should see four containers: `airstack-isaac-sim-livestream-1`, +`airstack-robot-desktop-1`, `airstack-gcs-1`, plus the AirStack CLI helper. + +
+Under the hood — raw port-forward + manual IDE attach + +`airstack osmo:ide` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_ide`) is equivalent to running the port-forward by hand: + +```bash +osmo workflow port-forward $WF workspace --port 2200:22 --connect-timeout 86400 +``` + +…then in the editor: + +- **VS Code:** Command Palette → **Remote-SSH: Connect to Host…** → pick + `airstack-osmo`. +- **Cursor:** the same flow under its remote-development menu. + +Add `--no-open` to `airstack osmo:ide` to only run the port-forward and +attach the IDE manually. + +
+ +## Step 5 — Pick a feature branch and start working + +The pod cloned `main` into `/root/AirStack` on startup. Treat it like any +git working tree: + +```bash +git checkout -b my-feature +# edit code in the IDE... +bws --packages-select # build inside the robot-desktop container per AGENTS.md +``` + +Standard ROS 2 commands work from the integrated terminal: + +```bash +docker exec airstack-robot-desktop-1 bash -c "ros2 node list" +docker exec airstack-robot-desktop-1 bash -c "ros2 topic hz /robot_1/odometry" +``` + +This is the same `docker exec` pattern documented in +[AGENTS.md](https://github.com/castacks/AirStack/blob/main/AGENTS.md) — the +fact that you're on a remote pod is invisible from inside the IDE. + +## Step 6 — View Isaac Sim (WebRTC livestream) + +Isaac Sim runs headless inside the pod with the Kit +`omni.kit.livestream.webrtc` extension enabled (configured by the +`isaac-sim-livestream` Compose profile). To view it locally: + +```bash +./airstack.sh osmo:webrtc +``` + +This spawns the UDP port-forward (media, `49099`) in the background and +runs the TCP port-forward (signaling, `49100`) in the foreground — leave +that terminal running. + +Then point the **Omniverse Streaming Client** (or a WebRTC-capable browser +client) at `http://localhost`. The simulation viewport shows up the same +way it would on a local Linux desktop. + +
+Under the hood — raw TCP + UDP port-forwards + +`airstack osmo:webrtc` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_webrtc`) is equivalent to running the two raw port-forwards +in separate terminals — Kit's WebRTC needs both TCP signaling and UDP +SRTP media, and the AirStack workflow pins both to single ports rather +than scanning the Kit default range: + +```bash +# Terminal A — TCP signaling (49100): +osmo workflow port-forward $WF workspace --port 49100 --connect-timeout 86400 + +# Terminal B — UDP media (49099, pinned by the Pegasus launch script): +osmo workflow port-forward $WF workspace --port 49099 --udp --connect-timeout 86400 +``` + +
+ +## Step 7 — View ROS topics in Foxglove + +The GCS container runs `foxglove_bridge` on container-port `8765`, +published as host-port `8766` on the workspace pod. To install the +AirStack Foxglove extensions locally and forward the websocket in one +step: + +```bash +./airstack.sh osmo:foxglove +``` + +This copies the AirStack Foxglove extensions (Robot Tasks, Waypoint +Editor, Polygon Editor) into your local Foxglove Desktop user-extensions +dir (default `~/.foxglove-studio/extensions`; override with +`OSMO_FOXGLOVE_EXT_DIR`, skip with `OSMO_FOXGLOVE_SKIP_EXTENSIONS=1` for +`app.foxglove.dev` which doesn't load local extensions), then runs the +`localhost:8766 → pod:8766` port-forward in the foreground — leave the +terminal running. + +Then in [https://app.foxglove.dev](https://app.foxglove.dev) (or Foxglove +Desktop): + +1. **Open connection** → `ws://localhost:8766`. +2. **Layouts** → **Import from file** → + [`gcs/foxglove_extensions/airstack_default.json`](https://github.com/castacks/AirStack/blob/main/gcs/foxglove_extensions/airstack_default.json) + from your AirStack clone. +3. Pick the imported layout from the layout dropdown in the top-right. + +The full Foxglove flow — layout import, panel customisation, DDS bridge +naming — is documented at +[Foxglove Visualization](../gcs/foxglove.md). The only OSMO-specific +difference is the `osmo:foxglove` line in front of it. + +
+Under the hood — raw `osmo workflow port-forward` + +`airstack osmo:foxglove` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_foxglove`) wraps the extension install plus: + +```bash +osmo workflow port-forward $WF workspace --port 8766:8766 --connect-timeout 86400 +``` + +Set `OSMO_FOXGLOVE_SKIP_EXTENSIONS=1` to only run the port-forward. + +
+ +## Step 8 — Commit and push from inside the IDE + +The pod's filesystem is **ephemeral**. The persistence boundary is git, not +disk. Commit and push every meaningful chunk of work — a Source Control +panel commit + push, or in the integrated terminal: + +```bash +git add -A +git commit -m "WIP: feature X" +git push -u origin my-feature +``` + +Once your branch is on the remote, you can pull it from anywhere — your +laptop, a fresh pod tomorrow, a colleague's machine. + +> **Configuring git auth in the pod.** The pod is yours for the session. +> Inside the IDE's integrated terminal, set `git config user.name`, +> `user.email`, and configure your push auth (HTTPS + a GitHub PAT, or a +> per-pod SSH key the IDE forwards via `AllowAgentForwarding yes`). The +> `airstack-osmo-workspace` image deliberately does not bake any one +> student's git creds. + +## Step 9 — Tearing down + +When you're done: + +```bash +./airstack.sh osmo:down +``` + +This prints a 5-second warning then cancels the workflow stored in +`~/.airstack/osmo-state`. Hit Ctrl-C in the grace window if you submitted +by accident. + +> **Push first.** Anything that's still in your working tree, in `.git/` +> but not pushed, in `build/`, in `bags/`, or in `/root/` outside the repo +> **will be lost** on cancel. The pod is cattle. If you forget and need +> something pulled out, see "I forgot to push before tearing down" below +> *before* hitting cancel. + +
+Under the hood — raw `osmo workflow cancel` + +`airstack osmo:down` (defined in +[`.airstack/modules/osmo.sh`](https://github.com/castacks/AirStack/blob/main/.airstack/modules/osmo.sh) +as `cmd_osmo_down`) is equivalent to: + +```bash +osmo workflow cancel $WF +``` + +
+ +## Troubleshooting + +| Symptom | Likely cause | Fix | +|---|---|---| +| `Remote-SSH: Connection refused` after a working session | Port-forward died (laptop slept, network blip) | Re-run `./airstack.sh osmo:ide` | +| `Permission denied (publickey)` on Remote-SSH | The pod authorised a different pubkey than the one your local SSH client is offering | Confirm `cat ~/.ssh/id_ed25519.pub` matches the key that was injected at submit time. Re-submit with `./airstack.sh osmo:down && ./airstack.sh osmo:up --pool airstack`. | +| `airstack osmo:logs` shows `ERROR: SSH_PUB_KEY not set` | The submit didn't inject a pubkey (e.g. you ran raw `osmo workflow submit` without `--set-env`) | `./airstack.sh osmo:down`, then resubmit with `./airstack.sh osmo:up --pool airstack` (it injects `SSH_PUB_KEY` automatically). | +| `docker pull` fails inside the pod with `unauthorized` | Your `airlab-docker-login` credential is missing or has the wrong Andrew ID/password | Re-run `./airstack.sh osmo:setup`. | +| Logs show `WARN: airlab-nucleus OSMO credential not set` and Isaac Sim asset loads fail, **or** Isaac Sim shows "Login Required: Unable to connect server omniverse://airlab-nucleus..." with the auth-service log showing `InternalCredentials.auth … 'username': '' … status: 'DENIED'` (no `Tokens.auth_with_api_token` call) | The pod is doing **password auth** instead of **API-token auth**. Inside the pod, `simulation/isaac-sim/docker/omni_pass.env` must have `OMNI_USER=$$omni-api-token` (literal `$$`, the sentinel for API-token auth — docker-compose v2 collapses `$$` to `$` on its way to the container). The OSMO entrypoint sets this automatically when `OMNI_PASS` looks like a JWT; if you see `OMNI_USER=` in the file, recreate the container with `docker compose --profile desktop --profile isaac-sim-livestream up -d isaac-sim-livestream` (`restart` does NOT re-read `env_file`). | +| Logs show `WARN: airlab-nucleus OSMO credential not set` and Isaac Sim asset loads fail, **or** Isaac Sim shows "Login Required: Unable to connect server omniverse://airlab-nucleus..." with the auth-service log showing `Tokens.auth_with_api_token … status: 'DENIED'` | Your `airlab-nucleus` API token is missing, expired, or revoked (rotation invalidates the predecessor). Confirm by SSH'ing the Nucleus host and running `sudo docker logs --tail 200 base_stack-nucleus-auth-1`. Regenerate the token at , then `./airstack.sh osmo:setup` and `./airstack.sh osmo:down && ./airstack.sh osmo:up --pool airstack` to resubmit (or live-edit `simulation/isaac-sim/docker/omni_pass.env` in the pod and recreate the `isaac-sim-livestream` container — see row above). | +| Isaac Sim container restarts repeatedly | GPU not visible to the inner Docker daemon (toolkit not configured on the node) | Lab admin task. From inside the pod: `docker info \| grep -i runtime` should list `nvidia`. | +| Isaac Sim is up but the WebRTC stream is blank | The Pegasus script isn't getting `--/app/livestream/enabled=true`, or the wrong Compose profile is active | In the integrated terminal: `docker logs airstack-isaac-sim-livestream-1`. Confirm `ISAAC_SIM_LIVESTREAM=true` and that the `isaac-sim-livestream` profile is the one running (`docker ps`). | +| Foxglove "no connection" | Port-forward died, GCS container hasn't started yet, or browser is caching an old connection | Re-run `./airstack.sh osmo:foxglove`; check `docker ps` shows `airstack-gcs-1` Up; try `ws://127.0.0.1:8766` instead of `ws://localhost:8766`. | +| First Remote-SSH connect takes forever | VS Code / Cursor downloading its remote server (~50 MB) into the fresh pod | Wait it out the first time. Subsequent connects to the same pod hit the cache. | +| **I forgot to push before tearing down** | The pod is still up; cancel hasn't fired yet | Don't run `./airstack.sh osmo:down`. SSH in via the existing port-forward (`./airstack.sh osmo:ide --no-open` if the tunnel is gone), push from the IDE terminal, *then* tear down. If the workflow has already terminated and the pod is gone, the work is gone — git is the only persistence layer. | + +## What survives `airstack osmo:down`? + +| Artifact | Lives in | Survives? | +|---|---|---| +| Code committed and pushed to a feature branch | GitHub | **Yes** | +| Code committed but not pushed | Pod-local `.git` | **No** | +| Uncommitted edits in the IDE | Pod-local working tree | **No** | +| `colcon build` outputs (`build/`, `install/`, `log/`) | `/root/AirStack/**/ros_ws/...` | **No** (gitignored Linux x86_64 binaries; rebuild trivially) | +| Inner-dockerd image cache | Pod-local Docker layer cache | **No** | +| Bag files, sim recordings, debug screenshots | `/root/AirStack/bags/`, etc. | **No** — pull selectively via `osmo workflow rsync download "$(cat ~/.airstack/osmo-state)" :` *before* tearing down | + +The rule of thumb: **commit + push every time you'd save a file in a +git-tracked sense.** The Source Control panel is the persistence boundary. + +## See also + +- [`osmo/README.md`](https://github.com/castacks/AirStack/blob/main/osmo/README.md) + — lab-admin reference (pool prerequisites, OSMO credential registration, + workspace image build, validation stages). +- [Foxglove Visualization](../gcs/foxglove.md) — full layout import + + panel-customisation flow once your `airstack osmo:foxglove` is up. +- [AGENTS.md](https://github.com/castacks/AirStack/blob/main/AGENTS.md) — + inside-the-pod workflow once you're attached: `bws`, `sws`, `docker exec`, + ROS 2 commands. +- [Getting Started](../getting_started/index.md) — the local-Linux-GPU + alternative. diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index d1606f4e3..08dbbdaee 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -5,6 +5,7 @@ Step-by-step guides for common AirStack workflows. If you are new, start with ** | Tutorial | Description | |---|---| | [Getting Started](../getting_started.md) | Install AirStack, pull Docker images, launch a simulated robot, and fly it for the first time. | +| [AirStack on OSMO (Mac/Windows OK)](airstack_on_osmo.md) | Develop on AirStack from a Mac, Windows, or no-GPU Linux laptop using NVIDIA OSMO + VS Code/Cursor Remote-SSH. No local Docker or local `airstack install`; use a local repo clone for the `airstack osmo:*` wrappers and workflow YAML. | | [Multi-Robot Simulation](multi_robot_simulation.md) | Spin up multiple simulated robots in Isaac Sim and verify independent ROS 2 namespaces. | | [Autonomy Modes](autonomy_modes.md) | Understand `onboard_all`, `onboard_local`, and `offboard_global` modes and the commands to run each. | | [Deploying to Hardware](deploying_to_hardware.md) | Flash a Jetson or VOXL device, configure the robot hostname, and run the autonomy stack on a real drone. | diff --git a/gcs/foxglove_extensions/install.py b/gcs/foxglove_extensions/install.py index f948cac54..fc28de102 100644 --- a/gcs/foxglove_extensions/install.py +++ b/gcs/foxglove_extensions/install.py @@ -1,11 +1,34 @@ #!/usr/bin/env python3 +""" +Install AirStack Foxglove extensions into a Foxglove user-extensions dir. + +By default this targets the GCS container's bundled Foxglove app +(/root/.foxglove-studio/extensions), which is the entrypoint that +gcs/docker/gcs-base-docker-compose.yaml runs on container start. + +The src/dst paths can be overridden via env vars, which is how the +`airstack osmo:foxglove` wrapper reuses this same script to install the +extensions into the laptop's local Foxglove Desktop app before +port-forwarding the GCS bridge — that way the laptop's Foxglove sees +"Robot Tasks" / "Waypoint Editor" / "Polygon Editor" instead of the +"Unknown panel type: ..." placeholders. + +Env vars: + FOXGLOVE_EXT_SRC directory containing the extension subdirectories + (each with a package.json + dist/extension.js) + FOXGLOVE_EXT_DST target user-extensions directory, e.g. + ~/.foxglove-studio/extensions on Linux/macOS. +""" + import json import os import re import shutil -src = '/root/AirStack/gcs/foxglove_extensions' -dst = '/root/.foxglove-studio/extensions' +src = os.environ.get( + 'FOXGLOVE_EXT_SRC', '/root/AirStack/gcs/foxglove_extensions') +dst = os.path.expanduser(os.environ.get( + 'FOXGLOVE_EXT_DST', '/root/.foxglove-studio/extensions')) os.makedirs(dst, exist_ok=True) @@ -13,11 +36,16 @@ def _slug(s: str) -> str: return re.sub(r'[^a-z0-9-]+', '-', s.lower()).strip('-') -for ext in os.listdir(src): +installed = 0 +for ext in sorted(os.listdir(src)): pkg_path = os.path.join(src, ext, 'package.json') if not os.path.exists(pkg_path): continue pkg = json.load(open(pkg_path)) name = '{}.{}-{}'.format(_slug(pkg['publisher']), pkg['name'], pkg['version']) shutil.copytree(os.path.join(src, ext), os.path.join(dst, name), dirs_exist_ok=True) - print('Installed Foxglove extension:', name) + print('Installed Foxglove extension:', name, '->', os.path.join(dst, name)) + installed += 1 + +if installed == 0: + print('No Foxglove extensions found under', src) diff --git a/mkdocs.yml b/mkdocs.yml index 00a1aee14..c4d92fede 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -52,6 +52,7 @@ nav: - Home: docs/index.md - Getting Started: - docs/getting_started/index.md + - docs/tutorials/airstack_on_osmo.md - docs/getting_started/tutorials_reference.md - Development: - docs/development/index.md diff --git a/osmo/README.md b/osmo/README.md new file mode 100644 index 000000000..91b41dbd5 --- /dev/null +++ b/osmo/README.md @@ -0,0 +1,301 @@ +# AirStack on OSMO + +This directory holds the bits that let students develop on AirStack remotely +through [NVIDIA OSMO](https://github.com/NVIDIA/OSMO): + +``` +osmo/ +├── README.md # This file (admin / operator reference) +├── workflows/ +│ └── airstack-dev.yaml # The OSMO workflow students submit +└── workspace/ + ├── Dockerfile # The airstack-osmo-workspace image + ├── sshd_config # Pubkey-only sshd config baked into the image + └── entrypoint.sh # Pod startup: sshd, dockerd, clone, airstack up +``` + +The student-facing walkthrough lives in +[`docs/tutorials/airstack_on_osmo.md`](../docs/tutorials/airstack_on_osmo.md) +— including the per-user **Step 0** for registering OSMO credentials. This +README is the **lab admin / operator** reference: pool requirements, +workspace image build & push, validation stages, plus a credential summary +for context. + +> **Scope:** developer workflow only. CI/CD on OSMO is **not** part of this +> integration — the existing `system-tests.yml` + OpenStack orchestrator path +> is unchanged. + +## Architecture in one minute + +A student submits one OSMO task that runs a Docker-in-Docker (DinD) pod with +sshd. Inside that pod, `airstack.sh up` brings up the regular +three-container AirStack stack (Isaac Sim, robot-desktop, GCS) on the inner +Docker daemon. The student attaches VS Code or Cursor over Remote-SSH and +streams Isaac Sim (WebRTC) and the GCS Foxglove bridge (websocket) back to +their laptop via `osmo workflow port-forward`. + +``` +Student laptop OSMO workspace pod (GPU) +───────────────── ───────────────────────────────────── +VS Code / Cursor ── ssh ──► port-forward 2200:22 ──► sshd +Isaac Sim WebRTC ── webrtc ► port-forward 47995… ──► inner isaac-sim ctnr +app.foxglove.dev ── ws ────► port-forward 8766 ────► inner gcs ctnr (8765) + ▲ + │ inner dockerd + │ (NVIDIA runtime) + │ + airstack.sh up brings these 3 up +``` + +## Pool requirements + +The OSMO pool the workflow runs on must satisfy: + +| Requirement | Why | +|---|---| +| GPU pool with NVIDIA driver + `nvidia-container-toolkit` on each node | Isaac Sim needs the GPU. The toolkit must be on the node so the inner `dockerd` (configured with `--add-runtime nvidia=...`, `default-runtime: nvidia`) can hand the device to the inner Isaac Sim container. | +| No NetworkPolicy blocking pod-namespace ports `47995–48012/tcp+udp`, `49099/udp`, `49100/tcp`, `8766/tcp`, `22/tcp` | These are the ports `osmo workflow port-forward` reaches inside the pod NS for Isaac Sim WebRTC, GCS Foxglove websocket, and sshd. | +| Resource limits ≥ `cpu: 16`, `memory: 64Gi`, `storage: 200Gi`, `gpu: 1` | Isaac Sim + AirStack images + `colcon build` working tree. Adjust upward if running multiple robots or heavy bag recording. | + +`hostNetwork: true` is **not** required. `osmo workflow port-forward` reaches +the pod's network namespace, which is where the inner `dockerd` publishes +ports via standard NAT (or `network_mode: host` on individual inner +containers, both of which terminate at the pod NS, not the cluster node). + +## OSMO credentials (per user, one time) + +OSMO credentials live in **each user's** OSMO profile, not in a lab-wide +store. Every student registers their own three credentials with `osmo +credential set` once on their laptop. The full walkthrough — including the +exact `osmo credential set ...` commands and how to obtain a Nucleus API +token — lives in +[`docs/tutorials/airstack_on_osmo.md` Step 0](../docs/tutorials/airstack_on_osmo.md#step-0--register-your-osmo-credentials-one-time). + +The three credentials, summarized for quick reference: + +| Name | Type | Used for | Referenced in workflow YAML? | +|---|---|---|---| +| `airlab-docker-registry` | `REGISTRY` | OSMO's automatic pull of the workspace image (`airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:...`) | No — OSMO auto-attaches it to any image whose hostname matches the credential's `registry=` field. | +| `airlab-docker-login` | `GENERIC` | `entrypoint.sh` calls `docker login airlab-docker.andrew.cmu.edu` on the **inner** dockerd before `airstack up`, so the inner Compose stack can pull AirStack images | Yes — exposed as env vars `AIRLAB_REGISTRY_USER`/`AIRLAB_REGISTRY_PASS`. | +| `airlab-nucleus` | `GENERIC` | `entrypoint.sh` materializes `simulation/isaac-sim/docker/omni_pass.env` from it so Compose can env-file it into the Isaac Sim container | Yes — exposed as env vars `OMNI_USER`/`OMNI_PASS`/`OMNI_SERVER`. | + +The convenience helper `airstack osmo:setup` in +[`.airstack/modules/osmo.sh`](../.airstack/modules/osmo.sh) prompts for the +underlying values (Andrew ID, AirLab password, Nucleus API token) and runs +all three `osmo credential set` commands. + +> **Why a `REGISTRY` and a `GENERIC` credential for the same registry?** +> OSMO `REGISTRY` credentials drive Kubernetes `imagePullSecrets` — +> auto-attached but not exposed to the container as env vars. The +> **inner** dockerd (DinD) that `entrypoint.sh` starts is a separate +> Docker daemon and needs its own `docker login`. Hence the two-credential +> split. + +## Build & push the workspace image + +The workspace image is built once and pushed to the AirLab registry; students +never build it themselves. + +> **Always use `docker buildx build --platform linux/amd64 --push`.** +> OSMO pool workers are linux/amd64. Building with plain `docker build` on +> an Apple Silicon Mac silently produces a `linux/arm64` image and the +> resulting `latest` tag will fail every workflow with +> `no match for platform in manifest ...: not found` (the outer pod's +> image-pull bails before the entrypoint even runs). Forcing `--platform +> linux/amd64` cross-compiles for amd64 even on an arm64 host. `--push` +> is required because buildx cross-platform builds can't be loaded into a +> local Docker daemon — they live only in the build cache or the +> registry. Linux/amd64 admins can use plain `docker build && docker push`. + +```bash +cd osmo/workspace + +# One-time builder setup (skip if `docker buildx ls` already shows a builder): +docker buildx create --use --name airstack-builder + +docker buildx build \ + --platform linux/amd64 \ + -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest \ + --push \ + . +``` + +Verify the manifest has `linux/amd64` after pushing: + +```bash +docker manifest inspect airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest \ + | grep -A2 architecture +# → "architecture": "amd64" +``` + +Tag a versioned release alongside `latest` if you change anything in +`Dockerfile`, `sshd_config`, or `entrypoint.sh`: + +```bash +docker buildx build \ + --platform linux/amd64 \ + -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest \ + -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:v0.1.0 \ + --push \ + . +``` + +Then update the `image:` field in +[`workflows/airstack-dev.yaml`](workflows/airstack-dev.yaml) to match. + +The image bakes: + +- Ubuntu 24.04 base with `docker-ce`, `docker-compose-plugin`, `nvidia-container-toolkit` +- `git`, `python3`, `curl` +- `openssh-server` with **password auth permanently disabled** (pubkey only) via the baked `sshd_config` +- The AirStack `airstack.sh` CLI script on `PATH` + +The image does **not** bake the AirStack source tree. `entrypoint.sh` clones +it on first start (and skips re-cloning across pod restarts). + +## Validation stages + +Run these in order against a fresh submission. Each unlocks the next; if (a) +fails don't bother trying (b). + +### (a) sshd reachable, key auth works + +```bash +osmo workflow submit osmo/workflows/airstack-dev.yaml \ + --pool \ + --set-env "SSH_PUB_KEY=$(cat ~/.ssh/id_ed25519.pub)" +# → record + +osmo workflow port-forward workspace --port 2200:22 --connect-timeout 86400 & +# StrictHostKeyChecking=no + UserKnownHostsFile=/dev/null because every +# fresh pod has a different sshd host key — the previous workflow's +# fingerprint will always look like a "host key changed" attack +# otherwise. +ssh -p 2200 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \ + root@localhost 'echo ok && whoami' +# → "ok\nroot" +``` + +If SSH fails: check `osmo workflow logs workspace` for the +`SSH_PUB_KEY not set` error or for `sshd` failing to start. + +### (b) VS Code / Cursor Remote-SSH attaches and opens `/root/AirStack` + +Add to `~/.ssh/config`: + +``` +Host airstack-osmo + HostName localhost + Port 2200 + User root + # Each fresh pod has a new sshd host key, so accept-new doesn't help + # — the second workflow always trips the "host key changed" check. + # Bypass host-key checks for this loopback alias only; the security + # boundary is OSMO's authenticated port-forward, not the local key. + StrictHostKeyChecking no + UserKnownHostsFile /dev/null + LogLevel ERROR +``` + +Then in VS Code: Command Palette → **Remote-SSH: Connect to Host…** → +`airstack-osmo` → open folder `/root/AirStack`. The IDE will install its +remote server in the pod on first connect (~50 MB download, slow on a fresh +pod; cached afterwards). + +### (c) `airstack up` brings the three containers Up + +In the IDE's integrated terminal (or `osmo workflow exec`): + +```bash +docker ps +# → expect: airstack-isaac-sim-1, airstack-robot-desktop-1, airstack-gcs-1 +``` + +If any container is missing or restarting, the most common causes (in order): + +1. The user's `airlab-docker-login` GENERIC credential is wrong / unset → + inner `docker pull` from `airlab-docker.andrew.cmu.edu` failed. + Re-run `airstack osmo:setup` (or the explicit `osmo credential set + airlab-docker-login ...` command in the tutorial Step 0). +2. `nvidia-container-toolkit` is not configured on the node → inner Isaac Sim + can't see the GPU. Check `docker info | grep -i runtime` inside the + workspace pod; you should see `nvidia` in the runtime list. +3. The pod ran out of `storage:` quota during the image pull. Bump it. + +### (d) Isaac Sim WebRTC client renders + +Two port-forwards (TCP + UDP): + +```bash +osmo workflow port-forward workspace \ + --port 47995-48012,49000-49007,49100 --connect-timeout 86400 & +osmo workflow port-forward workspace \ + --port 47995-48012,49000-49007,49099 --udp --connect-timeout 86400 & +``` + +Open the Omniverse Streaming Client (or a browser WebRTC client) at +`http://localhost`. + +If the stream is blank: check that the Pegasus standalone script was launched +with `--/app/livestream/enabled=true`. The +[`isaac-sim-livestream`](../simulation/isaac-sim/docker/docker-compose.yaml) +Compose profile is what wires that argument; verify the workflow YAML has +`ISAAC_SIM_LIVESTREAM=true` in `environment:`. + +### (e) Foxglove websocket loads the AirStack layout + +```bash +osmo workflow port-forward workspace --port 8766:8766 --connect-timeout 86400 & +``` + +Open [https://app.foxglove.dev](https://app.foxglove.dev) → **Open +connection** → `ws://localhost:8766` → **Layouts** → **Import from file** → +[`gcs/foxglove_extensions/airstack_default.json`](../gcs/foxglove_extensions/airstack_default.json). + +The wider Foxglove layout / panel-import flow is documented in +[`docs/gcs/foxglove.md`](../docs/gcs/foxglove.md); the only OSMO-specific +piece is the `port-forward` line in front of it. + +## Nucleus connectivity from OSMO + +`airlab-nucleus.andrew.cmu.edu` runs the standard Omniverse Enterprise +Nucleus stack with TLS termination at its Ingress Router (NGINX) on **port +443**. Per [NVIDIA's TLS doc](https://docs.omniverse.nvidia.com/nucleus/latest/enterprise/installation/tls.html), +clients only need outbound TCP **443** — the Ingress Router path-based- +routes requests (`/omni/api`, `/omni/auth`, `/omni/lft`, `/omni/conn`, +`/omni/web3/...`) to the internal service ports (3009, 3100, 3030, 3019, +3400). Omniclient detects SSL/TLS and prefers it, so the OSMO pod (whose +egress allows 80/443/22) reaches Nucleus over the same single 443 the +Web3 navigator uses. **The native protocol ports 3009–3180 do NOT need to +be open from OSMO** as long as TLS is configured on the Nucleus side. + +If you see Isaac Sim's "Login Required" popup at startup: + +1. **Check the auth-service log on the Nucleus host** (`ssh + ubuntu@; sudo docker logs --tail 200 + base_stack-nucleus-auth-1`). Look for `InternalCredentials.auth: + {... 'username': ''} → status: 'DENIED'` lines. That + means the API token in your `airlab-nucleus` OSMO credential is + revoked, expired, or has whitespace/quoting damage. +2. **Regenerate the token** at + → right-click the + cloud icon → **API Tokens** → create a new one. +3. **Update the OSMO credential** with `airstack osmo:setup` (or the + raw `osmo credential set airlab-nucleus ...` command from the + tutorial Step 0) and **resubmit the workflow** so the new token + lands in `omni_pass.env` on pod boot. To live-patch a running pod + instead, edit `simulation/isaac-sim/docker/omni_pass.env` inside + the workspace and `docker compose --profile isaac-sim-livestream + restart isaac-sim-livestream`. + +## Out of scope (followups) + +- **OSMO-native split** — three separate OSMO tasks for `isaac-sim` / + `robot-desktop` / `gcs` instead of one DinD pod. Larger refactor of + Compose, DDS networking, and `tests/conftest.py`. The `osmo/workflows/` + layout leaves room for additional workflow files when this is done. +- **Persistent workspace** — mount `/root/AirStack` to a PVC so uncommitted + edits survive `osmo workflow cancel`. Pool-policy dependent. +- **CI/CD on OSMO** — the existing `.github/workflows/system-tests.yml` + + OpenStack ephemeral runner path is unchanged. Migrating CI to OSMO is a + separate effort. diff --git a/osmo/workflows/airstack-dev.yaml b/osmo/workflows/airstack-dev.yaml new file mode 100644 index 000000000..a38ee91e2 --- /dev/null +++ b/osmo/workflows/airstack-dev.yaml @@ -0,0 +1,99 @@ +# AirStack remote developer workflow on OSMO. +# +# Submits a single GPU task ("workspace") that runs Docker-in-Docker +# (DinD) and brings up the regular AirStack three-container stack (Isaac Sim +# with WebRTC livestream, robot-desktop, GCS) on the inner Docker daemon. The +# task also runs sshd so a student can attach VS Code or Cursor over Remote-SSH +# from their laptop (Mac, Windows, or Linux — no local Docker / NVIDIA driver +# required). +# +# To submit (replace and substitute your actual pubkey): +# +# osmo workflow submit osmo/workflows/airstack-dev.yaml \ +# --pool \ +# --set-env "SSH_PUB_KEY=$(cat ~/.ssh/id_ed25519.pub)" +# +# To stream the IDE, Isaac Sim, and Foxglove back to the laptop: +# +# # SSH so VS Code / Cursor Remote-SSH can attach (1 terminal): +# osmo workflow port-forward workspace --port 2200:22 --connect-timeout 86400 +# +# # Isaac Sim WebRTC (2 terminals — TCP + UDP): +# osmo workflow port-forward workspace \ +# --port 47995-48012,49000-49007,49100 --connect-timeout 86400 +# osmo workflow port-forward workspace \ +# --port 49099 --udp --connect-timeout 86400 +# +# # GCS Foxglove websocket (1 terminal): +# osmo workflow port-forward workspace --port 8766:8766 --connect-timeout 86400 +# +# See docs/tutorials/airstack_on_osmo.md for the full walkthrough and +# osmo/README.md for the lab-admin setup (pool prerequisites, OSMO credential +# registration, workspace image build). + +workflow: + name: airstack-dev + groups: # `groups:` keeps room to add a separate + # dind sidecar or split out isaac-sim / + # robot-desktop / gcs as their own + # tasks later. For now, one lead task. + - name: airstack + tasks: + - name: workspace + lead: true + image: airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest + # Required so the inner dockerd can run. hostNetwork is intentionally + # NOT set — osmo workflow port-forward reaches the pod NS, where the + # inner dockerd publishes ports via standard NAT. + privileged: true + command: ["bash"] + args: ["/usr/local/bin/entrypoint.sh"] + environment: + # Behaviour switches consumed by entrypoint.sh and airstack.sh: + AUTOLAUNCH: "true" # boot AirStack on startup + ISAAC_SIM_LIVESTREAM: "true" # use the isaac-sim-livestream profile + NUM_ROBOTS: "1" + AIRSTACK_BRANCH: "main" # branch entrypoint.sh clones + AIRSTACK_REPO_URL: "https://github.com/castacks/AirStack.git" + # SSH_PUB_KEY is supplied at submit time: + # --set-env "SSH_PUB_KEY=$(cat ~/.ssh/id_ed25519.pub)" + credentials: + # Each student registers these in their own OSMO profile once. + # See docs/tutorials/airstack_on_osmo.md "Step 0". + # + # airlab-nucleus (GENERIC) — materialized into omni_pass.env by + # entrypoint.sh so Compose env_files it into the Isaac Sim ctnr. + airlab-nucleus: + OMNI_USER: omni_user + OMNI_PASS: omni_pass + OMNI_SERVER: omni_server + # airlab-docker-login (GENERIC) — exposed as env vars so the + # inner dockerd can authenticate to airlab-docker.andrew.cmu.edu + # when `airstack up` triggers an AirStack image-pull. (The + # *outer* pod's image-pull of airstack-osmo-workspace is handled + # automatically by a sibling REGISTRY-type credential which does + # not need a reference here — OSMO auto-attaches it on submit.) + airlab-docker-login: + AIRLAB_REGISTRY_USER: username + AIRLAB_REGISTRY_PASS: password + + resources: + default: + cpu: 16 + gpu: 1 + memory: 64Gi + # AirStack image set is large: airstack-dev-9 (2026-05-14) hit the + # 100Gi container ephemeral cap during inner Compose's first image + # extract, before the second image even started downloading. So the + # full set of inner images alone exceeds 100Gi extracted. Going to + # 500Gi to comfortably hold isaac-sim + robot-desktop + gcs images, + # plus the AirStack source clone, colcon build output, and bag + # recordings. The airstack pool's workers have 4.2Ti of ephemeral + # capacity each (root disks resized 2026-05-14), so this leaves + # plenty of room for other tenants on the shared workers. + storage: 500Gi + + timeout: + # 8h covers a normal dev session. Bump for longer runs; cancel manually + # before the timeout if you want to free the GPU early. + exec_timeout: 8h diff --git a/osmo/workspace/Dockerfile b/osmo/workspace/Dockerfile new file mode 100644 index 000000000..e80f3be59 --- /dev/null +++ b/osmo/workspace/Dockerfile @@ -0,0 +1,112 @@ +# airstack-osmo-workspace +# +# Image used by the OSMO airstack-dev workflow. Boots into a Docker-in-Docker +# (DinD) pod with sshd on :22 so VS Code / Cursor Remote-SSH can attach. The +# inner dockerd then runs the regular AirStack docker-compose stack (Isaac +# Sim, robot-desktop, GCS) on the GPU forwarded into the pod. +# +# Built and pushed by the lab admin (see osmo/README.md): +# +# cd osmo/workspace +# docker buildx build --platform linux/amd64 \ +# -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest \ +# --push . +# +# Use `docker buildx build --platform linux/amd64 --push` (not plain +# `docker build && docker push`) so an Apple Silicon Mac doesn't silently +# push an arm64 image; OSMO workers are amd64 and would fail every +# workflow with "no match for platform in manifest". Students never +# build this image. + +FROM ubuntu:24.04 + +ENV DEBIAN_FRONTEND=noninteractive +ENV LANG=C.UTF-8 +ENV LC_ALL=C.UTF-8 + +# Base utilities + sshd + dev ergonomics. python3 is here so the airstack.sh +# CLI's helper scripts can shell out to python3 without extra installs. +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates \ + curl \ + git \ + git-lfs \ + gnupg \ + iproute2 \ + iptables \ + jq \ + less \ + locales \ + lsb-release \ + openssh-server \ + procps \ + python3 \ + python3-pip \ + rsync \ + sudo \ + tmux \ + tzdata \ + vim-tiny \ + wget \ + && locale-gen C.UTF-8 \ + && rm -rf /var/lib/apt/lists/* + +# Docker CE + Compose plugin (required for `airstack up` to work inside the +# pod). The same install procedure as get.docker.com but pinned to apt repos +# so we can upgrade explicitly. +RUN install -m 0755 -d /etc/apt/keyrings \ + && curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ + | gpg --dearmor -o /etc/apt/keyrings/docker.gpg \ + && chmod a+r /etc/apt/keyrings/docker.gpg \ + && echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ + https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \ + > /etc/apt/sources.list.d/docker.list \ + && apt-get update && apt-get install -y --no-install-recommends \ + docker-ce \ + docker-ce-cli \ + containerd.io \ + docker-buildx-plugin \ + docker-compose-plugin \ + fuse-overlayfs \ + && rm -rf /var/lib/apt/lists/* + +# Note: dockerd inside an OSMO/k8s pod is running on top of an overlayfs +# rootfs (the pod's own /). Docker refuses overlay2-on-overlayfs, so without +# fuse-overlayfs the entrypoint's storage-driver fallback chain lands on +# vfs, which has no copy-on-write and bloats the AirStack image set ~3x +# (airstack-dev-10, 2026-05-14 burned 270Gi for 2 of 3 inner images). +# fuse-overlayfs gives proper CoW for DinD over overlayfs. + +# NVIDIA Container Toolkit, configured to register the `nvidia` runtime with +# dockerd. This is what lets the inner Isaac Sim container see the GPU +# forwarded into the workspace pod. The pool's nodes still need the host-side +# NVIDIA driver + nvidia-container-toolkit; this just configures the inner +# dockerd. +RUN curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ + && curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + > /etc/apt/sources.list.d/nvidia-container-toolkit.list \ + && apt-get update && apt-get install -y --no-install-recommends \ + nvidia-container-toolkit \ + && rm -rf /var/lib/apt/lists/* + +# sshd: pubkey-only, no password auth ever. Host keys generated at runtime in +# entrypoint.sh so each pod has unique keys (good practice; harmless cost). +COPY sshd_config /etc/ssh/sshd_config +RUN chmod 644 /etc/ssh/sshd_config && mkdir -p /var/run/sshd + +# entrypoint: starts sshd, dockerd, clones AirStack, materializes secrets, +# runs `airstack up`, then sleeps so port-forwards keep working. +COPY entrypoint.sh /usr/local/bin/entrypoint.sh +RUN chmod 0755 /usr/local/bin/entrypoint.sh + +# Symlink airstack.sh from the cloned repo into PATH on first run; for now +# expose a placeholder so command lookups don't fail before the clone. +WORKDIR /root + +# Default to a long-running entrypoint. The OSMO workflow overrides command +# and args to invoke /tmp/entry.sh which sources this image's entrypoint.sh +# logic. Either path works. +CMD ["/usr/local/bin/entrypoint.sh"] diff --git a/osmo/workspace/entrypoint.sh b/osmo/workspace/entrypoint.sh new file mode 100755 index 000000000..bef9d9ba8 --- /dev/null +++ b/osmo/workspace/entrypoint.sh @@ -0,0 +1,281 @@ +#!/usr/bin/env bash +# entrypoint.sh — airstack-osmo-workspace pod startup. +# +# Order of operations: +# 1. Install SSH_PUB_KEY into authorized_keys, generate sshd host keys, +# start sshd. Done first so the student can SSH in even if a later step +# fails (huge debugging accelerator). +# 2. Start the inner Docker daemon (DinD) with the NVIDIA runtime so Isaac +# Sim sees the GPU forwarded into the pod. +# 3. Clone AirStack into /root/AirStack (skipped if already cloned by a +# previous pod incarnation). +# 4. Materialize simulation/isaac-sim/docker/omni_pass.env from the +# `airlab-nucleus` OSMO GENERIC credential. +# 5. docker login airlab-docker.andrew.cmu.edu using the +# `airlab-docker-login` OSMO GENERIC credential. +# 6. cd /root/AirStack && ./airstack.sh up +# 7. sleep infinity so port-forwards keep working. +# +# All steps are idempotent across pod restarts: re-running this script +# inside the same pod is safe. + +set -uo pipefail + +log() { echo "[entrypoint] $*"; } +fail() { echo "[entrypoint] ERROR: $*" >&2; exit 1; } + +# ─── 0. Stale-state cleanup ──────────────────────────────────────────────── +# +# Cursor / VS Code Remote-SSH guards its server install with a file lock +# at /tmp/cursor-remote-lock.* (and a sibling .target file naming the PIDs +# that hold it). If a previous connect attempt crashed mid-install +# (e.g. the port-forward died while the install was in flight, as +# happened on airstack-dev-13 / 2026-05-14), the lock file outlives the +# dead PIDs and every subsequent IDE retry bails out *silently* at the +# lock check — leaving an empty bin// dir and the user staring at +# a "Connecting to remote host (attempt 1)..." spinner forever. +# +# A fresh pod has nothing to preserve here, so clearing these on startup +# is always safe. +rm -f /tmp/cursor-remote-lock.* /tmp/vscode-remote-lock.* 2>/dev/null || true + +# ─── 1. SSHD ─────────────────────────────────────────────────────────────── + +log "configuring sshd" + +mkdir -p /root/.ssh && chmod 700 /root/.ssh + +if [ -z "${SSH_PUB_KEY:-}" ]; then + fail "SSH_PUB_KEY not set. Re-submit with --set-env \"SSH_PUB_KEY=\$(cat ~/.ssh/id_ed25519.pub)\"" +fi + +# Always overwrite — last submit wins. Single-user dev pod. +echo "${SSH_PUB_KEY}" > /root/.ssh/authorized_keys +chmod 600 /root/.ssh/authorized_keys + +# Generate fresh host keys if missing (first boot of this pod). +ssh-keygen -A + +mkdir -p /var/run/sshd +/usr/sbin/sshd +log "sshd listening on :22" + +# ─── 2. Inner dockerd (DinD) ─────────────────────────────────────────────── + +log "starting inner dockerd (DinD with NVIDIA runtime)" + +# nvidia-container-toolkit ships a CLI that registers the nvidia runtime in +# the dockerd config and (optionally) sets it as the default. We want it as +# the default so `airstack up` doesn't have to specify --runtime. +nvidia-ctk runtime configure --runtime=docker --set-as-default || \ + log "WARN: nvidia-ctk runtime configure failed — Isaac Sim probably won't see the GPU" + +# Pre-flight diagnostics so failures surface in OSMO logs (the pod is gone +# by the time anyone reads /var/log/dockerd.log otherwise). +log "diagnostics: kernel=$(uname -r) cgroups=$(stat -fc %T /sys/fs/cgroup 2>/dev/null) rootfs=$(stat -fc %T / 2>/dev/null)" +log "diagnostics: /var/lib/docker fs=$(stat -fc %T /var/lib/docker 2>/dev/null || echo absent)" + +# Inner dockerd setup. We try storage drivers in order: overlay2 (fastest, +# works on most modern hosts) → fuse-overlayfs (rootless-friendly, may not be +# present) → vfs (always works, slowest). Falling back avoids the +# overlay-on-overlay failure that bites DinD on some kernel/storage +# combinations. +# +# data-root: the OSMO pod's `/` is itself an overlay (containerd's +# snapshot), and Linux refuses to stack a second overlayfs on top of an +# overlay rootfs — that's exactly why dockerd here used to fall through +# to fuse-overlayfs. fuse-overlayfs is a userspace FUSE driver, and every +# `creat()` during layer extraction pays a kernel↔userspace round-trip, +# which crushes throughput on the apt/pip/ROS layers (observed: ~30-50 +# MB/s vs. ~480 MB/s on layers with few large files). Pointing data-root +# at /osmo/run/docker (the kubelet emptyDir bind-mount, backed by ext4 on +# /dev/vda3) lets us use kernel overlay2 instead, restoring the 10× +# extraction speed-up. emptyDir lives for the workflow's lifetime, which +# is exactly the docker-cache lifetime we want anyway. +DOCKERD_DATA_ROOT="${DOCKERD_DATA_ROOT:-}" +if [ -z "$DOCKERD_DATA_ROOT" ]; then + if [ -d /osmo/run ] && [ -w /osmo/run ]; then + DOCKERD_DATA_ROOT=/osmo/run/docker + else + DOCKERD_DATA_ROOT=/var/lib/docker + fi +fi +mkdir -p "$DOCKERD_DATA_ROOT" +log "dockerd data-root: $DOCKERD_DATA_ROOT (fs=$(stat -fc %T "$DOCKERD_DATA_ROOT" 2>/dev/null))" + +# Concurrency: dockerd's defaults are --max-concurrent-downloads=3 and +# --max-concurrent-uploads=5. With 2 GB+ AirStack image blobs on a 10 GbE +# pool, a single TLS pull stream tops out around 300-500 MiB/s (CPU-bound +# on the registry-side TLS encryption), so 3 parallel streams cap the +# whole bring-up around the 300 MiB/s mark seen empirically against the +# airlab-backup-10g registry — even though Ceph + 10 GbE can do far more. +# Bumping to 10 streams overlaps blob downloads enough to saturate the +# pipe without overwhelming the registry. Override with DOCKERD_MAX_* +# env vars at submit time if a particular pool needs different tuning. +DOCKERD_MAX_DOWNLOADS="${DOCKERD_MAX_DOWNLOADS:-10}" +DOCKERD_MAX_UPLOADS="${DOCKERD_MAX_UPLOADS:-10}" + +_start_dockerd() { + local driver="$1" + : > /var/log/dockerd.log + nohup dockerd \ + --host=unix:///var/run/docker.sock \ + --data-root="$DOCKERD_DATA_ROOT" \ + --storage-driver="$driver" \ + --max-concurrent-downloads="$DOCKERD_MAX_DOWNLOADS" \ + --max-concurrent-uploads="$DOCKERD_MAX_UPLOADS" \ + > /var/log/dockerd.log 2>&1 & + DOCKERD_PID=$! + log "dockerd started (pid=$DOCKERD_PID, data-root=$DOCKERD_DATA_ROOT, storage-driver=$driver); waiting for socket" + for i in $(seq 1 30); do + if docker info >/dev/null 2>&1; then + log "dockerd ready (storage-driver=$driver)" + return 0 + fi + if ! kill -0 "$DOCKERD_PID" 2>/dev/null; then + log "dockerd exited; tailing /var/log/dockerd.log:" + tail -40 /var/log/dockerd.log | sed 's/^/[dockerd] /' + return 1 + fi + sleep 1 + done + log "dockerd unresponsive after 30s; tailing /var/log/dockerd.log:" + tail -40 /var/log/dockerd.log | sed 's/^/[dockerd] /' + kill "$DOCKERD_PID" 2>/dev/null || true + return 1 +} + +DOCKERD_OK=false +for drv in overlay2 fuse-overlayfs vfs; do + if _start_dockerd "$drv"; then + DOCKERD_OK=true + break + fi + log "WARN: dockerd failed with storage-driver=$drv; trying next" +done +if [ "$DOCKERD_OK" != "true" ]; then + fail "dockerd refused to start with any of overlay2 / fuse-overlayfs / vfs" +fi + +# ─── 3. Clone AirStack ───────────────────────────────────────────────────── + +AIRSTACK_REPO_URL="${AIRSTACK_REPO_URL:-https://github.com/castacks/AirStack.git}" +AIRSTACK_BRANCH="${AIRSTACK_BRANCH:-main}" +AIRSTACK_ROOT=/root/AirStack + +if [ ! -d "$AIRSTACK_ROOT/.git" ]; then + log "cloning $AIRSTACK_REPO_URL ($AIRSTACK_BRANCH) -> $AIRSTACK_ROOT" + git clone --recursive --branch "$AIRSTACK_BRANCH" "$AIRSTACK_REPO_URL" "$AIRSTACK_ROOT" \ + || fail "git clone failed" +else + log "$AIRSTACK_ROOT already cloned (skipping)" +fi + +# Make sure the airstack CLI is on PATH for interactive shells. +ln -sf "$AIRSTACK_ROOT/airstack.sh" /usr/local/bin/airstack +ln -sf "$AIRSTACK_ROOT/airstack.sh" /usr/local/bin/airstack.sh + +# ─── 4. omni_pass.env from airlab-nucleus credential ─────────────────────── + +OMNI_PASS_FILE="$AIRSTACK_ROOT/simulation/isaac-sim/docker/omni_pass.env" + +if [ -z "${OMNI_USER:-}" ] || [ -z "${OMNI_PASS:-}" ]; then + log "WARN: airlab-nucleus OSMO credential not set." + log "WARN: Run on your laptop:" + log "WARN: osmo credential set airlab-nucleus --type GENERIC \\" + log "WARN: --payload omni_user= omni_pass= \\" + log "WARN: omni_server=omniverse://airlab-nucleus.andrew.cmu.edu/NVIDIA/Assets/Isaac/5.1" + log "WARN: Falling back to guest/guest (read-only Nucleus) — Isaac Sim assets may fail to load." +fi + +# Default to read-only Nucleus access so a missing credential degrades +# instead of crashing the pod. +: "${OMNI_USER:=guest}" +: "${OMNI_PASS:=guest}" +: "${OMNI_SERVER:=omniverse://airlab-nucleus.andrew.cmu.edu/NVIDIA/Assets/Isaac/5.1}" + +# If OMNI_PASS looks like a Nucleus API JWT (header starts with `eyJ`), +# switch to API-token auth: omniclient expects the literal sentinel +# username `$omni-api-token` paired with the JWT as the password. +# Setting OMNI_USER to the actual Andrew ID would route the JWT through +# the password-verification path instead and Nucleus would silently +# DENY (visible only in the auth-service log as +# `InternalCredentials.auth … 'username': '' … status: DENIED`). +# +# docker-compose v2 interpolates env_file values, so the literal `$` +# must be doubled to `$$` to survive Compose's parser. The container +# ultimately sees `OMNI_USER=$omni-api-token`. +case "$OMNI_PASS" in + eyJ*.*.*) + log "OMNI_PASS looks like a JWT — using API-token auth (OMNI_USER=\$omni-api-token)" + OMNI_USER_LINE='OMNI_USER=$$omni-api-token' + ;; + *) + OMNI_USER_LINE="OMNI_USER=${OMNI_USER}" + ;; +esac + +log "writing $OMNI_PASS_FILE (${OMNI_USER_LINE}, omni_server=${OMNI_SERVER})" +cat > "$OMNI_PASS_FILE" < password=" +fi + +# ─── 6. airstack up ──────────────────────────────────────────────────────── + +# Honor optional overrides passed in via OSMO env. Defaults match a "single +# robot, Isaac Sim with WebRTC livestream" dev session. +export AUTOLAUNCH="${AUTOLAUNCH:-true}" +export NUM_ROBOTS="${NUM_ROBOTS:-1}" +export ISAAC_SIM_LIVESTREAM="${ISAAC_SIM_LIVESTREAM:-true}" + +# COMPOSE_PROFILES selection: the default `desktop,isaac-sim` from .env runs +# the standard isaac-sim service. If the student wants livestream, they (or +# we) swap to the isaac-sim-livestream profile, which is the OSMO-friendly +# variant defined in simulation/isaac-sim/docker/docker-compose.yaml. +if [ "$ISAAC_SIM_LIVESTREAM" = "true" ]; then + export COMPOSE_PROFILES="${COMPOSE_PROFILES:-desktop,isaac-sim-livestream}" +else + export COMPOSE_PROFILES="${COMPOSE_PROFILES:-desktop,isaac-sim}" +fi + +cd "$AIRSTACK_ROOT" +if [ "${OSMO_AIRSTACK_UP:-true}" = "true" ]; then + log "airstack up (COMPOSE_PROFILES=$COMPOSE_PROFILES, NUM_ROBOTS=$NUM_ROBOTS, livestream=$ISAAC_SIM_LIVESTREAM)" + ./airstack.sh up || log "WARN: airstack up exited non-zero — pod stays alive for debugging via SSH" +else + log "OSMO_AIRSTACK_UP=false — skipping airstack up; SSH in and run ./airstack.sh up manually" +fi + +# ─── 7. Sleep ────────────────────────────────────────────────────────────── + +log "entrypoint complete; sleeping forever so port-forwards keep working" +if [ "$ISAAC_SIM_LIVESTREAM" = "true" ]; then + isaac_sim_log_container="isaac-sim-livestream" +else + isaac_sim_log_container="airstack-isaac-sim-1" +fi +log "pod-side log paths:" +log " - dockerd: /var/log/dockerd.log" +log " - airstack: docker logs ${isaac_sim_log_container} / airstack-robot-desktop-1 / airstack-gcs-1" +exec sleep infinity diff --git a/osmo/workspace/sshd_config b/osmo/workspace/sshd_config new file mode 100644 index 000000000..097efdb21 --- /dev/null +++ b/osmo/workspace/sshd_config @@ -0,0 +1,41 @@ +# sshd_config baked into airstack-osmo-workspace. +# +# Permanently disables password auth — the only way in is via a pubkey +# installed by entrypoint.sh from the SSH_PUB_KEY env var (passed at submit +# time with `osmo workflow submit ... --set-env "SSH_PUB_KEY=$(cat ~/.ssh/id_ed25519.pub)"`). + +Port 22 +AddressFamily any +ListenAddress 0.0.0.0 +ListenAddress :: + +# Pubkey only. +PasswordAuthentication no +PubkeyAuthentication yes +ChallengeResponseAuthentication no +KbdInteractiveAuthentication no +PermitEmptyPasswords no + +# Single-user dev pod; the IDE attaches as root because /root/AirStack and +# the existing AirStack ergonomics expect it. PermitRootLogin +# prohibit-password forbids password root login but allows pubkey root login. +PermitRootLogin prohibit-password + +# Standard auth path. +AuthorizedKeysFile .ssh/authorized_keys + +UsePAM yes + +# Performance / VS Code-Remote-SSH friendliness: +# - AcceptEnv lets the IDE forward LANG, ENV, etc. +# - X11 forwarding off (no display in the pod). +# - Allow forwarding so port-forwards through SSH are permitted if students +# want to layer their own. +AcceptEnv LANG LC_* +X11Forwarding no +AllowAgentForwarding yes +AllowTcpForwarding yes +PrintMotd no + +# Subsystems VS Code Remote needs for sftp/file ops. +Subsystem sftp /usr/lib/openssh/sftp-server diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index 55b1ad40f..4bd86d0c6 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -78,6 +78,7 @@ RUN apt update && apt install -y --no-install-recommends \ python3-rosdep \ tmux \ gdb \ + xvfb \ && rm -rf /var/lib/apt/lists/* # Install any additional ROS2 packages @@ -277,6 +278,7 @@ RUN apt update && apt install -y --no-install-recommends \ python3-pip \ python3-rosdep \ tmux \ + xvfb \ && rm -rf /var/lib/apt/lists/* # Install runtime ROS2 packages (no libcgal-dev) diff --git a/robot/docker/docker-compose.yaml b/robot/docker/docker-compose.yaml index dfbe71fd3..2ae1c9a32 100644 --- a/robot/docker/docker-compose.yaml +++ b/robot/docker/docker-compose.yaml @@ -30,6 +30,11 @@ services: # 'command' uses variables so that it can be shared across robot-desktop and robot-l4t, with different launch packages and roles. command: > bash -c " + if [ -z \"$$DISPLAY\" ] && command -v Xvfb >/dev/null 2>&1; then + tmux new -d -s xvfb 'Xvfb :99 -screen 0 1280x720x24 -ac +extension GLX +render -noreset 2>&1 | tee /tmp/xvfb.log'; + export DISPLAY=:99; + for i in 1 2 3 4 5 6 7 8 9 10; do [ -e /tmp/.X11-unix/X99 ] && break; sleep 1; done; + fi; service ssh restart; tmux new -d -s bringup; if [ $$AUTOLAUNCH == 'true' ]; then diff --git a/simulation/isaac-sim/docker/docker-compose.yaml b/simulation/isaac-sim/docker/docker-compose.yaml index a1b40ae56..dfd699aa4 100644 --- a/simulation/isaac-sim/docker/docker-compose.yaml +++ b/simulation/isaac-sim/docker/docker-compose.yaml @@ -94,3 +94,61 @@ services: tmux send-keys -t isaac '/isaac-sim/runapp.sh' ENTER; sleep infinity" networks: !reset null + + # =================================================================================================================== + # WebRTC livestream variant for OSMO / remote dev. Headless: no X server, + # no display, no GUI window. Kit's `omni.kit.livestream.webrtc` extension + # serves WebSocket signaling on TCP 49100 and SRTP media on UDP 49099 (the + # latter pinned by `app.livestream.fixedHostPort/minHostPort/maxHostPort=49099` + # in the Pegasus launch script — see example_one_px4_pegasus_launch_script.py). + # Those two ports are published into the host (the OSMO workspace pod's + # network namespace) so `osmo workflow port-forward` can reach them. Kit + # 107's default media-port range is wider and dynamic, so pinning to a + # known value is the only way to keep the forward surface to two ports. + # + # Selected via COMPOSE_PROFILES (e.g. `desktop,isaac-sim-livestream`) in + # osmo/workspace/entrypoint.sh, or by setting ISAAC_SIM_LIVESTREAM=true. + isaac-sim-livestream: + extends: + service: isaac-sim + container_name: isaac-sim-livestream + profiles: !override + - isaac-sim-livestream + # Always run the Pegasus standalone path; the livestream branch in the + # script is gated on ISAAC_SIM_LIVESTREAM=true (env-driven, additive + # to the existing script behavior). + command: > + bash -c " + tmux new -d -s isaac; + tmux send-keys -t isaac 'PYTHONPATH=\"$$ISAAC_SIM_PYTHONPATH\" /isaac-sim/python.sh /isaac-sim/AirStack/simulation/isaac-sim/launch_scripts/${ISAAC_SIM_SCRIPT_NAME} --ext-folder ~/.local/share/ov/data/documents/Kit/shared/exts --/app/livestream/enabled=true' ENTER; + sleep infinity" + environment: + # Inherit everything from isaac-sim and append: + - ISAAC_SIM_LIVESTREAM=true + - ISAAC_SIM_USE_STANDALONE=true + - ISAAC_SIM_HEADLESS=true + # Publish the WebRTC livestream ports to the pod NS. Bridge-mode + + # publish (the conservative choice) keeps the rest of the stack on + # airstack_network for DDS multicast. + ports: + - "49100:49100/tcp" # WebSocket signaling (omni.kit.livestream.webrtc, app.livestream.port=49100) + - "49099:49099/udp" # SRTP media (pinned via app.livestream.fixedHostPort=49099 in the launch script) + # Drop X11-specific volume mounts inherited from the isaac-sim service — + # there is no X server in an OSMO pod. + volumes: !override + - $HOME/docker/isaac-sim/cache/main:/isaac-sim/.cache:rw + - $HOME/docker/isaac-sim/cache/computecache:/isaac-sim/.nv/ComputeCache:rw + - $HOME/docker/isaac-sim/logs:/isaac-sim/.nvidia-omniverse/logs:rw + - $HOME/docker/isaac-sim/config:/isaac-sim/.nvidia-omniverse/config:rw + - $HOME/docker/isaac-sim/data:/isaac-sim/.local/share/ov/data:rw + - $HOME/docker/isaac-sim/pkg:/isaac-sim/.local/share/ov/pkg:rw + - ../extensions/PegasusSimulator/extensions/pegasus.simulator:/isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/pegasus.simulator/:rw + - ./omniverse.toml:/isaac-sim/.nvidia-omniverse/config/omniverse.toml:rw + - ./user.config.json:/isaac-sim/.local/share/ov/data/Kit/Isaac-Sim Full/5.1/user.config.json:rw + - .dev:/isaac-sim/.dev:rw + - .bashrc:/isaac-sim/.bashrc:rw + - ../../../common/inputrc:/etc/inputrc:rw + - ../../../common/.tmux.conf:/isaac-sim/.tmux.conf:rw + - ../../..:/isaac-sim/AirStack:rw + - ../../../.devcontainer/isaac-sim/launch.json:/isaac-sim/AirStack/.vscode/launch.json:rw + - ../../../.devcontainer/isaac-sim/tasks.json:/isaac-sim/AirStack/.vscode/tasks.json:rw diff --git a/simulation/isaac-sim/docker/omni_pass_TEMPLATE.env b/simulation/isaac-sim/docker/omni_pass_TEMPLATE.env index 45bf2fc71..e010e0db4 100644 --- a/simulation/isaac-sim/docker/omni_pass_TEMPLATE.env +++ b/simulation/isaac-sim/docker/omni_pass_TEMPLATE.env @@ -6,15 +6,28 @@ ######################################################################### ## Nucleus Login information -# This can either be your username and password or the nucleus login token -# The login token method is preferred. You can get the token by going to -# the nucleus server website. For us -# https://airlab-nucleus.andrew.cmu.edu/omni/web3/ -# logging in. -# Then right clicking on the cloud and click the "API Tokens" window -# to generate an API token and copy it to "OMNI_PASS". -# If you skip that step, leave both values at their guest defaults. - +# +# Recommended: API-token auth. Generate a token at +# https://airlab-nucleus.andrew.cmu.edu/omni/web3/ +# → right-click cloud icon → API Tokens → Create +# Then set: +# OMNI_USER=$$omni-api-token ← literal sentinel value +# OMNI_PASS= ← the JWT (~1 KB, starts with eyJ) +# +# IMPORTANT: the `$$` is intentional. docker-compose v2 interpolates +# env_file values, and the literal `$` must be doubled to survive +# Compose's parser. The container ultimately sees `OMNI_USER=$omni-api-token`, +# which is what omniclient expects for API-token auth (anything else, e.g. +# your Andrew ID, routes the JWT through the password-verification path +# and Nucleus silently DENIES the request). +# +# Fallback: username/password auth. Set OMNI_USER to your Nucleus username +# and OMNI_PASS to your Nucleus password (NOT your Andrew password unless +# Nucleus is SSO-linked to it). +# +# If you skip Nucleus auth entirely, leave both at the guest defaults +# (read-only access; Isaac Sim asset loads from Nucleus may fail). +# ######################################################################### OMNI_USER=guest diff --git a/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_launch_script.py b/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_launch_script.py index f53ea0993..11819fc2f 100755 --- a/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_launch_script.py +++ b/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_launch_script.py @@ -10,17 +10,77 @@ - Optionally saving the prepared scene as a self-contained USD """ -import carb -from isaacsim import SimulationApp - -# Must be created before any omni imports -simulation_app = SimulationApp({"headless": False}) - import os import sys import time import asyncio +import carb +from isaacsim import SimulationApp + +_LIVESTREAM = os.environ.get("ISAAC_SIM_LIVESTREAM", "").lower() == "true" + +# Must be created before any omni imports. +# +# When livestreaming, mirror the NVIDIA reference config from +# simulation/isaac-sim/standalone_examples/api/isaacsim.simulation_app/livestream.py +# so the Kit GUI (menu bar, toolbar, viewport, status bar) actually gets +# rendered into the WebRTC stream instead of just the bare 3D viewport. +# Key field: `hide_ui: False` — SimulationApp's default when `headless=True` +# is to also hide the UI; the livestream reference opts back into showing +# it. `display_options=3286` is the same bitmask the reference uses to keep +# the default grid + axes visible at scene start. +if _LIVESTREAM: + _SIM_APP_CONFIG = { + "width": 1280, + "height": 720, + "window_width": 1920, + "window_height": 1080, + "headless": True, + "hide_ui": False, + "renderer": "RaytracedLighting", + "display_options": 3286, + } +else: + _SIM_APP_CONFIG = {"headless": False} + +simulation_app = SimulationApp(launch_config=_SIM_APP_CONFIG) + +if _LIVESTREAM: + # Headless + WebRTC livestream when ISAAC_SIM_LIVESTREAM=true (set by the + # OSMO airstack-osmo-workspace entrypoint and the isaac-sim-livestream + # Compose profile). Local desktop dev keeps the original windowed behavior. + # Mirrors AirStack's standalone livestream reference at + # simulation/isaac-sim/standalone_examples/api/isaacsim.simulation_app/livestream.py + from isaacsim.core.utils.extensions import enable_extension + simulation_app.set_setting("/app/window/drawMouse", True) + simulation_app.set_setting("/app/livestream/enabled", True) + + # Pin the UDP media port so it stays inside the narrow set of ports we + # publish from this container and that `airstack osmo:webrtc` forwards. + # + # Kit 107's WebRTC livestream picks a UDP media port dynamically. The + # documented `omni.services.livestream.nvcf` defaults were + # minHostPort=47998 / maxHostPort=48020 / fixedHostPort=0, but the + # actual Kit binary ignored that range on airstack-dev-13 and bound to + # UDP 49042 — outside both the Compose-published port range AND the + # default osmo `--udp` forward (47995-48012,49000-49007). Result: + # signaling worked (TCP 49100), the WebRTC Streaming Client window + # opened, but every media packet was dropped → black viewport + + # the `NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1` underflow + # storm in the Kit log. + # + # Set all three settings so whichever code path the plugin reads, it + # lands on UDP 49099. The value of 49099 is picked as one-off from the + # 49100 signaling port — same range, easy to remember, and TCP/UDP can + # coexist on the same number if anyone later wants a single port. + LIVESTREAM_UDP_PORT = int(os.environ.get("ISAAC_SIM_LIVESTREAM_UDP_PORT", "49099")) + simulation_app.set_setting("/app/livestream/fixedHostPort", LIVESTREAM_UDP_PORT) + simulation_app.set_setting("/app/livestream/minHostPort", LIVESTREAM_UDP_PORT) + simulation_app.set_setting("/app/livestream/maxHostPort", LIVESTREAM_UDP_PORT) + + enable_extension("omni.kit.livestream.webrtc") + import omni.kit.app import omni.timeline import omni.usd From dff3dc6f765ec23ca3fbeb324ef83358b989dabc Mon Sep 17 00:00:00 2001 From: Andrew Jong Date: Thu, 28 May 2026 14:23:18 -0700 Subject: [PATCH 03/21] fix(isaac-sim): pegasus drone retains PX4 state across Stop/Play (#363) * Update submodule to point to pegasus fix fixing start/stop behavior * Bump VERSION to 0.19.0-alpha.2 Co-Authored-By: Claude Opus 4.7 (1M context) --------- Co-authored-by: Claude Opus 4.7 (1M context) --- .env | 2 +- simulation/isaac-sim/extensions/PegasusSimulator | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.env b/.env index aab86845e..e368cd50b 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.1" +VERSION="0.19.0-alpha.2" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/simulation/isaac-sim/extensions/PegasusSimulator b/simulation/isaac-sim/extensions/PegasusSimulator index 8e01d0138..fe8b5a101 160000 --- a/simulation/isaac-sim/extensions/PegasusSimulator +++ b/simulation/isaac-sim/extensions/PegasusSimulator @@ -1 +1 @@ -Subproject commit 8e01d01380cdb4f9bd5514fe3eb952b6821b6147 +Subproject commit fe8b5a101857f2cda290b9b677b3a95c4cca6b09 From 8b927e465c164039fd00c1b9525bb6e7e36629d1 Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Fri, 29 May 2026 12:02:25 -0400 Subject: [PATCH 04/21] Johnliu/optitrack autonomy (#359) * incremented version tag * docker image builds on l4t with generalizability features for other ros and linux versions * documentation and claude skills for developing a new profile. * initial natnet implementation * deployment to jetson with ros2 jazzy now fixed * unit testing dependency fix * added optitrack perception to launch * put tag version back in * added instructions for Agents to run tests * attempt at completely custom Optitrack Parser (Not working) * fully implemented NatNetSDK natnet ros2 wrapper natively in AirStack. Hand test in mocap room successful * unit test restructuring * reorganized natnet logic for unit-testability * unit testing restructuring to have unit tests in src and proxies in test. Unit tests workflows created * reupdated documentation for current state of testing * change unit tests to occur with system tests so that environment is builtgit status * generalizes natnet parameters and disables natnet automatically for launch * natnet client adaptor now references correct error code from NatNet SDK 4.4.0.0 * increment version tag * bug fixes to natnet launching from env file * fixed failing systems test due to depends issue and specifying unit tests via yaml * Use NatNet callback context instead of thread-local dispatch * addressing Krrish' documentation comments * incrementing version tag after osmo PR merge * documentation corrections * Bump VERSION Update .env --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Andrew Jong --- .../assets/package_template/setup.py | 4 +- .agents/skills/add-unit-tests/SKILL.md | 309 +++++++ .agents/skills/configure-multi-robot/SKILL.md | 2 +- .agents/skills/docker-build-profiles/SKILL.md | 134 ++++ .agents/skills/run-system-tests/SKILL.md | 60 +- .env | 4 +- AGENTS.md | 27 +- airstack.sh | 64 +- .../msgs/airstack_msgs/package.xml | 3 +- .../ros_packages/msgs/task_msgs/package.xml | 3 +- .../beginner/airstack-cli/docker_usage.md | 2 + docs/development/index.md | 2 + .../intermediate/docker-build-profiles.md | 126 +++ .../development/intermediate/testing/index.md | 51 +- .../intermediate/testing/unit_testing.md | 206 +++++ docs/robot/autonomy/perception/index.md | 6 +- docs/robot/docker/index.md | 2 + mkdocs.yml | 9 +- robot/docker/Dockerfile.l4t-stack-base | 50 ++ robot/docker/Dockerfile.robot | 138 +++- robot/docker/docker-compose.yaml | 37 +- robot/docker/zed/Dockerfile.zed-l4t | 2 +- .../controls/pid_controller_msgs/package.xml | 3 +- .../src/perception/natnet_ros2/.gitignore | 27 + .../src/perception/natnet_ros2/CMakeLists.txt | 99 +++ .../src/perception/natnet_ros2/README.md | 199 +++++ .../natnet_ros2/config/natnet_config.yaml | 51 ++ .../config/vision_pose_converter.yaml | 12 + .../env-hooks/natnet_library_path.dsv.in | 1 + .../natnet_ros2/natnet_client_adapter.hpp | 60 ++ .../include/natnet_ros2/natnet_logic.hpp | 354 ++++++++ .../natnet_ros2/launch/natnet_ros2.launch.py | 117 +++ .../launch/vision_pose_converter.launch.xml | 45 ++ .../src/perception/natnet_ros2/package.xml | 46 ++ .../scripts/download-natnet-sdk.sh | 167 ++++ .../natnet_ros2/src/natnet_client_adapter.cpp | 176 ++++ .../natnet_ros2/src/natnet_ros2_node.cpp | 325 ++++++++ .../src/vision_pose_converter_node.py | 116 +++ .../natnet_ros2/test/fake_natnet_client.hpp | 163 ++++ .../natnet_ros2/test/test_natnet_logic.cpp | 757 ++++++++++++++++++ .../natnet_ros2/test/test_natnet_ros2.py | 152 ++++ .../launch/perception.launch.xml | 7 + .../lidar_point_cloud_filter/README.md | 4 +- .../validation_core.py | 73 ++ .../scripts/validate_lidar_filter_clouds.py | 45 +- .../lidar_point_cloud_filter/setup.cfg | 8 + .../sensors/lidar_point_cloud_filter/setup.py | 4 +- .../test/test_validation_core.py | 75 ++ .../src/sensors/sensor_interfaces/package.xml | 5 +- tests/README.md | 69 +- tests/colcon_unit_test_packages.yaml | 13 + tests/conftest.py | 74 +- tests/parse_metrics.py | 12 +- tests/pytest.ini | 1 + tests/requirements.txt | 2 + tests/robot/README.md | 37 + tests/robot/behavior/README.md | 3 + tests/robot/global/README.md | 3 + tests/robot/interface/README.md | 3 + tests/robot/local/README.md | 3 + tests/robot/perception/README.md | 3 + .../natnet_ros2/test_natnet_ros2.py | 32 + tests/robot/sensors/README.md | 4 + .../test_validation_core.py | 38 + tests/sensor_probes.py | 6 +- tests/sim/README.md | 14 + tests/sim/motive_emulator/README.md | 61 ++ tests/system/__init__.py | 1 + tests/{ => system}/test_build_docker.py | 0 tests/{ => system}/test_build_packages.py | 41 +- tests/{ => system}/test_liveliness.py | 2 +- tests/{ => system}/test_sensors.py | 4 +- tests/{ => system}/test_takeoff_hover_land.py | 0 73 files changed, 4599 insertions(+), 159 deletions(-) create mode 100644 .agents/skills/add-unit-tests/SKILL.md create mode 100644 .agents/skills/docker-build-profiles/SKILL.md create mode 100644 docs/development/intermediate/docker-build-profiles.md create mode 100644 robot/docker/Dockerfile.l4t-stack-base create mode 100644 robot/ros_ws/src/perception/natnet_ros2/.gitignore create mode 100644 robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt create mode 100644 robot/ros_ws/src/perception/natnet_ros2/README.md create mode 100644 robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/env-hooks/natnet_library_path.dsv.in create mode 100644 robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp create mode 100644 robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp create mode 100644 robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py create mode 100644 robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/package.xml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/scripts/download-natnet-sdk.sh create mode 100644 robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp create mode 100644 robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp create mode 100755 robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py create mode 100644 robot/ros_ws/src/perception/natnet_ros2/test/fake_natnet_client.hpp create mode 100644 robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp create mode 100644 robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py create mode 100644 robot/ros_ws/src/sensors/lidar_point_cloud_filter/lidar_point_cloud_filter/validation_core.py create mode 100644 robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py create mode 100644 tests/colcon_unit_test_packages.yaml create mode 100644 tests/robot/README.md create mode 100644 tests/robot/behavior/README.md create mode 100644 tests/robot/global/README.md create mode 100644 tests/robot/interface/README.md create mode 100644 tests/robot/local/README.md create mode 100644 tests/robot/perception/README.md create mode 100644 tests/robot/perception/natnet_ros2/test_natnet_ros2.py create mode 100644 tests/robot/sensors/README.md create mode 100644 tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py create mode 100644 tests/sim/README.md create mode 100644 tests/sim/motive_emulator/README.md create mode 100644 tests/system/__init__.py rename tests/{ => system}/test_build_docker.py (100%) rename tests/{ => system}/test_build_packages.py (64%) rename tests/{ => system}/test_liveliness.py (99%) rename tests/{ => system}/test_sensors.py (97%) rename tests/{ => system}/test_takeoff_hover_land.py (100%) diff --git a/.agents/skills/add-ros2-package/assets/package_template/setup.py b/.agents/skills/add-ros2-package/assets/package_template/setup.py index 4056e5d5f..3982cc356 100644 --- a/.agents/skills/add-ros2-package/assets/package_template/setup.py +++ b/.agents/skills/add-ros2-package/assets/package_template/setup.py @@ -26,7 +26,9 @@ maintainer_email='your.email@example.com', # TODO: Update description='Brief description of your module', # TODO: Update license='Apache-2.0', - tests_require=['pytest'], + extras_require={ + 'test': ['pytest'], + }, entry_points={ 'console_scripts': [ # TODO: Add your node executables here diff --git a/.agents/skills/add-unit-tests/SKILL.md b/.agents/skills/add-unit-tests/SKILL.md new file mode 100644 index 000000000..7d1d3b582 --- /dev/null +++ b/.agents/skills/add-unit-tests/SKILL.md @@ -0,0 +1,309 @@ +--- +name: add-unit-tests +description: Add Python or C++ unit tests to an AirStack ROS 2 package. Covers the co-location pattern (test source in package/test/), the thin proxy that makes tests discoverable by pytest tests/ and airstack test -m unit, and how to extend the pattern to sim and GCS modules. +license: MIT +metadata: + author: AirLab CMU + repository: AirStack +--- + +# Skill: Add Unit Tests to an AirStack Module + +## When to Use + +Use this skill when: + +- Adding Python unit tests for a ROS 2 package (perception, sensors, local, global, behavior, interface) +- Adding C++ unit tests (`gtest`) to a package already using `ament_cmake` +- Extending unit tests to sim-side Python (`tests/sim/`) or GCS modules (`tests/gcs/`) +- Verifying that `airstack test -m unit` and `pytest tests/` (CI) pick up your new tests + +For system tests (full Docker stack, sim, sensors, takeoff/hover/land) see the +`run-system-tests` skill instead. + +## Architecture Overview + +Unit tests follow a **co-location + proxy** pattern: + +``` +robot/ros_ws/src/// +├── src/ # production source (Python or C++) +├── test/ +│ ├── test_.py # ← unit test SOURCE (canonical location) +│ ├── test_.cpp # ← C++ gtest SOURCE (optional) +│ └── fake_.hpp # ← C++ test doubles (optional) +└── CMakeLists.txt # wires ament_add_gtest under BUILD_TESTING + +tests/robot/// +└── test_.py # ← thin PROXY (re-exports tests from above) +``` + +The **proxy** is a one-file shim that loads the real test module with `importlib` +and re-exports every `test_*` function. This means: + +| Invocation | What runs | +|---|---| +| `pytest tests/ -m unit` | Proxy in `tests/robot/` → loads real test from package | +| `airstack test -m unit` | Same path | +| CI `system-tests.yml` (PR open / approved) | Same path via `pytest tests/` | +| `colcon test --packages-select ` | Real test in `package/test/` directly | + +## Step-by-Step: Adding a Python Unit Test + +### 1. Identify pure-Python logic to test + +Good candidates are functions/classes with **no ROS or hardware dependencies**: +- Pure math / geometry helpers +- Protocol parsers +- Data-structure converters +- Any function that takes plain Python types and returns plain Python types + +If the code imports ROS types, stub them out at the import boundary +(see `test_natnet_ros2.py` for the `sys.modules` stub pattern). + +### 2. Write the test source in the package + +Create `robot/ros_ws/src///test/test_.py`: + +```python +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Unit tests for — .""" + +import sys +from pathlib import Path +import pytest + +# Add the package src/ dir so the production module is importable +# without colcon installing the package first. +_src = Path(__file__).resolve().parent.parent / "src" +if str(_src) not in sys.path: + sys.path.insert(0, str(_src)) + +from my_module import my_function # noqa: E402 + + +@pytest.mark.unit +def test_my_function_basic(): + assert my_function(1, 2) == 3 +``` + +**Key points:** +- Always decorate with `@pytest.mark.unit` — this is the filter for fast runs. +- Compute paths relative to `__file__` (`parent.parent / "src"`) — never hardcode + absolute paths. +- For packages with a Python module directory (`//`), add the package + root (`parent.parent`) to `sys.path` and import as + `from . import ...`. +- If the code uses ROS types, stub `sys.modules` before importing: + +```python +import sys +from unittest.mock import MagicMock + +sys.modules.setdefault("rclpy", MagicMock()) +sys.modules.setdefault("rclpy.node", MagicMock()) +sys.modules.setdefault("geometry_msgs", MagicMock()) +sys.modules.setdefault("geometry_msgs.msg", MagicMock()) +# ... then import your module +``` + +For `rclpy.node.Node` subclasses use a real dummy base class instead of a +`MagicMock()` to ensure `__init_subclass__` fires and method bodies are defined +(see `test_natnet_ros2.py` for the full pattern). + +### 3. Write the thin proxy in tests/robot/ + +Create `tests/robot///test_.py`: + +```python +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Proxy: re-exposes unit tests from the package source tree. + +Unit test logic lives co-located with the package source (ROS 2 / colcon convention): + robot/ros_ws/src///test/test_.py + +This file makes those tests discoverable by ``pytest tests/`` (CI) and +``airstack test -m unit`` without any changes to the CI workflow. +""" +import importlib.util +import sys +from pathlib import Path + +_repo_root = Path(__file__).resolve().parents[N] # adjust N so this resolves to repo root +_pkg_test = _repo_root / "robot/ros_ws/src///test" +_real_file = _pkg_test / "test_.py" + +# If the test imports from a package module, ensure the package root is on sys.path. +# Example: _pkg_root = _pkg_test.parent; sys.path.insert(0, str(_pkg_root)) + +# Load the real module under a unique name to avoid the circular import that +# would occur if we used `from test_ import *` (this file has the same +# name, and pytest adds its directory to sys.path at collection time). +_spec = importlib.util.spec_from_file_location("__unit_tests", _real_file) +_real = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_real) + +# Re-export every test_* symbol so pytest collects them from this proxy. +for _name in dir(_real): + if _name.startswith("test_"): + globals()[_name] = getattr(_real, _name) +``` + +**Counting `parents[N]` to reach the repo root:** + +| Proxy location | `parents[N]` for repo root | +|---|---| +| `tests/robot///` | `parents[4]` | +| `tests/sim//` | `parents[3]` | +| `tests/gcs//` | `parents[3]` | + +### 4. Ensure the tests/ directory structure exists + +```bash +mkdir -p tests/robot// +touch tests/robot///__init__.py # only if needed for conftest path discovery +``` + +The READMEs in `tests/robot/behavior/`, `tests/robot/global/`, etc. describe the +purpose of each layer mirror. Update the layer README when you add a new package. + +### 5. Run locally to verify + +```bash +# From repo root — no container needed +cd tests +pytest -m unit -v +# or +airstack test -m unit -v +``` + +All 14+ existing tests plus your new ones should pass. The proxy output shows: +``` +robot///test_.py::test_my_function_basic + <- ../robot/ros_ws/src///test/test_.py PASSED +``` + +### 6. CI picks it up automatically + +Unit tests are discovered by `pytest tests/` and run as part of `system-tests.yml` +(triggered on PR open) — no changes to CI needed. + +--- + +## Step-by-Step: Adding a C++ gtest + +C++ tests don't use the proxy pattern — they live entirely within the package and +run exclusively via `colcon test`. + +### 1. Write the test in `package/test/` + +```cpp +// Copyright (c) 2024 Carnegie Mellon University +// MIT License - see LICENSE in the repository root for full text. +#include +#include "my_package/my_header.hpp" + +TEST(MyGroup, BasicCase) { + EXPECT_EQ(my_function(1, 2), 3); +} +``` + +### 2. Wire `ament_add_gtest` in `CMakeLists.txt` + +```cmake +if(BUILD_TESTING) + find_package(ament_lint_auto REQUIRED) + ament_lint_auto_find_test_dependencies() + + find_package(ament_cmake_gtest REQUIRED) + ament_add_gtest(test_my_name test/test_my_name.cpp) + target_include_directories(test_my_name PRIVATE + $ + $) + # Link any production library targets here if needed: + # target_link_libraries(test_my_name my_lib) +endif() +``` + +### 3. Add test depend in `package.xml` + +```xml +ament_cmake_gtest +``` + +### 4. Build and run + +```bash +# Inside the robot container: +docker exec airstack-robot-desktop-1 bash -c \ + "bws --cmake-args '-DBUILD_TESTING=ON' --packages-select " +docker exec airstack-robot-desktop-1 bash -c \ + "colcon test --packages-select --event-handlers console_direct+" +docker exec airstack-robot-desktop-1 bash -c \ + "colcon test-result --all" +``` + +The `build_packages` system test in CI (`tests/system/test_build_packages.py`) also +runs `colcon test` with `BUILD_TESTING=ON` for the robot container. Packages gated +there are listed in [`tests/colcon_unit_test_packages.yaml`](../../../tests/colcon_unit_test_packages.yaml) +— add your package under `robot.packages` when it has gtests or pytest tests in +`package/test/`. + +--- + +## Extending to sim and GCS + +The same proxy pattern applies verbatim: + +**Sim-side Python** (e.g. motive emulator protocol logic): +``` +simulation/...//test/test_.py ← source +tests/sim//test_.py ← proxy (parents[3] = repo root) +``` + +**GCS modules**: +``` +gcs/...//test/test_.py ← source +tests/gcs//test_.py ← proxy (parents[3] = repo root) +``` + +`pytest tests/ -m unit` discovers them through the proxy without any +pytest.ini or CI changes needed. + +--- + +## Pattern Summary + +| Concern | Answer | +|---|---| +| Where does test source live? | `/…//test/` (co-located with the package) | +| Where does pytest discover tests? | `tests/robot/` (or `tests/sim/`, `tests/gcs/`) via thin proxy | +| How does the proxy avoid circular import? | `importlib.util.spec_from_file_location` with a unique module name | +| What mark do all unit tests use? | `@pytest.mark.unit` | +| What CI workflow runs them? | `system-tests.yml` — runs `pytest tests/` which includes unit tests | +| When does that workflow trigger? | PR opened, `/pytest` comment, `workflow_dispatch` | +| Do system tests (`liveliness`, etc.) run too? | No — `-m unit` filters to hermetic tests only | +| Does `colcon test` also run these? | Yes — Python tests in `package/test/` are discovered by colcon's pytest runner | +| Can I add pure C++ gtests? | Yes — `ament_add_gtest` in CMakeLists.txt, no proxy needed | + +## Reference Implementations + +| Package | Python test | What it covers | +|---|---|---| +| `natnet_ros2` | `robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py` | `VisionPoseConverterNode._canonical_quaternion` (ROS-stubbed) | +| `natnet_ros2` (C++) | `robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp` | `build_covariance_6x6`, `negotiate()`, `INatNetClient` seam | +| `lidar_point_cloud_filter` | `robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py` | Pure-numpy range validation rules | + +Corresponding proxies: `tests/robot/perception/natnet_ros2/test_natnet_ros2.py`, +`tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py`. + +## Files to Know + +- `.github/workflows/system-tests.yml` — CI workflow (runs `pytest tests/` including unit tests) +- `tests/pytest.ini` — mark registration (`unit`, `build_docker`, etc.) +- `tests/robot/` — proxy layer mirroring `robot/ros_ws/src/` +- `tests/sim/` — proxy layer for sim-side code (future) +- `tests/gcs/` — proxy layer for GCS code (future) +- `tests/README.md` — full test harness reference diff --git a/.agents/skills/configure-multi-robot/SKILL.md b/.agents/skills/configure-multi-robot/SKILL.md index 4fc977472..d147fdcf2 100644 --- a/.agents/skills/configure-multi-robot/SKILL.md +++ b/.agents/skills/configure-multi-robot/SKILL.md @@ -242,7 +242,7 @@ env_overrides = { } ``` -Tests that act on robots iterate `n=1..num_robots` and address them as `/robot_{n}/...` directly (see `_takeoff_one_robot` in `tests/test_takeoff_hover_land.py`). The test sets `ROS_DOMAIN_ID=n` for each per-robot subprocess (`domain_id=n` in `ros2_exec(...)`), matching what the resolver assigned inside the container. **If you write a new test that talks to a robot, follow this same `domain_id=n` + `/robot_{n}/...` pattern.** +Tests that act on robots iterate `n=1..num_robots` and address them as `/robot_{n}/...` directly (see `_takeoff_one_robot` in `tests/system/test_takeoff_hover_land.py`). The test sets `ROS_DOMAIN_ID=n` for each per-robot subprocess (`domain_id=n` in `ros2_exec(...)`), matching what the resolver assigned inside the container. **If you write a new test that talks to a robot, follow this same `domain_id=n` + `/robot_{n}/...` pattern.** CLI passthrough: diff --git a/.agents/skills/docker-build-profiles/SKILL.md b/.agents/skills/docker-build-profiles/SKILL.md new file mode 100644 index 000000000..64b579a01 --- /dev/null +++ b/.agents/skills/docker-build-profiles/SKILL.md @@ -0,0 +1,134 @@ +# docker-build-profiles SKILL + +Summary +- Purpose: Provide actionable build-time validation snippets and YAML guidance for AirStack Docker builds. Designed for Claude/GPT-style agents that automate repo changes, CI checks, or PR review suggestions. +- Location: .agents/skills/docker-build-profiles/SKILL.md + +When to use +- When adding or updating a `docker-compose` profile that passes `PYTHON_VERSION`, `ROS_DISTRO`, or other numeric-like build args. +- When an automated agent needs to verify a new profile will produce a correct `PYTHONPATH` and avoid YAML float-parsing bugs. + +Actions the agent can perform +1. Validate `docker-compose.yaml` args are quoted when numeric-like (e.g. `PYTHON_VERSION: "3.10"`). +2. Insert a build-time validation `RUN` into `robot/docker/Dockerfile.robot` to fail early when the ROS Python path does not exist. +3. Add or update a short test in documentation showing how to build the `builder` stage and check `ament_package` import. +4. Suggest `network: host` under `build:` for L4T/Jetson profiles only when necessary (kernel iptables workarounds). + +Snippets (copyable) + +- YAML-check rule (agent pseudocode): + + - If a `build.args` key named `PYTHON_VERSION` exists and the value matches `/^\d+\.\d+$/`, ensure it's a quoted string in YAML; otherwise update to `""`. + +- Dockerfile validation snippet (recommended, place before using `PYTHON_VERSION` to compose `PYTHONPATH`): + +```dockerfile +RUN test -d /opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION} \ + || (echo "Invalid PYTHON_VERSION=${PYTHON_VERSION} or missing ROS python path" && exit 1) +``` + +- Quick builder-stage test commands (agent can run or instruct user to run): + +```bash +DOCKER_BUILDKIT=1 docker build --target builder \ + -f robot/docker/Dockerfile.robot \ + --build-arg BASE_IMAGE= \ + --build-arg ROS_DISTRO= \ + --build-arg PYTHON_VERSION="" \ + -t airstack-builder-test:local robot/docker + +docker run --rm -it airstack-builder-test:local bash -c "python3 -c 'import ament_package; print(ament_package.__file__)'" +``` + +Guidance for agents when editing the repo +- Prefer making minimal, reversible changes: add the `RUN test -d ...` check early in the Dockerfile and gate it with informative message text. +- When updating `docker-compose.yaml`, only quote the numeric-like values; do not change unrelated fields. +- If creating PRs, include a short note in the PR description instructing maintainers to run the builder-stage sanity build on both an amd64 desktop profile and an arm64 L4T profile. + +Troubleshooting notes +- YAML quirk: unquoted `3.10` may be parsed as float `3.1` — this changes path strings and breaks imports (e.g., `python3.1` instead of `python3.10`). +- Jetson/L4T builds may require `network: host` during the build to avoid kernel iptables/raw table missing-module errors. +- Jetson **`robot-l4t`** builds from **`robot-l4t-stack-base`** (`robot/docker/Dockerfile.l4t-stack-base`), not raw dustynv, so **`Dockerfile.robot` stays Ubuntu-shaped.** `airstack image-build --profile l4t robot-l4t` triggers **`robot-l4t-stack-base`** first (`airstack.sh`); bare `compose build robot-l4t` can still parallelize badly, so list stack-base explicitly if not using AirStack CLI. + +Examples of agent prompts +- "Check `robot/docker/docker-compose.yaml` for `PYTHON_VERSION` entries and quote any unquoted numeric values; open a PR with the fixes and include a test log from a builder-stage build." +- "Insert a build-time validation `RUN` in `robot/docker/Dockerfile.robot` that ensures `/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION}` exists; push as a separate small commit." + +Notes +- This SKILL is intended for agent workflows (automated PRs, repo fixes, review suggestions). Keep changes explicit and reversible. +- For human-facing docs, maintain a high-level page in `docs/` that links to this SKILL for actionable snippets and agent tasks. + +SKILL vs human docs + +- Keep SKILLs low-level and exact: this file contains raw `docker` commands and copyable build-time snippets intended for agents and automation. +- Keep human-facing docs (`docs/`) showing the `airstack` CLI equivalents and higher-level workflows. This reduces cognitive load for maintainers while preserving exact commands in SKILLs for automation and debugging. +- For the robot profile, human docs should prefer `airstack image-build --target builder --progress=plain ` when showing how to inspect build output. + +Creating a new profile (step-by-step) + +This section shows the minimal, recommended steps an agent or maintainer should perform to add a new `docker-compose` profile that builds from `Dockerfile.robot`. + +1. Pick a sensible service name and base image + + - Choose a service name that clearly indicates the platform, e.g. `robot-desktop`, `robot-l4t`, or `robot-myboard`. + - Select an appropriate `BASE_IMAGE` (amd64 desktop base or `nvcr.io/nvidia/l4t-jetpack:...` for Jetson). + +2. Add the profile with quoted numeric args + + - Add a service block in `robot/docker/docker-compose.yaml` (or an override file) and set `build.args` for the profile. + - Always quote `PYTHON_VERSION` values (e.g. `"3.10"`) so YAML does not convert them to floats. + + Example snippet to add: + + ```yaml + robot-myboard: + build: + context: ./robot/docker + dockerfile: ./Dockerfile.robot + args: + BASE_IMAGE: nvcr.io/nvidia/l4t-jetpack:r36.4.0 + ROS_DISTRO: humble + PYTHON_VERSION: "3.10" + REAL_ROBOT: true + SKIP_MACVO: true + # for L4T builds only when necessary + # network: host + ``` + +3. Add an optional validate-early check (recommended) + + - Insert the `RUN test -d /opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION}` check near the top of `Dockerfile.robot` (before `ENV PYTHONPATH` or any Python-dependent operations). This ensures the build fails fast with a clear message. + +4. Run the builder-stage sanity build + + - Run the builder-target build locally (or in CI) to verify the image picks up the correct Python/ROS paths and that `ament_package` imports: + + ```bash + DOCKER_BUILDKIT=1 docker build --target builder \ + -f robot/docker/Dockerfile.robot \ + --build-arg BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0 \ + --build-arg ROS_DISTRO=humble \ + --build-arg PYTHON_VERSION="3.10" \ + -t airstack-builder-test:local robot/docker + + docker run --rm airstack-builder-test:local python3 -c "import ament_package; print('ok', ament_package.__file__)" + ``` + +5. Smoke-run the full compose build (optional but recommended) + + - Use `docker compose -f robot/docker/docker-compose.yaml build robot-myboard` to ensure compose passes the args correctly. + +6. Prepare the PR with clear validation notes + + - Make the code change small and focused (one commit to `docker-compose.yaml`, one optional commit for the `Dockerfile` validation line). + - In the PR description include the builder-stage test command output and request a reviewer to run the builder-stage test on both an amd64 and arm64 profile if possible. + +7. Merge and monitor + + - After merge, ensure CI (if configured) runs the sanity build or that maintainers run the checks on the target hardware. + +Agent implementation tips + +- When automating the change, produce a single commit that updates only the new service block and, if needed, a second commit that adds the `RUN` check to `Dockerfile.robot`. +- If the target is Jetson/L4T, add `network: host` under `build:` only when prior builds show iptables/kernel errors; do not enable it by default. +- If you detect a pre-existing unquoted `PYTHON_VERSION` in the repo, prefer to update that entry in-place and include an explanatory commit message about YAML float parsing. diff --git a/.agents/skills/run-system-tests/SKILL.md b/.agents/skills/run-system-tests/SKILL.md index 868d8c695..453bbc953 100644 --- a/.agents/skills/run-system-tests/SKILL.md +++ b/.agents/skills/run-system-tests/SKILL.md @@ -22,15 +22,39 @@ This skill is about the **test harness itself** — pytest marks, fixtures, the ## Test Suite Overview -The suite lives at `tests/` (repo root) and is fully pytest-based. Configuration is in `tests/pytest.ini` and shared infrastructure in `tests/conftest.py`. Marks include `build_docker`, `build_packages`, `liveliness`, `sensors`, and `takeoff_hover_land`: +The suite lives at `tests/` (repo root) and is fully pytest-based. Configuration is in `tests/pytest.ini` and shared infrastructure in `tests/conftest.py`. + +- **`tests/system/`** — Docker stack integration tests. Marks: `build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`. +- **`tests/robot/`** and **`tests/sim/`** — Hermetic **unit** tests (`@pytest.mark.unit`). These are **thin proxy files** that re-export tests from each ROS 2 package's own `test/` directory (co-located with the source, the ROS 2 / colcon convention). The proxy pattern keeps test source next to the code it tests while making tests discoverable by `pytest tests/`. + +### Unit tests vs system tests + +| Concern | Unit (`-m unit`) | System (`-m liveliness` etc.) | +|---|---|---| +| Hardware required | None — pure Python | Docker daemon, NVIDIA GPU, sim license | +| CI workflow | `system-tests.yml` (included in `pytest tests/`) | `system-tests.yml` (GPU OpenStack VM) | +| Trigger | Every push + PR (automatic) | PR opened, `/pytest` comment, `workflow_dispatch` | +| Source location | `/test/test_*.py` (proxied via `tests/robot/`) | `tests/system/` | +| How to add | See `add-unit-tests` skill | See *Adding a New System Test* below | + +Run unit tests without any Docker stack: + +```bash +airstack test -m unit -v +# or +pytest tests/ -m unit -v # AIRSTACK_ROOT=$(pwd) for direct pytest +``` + +For details on the proxy pattern and adding new unit tests, see the +`add-unit-tests` skill. | File | Mark | What it tests | Hardware required | |------|------|---------------|-------------------| -| `tests/test_build_docker.py` | `build_docker` | `airstack image-build` for `robot-desktop`, `gcs`, `isaac-sim`, `ms-airsim`; records image size to `metrics.json` | Docker daemon | -| `tests/test_build_packages.py` | `build_packages` | `colcon build` (`bws`) inside the robot, GCS, and ms-airsim ROS workspaces — brought up with `AUTOLAUNCH=false` | Docker daemon | -| `tests/test_liveliness.py` | `liveliness` | Stack bring-up: containers Running, `/clock` readiness, tmux panes, sentinel ROS 2 nodes, compute, infra-only `test_stable` | Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, sim license / Omniverse creds | -| `tests/test_sensors.py` | `sensors` | Topic Hz (Isaac: batched on sim + robot; LiDAR `echo-once` + cloud sanity), RTF, `test_sensor_streams_stable` | Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, sim license / Omniverse creds | -| `tests/test_takeoff_hover_land.py` | `takeoff_hover_land` | 4-phase flight chain per `(sim, num_robots, iteration, velocity)`: `test_px4_ready` → `test_takeoff` → `test_hover` → `test_landing`. Records altitude error, overshoot, hover stability, landing accuracy, odometry drift | Docker daemon, NVIDIA GPU, sim license | +| `tests/system/test_build_docker.py` | `build_docker` | `airstack image-build` for `robot-desktop`, `gcs`, `isaac-sim`, `ms-airsim`; records image size to `metrics.json` | Docker daemon | +| `tests/system/test_build_packages.py` | `build_packages` | `colcon build` (`bws`) inside the robot, GCS, and ms-airsim ROS workspaces — brought up with `AUTOLAUNCH=false` | Docker daemon | +| `tests/system/test_liveliness.py` | `liveliness` | Stack bring-up: containers Running, `/clock` readiness, tmux panes, sentinel ROS 2 nodes, compute, infra-only `test_stable` | Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, sim license / Omniverse creds | +| `tests/system/test_sensors.py` | `sensors` | Topic Hz (Isaac: batched on sim + robot; LiDAR `echo-once` + cloud sanity), RTF, `test_sensor_streams_stable` | Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, sim license / Omniverse creds | +| `tests/system/test_takeoff_hover_land.py` | `takeoff_hover_land` | 4-phase flight chain per `(sim, num_robots, iteration, velocity)`: `test_px4_ready` → `test_takeoff` → `test_hover` → `test_landing`. Records altitude error, overshoot, hover stability, landing accuracy, odometry drift | Docker daemon, NVIDIA GPU, sim license | The marks are declared in `tests/pytest.ini`. **Do not invent new marks ad-hoc** — register any new mark there or pytest will warn about unknown marks. @@ -39,10 +63,10 @@ The marks are declared in `tests/pytest.ini`. **Do not invent new marks ad-hoc** `conftest.py` enforces a deterministic global order so cheap-and-fast-failing tests surface first: ``` -test_build_docker → test_build_packages → test_liveliness → test_sensors → test_takeoff_hover_land +system.test_build_docker → system.test_build_packages → system.test_liveliness → system.test_sensors → system.test_takeoff_hover_land ``` -Within `test_takeoff_hover_land`, items are re-sorted to `(airstack_env, velocity, phase)` so each `(sim, robots, iter)` env brings the stack up once and the drone goes ground → air → ground per velocity before pytest moves to the next velocity. +Within `system.test_takeoff_hover_land`, items are re-sorted to `(airstack_env, velocity, phase)` so each `(sim, robots, iter)` env brings the stack up once and the drone goes ground → air → ground per velocity before pytest moves to the next velocity. ### Isaac Sim (`sensors`): why Hz is batched and LiDAR uses `echo --once` @@ -134,10 +158,10 @@ The `airstack_env` fixture is parametrized over `(sim, num_robots, iteration)` t | `--sim` | `msairsim,isaacsim` | `airstack_env` | One env-tuple per sim | | `--num-robots` | `1,3` | `airstack_env` | Cross-product with sim | | `--stress-iterations` | `1` | `airstack_env` | Up/down cycles per `(sim, num_robots)` | -| `--stable-duration` | `120` | `test_liveliness::test_stable` and `test_sensors::test_sensor_streams_stable` | Total seconds polled | -| `--stable-interval` | `10` | `test_liveliness::test_stable` and `test_sensors::test_sensor_streams_stable` | Seconds between polls | +| `--stable-duration` | `120` | `system.test_liveliness::test_stable` and `system.test_sensors::test_sensor_streams_stable` | Total seconds polled | +| `--stable-interval` | `10` | `system.test_liveliness::test_stable` and `system.test_sensors::test_sensor_streams_stable` | Seconds between polls | | `--gui` | off (headless) | `airstack_env` | Sets `QT_QPA_PLATFORM=offscreen` when off | -| `--takeoff-velocities` | `0.5` (current default) | `test_takeoff_hover_land` | One full 4-phase chain per velocity | +| `--takeoff-velocities` | `0.5` (current default) | `system.test_takeoff_hover_land` | One full 4-phase chain per velocity | Total parametrize cardinality for sim tests = `len(sims) × len(num_robots) × stress_iterations × len(velocities for takeoff)`. Keep this small locally — a 2×2×3×3 sweep on a workstation is several hours. @@ -198,10 +222,10 @@ tests/results/2025-04-21_14-30-00/ ├── results.xml # JUnit XML — durations + pass/fail per test ├── metrics.json # Custom metrics keyed by test_node_id → metric_key └── logs/ - ├── test_build_docker.TestDockerBuilds.test_build_robot_desktop.log - ├── test_sensors.TestSensors.test_sensor_streams_stable[msairsim-rob#1-iter0].log - ├── test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0].log - ├── airstack_env.test_liveliness.TestLiveliness.test_robot_containers_running[...].log + ├── system.test_build_docker.TestDockerBuilds.test_build_robot_desktop.log + ├── system.test_sensors.TestSensors.test_sensor_streams_stable[msairsim-rob#1-iter0].log + ├── system.test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0].log + ├── airstack_env.system.test_liveliness.TestLiveliness.test_robot_containers_running[...].log └── ... ``` @@ -211,7 +235,7 @@ tests/results/2025-04-21_14-30-00/ ```json { - "test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0]": { + "system.test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0]": { "airstack_up_duration_s": {"value": 42.7, "unit": "s", "direction": "lower_is_better"}, "robot.sensors.front_stereo.left.image_rect.hz_samples": { "samples": [{"t": 10, "value": 19.27}, {"t": 20, "value": 19.31}, ...] @@ -266,7 +290,7 @@ If your test... ### 2. File location and naming -- File: `tests/test_.py` — matches pytest's default test discovery (`test_*.py`) +- File: `tests/system/test_.py` — matches pytest's default test discovery (`test_*.py`) under the system suite - Class: `Test` with the mark applied at the class level: `@pytest.mark.` - Add a class-level `@pytest.mark.timeout()` — long-running sim tests need it - Imports: pull helpers from `conftest` directly (`from conftest import ...`); `tests/` is on `sys.path` because `testpaths = .` in pytest.ini @@ -274,7 +298,7 @@ If your test... ### 3. Decide if you need `airstack_env` - **Need full stack up (sim + robot + GCS)?** Take `airstack_env` as a fixture argument. You'll automatically be parametrized over `(sim, num_robots, iteration)` from CLI flags — `pytest_generate_tests` in conftest activates this only for tests that name the fixture. -- **Just need one container or no containers?** Don't take `airstack_env` — bring up only what you need with `airstack_cmd("up", "", env_overrides={"AUTOLAUNCH": "false"})` and tear down in a `try/finally`, the way `test_build_packages.py` does. +- **Just need one container or no containers?** Don't take `airstack_env` — bring up only what you need with `airstack_cmd("up", "", env_overrides={"AUTOLAUNCH": "false"})` and tear down in a `try/finally`, the way `tests/system/test_build_packages.py` does. - **Need extra parametrization** (e.g. velocity for `takeoff_hover_land`)? Add a module-level `pytest_generate_tests(metafunc)` in your test file. Don't put it in `conftest.py` unless it applies broadly. ### 4. Use the existing helpers diff --git a/.env b/.env index e368cd50b..82cc01ccb 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.2" +VERSION="0.19.0-alpha.3" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. @@ -53,4 +53,4 @@ DEBUG_RVIZ="false" # "true" or "false". If true, launches RViz alongside the ro # offboard API streaming out. this is so that ports don't conflict for multi-agent FCU communication. OFFBOARD_BASE_PORT=14540 -ONBOARD_BASE_PORT=14580 \ No newline at end of file +ONBOARD_BASE_PORT=14580 diff --git a/AGENTS.md b/AGENTS.md index f9eed977c..b53069e75 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,6 +93,7 @@ For detailed step-by-step instructions, refer to the **`.agents/skills/`** direc | [debug-module](.agents/skills/debug-module) | Autonomous debugging of ROS 2 modules | | [update-documentation](.agents/skills/update-documentation) | Documenting new modules and updating mkdocs | | [test-in-simulation](.agents/skills/test-in-simulation) | End-to-end simulation testing of a module | +| [add-unit-tests](.agents/skills/add-unit-tests) | Adding Python or C++ unit tests to a ROS 2 package (co-location + proxy pattern, CI workflow, extending to sim/GCS) | | [run-system-tests](.agents/skills/run-system-tests) | Running the pytest system test harness (marks, MetricsRecorder, /pytest PR trigger) | | [add-behavior-tree-node](.agents/skills/add-behavior-tree-node) | Creating behavior tree nodes | | [use-airstack-cli](.agents/skills/use-airstack-cli) | Using the `airstack` CLI and the non-interactive `docker exec` pattern | @@ -195,30 +196,38 @@ docker exec airstack-robot-desktop-1 bash -c "ros2 topic echo --onc - Verify module behavior in isolation - Test with synthetic data - Located in module's `test/` directory + - **Run in the robot container** with `colcon test` (after `bws`), not via `airstack test -m unit`. The root [`tests/`](tests/) suite does **not** register a `unit` pytest mark; `airstack test -m ` only selects marks declared in [`tests/pytest.ini`](tests/pytest.ini) (`build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`). -2. **System Level:** Full simulation tests (Isaac Sim or Microsoft AirSim legacy) + ```bash + docker exec airstack-robot-desktop-1 bash -c "sws && colcon test --packages-select natnet_ros2 --event-handlers console_direct+" + ``` + +2. **Unit tests (`pytest`, `unit` mark):** Fast, hermetic checks. Test **source** lives co-located with each ROS 2 package in `/test/` (standard colcon convention). Thin **proxy** files in [`tests/robot/`](tests/robot/) and [`tests/sim/`](tests/sim/) re-export those tests so `pytest tests/` discovers them. Unit tests run as part of the `system-tests.yml` suite. Example: `airstack test -m unit -v`. See `add-unit-tests` skill. + +3. **System Level (`tests/system/`):** Full simulation tests (Isaac Sim or Microsoft AirSim legacy) - End-to-end autonomy stack testing - Real sensor simulation - Multi-robot scenarios - - Implemented in [`tests/`](tests/) — see below + - Pytest modules in [`tests/system/`](tests/system/) — see below -### System Test Suite (`tests/`) +### System Test Suite (`tests/system/`) -Pytest-based system tests live at the repo root in [`tests/`](tests/). They bring up the full Docker stack (sim + robot + GCS) and verify container health, ROS 2 node presence, compute usage, sensor topic streams (``sensors`` mark), and end-to-end flight behavior. +Pytest-based system tests live under [`tests/system/`](tests/system/). They bring up the full Docker stack (sim + robot + GCS) and verify container health, ROS 2 node presence, compute usage, sensor topic streams (``sensors`` mark), and end-to-end flight behavior. | File | Mark | What it tests | Hardware | |------|------|---------------|----------| -| [`tests/test_build_docker.py`](tests/test_build_docker.py) | `build_docker` | Docker image builds (robot-desktop, gcs, isaac-sim, ms-airsim) | Docker | -| [`tests/test_build_packages.py`](tests/test_build_packages.py) | `build_packages` | `colcon build` inside each container | Docker | -| [`tests/test_liveliness.py`](tests/test_liveliness.py) | `liveliness` | Stack bring-up: containers, ``/clock`` readiness, tmux, sentinel ROS 2 nodes, compute, infra-only stability poll | Docker, GPU, sim license | -| [`tests/test_sensors.py`](tests/test_sensors.py) | `sensors` | Topic Hz (Isaac: batched sim + robot ``ros2 topic hz``; filtered LiDAR ``echo-once`` + validation script), RTF, sensor stability time-series | Docker, GPU, sim license | -| [`tests/test_takeoff_hover_land.py`](tests/test_takeoff_hover_land.py) | `takeoff_hover_land` | 4-phase flight chain (PX4 ready → takeoff → hover → land) per (sim, num_robots, iter, velocity) | Docker, GPU, sim license | +| [`tests/system/test_build_docker.py`](tests/system/test_build_docker.py) | `build_docker` | Docker image builds (robot-desktop, gcs, isaac-sim, ms-airsim) | Docker | +| [`tests/system/test_build_packages.py`](tests/system/test_build_packages.py) | `build_packages` | `colcon build` inside each container | Docker | +| [`tests/system/test_liveliness.py`](tests/system/test_liveliness.py) | `liveliness` | Stack bring-up: containers, ``/clock`` readiness, tmux, sentinel ROS 2 nodes, compute, infra-only stability poll | Docker, GPU, sim license | +| [`tests/system/test_sensors.py`](tests/system/test_sensors.py) | `sensors` | Topic Hz (Isaac: batched sim + robot ``ros2 topic hz``; filtered LiDAR ``echo-once`` + validation script), RTF, sensor stability time-series | Docker, GPU, sim license | +| [`tests/system/test_takeoff_hover_land.py`](tests/system/test_takeoff_hover_land.py) | `takeoff_hover_land` | 4-phase flight chain (PX4 ready → takeoff → hover → land) per (sim, num_robots, iter, velocity) | Docker, GPU, sim license | Shared fixtures, the `airstack_env` parametrized fixture, and `MetricsRecorder` live in [`tests/conftest.py`](tests/conftest.py). Each run produces a timestamped directory under `tests/results//` with `results.xml`, `metrics.json`, and per-test logs. [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). **Run via the CLI** (containerized runner — no local Python needed): ```bash +airstack test -m unit -v airstack test -m "build_docker or build_packages" -v airstack test -m liveliness --sim msairsim --num-robots 1 --stress-iterations 1 -v airstack test -m sensors --sim isaacsim --num-robots 1 --stress-iterations 1 -v diff --git a/airstack.sh b/airstack.sh index 78b475c07..1c57c47cb 100755 --- a/airstack.sh +++ b/airstack.sh @@ -149,6 +149,7 @@ function print_command_help { echo "Options:" echo " --no-shell Don't modify shell configuration" echo " --no-config Skip configuration tasks (Isaac Sim, Nucleus, Git hooks)" + echo " --no-natnet Skip NatNet SDK installation for OptiTrack motion capture support" echo "" echo "This command adds an 'airstack' function to your shell profile that will" echo "automatically find and use the airstack.sh script from the current directory" @@ -235,10 +236,11 @@ function print_command_help { echo "Results are written to tests/results//." echo "" echo "Test marks (-m):" + echo " unit Fast hermetic tests (robot/sim mirrored layout; no Docker stack)" echo " build_docker Docker image build tests (no GPU needed)" echo " build_packages colcon workspace build tests (no GPU needed)" echo " liveliness Full stack up: nodes, topics, compute, stability" - echo " autonomy Takeoff / hover / land flight chain" + echo " takeoff_hover_land Takeoff / hover / land flight chain" echo "" echo "AirStack-specific options:" echo " --sim=TARGETS Comma-separated sim targets" @@ -618,18 +620,33 @@ function cmd_install { log_info "Installation complete!" } +function cmd_setup_natnet_sdk { + local sdk_script="$PROJECT_ROOT/robot/ros_ws/src/perception/natnet_ros2/scripts/download-natnet-sdk.sh" + + if [ ! -f "$sdk_script" ]; then + log_warn "NatNet SDK installer not found at $sdk_script" + return 0 + fi + + log_info "Checking NatNet SDK installation..." + bash "$sdk_script" +} + function cmd_setup { log_info "Setting up AirStack environment..." # Check for --no-shell flag local modify_shell=true local skip_config=false + local skip_natnet=false # initially set to false so that the --no-natnet flag can determine whether to skip the NatNet SDK installation for arg in "$@"; do if [ "$arg" == "--no-shell" ]; then modify_shell=false elif [ "$arg" == "--no-config" ]; then skip_config=true + elif [ "$arg" == "--no-natnet" ]; then + skip_natnet=true fi done @@ -694,6 +711,15 @@ function cmd_setup { fi fi + # Install NatNet SDK unless explicitly skipped. + if [ "$skip_natnet" = false ]; then + if declare -f "cmd_setup_natnet_sdk" > /dev/null; then + cmd_setup_natnet_sdk + else + log_warn "NatNet SDK setup helper not loaded. Skipping NatNet SDK installation." + fi + fi + # Run configuration tasks if not skipped if [ "$skip_config" = false ]; then # Check if the config module is available @@ -769,6 +795,35 @@ function classify_compose_args { done } +# robot-l4t uses Dockerfile.robot with BASE_IMAGE=robot-l4t-stack-base. Compose v2+ may schedule +# service builds in parallel, so BUILD FROM that tag can race before stack-base finishes. Ensure +# the intermediary image exists first (still requires --profile l4t like the robot-l4t build). +function ensure_robot_l4t_stack_base() { + local -n _ga="$1" + local -n _sc="$2" + local wants_l4t=false + local has_stack=false + for arg in "${_sc[@]}"; do + if [[ "$arg" == robot-l4t ]]; then + wants_l4t=true + fi + if [[ "$arg" == robot-l4t-stack-base ]]; then + has_stack=true + fi + done + if [[ "$wants_l4t" != true ]] || [[ "$has_stack" == true ]]; then + return 0 + fi + log_info "Building robot-l4t-stack-base before robot-l4t..." + local build_opts=() + for arg in "${_sc[@]}"; do + if [[ "$arg" == -* ]]; then + build_opts+=("$arg") + fi + done + run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${_ga[@]}" build "${build_opts[@]}" robot-l4t-stack-base +} + function cmd_up { check_docker @@ -812,6 +867,13 @@ function cmd_image_build { local global_args=() local subcmd_args=() classify_compose_args global_args subcmd_args "$@" + # Jetson only: building robot-l4t needs robot-l4t-stack-base first + for arg in "${subcmd_args[@]}"; do + if [[ "$arg" == robot-l4t ]]; then + ensure_robot_l4t_stack_base global_args subcmd_args + break + fi + done # Registry-cache mode (CI / opt-in): pre-pull existing images to seed the # local cache, build with BUILDKIT_INLINE_CACHE=1 so the resulting image diff --git a/common/ros_packages/msgs/airstack_msgs/package.xml b/common/ros_packages/msgs/airstack_msgs/package.xml index c040458b1..06c93abe6 100644 --- a/common/ros_packages/msgs/airstack_msgs/package.xml +++ b/common/ros_packages/msgs/airstack_msgs/package.xml @@ -16,11 +16,12 @@ rosidl_default_generators rosidl_default_runtime - rosidl_interface_packages ament_lint_auto ament_lint_common + rosidl_interface_packages + ament_cmake diff --git a/common/ros_packages/msgs/task_msgs/package.xml b/common/ros_packages/msgs/task_msgs/package.xml index 8440930c2..8e97d3397 100644 --- a/common/ros_packages/msgs/task_msgs/package.xml +++ b/common/ros_packages/msgs/task_msgs/package.xml @@ -17,11 +17,12 @@ airstack_msgs rosidl_default_runtime - rosidl_interface_packages ament_lint_auto ament_lint_common + rosidl_interface_packages + ament_cmake diff --git a/docs/development/beginner/airstack-cli/docker_usage.md b/docs/development/beginner/airstack-cli/docker_usage.md index d3e1fe3eb..9e504d6bb 100644 --- a/docs/development/beginner/airstack-cli/docker_usage.md +++ b/docs/development/beginner/airstack-cli/docker_usage.md @@ -38,6 +38,8 @@ The available image tags are listed [here](https://airlab-docker.andrew.cmu.edu/ ## Build Images +For an overview of build-time options (`BASE_IMAGE`, `ROS_DISTRO`, platform profiles), see [Docker Build Profiles](../../intermediate/docker-build-profiles.md). For runtime container operations, continue below. + ```bash # Build all images from scratch docker compose build diff --git a/docs/development/index.md b/docs/development/index.md index 656550f0d..a1ccd4663 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -31,6 +31,7 @@ Welcome to AirStack development! This guide will help you extend and customize t - [System Testing](intermediate/testing/system_testing.md) - [CI/CD](intermediate/testing/ci_cd.md) - **[Frame Conventions](intermediate/frame_conventions.md)** - Coordinate frame standards +- **[Docker Build Profiles](intermediate/docker-build-profiles.md)** - Robot image build args and platform profiles (`robot-desktop`, `robot-l4t`, etc.) - **[Contributing](intermediate/contributing.md)** - Contribute to AirStack - **[Documentation Guide](intermediate/documentation.md)** - Write great documentation @@ -52,6 +53,7 @@ Welcome to AirStack development! This guide will help you extend and customize t | Add a world model | [Integration Checklist](../robot/autonomy/integration_checklist.md) | | Create simulation scene | [Isaac Sim Setup](../simulation/isaac_sim/pegasus_scene_setup.md) | | Debug a module | [VSCode Debugging](beginner/vscode/vscode_debug.md) | +| Build robot images for a platform | [Docker Build Profiles](intermediate/docker-build-profiles.md) | | Write tests | [Testing Guide](intermediate/testing/index.md) | ### Essential Commands diff --git a/docs/development/intermediate/docker-build-profiles.md b/docs/development/intermediate/docker-build-profiles.md new file mode 100644 index 000000000..bbd3ebff1 --- /dev/null +++ b/docs/development/intermediate/docker-build-profiles.md @@ -0,0 +1,126 @@ +# Docker build profiles and build-args + +High-level overview of the build-time knobs used to create AirStack robot images. For step-by-step profile creation, validation snippets, and YAML quoting rules, use the **`docker-build-profiles` agent skill** at `.agents/skills/docker-build-profiles/` — it is intended for maintainers and AI agents generating or reviewing new compose profiles. + +## What this covers + +- AirStack builds robot images from one Dockerfile: `robot/docker/Dockerfile.robot` +- The active variant is selected by `build.args` on each service in `robot/docker/docker-compose.yaml` +- Shared defaults live in `robot/docker/robot-base-docker-compose.yaml` + +Key inputs: + +| Arg | Purpose | +|-----|---------| +| `BASE_IMAGE` | OS / vendor base image | +| `ROS_DISTRO` | ROS 2 distro to install (e.g. `jazzy`) | +| `PYTHON_VERSION` | Must match the ROS Python path; **quote in YAML** (e.g. `"3.12"`) | +| `REAL_ROBOT` | Platform-specific content toggles | +| `SKIP_MACVO` | Skip MACVO in image when set | +| `SKIP_TENSORRT` | Skip TensorRT in image when set | + +## Example profiles + +**Desktop** (`robot-desktop`) — x86-64 dev / simulation: + +```yaml +# robot/docker/docker-compose.yaml (excerpt) +robot-desktop: + build: + dockerfile: ./Dockerfile.robot + args: + BASE_IMAGE: nvidia/cuda:13.0.2-base-ubuntu24.04 + ROS_DISTRO: jazzy +``` + +**Jetson L4T** (`robot-l4t`) — uses `network: host` during build: + +```yaml +# robot/docker/docker-compose.yaml (excerpt) +robot-l4t: + build: + dockerfile: ./Dockerfile.robot + network: host + args: + BASE_IMAGE: *l4t_stack_base_image # from robot-l4t-stack-base + REAL_ROBOT: true + SKIP_MACVO: true + SKIP_TENSORRT: true + ROS_DISTRO: jazzy +``` + +**New profile skeleton** — when adding a platform, follow the agent skill for full steps: + +```yaml +robot-myboard: + build: + context: ./robot/docker + dockerfile: ./Dockerfile.robot + args: + BASE_IMAGE: nvcr.io/nvidia/l4t-jetpack:r36.4.0 + ROS_DISTRO: jazzy + PYTHON_VERSION: "3.12" + REAL_ROBOT: true + SKIP_MACVO: true + # L4T only, when needed: + # network: host +``` + +## YAML quoting + +Unquoted `3.12` can be parsed as float `3.1`, which breaks `PYTHONPATH` paths. Always quote numeric Python versions: + +```yaml +PYTHON_VERSION: "3.12" # correct +PYTHON_VERSION: 3.12 # avoid +``` + +## Build-time validation (optional) + +The agent skill recommends an early Dockerfile check before composing `PYTHONPATH`: + +```dockerfile +# robot/docker/Dockerfile.robot +RUN test -d /opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION} \ + || (echo "Invalid PYTHON_VERSION=${PYTHON_VERSION} or missing ROS python path" && exit 1) +``` + +## Network mode + +- `network: host` under `build:` is a workaround for Jetson/L4T build issues (Docker networking / host kernel modules). +- Generally **not** needed for the desktop profile. +- L4T stack-base images are built via `robot/docker/Dockerfile.l4t-stack-base` before `Dockerfile.robot` runs on Jetson targets. + +## Human-friendly `airstack` commands + +Prefer the `airstack` CLI over raw `docker compose` for day-to-day use: + +```bash +# Build a compose service image +airstack image-build robot-desktop + +# Build and start a service +airstack image-build robot-desktop +airstack up robot-desktop + +# Inspect the robot image build with full output +airstack image-build --target builder --progress=plain robot-desktop + +# Jetson: stack-base is built first automatically +airstack image-build --profile l4t robot-l4t + +# Open a shell in a running container +airstack connect robot-desktop --command=bash + +# View container logs +airstack logs robot-desktop +``` + +## Adding a new profile + +1. Pick a service name and `BASE_IMAGE` for the target platform. +2. Add a service block in `robot/docker/docker-compose.yaml` with quoted `build.args`. +3. For L4T/Jetson, consider `network: host` on `build:` and the stack-base image pattern. +4. Run `airstack image-build ` and verify the builder stage imports ROS Python packages. + +For copy-paste validation commands, compose templates, and agent-oriented checklists, see `.agents/skills/docker-build-profiles/SKILL.md`. diff --git a/docs/development/intermediate/testing/index.md b/docs/development/intermediate/testing/index.md index 9279f96c4..0769e2b09 100644 --- a/docs/development/intermediate/testing/index.md +++ b/docs/development/intermediate/testing/index.md @@ -1,23 +1,56 @@ # Testing -AirStack uses several test layers: ROS 2 package tests (`colcon test`), and **system tests** under [`tests/`](../../../../tests/) at the repo root (pytest, full Docker stack). +AirStack uses three complementary test layers, each with a distinct scope and +hardware requirement: -## System tests (`tests/`) +| Layer | Where | Mark / Tool | Hardware | +|---|---|---|---| +| **Unit tests** | `tests/robot/`, `tests/sim/` | `pytest -m unit` | None — pure Python | +| **Package tests** | `/test/` | `colcon test` | Robot container | +| **System tests** | `tests/system/` | `pytest -m liveliness` etc. | Docker, GPU, sim license | -The canonical reference is **[`tests/README.md`](../../../../tests/README.md)** (also included in the MkDocs site). In short: +## Unit tests (`pytest -m unit`) + +Fast, hermetic Python tests that run in seconds with no Docker or GPU. Test source +lives **co-located with its ROS 2 package** (`/test/`) and is re-exported +through thin proxy files in `tests/robot/` for centralized discovery. + +```bash +airstack test -m unit -v +# or directly: +pytest tests/ -m unit -v +``` + +Unit tests run as part of `system-tests.yml` via `pytest tests/` and can also be +run locally with no Docker or GPU needed. + +→ **[Unit Testing Guide](unit_testing.md)** — patterns, proxy layout, CI workflow, + how to add tests for new packages (Python and C++ gtest). + +## System tests (`tests/system/`) + +Full Docker-stack integration tests. The canonical reference is +**[`tests/README.md`](../../../../tests/README.md)**. In short: | Mark | Module | Role | -|------|--------|------| -| `liveliness` | `test_liveliness.py` | Containers, `/clock` readiness, tmux, sentinel ROS 2 nodes, compute, infra-only stability poll | -| `sensors` | `test_sensors.py` | Sim + robot stereo/depth Hz, filtered LiDAR (`echo --once` + validation script on Isaac), sim RTF, sensor stability time-series | -| `takeoff_hover_land` | `test_takeoff_hover_land.py` | Four-phase flight chain per configuration | +|---|---|---| +| `liveliness` | `system/test_liveliness.py` | Containers, `/clock` readiness, tmux, sentinel ROS 2 nodes, compute, infra-only stability poll | +| `sensors` | `system/test_sensors.py` | Sim + robot stereo/depth Hz, filtered LiDAR (`echo --once` + validation script on Isaac), sim RTF, sensor stability time-series | +| `takeoff_hover_land` | `system/test_takeoff_hover_land.py` | Four-phase flight chain per configuration | -Collection order is defined in `tests/conftest.py` (`liveliness` before `sensors` before `takeoff_hover_land`). Each mark’s test **class** uses **class-scoped** `airstack_env`, so combining marks with **`and`** runs multiple full stack bring-ups per `(sim, num_robots, iteration)` — see *Bring-up scope* in `tests/README.md`. +Collection order is defined in `tests/conftest.py` (`liveliness` before `sensors` +before `takeoff_hover_land`). Each mark's test class uses **class-scoped** +`airstack_env`, so combining marks with `and` runs multiple full stack bring-ups +per `(sim, num_robots, iteration)` — see *Bring-up scope* in `tests/README.md`. -**Isaac Sim:** the `sensors` implementation batches `ros2 topic hz` on sim and robot paths and avoids `hz` on filtered `PointCloud2`; pytest enables `ENABLE_LIDAR` for the multi-drone Pegasus script. Details: **`tests/README.md`** → *Isaac Sim and the sensors mark*. +**Isaac Sim:** the `sensors` implementation batches `ros2 topic hz` on sim and +robot paths and avoids `hz` on filtered `PointCloud2`; pytest enables `ENABLE_LIDAR` +for the multi-drone Pegasus script. Details: **`tests/README.md`** → *Isaac Sim and +the sensors mark*. ## Other testing docs +- [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, proxy pattern, CI workflow - [Testing frameworks](testing_frameworks.md) — `colcon test`, rostest patterns - [Integration testing](integration_testing.md) - [CI/CD](ci_cd.md) — pipeline overview diff --git a/docs/development/intermediate/testing/unit_testing.md b/docs/development/intermediate/testing/unit_testing.md index e69de29bb..f69056008 100644 --- a/docs/development/intermediate/testing/unit_testing.md +++ b/docs/development/intermediate/testing/unit_testing.md @@ -0,0 +1,206 @@ +# Unit Testing + +AirStack unit tests are **fast, hermetic, and purely Python** — no Docker stack, no GPU, no running containers. They run locally in seconds and gate every pull request via a dedicated GitHub Actions workflow on a standard `ubuntu-latest` runner. + +## Design principles + +- **Co-located with source.** Test files live in `/test/` alongside the code they test. This is the standard ROS 2 / colcon convention and ensures tests are discovered by both `colcon test` and `pytest`. +- **Proxy for centralized discovery.** A thin shim in `tests/robot///` re-exports the test functions so `pytest tests/` (the CI command) and `airstack test -m unit` discover them without any changes to the CI workflow. +- **`@pytest.mark.unit` on every test.** The `unit` mark is the filter that keeps unit tests isolated from system tests that need Docker, GPUs, and sim licenses. + +## Repository layout + +``` +robot/ros_ws/src/ +└── // + ├── src/ # production source + └── test/ + ├── test_.py # unit test source ← canonical location + ├── test_.cpp # C++ gtest source (optional) + └── fake_.hpp # C++ test doubles (optional) + +tests/ +├── robot/ +│ └── // +│ └── test_.py # thin proxy → package test/ +├── sim/ # future: sim-side unit tests +└── gcs/ # future: GCS unit tests +``` + +When pytest collects `tests/robot/…/test_.py`, the `<-` annotation in the +output shows the actual source location: + +``` +robot/perception/natnet_ros2/test_natnet_ros2.py::test_canonical_quaternion_identity + <- ../robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py PASSED +``` + +## Running unit tests + +```bash +# Locally — no container or Docker stack required +airstack test -m unit -v + +# Or directly with pytest (AIRSTACK_ROOT must point to the repo root) +export AIRSTACK_ROOT=$(pwd) +pip install pytest numpy +pytest tests/ -m unit -v +``` + +Unit tests complete in under one second for the current suite. + +## CI + +Unit tests are collected and run as part of `system-tests.yml` via `pytest tests/` +(no marks specified on PR open = all tests including `unit`). Run them locally at +any time with no infrastructure required: + +```bash +airstack test -m unit -v +# or directly (requires tests/requirements.txt installed): +AIRSTACK_ROOT=$(pwd) pytest tests/ -m unit -v +``` + +## Current test coverage + +| Package | Test file | What is covered | +|---|---|---| +| `natnet_ros2` | `perception/natnet_ros2/test/test_natnet_ros2.py` | `VisionPoseConverterNode._canonical_quaternion` (ROS stubbed with `sys.modules`) | +| `natnet_ros2` (C++) | `perception/natnet_ros2/test/test_natnet_logic.cpp` | `build_covariance_6x6`, topic name helpers, `ConnectConfig`, `negotiate()` via `FakeNatNetClient` | +| `lidar_point_cloud_filter` | `sensors/lidar_point_cloud_filter/test/test_validation_core.py` | Pure-numpy LiDAR range validation rules | + +## Adding a new unit test + +### Python + +**1. Write the test source in the package:** + +```python +# robot/ros_ws/src///test/test_my_module.py +import sys +from pathlib import Path +import pytest + +# Make the package importable without a colcon install +_src = Path(__file__).resolve().parent.parent / "src" +if str(_src) not in sys.path: + sys.path.insert(0, str(_src)) + +from my_module import my_function # noqa: E402 + + +@pytest.mark.unit +def test_basic(): + assert my_function(1, 2) == 3 +``` + +If the production code inherits from `rclpy.node.Node`, stub ROS at the import +boundary: + +```python +import sys +from unittest.mock import MagicMock + +class _FakeNode: + def __init__(self, name): pass + def get_logger(self): return MagicMock() + def declare_parameter(self, *a, **kw): pass + def get_parameter(self, name): + m = MagicMock(); m.value = MagicMock(); return m + def create_subscription(self, *a, **kw): return MagicMock() + def create_publisher(self, *a, **kw): return MagicMock() + +_rclpy_node_mod = MagicMock() +_rclpy_node_mod.Node = _FakeNode +sys.modules.setdefault("rclpy", MagicMock()) +sys.modules["rclpy.node"] = _rclpy_node_mod +# ... then import your module +``` + +**2. Write the thin proxy in `tests/robot/`:** + +```python +# tests/robot///test_my_module.py +"""Proxy: re-exposes unit tests from the package source tree.""" +import importlib.util +from pathlib import Path + +_repo_root = Path(__file__).resolve().parents[4] # adjust depth if needed +_real_file = _repo_root / "robot/ros_ws/src///test/test_my_module.py" + +_spec = importlib.util.spec_from_file_location("__unit_tests", _real_file) +_real = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_real) + +for _name in dir(_real): + if _name.startswith("test_"): + globals()[_name] = getattr(_real, _name) +``` + +The unique module name (e.g. `"__unit_tests"`) prevents a circular import: +pytest adds the proxy's directory to `sys.path` at collection time, which would +cause `from test_my_module import *` to import the proxy itself. + +**3. Verify:** + +```bash +pytest tests/ -m unit -v +``` + +### C++ (gtest) + +C++ tests live entirely in the package and run via `colcon test` — no proxy needed. + +**`CMakeLists.txt`:** + +```cmake +if(BUILD_TESTING) + find_package(ament_cmake_gtest REQUIRED) + ament_add_gtest(test_my_name test/test_my_name.cpp) + target_include_directories(test_my_name PRIVATE + $ + $) +endif() +``` + +**`package.xml`:** + +```xml +ament_cmake_gtest +``` + +**Run inside the robot container:** + +```bash +docker exec airstack-robot-desktop-1 bash -c \ + "bws --cmake-args '-DBUILD_TESTING=ON' --packages-select " +docker exec airstack-robot-desktop-1 bash -c \ + "colcon test --packages-select --event-handlers console_direct+" +``` + +The `build_packages` CI job (`tests/system/test_build_packages.py`) also runs +`colcon test` with `BUILD_TESTING=ON` so C++ gtests are gated in CI as well. + +## Extending to sim and GCS + +The proxy pattern extends to other components. As sim-side Python logic (e.g. the +[Motive emulator](../../../../tests/sim/motive_emulator/README.md)) and GCS modules +acquire unit-testable code, follow the same layout: + +``` +simulation/...//test/test_.py ← source +tests/sim//test_.py ← proxy (parents[3] to reach repo root) + +gcs/...//test/test_.py ← source +tests/gcs//test_.py ← proxy +``` + +`pytest tests/ -m unit` discovers them automatically — no changes to `pytest.ini` +or CI changes needed. + +## See also + +- [`.agents/skills/add-unit-tests`](../../../../.agents/skills/add-unit-tests/SKILL.md) — step-by-step agent workflow +- [System tests](../../../../tests/README.md) — full Docker-stack integration tests +- [CI/CD](ci_cd.md) — pipeline overview and ephemeral runner architecture +- [Testing frameworks](testing_frameworks.md) — `colcon test`, ament linters diff --git a/docs/robot/autonomy/perception/index.md b/docs/robot/autonomy/perception/index.md index 789fce478..87a113c15 100644 --- a/docs/robot/autonomy/perception/index.md +++ b/docs/robot/autonomy/perception/index.md @@ -31,7 +31,11 @@ ros2 launch perception_bringup perception.launch.xml ## Modules -- [**State Estimation**](state_estimation.md) - Overview of state estimation approaches and implementations +State estimation and related perception packages live under `robot/ros_ws/src/perception/`. Only external motion capture is documented below today; other approaches (onboard sensor fusion, visual-inertial odometry, etc.) will be added here in future releases. + +### External pose (motion capture) + +- [**NatNet (OptiTrack)**](../../../../robot/ros_ws/src/perception/natnet_ros2/README.md) — Receives rigid-body poses from an external Motive PC over NatNet UDP and publishes `/{robot_name}/perception/optitrack/...` topics. Optional MAVROS bridge for PX4 vision pose. Enabled with `LAUNCH_NATNET=true` in `.env` (off by default). ## Configuration diff --git a/docs/robot/docker/index.md b/docs/robot/docker/index.md index db2fd8bcd..1502fdeb6 100644 --- a/docs/robot/docker/index.md +++ b/docs/robot/docker/index.md @@ -43,6 +43,8 @@ robot_base (robot-base-docker-compose.yaml) ## Platform Profiles +Build-time args (`BASE_IMAGE`, `ROS_DISTRO`, `PYTHON_VERSION`, and platform toggles) are set per service in `docker-compose.yaml`. See [Docker Build Profiles](../../development/intermediate/docker-build-profiles.md) for how those args map to image variants. + Select a profile by passing `--profile ` to `docker compose` (or via the `airstack` CLI). ### `desktop` — x86-64 development machine diff --git a/mkdocs.yml b/mkdocs.yml index c4d92fede..4dc57b961 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -66,9 +66,12 @@ nav: - docs/development/beginner/fork_your_own_project.md - Intermediate Tutorials: - Testing: - - tests/README.md - - tests/ci-cd-orchestrator.md + - Overview: docs/development/intermediate/testing/index.md + - Unit Testing: docs/development/intermediate/testing/unit_testing.md + - System Tests: tests/README.md + - CI/CD Orchestrator: tests/ci-cd-orchestrator.md - Frame Conventions: docs/development/intermediate/frame_conventions.md + - Docker Build Profiles: docs/development/intermediate/docker-build-profiles.md - Contributing: - docs/development/intermediate/contributing.md - docs/development/intermediate/documentation.md @@ -111,7 +114,7 @@ nav: - Gimbal: docs/robot/autonomy/sensors/gimbal.md - Perception: - docs/robot/autonomy/perception/index.md - - State Estimation: docs/robot/autonomy/perception/state_estimation.md + - NatNet (OptiTrack): robot/ros_ws/src/perception/natnet_ros2/README.md - Local: - docs/robot/autonomy/local/index.md - World Model: diff --git a/robot/docker/Dockerfile.l4t-stack-base b/robot/docker/Dockerfile.l4t-stack-base new file mode 100644 index 000000000..0ebb8cd14 --- /dev/null +++ b/robot/docker/Dockerfile.l4t-stack-base @@ -0,0 +1,50 @@ +# Thin intermediary for Jetson: dustynv Jazzy (Noble) + quirks so Dockerfile.robot matches Ubuntu/desktop flow. +# +# Builds: PROJECT_DOCKER_REGISTRY/...:_robot-l4t-stack-base_${DOCKER_IMAGE_BUILD_MODE} +# Build before robot-l4t: compose build robot-l4t-stack-base robot-l4t +# +# - Refreshes ROS apt keyring (dusty snapshots can carry expired signatures). +# - Sets pip defaults to PyPI and clears dusty PIP_CONSTRAINT / PIP_EXTRA_INDEX_URL (mirror hash/index surprises during build). +# - Reconciles OpenCV CUDA overlay vs Ubuntu libopencv*-dev pulls from ROS stacks. +# - Rewrites prebuilt /opt/ros to use /usr/bin/python3 instead of dusty /opt/venv, then removes the venv +# so colcon/rosidl use the same system Python path as Dockerfile.robot Ubuntu images. + +ARG DUSTYNV_IMAGE=dustynv/ros:jazzy-ros-base-r36.4.0-cu128-24.04 +ARG ROS_DISTRO=jazzy +FROM ${DUSTYNV_IMAGE} + +ARG ROS_DISTRO=jazzy +ENV ROS_DISTRO=${ROS_DISTRO} +ENV DEBIAN_FRONTEND=noninteractive + +RUN curl -fsSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key \ + -o /usr/share/keyrings/ros-archive-keyring.gpg \ + && chmod 644 /usr/share/keyrings/ros-archive-keyring.gpg + +ENV PIP_INDEX_URL=https://pypi.org/simple \ + PIP_CONSTRAINT= \ + PIP_EXTRA_INDEX_URL= + +RUN apt-get update \ + && apt-get install -y -o Dpkg::Options::=--force-overwrite --no-install-recommends \ + libopencv-dev \ + python3-empy \ + python3-yaml \ + && rm -rf /var/lib/apt/lists/* + +# Point all dusty /opt/venv/python references at system Python (matches Ubuntu rosidl behavior). +RUN ros="/opt/ros/${ROS_DISTRO}"; \ + if [ -d "$ros" ]; then \ + ( grep -rlIZ '/opt/venv' "$ros" 2>/dev/null || true ) \ + | xargs -0 -r sed -i \ + -e 's#/opt/venv/bin/python3#/usr/bin/python3#g' \ + -e 's#/opt/venv/bin:##g' \ + -e 's#:/opt/venv/bin##g'; \ + fi; \ + rm -rf /opt/venv + +ENV AMENT_PYTHON_EXECUTABLE=/usr/bin/python3 +ENV PYTHON_EXECUTABLE=/usr/bin/python3 + +# Prefer system/cuda tooling over any removed venv prefix (CUDA symlink is usually /usr/local/cuda). +ENV PATH="/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index 4bd86d0c6..894874961 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -1,4 +1,4 @@ -# either ubuntu:24.04 or l4t. ubuntu:24.04 is default +# either ubuntu:24.04 / nvidia CUDA, or Jetson intermediary from Dockerfile.l4t-stack-base (see robot-l4t in compose). ARG BASE_IMAGE # ============================================================ # Stage 1 — builder: compile/download everything @@ -13,6 +13,12 @@ ARG INSTALL_FLAGS="-o APT::Get::AllowUnauthenticated=true" ARG SKIP_MACVO=false ARG SKIP_TENSORRT=false +ARG PIP_VERSION=24.0 +ARG PYTHON_VERSION=3.12 + +ARG ROS_DISTRO=jazzy +ENV ROS_DISTRO=${ROS_DISTRO} + # from https://github.com/athackst/dockerfiles/blob/main/ros2/jazzy.Dockerfile ENV DEBIAN_FRONTEND=noninteractive @@ -53,16 +59,15 @@ RUN sudo add-apt-repository universe \ && curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg \ && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null \ && apt-get ${UPDATE_FLAGS} update -y && apt-get ${INSTALL_FLAGS} install -y --no-install-recommends \ - ros-jazzy-desktop \ + ros-${ROS_DISTRO}-desktop \ python3-argcomplete \ && rm -rf /var/lib/apt/lists/* -ENV ROS_DISTRO=jazzy -ENV AMENT_PREFIX_PATH=/opt/ros/jazzy -ENV COLCON_PREFIX_PATH=/opt/ros/jazzy -ENV LD_LIBRARY_PATH=/opt/ros/jazzy/lib/x86_64-linux-gnu:/opt/ros/jazzy/lib -ENV PATH=/opt/ros/jazzy/bin:$PATH -ENV PYTHONPATH=/opt/ros/jazzy/local/lib/python3.12/dist-packages:/opt/ros/jazzy/lib/python3.12/site-packages +ENV AMENT_PREFIX_PATH=/opt/ros/${ROS_DISTRO} +ENV COLCON_PREFIX_PATH=/opt/ros/${ROS_DISTRO} +ENV LD_LIBRARY_PATH=/opt/ros/${ROS_DISTRO}/lib/x86_64-linux-gnu:/opt/ros/${ROS_DISTRO}/lib +ENV PATH=/opt/ros/${ROS_DISTRO}/bin:$PATH +ENV PYTHONPATH=/opt/ros/${ROS_DISTRO}/local/lib/python${PYTHON_VERSION}/dist-packages:/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION}/site-packages ENV ROS_PYTHON_VERSION=3 ENV ROS_VERSION=2 ENV ROS_AUTOMATIC_DISCOVERY_RANGE=SUBNET @@ -81,24 +86,31 @@ RUN apt update && apt install -y --no-install-recommends \ xvfb \ && rm -rf /var/lib/apt/lists/* +# Freeze pip and setuptools versions (ignore-installed: apt-shipped wheel/setuptools have no pip RECORD). +RUN python3 -m pip install --no-cache-dir --break-system-packages --ignore-installed --upgrade \ + "pip==${PIP_VERSION}" \ + "setuptools==79.0.1" \ + wheel + # Install any additional ROS2 packages RUN apt update -y && apt install -y --no-install-recommends \ ros-dev-tools \ - ros-jazzy-mavros \ - ros-jazzy-tf2* \ - ros-jazzy-stereo-image-proc \ - ros-jazzy-image-view \ - ros-jazzy-topic-tools \ - ros-jazzy-grid-map \ - ros-jazzy-domain-bridge \ - ros-jazzy-rosbag2-storage-mcap \ - ros-jazzy-xacro \ - ros-jazzy-foxglove-bridge \ + ros-${ROS_DISTRO}-mavros \ + ros-${ROS_DISTRO}-tf2* \ + ros-${ROS_DISTRO}-stereo-image-proc \ + ros-${ROS_DISTRO}-image-view \ + ros-${ROS_DISTRO}-topic-tools \ + ros-${ROS_DISTRO}-grid-map \ + ros-${ROS_DISTRO}-domain-bridge \ + ros-${ROS_DISTRO}-rosbag2-storage-mcap \ + ros-${ROS_DISTRO}-xacro \ + ros-${ROS_DISTRO}-ament-package \ + ros-${ROS_DISTRO}-foxglove-bridge \ libcgal-dev \ python3-colcon-common-extensions \ && rm -rf /var/lib/apt/lists/* -RUN /opt/ros/jazzy/lib/mavros/install_geographiclib_datasets.sh +RUN /opt/ros/${ROS_DISTRO}/lib/mavros/install_geographiclib_datasets.sh # Install TensorRT (NVIDIA/L4T images only, unless SKIP_TENSORRT=true) # Note: TensorRT 8 packages may not be available for Ubuntu 24.04, so this is optional @@ -182,6 +194,25 @@ RUN if [ "${SKIP_MACVO}" != "true" ]; then \ # TMux config RUN git clone --depth 1 https://github.com/tmux-plugins/tpm /root/.tmux/plugins/tpm +# Diagnostic: Check Python environment before DDS Router build +RUN echo "=== Python version ===" && \ + python3 --version && \ + echo "" && \ + echo "=== PYTHONPATH ===" && \ + echo "$PYTHONPATH" && \ + echo "" && \ + echo "=== sys.path ===" && \ + python3 -c "import sys; print('\n'.join(sys.path))" && \ + echo "" && \ + echo "=== Checking ament_package ===" && \ + python3 -c "import ament_package; print('✓ ament_package found at:', ament_package.__file__)" || echo "✗ ament_package NOT found" && \ + echo "" && \ + echo "=== Checking dpkg for ament packages ===" && \ + dpkg -l | grep -i ament || echo "No ament packages found in dpkg" && \ + echo "" && \ + echo "=== ROS Python packages ===" && \ + ls -la /opt/ros/${ROS_DISTRO}/lib/python*/dist-packages/ 2>/dev/null | head -20 || echo "No ROS python packages found" + # Install eProsima DDS Router # System library dependencies (Asio, TinyXML2, OpenSSL, yaml-cpp) RUN apt update && apt install -y --no-install-recommends \ @@ -215,6 +246,12 @@ ARG INSTALL_FLAGS="-o APT::Get::AllowUnauthenticated=true" ARG SKIP_MACVO=false ARG SKIP_TENSORRT=false +ARG PIP_VERSION=24.0 +ARG PYTHON_VERSION=3.12 + +ARG ROS_DISTRO +ENV ROS_DISTRO=${ROS_DISTRO} + ENV DEBIAN_FRONTEND=noninteractive SHELL ["/bin/bash", "-c"] @@ -254,17 +291,15 @@ RUN sudo add-apt-repository universe \ && curl -sSL https://raw.githubusercontent.com/ros/rosdistro/master/ros.key -o /usr/share/keyrings/ros-archive-keyring.gpg \ && echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/ros-archive-keyring.gpg] http://packages.ros.org/ros2/ubuntu $(. /etc/os-release && echo $UBUNTU_CODENAME) main" | sudo tee /etc/apt/sources.list.d/ros2.list > /dev/null \ && apt-get ${UPDATE_FLAGS} update -y && apt-get ${INSTALL_FLAGS} install -y --no-install-recommends \ - ros-jazzy-desktop \ + ros-${ROS_DISTRO}-desktop \ python3-argcomplete \ && rm -rf /var/lib/apt/lists/* -# Carry over all ROS2 ENV vars from the builder stage -ENV ROS_DISTRO=jazzy -ENV AMENT_PREFIX_PATH=/opt/ros/jazzy -ENV COLCON_PREFIX_PATH=/opt/ros/jazzy -ENV LD_LIBRARY_PATH=/opt/ros/jazzy/lib/x86_64-linux-gnu:/opt/ros/jazzy/lib -ENV PATH=/opt/ros/jazzy/bin:$PATH -ENV PYTHONPATH=/opt/ros/jazzy/local/lib/python3.12/dist-packages:/opt/ros/jazzy/lib/python3.12/site-packages +ENV AMENT_PREFIX_PATH=/opt/ros/${ROS_DISTRO} +ENV COLCON_PREFIX_PATH=/opt/ros/${ROS_DISTRO} +ENV LD_LIBRARY_PATH=/opt/ros/${ROS_DISTRO}/lib/x86_64-linux-gnu:/opt/ros/${ROS_DISTRO}/lib +ENV PATH=/opt/ros/${ROS_DISTRO}/bin:$PATH +ENV PYTHONPATH=/opt/ros/${ROS_DISTRO}/local/lib/python${PYTHON_VERSION}/dist-packages:/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION}/site-packages ENV ROS_PYTHON_VERSION=3 ENV ROS_VERSION=2 ENV ROS_AUTOMATIC_DISCOVERY_RANGE=SUBNET @@ -276,27 +311,38 @@ RUN apt update && apt install -y --no-install-recommends \ vim nano tree \ less htop jq \ python3-pip \ + python3-yaml \ + python3-empy \ python3-rosdep \ tmux \ xvfb \ && rm -rf /var/lib/apt/lists/* +# Freeze pip and setuptools versions (ignore-installed: apt-shipped wheel/setuptools have no pip RECORD). +RUN python3 -m pip install --no-cache-dir --break-system-packages --ignore-installed --upgrade \ + "pip==${PIP_VERSION}" \ + "setuptools==79.0.1" \ + wheel + # Install runtime ROS2 packages (no libcgal-dev) RUN apt update -y && apt install -y --no-install-recommends \ ros-dev-tools \ - ros-jazzy-mavros \ - ros-jazzy-tf2* \ - ros-jazzy-stereo-image-proc \ - ros-jazzy-image-view \ - ros-jazzy-topic-tools \ - ros-jazzy-grid-map \ - ros-jazzy-domain-bridge \ - ros-jazzy-rosbag2-storage-mcap \ - ros-jazzy-xacro \ - ros-jazzy-foxglove-bridge \ + ros-${ROS_DISTRO}-mavros \ + ros-${ROS_DISTRO}-tf2* \ + ros-${ROS_DISTRO}-stereo-image-proc \ + ros-${ROS_DISTRO}-image-view \ + ros-${ROS_DISTRO}-topic-tools \ + ros-${ROS_DISTRO}-grid-map \ + ros-${ROS_DISTRO}-domain-bridge \ + ros-${ROS_DISTRO}-rosbag2-storage-mcap \ + ros-${ROS_DISTRO}-xacro \ + ros-${ROS_DISTRO}-ament-package \ + ros-${ROS_DISTRO}-foxglove-bridge \ python3-colcon-common-extensions \ && rm -rf /var/lib/apt/lists/* +# TODO: consider splitting this into a separate "desktop-plus" image, since foxglove-bridge is a large install and not strictly necessary for most robot use cases + # Install emoji font support and refresh font cache RUN apt-get update && apt-get install -y --no-install-recommends \ fonts-noto-color-emoji \ @@ -304,7 +350,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && fc-cache -f -v \ && rm -rf /var/lib/apt/lists/* -RUN /opt/ros/jazzy/lib/mavros/install_geographiclib_datasets.sh +RUN /opt/ros/${ROS_DISTRO}/lib/mavros/install_geographiclib_datasets.sh # Install DDS Router runtime library dependencies + OpenVDB RUN apt update && apt install -y --no-install-recommends \ @@ -326,12 +372,16 @@ RUN if echo "$BASE_IMAGE" | grep -qE "(nvidia|l4t)" && [ "${SKIP_TENSORRT}" != " && rm -rf /var/lib/apt/lists/*; \ fi -# Install Foxglove Studio desktop app -RUN wget -q https://get.foxglove.dev/desktop/latest/foxglove-studio-latest-linux-amd64.deb -O /tmp/foxglove-studio.deb \ - && apt-get ${UPDATE_FLAGS} update \ - && apt-get ${INSTALL_FLAGS} install -y --no-install-recommends /tmp/foxglove-studio.deb \ - && rm /tmp/foxglove-studio.deb \ - && rm -rf /var/lib/apt/lists/* +# Install Foxglove Studio desktop app only for non-real-robot images +RUN if [ "${REAL_ROBOT}" != "true" ] && [ "$(dpkg --print-architecture)" = "amd64" ]; then \ + wget -q https://get.foxglove.dev/desktop/latest/foxglove-studio-latest-linux-amd64.deb -O /tmp/foxglove-studio.deb && \ + apt-get ${UPDATE_FLAGS} update && \ + apt-get ${INSTALL_FLAGS} install -y --no-install-recommends /tmp/foxglove-studio.deb && \ + rm /tmp/foxglove-studio.deb; \ + else \ + echo "Skipping Foxglove Studio install (REAL_ROBOT=${REAL_ROBOT}, arch=$(dpkg --print-architecture))"; \ + fi && \ + rm -rf /var/lib/apt/lists/* # Add ability to SSH (libglfw3-dev and libglm-dev kept per spec) RUN apt-get ${UPDATE_FLAGS} update && apt-get ${INSTALL_FLAGS} install -y --no-install-recommends \ diff --git a/robot/docker/docker-compose.yaml b/robot/docker/docker-compose.yaml index 2ae1c9a32..f1799f48d 100644 --- a/robot/docker/docker-compose.yaml +++ b/robot/docker/docker-compose.yaml @@ -14,6 +14,7 @@ services: dockerfile: ./Dockerfile.robot args: BASE_IMAGE: nvidia/cuda:13.0.2-base-ubuntu24.04 + ROS_DISTRO: jazzy tags: - *desktop_image cache_from: @@ -26,6 +27,7 @@ services: - LAUNCH_PACKAGE=desktop_bringup # desktop_bringup adds RViz; real robots use autonomy_bringup - AUTONOMY_ROLE=full - SIM_IP=${SIM_IP:-172.31.0.200} + - LAUNCH_NATNET=${LAUNCH_NATNET:-false} # FCU_URL and TGT_SYSTEM not set, dynamically calculated in interface.launch.py # 'command' uses variables so that it can be shared across robot-desktop and robot-l4t, with different launch packages and roles. command: > @@ -120,6 +122,7 @@ services: REAL_ROBOT: true SKIP_MACVO: true SKIP_TENSORRT: true + ROS_DISTRO: jazzy tags: - *voxl_image cache_from: @@ -129,6 +132,7 @@ services: - AUTOLAUNCH=${AUTOLAUNCH:-true} - LAUNCH_PACKAGE=autonomy_bringup - AUTONOMY_ROLE=onboard # VOXL is always lite-only; never runs global planning + - LAUNCH_NATNET=${LAUNCH_NATNET:-false} command: > bash -c " tmux new -d -s bringup; @@ -139,6 +143,23 @@ services: network_mode: host deploy: !reset {} # remove nvidia driver for voxl + # =================================================================================================================== + # Intermediate Jetson stack: dusty Jazzy + ROS keyring / pip / OpenCV shim for Dockerfile.robot (same recipe as Ubuntu). + # `airstack image-build --profile l4t robot-l4t` builds this first automatically; raw `compose build robot-l4t` may parallelize incorrectly. + robot-l4t-stack-base: + profiles: + - l4t + image: &l4t_stack_base_image ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:v${VERSION}_robot-l4t-stack-base_${DOCKER_IMAGE_BUILD_MODE} + build: + dockerfile: ./Dockerfile.l4t-stack-base + network: host + args: + DUSTYNV_IMAGE: dustynv/ros:jazzy-ros-base-r36.4.0-cu128-24.04 + tags: + - *l4t_stack_base_image + cache_from: + - *l4t_stack_base_image + # =================================================================================================================== # for running on an NVIDIA jetson (linux for tegra) device robot-l4t: @@ -152,13 +173,18 @@ services: image: &l4t_image ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:v${VERSION}_robot-l4t_${DOCKER_IMAGE_BUILD_MODE} build: dockerfile: ./Dockerfile.robot + network: host args: - BASE_IMAGE: nvcr.io/nvidia/l4t-jetpack:r36.4.0 + BASE_IMAGE: *l4t_stack_base_image REAL_ROBOT: true + SKIP_MACVO: true + SKIP_TENSORRT: true + ROS_DISTRO: jazzy tags: - *l4t_image cache_from: - *l4t_image + - *l4t_stack_base_image # we use tmux send-keys so that the session stays alive ipc: host command: > @@ -176,11 +202,13 @@ services: - driver: nvidia count: 1 capabilities: [gpu] - environment: !override - - ROBOT_NAME_SOURCE=hostname # see .bashrc + runtime: nvidia + environment: + - ROBOT_NAME_SOURCE=hostname - AUTOLAUNCH=${AUTOLAUNCH:-true} - LAUNCH_PACKAGE=autonomy_bringup - AUTONOMY_ROLE=full # l4t profile: Jetson runs everything onboard + - LAUNCH_NATNET=${LAUNCH_NATNET:-false} # mavros mavlink settings - FCU_URL="/dev/ttyTHS4:115200" - TGT_SYSTEM=1 @@ -210,10 +238,12 @@ services: build: context: ./ dockerfile: zed/Dockerfile.zed-l4t + network: host args: L4T_MAJOR: 36 L4T_MINOR: 4 L4T_PATCH: 0 + IMAGE_NAME: dustynv/ros:jazzy-desktop-r36.4.0-cu128-24.04 cache_from: - *zed_l4t_image command: > @@ -233,6 +263,7 @@ services: privileged: true ipc: host pid: host + runtime: nvidia environment: - NVIDIA_DRIVER_CAPABILITIES=all - DISPLAY diff --git a/robot/docker/zed/Dockerfile.zed-l4t b/robot/docker/zed/Dockerfile.zed-l4t index 5f25ceaf3..b39ec2bc7 100644 --- a/robot/docker/zed/Dockerfile.zed-l4t +++ b/robot/docker/zed/Dockerfile.zed-l4t @@ -35,7 +35,7 @@ RUN echo "# R${L4T_MAJOR} (release), REVISION: ${L4T_MINOR}" > /etc/nv_tegra_rel apt-get -o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDowngradeToInsecureRepositories=true update -y || true && \ apt-get -o APT::Get::AllowUnauthenticated=true install -y --no-install-recommends zstd wget less cmake curl gnupg2 \ build-essential python3 python3-pip python3-dev python3-setuptools libusb-1.0-0-dev \ - libgeographic-dev libdraco-dev zlib1g-dev -y + libgeographiclib-dev libdraco-dev zlib1g-dev -y RUN pip install protobuf --index-url https://pypi.jetson-ai-lab.io/jp6/cu126 RUN wget -q --no-check-certificate -O ZED_SDK_Linux_JP.run \ ${ZED_SDK_URL} && \ diff --git a/robot/ros_ws/src/local/controls/pid_controller_msgs/package.xml b/robot/ros_ws/src/local/controls/pid_controller_msgs/package.xml index 8dd8872f5..ba8f3126e 100644 --- a/robot/ros_ws/src/local/controls/pid_controller_msgs/package.xml +++ b/robot/ros_ws/src/local/controls/pid_controller_msgs/package.xml @@ -14,11 +14,12 @@ rosidl_default_generators rosidl_default_runtime - rosidl_interface_packages ament_lint_auto ament_lint_common + rosidl_interface_packages + ament_cmake diff --git a/robot/ros_ws/src/perception/natnet_ros2/.gitignore b/robot/ros_ws/src/perception/natnet_ros2/.gitignore new file mode 100644 index 000000000..f4a20d2b9 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/.gitignore @@ -0,0 +1,27 @@ +# Ignore proprietary NatNetSDK (downloaded at setup time) +lib/libNatNet.so +include/natnet/ + +# Python runtime and cache +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python +env/ +venv/ +ENV/ +*.egg-info/ +dist/ +build/ + +# IDE +.vscode/ +.idea/ +*.swp +*.swo +*~ + +# ROS build artifacts +devel/ +install/ diff --git a/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt b/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt new file mode 100644 index 000000000..ff47a00da --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt @@ -0,0 +1,99 @@ +cmake_minimum_required(VERSION 3.8) +project(natnet_ros2) + +if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") + add_compile_options(-Wall -Wextra -Wpedantic) +endif() + +# ament / ROS 2 dependencies +find_package(ament_cmake REQUIRED) +find_package(rclcpp REQUIRED) +find_package(geometry_msgs REQUIRED) +find_package(nav_msgs REQUIRED) + +# --------------------------------------------------------------------------- +# NatNet SDK — pre-built shared library downloaded via `airstack setup`. +# Headers: include/natnet/ Library: lib/libNatNet.so +# +# The SDK is proprietary and is NOT committed to the repository. +# Run `airstack setup` (or `airstack setup --natnet`) to download and place +# the SDK files before building this package. +# +# If the SDK is absent the C++ node is skipped with a warning; the Python +# vision_pose_converter_node.py and all launch/config files are still installed +# so the rest of the autonomy stack builds cleanly. +# --------------------------------------------------------------------------- +set(_NATNET_LIB "${CMAKE_CURRENT_SOURCE_DIR}/lib/libNatNet.so") +set(_NATNET_INC "${CMAKE_CURRENT_SOURCE_DIR}/include/natnet") + +if(EXISTS "${_NATNET_LIB}" AND EXISTS "${_NATNET_INC}") + add_library(NatNet SHARED IMPORTED) + set_target_properties(NatNet PROPERTIES + IMPORTED_LOCATION "${_NATNET_LIB}" + INTERFACE_INCLUDE_DIRECTORIES "${_NATNET_INC}" + ) + + # C++ NatNet ROS 2 node + add_executable(natnet_ros2_node + src/natnet_ros2_node.cpp + src/natnet_client_adapter.cpp) + target_include_directories(natnet_ros2_node PUBLIC + $ + $) + ament_target_dependencies(natnet_ros2_node rclcpp geometry_msgs nav_msgs) + target_link_libraries(natnet_ros2_node NatNet) + + install(TARGETS natnet_ros2_node + DESTINATION lib/${PROJECT_NAME}) + + # Install libNatNet.so alongside the node and register an environment hook so + # that sourcing the workspace adds lib/natnet_ros2/ to LD_LIBRARY_PATH. + # Use PROGRAMS (not FILES) to preserve the execute bit — shared libraries + # must be executable for the dynamic linker to map them. + install(PROGRAMS "${_NATNET_LIB}" + DESTINATION lib/${PROJECT_NAME}) + + ament_environment_hooks( + "${CMAKE_CURRENT_SOURCE_DIR}/env-hooks/natnet_library_path.dsv.in" + ) +else() + message(WARNING + "[natnet_ros2] NatNet SDK not found — skipping natnet_ros2_node build.\n" + " Expected: ${_NATNET_LIB}\n" + " ${_NATNET_INC}/\n" + " Run 'airstack setup' to download the OptiTrack NatNet SDK.") +endif() + +# --------------------------------------------------------------------------- +# Python nodes (vision_pose_converter remains Python) +# --------------------------------------------------------------------------- +install(PROGRAMS + src/vision_pose_converter_node.py + DESTINATION lib/${PROJECT_NAME}) + +# --------------------------------------------------------------------------- +# Launch and config files +# --------------------------------------------------------------------------- +install(DIRECTORY launch/ + DESTINATION share/${PROJECT_NAME}/launch) + +install(DIRECTORY config/ + DESTINATION share/${PROJECT_NAME}/config) + +install(DIRECTORY include/ + DESTINATION include) + +# --------------------------------------------------------------------------- +if(BUILD_TESTING) + find_package(ament_lint_auto REQUIRED) + ament_lint_auto_find_test_dependencies() + + # --- gtest: pure-logic unit tests (no NatNet SDK, no rclcpp) --- + find_package(ament_cmake_gtest REQUIRED) + ament_add_gtest(test_natnet_logic test/test_natnet_logic.cpp) + target_include_directories(test_natnet_logic PRIVATE + $ + $) +endif() + +ament_package() diff --git a/robot/ros_ws/src/perception/natnet_ros2/README.md b/robot/ros_ws/src/perception/natnet_ros2/README.md new file mode 100644 index 000000000..eb44763b4 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/README.md @@ -0,0 +1,199 @@ +# NatNet ROS 2 Wrapper + +OptiTrack NatNet ROS 2 wrapper for motion capture integration in AirStack (optional). Receives rigid body pose data from an external Motive PC via NatNet UDP protocol and publishes into the AirStack perception layer. + +**Note:** This module is only required if you intend to use OptiTrack Motive motion capture systems. If you do not plan to use OptiTrack, you can skip the NatNet SDK setup with `airstack setup --no-natnet`. + +### OptiTrack room calibration + +If rigid bodies are jumping around or not tracking well, consider re-calibrating the capture volume in Motive. See the [OptiTrack Motive calibration guide](https://docs.optitrack.com/motive/calibration). + +## Overview + +This module provides a bridge between OptiTrack Motive motion capture systems and the AirStack autonomy stack. It: + +- Receives **NatNet UDP packets** from an external Motive PC (configurable IP/port) +- **Decodes motion capture frames** containing rigid body positions and orientations +- **Publishes pose data** to the AirStack perception layer in standard ROS 2 formats +- **Supports multi-robot** via ROBOT_NAME namespacing +- **Optionally bridges** to MAVROS for PX4 external pose feedback +- **Respects OptiTrack licensing** by keeping the NatNet SDK external (host-side download with explicit consent) + +## Architecture + +``` +Motive (External PC) + ↓ NatNet UDP (port 1511) + ↓ +NatNet ROS 2 Node + ├→ /robot_1/perception/optitrack/{body_name} (PoseStamped, optional) + ├→ /robot_1/perception/optitrack/{body_name}/pose_cov (PoseWithCovarianceStamped, always) + └→ (Optional, publish_to_mavros: true) + vision_pose_converter_node + ├→ /robot_1/mavros/vision_pose/pose + └→ /robot_1/mavros/vision_pose/pose_cov +``` + +## Interfaces + +### Inputs + +- **Network**: NatNet UDP stream from Motive PC (external network) +- **Configuration**: `natnet_config.yaml` with server IP, ports, `body_name`, and covariance + +### Outputs + +For each tracked rigid body `{body_name}` from Motive: + +#### Direct OptiTrack pose (optional) + +- **Topic**: `/{ROBOT_NAME}/perception/optitrack/{body_name}` +- **Type**: `geometry_msgs/PoseStamped` +- **Description**: Position and orientation only (no covariance) +- **Enabled by**: `publish_direct_optitrack: true` in config (default: `true`) + +#### Pose with covariance (always) + +- **Topic**: `/{ROBOT_NAME}/perception/optitrack/{body_name}/pose_cov` +- **Type**: `geometry_msgs/PoseWithCovarianceStamped` +- **Description**: Same pose as above plus a 6×6 covariance matrix (`position_covariance` and `orientation_covariance` from config). Published whenever the rigid body is tracked — independent of `publish_direct_optitrack` and `publish_to_mavros`. + +#### MAVROS vision pose bridge (optional) + +When `publish_to_mavros: true`, `vision_pose_converter_node` subscribes to `pose_cov` and republishes for PX4: + +- **Topic**: `/{ROBOT_NAME}/mavros/vision_pose/pose` — `geometry_msgs/PoseStamped` (pose extracted from the covariance message) +- **Topic**: `/{ROBOT_NAME}/mavros/vision_pose/pose_cov` — `geometry_msgs/PoseWithCovarianceStamped` (full message, quaternion optionally canonicalized) +- **Enabled by**: `publish_to_mavros: true` in config + +## Configuration + +Edit `config/natnet_config.yaml`: + +```yaml +/**: + ros__parameters: + server_ip: "192.168.1.100" # IP of the Motive PC + client_ip: "0.0.0.0" + command_port: 1510 + data_port: 1511 + connection_type: "unicast" # or "multicast" + + body_name: "Drone" # rigid body name in Motive (case-sensitive) + body_id: -1 # -1 = publish all bodies in the frame + + publish_direct_optitrack: true # PoseStamped on …/optitrack/{body_name} + publish_to_mavros: false # include vision_pose_converter → MAVROS + + frame_id: "world" + + position_covariance: [0.1, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1] + orientation_covariance: [0.01, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.01] +``` + +## Launch + +### Basic launch + +Parameters come from `config/natnet_config.yaml` (network, body, covariance). Optional overrides: + +```bash +ros2 launch natnet_ros2 natnet_ros2.launch.py \ + config_file:=/path/to/custom_natnet.yaml \ + vision_pose_config_file:=/path/to/custom_vision_pose.yaml \ + use_sim_time:=true +``` + +### MAVROS bridge + +Set `publish_to_mavros: true` in `natnet_config.yaml`. The launch file reads `publish_to_mavros` and `body_name` from that YAML to decide whether to include `vision_pose_converter.launch.xml`. + +### From perception bringup + +With `LAUNCH_NATNET=true` in `.env`, `perception.launch.xml` includes `natnet_ros2.launch.py`. + +## Dependencies + +### Runtime +- `rclpy` — ROS 2 Python client +- `geometry_msgs` — Standard pose message types +- `tf_transformations` — Quaternion and rotation utilities +- `mavros_msgs` — Optional, for MAVROS bridge + +### Required +- **OptiTrack NatNet SDK** (Linux SDK) — **REQUIRED**, downloaded via `airstack setup` + +### Installation +To install the NatNet SDK and accept the license: +```bash +airstack setup +``` +The SDK will be installed into `robot/ros_ws/src/perception/natnet_ros2/lib/` and `robot/ros_ws/src/perception/natnet_ros2/include/natnet/` after accepting the OptiTrack License Agreement. + +## Implementation Details + +### Protocol Support +- **NatNet Version**: 4.4+ (SDK handles protocol negotiation) +- **Packet Type**: Frame of Data with rigid bodies and markers +- **Transport**: UDP (configurable port, default 1511) +- **SDK**: OptiTrack NatNet SDK handles all protocol parsing + +### Multi-Robot Support +Each container instance gets its own `ROBOT_NAME` and `ROS_DOMAIN_ID`: +- Topics: `/{ROBOT_NAME}/perception/optitrack/{body_name}` and `/{ROBOT_NAME}/perception/optitrack/{body_name}/pose_cov` +- Supported via launch file argument forwarding + +### Error Handling +- Invalid/malformed packets are skipped with debug logging +- Lost connectivity logs warnings; gracefully recovers when stream resumes +- Covariance in config allows tuning uncertainty per deployment + +## Testing + +### With Real Motive +1. Ensure Motive PC and robot are on same network +2. Configure server IP in `natnet_config.yaml` +3. Launch the node: + ```bash + ros2 launch natnet_ros2 natnet_ros2.launch.py + ``` +4. Verify topics: + ```bash + ros2 topic echo /robot_1/perception/optitrack/Drone/pose_cov + ``` + +### Without Real Hardware (Mock) +TODO: Implement Motive simulator in Isaac Sim to generate fake NatNet packets + +## Known Limitations + +- When `body_id: -1`, all rigid bodies in the Motive frame get publishers; filter by subscribing to the `{body_name}` you care about +- MAVROS bridge applies frame_id override and quaternion canonicalization; full PX4 frame alignment may still need tuning per airframe +- No support for skeleton tracking or labeled markers yet (future enhancement) + +## References + +- [OptiTrack NatNet Protocol Documentation](https://docs.optitrack.com/developer-tools/natnet-sdk/natnet-4.0) +- [NatNet SDK Download](https://optitrack.com/software/natnet-sdk/) +- [MAVROS Vision Pose Plugin](https://docs.ros.org/en/melodic/api/mavros_extras/html/classmavros_1_1extra__plugins_1_1VisionPoseEstimatePlugin.html) + +## Troubleshooting + +### No data being received +- Check Motive PC IP address in config +- Verify UDP port is not blocked by firewall +- Use `ros2 topic hz` to check if data is arriving + +### Topics not published +- Check `ros2 node list` — should see `natnet_ros2_node` +- Check `ros2 topic list | grep optitrack` — should see published topics +- Look at logs: `ros2 node info natnet_ros2_node` + +### Low frame rate or dropped frames +- Reduce other network traffic +- Check NatNet streaming rate in Motive (default 120 Hz) +- Monitor CPU usage: `docker stats` + +## License + +**Note on NatNet SDK Licensing**: The OptiTrack NatNet SDK is proprietary software governed by the OptiTrack Software License Agreement. Users download and install the SDK locally under their own license compliance. AirStack does not redistribute the SDK and remains fully open-source. diff --git a/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml b/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml new file mode 100644 index 000000000..69fc11d1c --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml @@ -0,0 +1,51 @@ +# NatNet ROS 2 parameters — loaded by natnet_ros2.launch.py (NatNet node + MAVROS gate). +# publish_to_mavros / body_name are read by the launch file to decide vision_pose_converter include. +# +# Use /** so parameters apply regardless of namespace (e.g. /robot_1/perception/natnet_ros2_node). +# See: https://docs.ros.org/en/humble/Tutorials/Beginner-CLI-Tools/Understanding-ROS2-Parameters.html + +/**: + ros__parameters: + # IP address of the PC running Motive (OptiTrack server). + # Change this to match your local network before launching NatNet. + server_ip: "192.168.1.100" + # Motive learns unicast destination from outbound UDP source IP — bind explicitly when you have + # multiple NICs (e.g. Docker 172.17.* vs LAN). + client_ip: "0.0.0.0" + + command_port: 1510 + data_port: 1511 + + # "unicast" — point-to-point; Motive streams directly to this machine's IP. + # Requires Motive unicast streaming enabled and client_ip set + # to the correct NIC when multiple interfaces are present. + # "multicast" — Motive broadcasts to a multicast group; any machine on the + # subnet that joins the group receives all body data. + # Use for multi-robot setups where every robot receives the + # full frame and filters by body_id. + connection_type: "unicast" + + # Only used when connection_type = "multicast". + # Must match Motive > Edit > Preferences > Data Streaming > Multicast Interface. + # OptiTrack default is 239.255.42.99. + multicast_address: "239.255.42.99" + + # Name of the rigid body as defined in Motive. Must match exactly (case-sensitive). + body_name: "Drone" + body_id: -1 + + publish_direct_optitrack: true + publish_to_mavros: true + + frame_id: "world" + debug: false + + position_covariance: + [0.1, 0.0, 0.0, + 0.0, 0.1, 0.0, + 0.0, 0.0, 0.1] + + orientation_covariance: + [0.01, 0.0, 0.0, + 0.0, 0.01, 0.0, + 0.0, 0.0, 0.01] diff --git a/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml b/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml new file mode 100644 index 000000000..a52181786 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml @@ -0,0 +1,12 @@ +# Vision pose converter → MAVROS bridge parameters. +# Loaded by vision_pose_converter.launch.xml via . +# $(env ROBOT_NAME ...) is expanded by launch substitution. + +/**: + ros__parameters: + frame_id: "world" + child_frame_id: "$(env ROBOT_NAME robot_1)/base_link" + # Normalise quaternion to canonical form (qw >= 0) before publishing. + # Recommended for ArduPilot EKF3 and any consumer sensitive to sign flips. + # PX4 EKF2 handles either sign internally, so this is optional for PX4. + canonical_quaternion: true diff --git a/robot/ros_ws/src/perception/natnet_ros2/env-hooks/natnet_library_path.dsv.in b/robot/ros_ws/src/perception/natnet_ros2/env-hooks/natnet_library_path.dsv.in new file mode 100644 index 000000000..8046498e2 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/env-hooks/natnet_library_path.dsv.in @@ -0,0 +1 @@ +prepend-non-duplicate;LD_LIBRARY_PATH;lib/natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp new file mode 100644 index 000000000..04b3638ef --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp @@ -0,0 +1,60 @@ +// Copyright (c) 2024 Carnegie Mellon University +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in all +// copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +// SOFTWARE. +// +// natnet_client_adapter.hpp — declaration of NatNetClientAdapter. +// +// NatNetClientAdapter wraps the NatNet SDK's NatNetClient and implements +// INatNetClient. It is the only place in the codebase that includes NatNet +// SDK headers; unit tests use FakeNatNetClient instead. +// +// Implementation: src/natnet_client_adapter.cpp + +#pragma once + +#include "natnet_ros2/natnet_logic.hpp" + +#include +#include +#include + +// Forward-declare the SDK type so this header stays SDK-header-free. +class NatNetClient; + +namespace natnet_ros2 +{ + +class NatNetClientAdapter : public INatNetClient +{ +public: + NatNetClientAdapter(); + ~NatNetClientAdapter() override; + + NatNetResult connect(const ConnectConfig & cfg) override; + bool get_server_info(ServerInfo & out) override; + std::vector get_body_descriptors() override; + void set_frame_callback(std::function cb) override; + void disconnect() override; + +private: + std::unique_ptr client_; + std::function user_cb_; +}; + +} // namespace natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp new file mode 100644 index 000000000..f23216565 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp @@ -0,0 +1,354 @@ +// Copyright (c) 2024 Carnegie Mellon University +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in all +// copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +// SOFTWARE. + +// natnet_logic.hpp — pure C++ helpers for natnet_ros2 (no ROS, no NatNet SDK). +// +// Five responsibility areas: +// +// 1. Covariance assembly +// 2. Topic names +// 3. Connection-configuration helpers (SDK-independent) +// 4. Rigid-body frame helpers (SDK-independent) +// 5. Abstraction seam: INatNetClient interface + negotiation logic +// +// NatNet SDK types (sNatNetClientConnectParams, sRigidBodyData, …) are only +// used inside natnet_ros2_node.cpp and natnet_client_adapter.cpp. +// All logic here uses plain C++ so test_natnet_logic.cpp compiles with only gtest. + +#pragma once + +#include +#include +#include +#include +#include + +namespace natnet_ros2 +{ + +// =========================================================================== +// 1. Covariance +// =========================================================================== + +/// Build a row-major 36-element 6×6 covariance from two flat 3×3 blocks. +/// +/// pos_cov (up to 9 elements) fills the top-left 3×3 block (rows/cols 0-2). +/// ori_cov (up to 9 elements) fills the bottom-right 3×3 block (rows/cols 3-5). +/// All other entries are zero. +inline std::array build_covariance_6x6( + const std::vector & pos_cov, + const std::vector & ori_cov) +{ + std::array cov{}; + cov.fill(0.0); + const int np = static_cast(pos_cov.size()); + const int no = static_cast(ori_cov.size()); + for (int r = 0; r < 3; ++r) { + for (int c = 0; c < 3; ++c) { + const int idx = r * 3 + c; + if (idx < np) { cov[r * 6 + c] = pos_cov[idx]; } + if (idx < no) { cov[(r + 3) * 6 + (c + 3)] = ori_cov[idx]; } + } + } + return cov; +} + + +// =========================================================================== +// 2. Topic names +// =========================================================================== + +/// Base topic for a rigid body: /{robot_name}/perception/optitrack/{body_name} +inline std::string optitrack_topic_base( + const std::string & robot_name, + const std::string & body_name) +{ + return "/" + robot_name + "/perception/optitrack/" + body_name; +} + +/// PoseWithCovarianceStamped topic: …/{body_name}/pose_cov +inline std::string optitrack_pose_cov_topic( + const std::string & robot_name, + const std::string & body_name) +{ + return optitrack_topic_base(robot_name, body_name) + "/pose_cov"; +} + + +// =========================================================================== +// 3. Connection-configuration helpers +// =========================================================================== + +/// Return ct if it is "unicast" or "multicast"; otherwise return "unicast". +inline std::string validate_connection_type(const std::string & ct) +{ + if (ct == "unicast" || ct == "multicast") { return ct; } + return "unicast"; +} + +/// SDK-independent connection configuration aggregate. +/// +/// natnet_ros2_node.cpp converts this into sNatNetClientConnectParams; tests +/// exercise the pure logic without linking the NatNet SDK. +struct ConnectConfig +{ + std::string server_ip = "192.168.1.1"; + std::string client_ip = "0.0.0.0"; + uint16_t command_port = 1510u; + uint16_t data_port = 1511u; + std::string connection_type = "unicast"; ///< validated + std::string multicast_address = "239.255.42.99"; +}; + +/// Build a validated ConnectConfig from raw user-supplied strings. +/// connection_type is normalised via validate_connection_type(). +inline ConnectConfig make_connect_config( + const std::string & server_ip, + const std::string & client_ip, + uint16_t command_port, + uint16_t data_port, + const std::string & connection_type, + const std::string & multicast_address = "239.255.42.99") +{ + return ConnectConfig{ + server_ip, + client_ip, + command_port, + data_port, + validate_connection_type(connection_type), + multicast_address + }; +} + +/// Returns true when the config requests a multicast connection. +inline bool is_multicast(const ConnectConfig & cfg) +{ + return cfg.connection_type == "multicast"; +} + +/// Returns true when the multicast address should be used. +/// When false, the multicast_address field is irrelevant and should be nullptr +/// when passed to sNatNetClientConnectParams. +inline bool needs_multicast_address(const ConnectConfig & cfg) +{ + return is_multicast(cfg); +} + + +// =========================================================================== +// 4. Rigid-body frame helpers (SDK-independent) +// =========================================================================== + +/// Lightweight, SDK-free representation of a single rigid-body sample. +/// natnet_ros2_node.cpp converts sRigidBodyData → RigidBodySample. +struct RigidBodySample +{ + int32_t id = 0; + float x = 0.f; + float y = 0.f; + float z = 0.f; + float qx = 0.f; + float qy = 0.f; + float qz = 0.f; + float qw = 1.f; + int16_t params = 0; ///< NatNet rb.params bitmask +}; + +/// Lightweight, SDK-free representation of one frame of mocap data. +struct FrameSample +{ + int32_t frame_num = 0; + float timestamp = 0.f; + int16_t params = 0; ///< NatNet frame.params bitmask + std::vector bodies; +}; + +/// Returns true when bit 0 of rb.params is set (NatNet: tracking valid). +inline bool is_tracking_valid(int16_t rb_params) +{ + return (rb_params & 0x01) != 0; +} + +/// Returns true when bit 1 of frame.params is set (NatNet: model list changed). +inline bool model_list_changed(int16_t frame_params) +{ + return (frame_params & 0x02) != 0; +} + +/// Returns true when the rigid body should be published. +/// filter_id < 0 means "publish all bodies"; otherwise only the matching ID. +inline bool should_publish_body(int32_t filter_id, int32_t rb_id) +{ + return filter_id < 0 || rb_id == filter_id; +} + +/// Double-precision pose extracted from a RigidBodySample. +struct PoseData +{ + double x = 0.0; + double y = 0.0; + double z = 0.0; + double qx = 0.0; + double qy = 0.0; + double qz = 0.0; + double qw = 1.0; +}; + +/// Convert a RigidBodySample to a double-precision PoseData. +inline PoseData rb_to_pose(const RigidBodySample & rb) +{ + return PoseData{ + static_cast(rb.x), + static_cast(rb.y), + static_cast(rb.z), + static_cast(rb.qx), + static_cast(rb.qy), + static_cast(rb.qz), + static_cast(rb.qw) + }; +} + +/// Fill a 36-element covariance array into a pre-allocated ROS-style covariance +/// field from a pre-built std::array. +/// Returns a copy of the array (ROS msg.covariance = cov6x6_to_array(...)). +inline std::array cov6x6_to_array(const std::array & src) +{ + return src; +} + +// =========================================================================== +// 5. Abstraction seam: INatNetClient + negotiation logic +// =========================================================================== + +/// SDK-independent result codes for connection attempts. +enum class NatNetResult +{ + OK, + NetworkError, + InvalidAddress, + Timeout, + InternalError, +}; + +inline const char * natnet_result_str(NatNetResult r) +{ + switch (r) { + case NatNetResult::OK: return "OK"; + case NatNetResult::NetworkError: return "NetworkError"; + case NatNetResult::InvalidAddress: return "InvalidAddress"; + case NatNetResult::Timeout: return "Timeout"; + case NatNetResult::InternalError: return "InternalError"; + } + return "Unknown"; +} + +/// Server identity returned after a successful connection. +struct ServerInfo +{ + bool host_present = false; + std::string host_app_name; + int host_app_version[4] = {}; ///< major.minor.build.revision + int natnet_version[4] = {}; ///< major.minor.build.revision +}; + +/// SDK-independent description of one rigid-body asset. +struct BodyDescriptor +{ + int32_t id = 0; + std::string name; + int32_t parent_id = -1; ///< >= 0 → skeleton bone; skip for top-level publishing +}; + +/// Result of the connect + GetServerDescription handshake. +struct NegotiationResult +{ + bool ok = false; + ServerInfo server_info; + std::string log_message; ///< human-readable outcome for the ROS logger +}; + +/// Pure-virtual client interface — implemented by NatNetClientAdapter (production) +/// and FakeNatNetClient (unit tests). +/// +/// Depends only on natnet_logic.hpp types; never includes NatNet SDK headers. +class INatNetClient +{ +public: + virtual ~INatNetClient() = default; + + /// Attempt to connect to a Motive server. + virtual NatNetResult connect(const ConnectConfig & cfg) = 0; + + /// Populate \p out with server identity. Returns false when host info is + /// unavailable (HostPresent == false in the SDK's sServerDescription). + virtual bool get_server_info(ServerInfo & out) = 0; + + /// Return descriptions of all rigid-body assets currently known to Motive. + /// Returns empty on failure; callers should retry on model-list-changed. + virtual std::vector get_body_descriptors() = 0; + + /// Register a callback invoked on every incoming frame. + /// The callback is called from the SDK receive thread. + virtual void set_frame_callback(std::function cb) = 0; + + /// Disconnect from the server and release SDK resources. + virtual void disconnect() = 0; +}; + +/// Execute the connect + GetServerDescription handshake and return a structured +/// result. Pure logic: no ROS calls, no SDK types — fully testable with a fake. +inline NegotiationResult negotiate(INatNetClient & client, const ConnectConfig & cfg) +{ + NegotiationResult result; + + const NatNetResult err = client.connect(cfg); + if (err != NatNetResult::OK) { + result.ok = false; + result.log_message = std::string("NatNetClient::Connect failed (") + + natnet_result_str(err) + + ") — server=" + cfg.server_ip + + " port=" + std::to_string(cfg.command_port) + + " type=" + cfg.connection_type; + return result; + } + + result.ok = true; + + const bool host_ok = client.get_server_info(result.server_info); + if (!host_ok || !result.server_info.host_present) { + result.log_message = "Connected to " + cfg.server_ip + + " but GetServerDescription returned no host info."; + } else { + result.log_message = "Connected to Motive '" + + result.server_info.host_app_name + + "' v" + + std::to_string(result.server_info.host_app_version[0]) + + "." + + std::to_string(result.server_info.host_app_version[1]) + + " (NatNet " + + std::to_string(result.server_info.natnet_version[0]) + + "." + + std::to_string(result.server_info.natnet_version[1]) + + ") at " + cfg.server_ip; + } + return result; +} + +} // namespace natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py b/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py new file mode 100644 index 000000000..cb7c178d8 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py @@ -0,0 +1,117 @@ +#!/usr/bin/env python3 +"""Bring up NatNet node; optionally MAVROS bridge per natnet_config.yaml. + +natnet_ros2_node is a C++ executable that requires the OptiTrack NatNet SDK. +If the SDK was not installed (``airstack setup`` not run) and the workspace +has not been rebuilt, launching this file will raise a RuntimeError with +instructions. Set LAUNCH_NATNET=false in .env to disable OptiTrack entirely. +""" + +from __future__ import annotations + +import os +from pathlib import Path +from typing import cast + +import yaml +from ament_index_python.packages import get_package_share_directory +from launch import LaunchDescription +from launch.actions import DeclareLaunchArgument, IncludeLaunchDescription, OpaqueFunction +from launch.launch_description_sources import FrontendLaunchDescriptionSource +from launch.substitutions import LaunchConfiguration +from launch_ros.actions import Node +from launch_ros.parameter_descriptions import ParameterFile + + +def _ros_params_from_file(config_path: str) -> dict: + """Parse /** / ros__parameters block from a ROS 2 parameter YAML.""" + path = Path(config_path) + if not path.is_file(): + return {} + with path.open(encoding='utf-8') as f: + data = yaml.safe_load(f) + if not isinstance(data, dict): + return {} + block = data.get('/**') + if not isinstance(block, dict): + return {} + params = block.get('ros__parameters', {}) + return cast(dict, params) if isinstance(params, dict) else {} + + +def generate_launch_description() -> LaunchDescription: + pkg_share = get_package_share_directory('natnet_ros2') + default_natnet_yaml = os.path.join(pkg_share, 'config', 'natnet_config.yaml') + default_vp_yaml = os.path.join(pkg_share, 'config', 'vision_pose_converter.yaml') + + config_file = LaunchConfiguration('config_file') + vision_pose_config_file = LaunchConfiguration('vision_pose_config_file') + use_sim_time = LaunchConfiguration('use_sim_time') + + def launch_setup(context, *_args, **_kwargs): + cfg_path = config_file.perform(context) + vp_path = vision_pose_config_file.perform(context) + ust = use_sim_time.perform(context) + + ros_params = _ros_params_from_file(cfg_path) + publish_mavros = bool(ros_params.get('publish_to_mavros', False)) + body_name = str(ros_params.get('body_name', 'robot_1')) + + # pkg_share = /share/natnet_ros2 → go up two levels to reach , + # then down into lib/natnet_ros2/ where colcon installs executables. + pkg_share = get_package_share_directory('natnet_ros2') + node_path = Path(pkg_share).parent.parent / 'lib' / 'natnet_ros2' / 'natnet_ros2_node' + if not node_path.exists(): + raise RuntimeError( + 'natnet_ros2_node executable not found — NatNet SDK is not installed.\n' + "Run 'airstack setup' to download and install the OptiTrack NatNet SDK,\n" + 'then rebuild the workspace: bws --packages-select natnet_ros2\n' + 'Or set LAUNCH_NATNET=false in .env to disable OptiTrack.' + ) + + actions = [ + Node( + package='natnet_ros2', + executable='natnet_ros2_node', + name='natnet_ros2_node', + output='screen', + parameters=[ParameterFile(config_file, allow_substs=True)], + ), + ] + + if publish_mavros: + actions.append( + IncludeLaunchDescription( + FrontendLaunchDescriptionSource( + os.path.join(pkg_share, 'launch', 'vision_pose_converter.launch.xml'), + ), + launch_arguments=[ + ('config_file', vp_path), + ('body_name', body_name), + ('use_sim_time', ust), + ], + ), + ) + return actions + + return LaunchDescription( + [ + DeclareLaunchArgument( + 'config_file', + default_value=default_natnet_yaml, + description='NatNet parameter YAML (/** ros__parameters). ' + 'publish_to_mavros and body_name control MAVROS include.', + ), + DeclareLaunchArgument( + 'vision_pose_config_file', + default_value=default_vp_yaml, + description='vision_pose_converter parameter YAML.', + ), + DeclareLaunchArgument( + 'use_sim_time', + default_value='false', + description='Forwarded to vision_pose_converter.launch.xml.', + ), + OpaqueFunction(function=launch_setup), + ], + ) diff --git a/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml b/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml new file mode 100644 index 000000000..aad9c4474 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml @@ -0,0 +1,45 @@ + + + + + + + + + + + + + + + + + + + + + + diff --git a/robot/ros_ws/src/perception/natnet_ros2/package.xml b/robot/ros_ws/src/perception/natnet_ros2/package.xml new file mode 100644 index 000000000..f9632b0f7 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/package.xml @@ -0,0 +1,46 @@ + + + + natnet_ros2 + 0.1.0 + + NatNet ROS 2 wrapper for OptiTrack Motive motion capture integration. + Receives NatNet data from external Motive PC via the official NatNet SDK + and publishes pose data into the AirStack perception layer. + + + AirLab CMU + MIT + + ament_cmake + + + rclcpp + geometry_msgs + nav_msgs + + + rclpy + tf_transformations + airstack_msgs + + + mavros_msgs + + ament_index_python + launch + launch_ros + python3-yaml + + + ament_lint_auto + ament_copyright + ament_flake8 + ament_pep257 + python3-pytest + ament_cmake_gtest + + + ament_cmake + + diff --git a/robot/ros_ws/src/perception/natnet_ros2/scripts/download-natnet-sdk.sh b/robot/ros_ws/src/perception/natnet_ros2/scripts/download-natnet-sdk.sh new file mode 100644 index 000000000..e91360197 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/scripts/download-natnet-sdk.sh @@ -0,0 +1,167 @@ +#!/bin/bash + +set -euo pipefail + +module_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +sdk_lib_dir="${module_root}/lib" +sdk_include_dir="${module_root}/include/natnet" + +sdk_archive_name="" +sdk_download_url="" +temp_dir="" + +info() { + printf '[INFO] %s\n' "$1" +} + +warn() { + printf '[WARN] %s\n' "$1" >&2 +} + +fail() { + printf '[ERROR] %s\n' "$1" >&2 + exit 1 +} + +have() { + command -v "$1" >/dev/null 2>&1 +} + +cleanup() { + if [ -n "$temp_dir" ] && [ -d "$temp_dir" ]; then + rm -rf "$temp_dir" + fi +} + +trap cleanup EXIT + +sdk_is_installed() { + [ -f "${sdk_lib_dir}/libNatNet.so" ] || return 1 + [ -n "$(find "${sdk_include_dir}" -maxdepth 1 -type f -name 'NatNet*' -print -quit 2>/dev/null)" ] +} + +choose_archive() { + case "$(uname -m)" in + x86_64) + sdk_archive_name="NatNet_SDK_4.4_ubuntu.tar" + sdk_download_url="https://d2mzlempwep3hb.cloudfront.net/NatNetSDKLinux/ubuntu/${sdk_archive_name}" + info "Using the x86_64 NatNet SDK package" + ;; + aarch64|arm*) + sdk_archive_name="NatNet_SDK_4.4_ubuntu_ARM.tar" + sdk_download_url="https://d2mzlempwep3hb.cloudfront.net/NatNetSDKLinux/ubuntu_arm/${sdk_archive_name}" + info "Using the ARM NatNet SDK package" + ;; + *) + fail "Unsupported architecture: $(uname -m)" + ;; + esac +} + +show_license_notice() { + cat <<'EOF' +=============================================================================== +OptiTrack NatNet SDK License +=============================================================================== + +The NatNet SDK is used for OptiTrack Motive motion capture integration. +It is only required if you intend to use OptiTrack with AirStack. + +The SDK is proprietary and AirStack does not redistribute it. +Please review the OptiTrack terms before continuing: +https://optitrack.com/about/legal/eula + +This installer will download the SDK into the natnet_ros2 module tree: + - lib/libNatNet.so + - include/natnet/NatNet* + +If you do not plan to use OptiTrack, you can skip this step by pressing 'N' or by running: + airstack setup --no-natnet + +=============================================================================== +EOF +} + +download_archive() { + info "Downloading ${sdk_archive_name}" + + if have wget; then + wget -O "${temp_dir}/${sdk_archive_name}" "${sdk_download_url}" + return 0 + fi + + if have curl; then + curl -fsSL -o "${temp_dir}/${sdk_archive_name}" "${sdk_download_url}" + return 0 + fi + + fail "Neither wget nor curl is available. Install one of them and try again." +} + +extract_archive() { + info "Extracting archive" + mkdir -p "${temp_dir}/extract" + tar -xf "${temp_dir}/${sdk_archive_name}" -C "${temp_dir}/extract" +} + +install_files() { + local source_lib + local source_include + + source_lib="$(find "${temp_dir}/extract" -type f -name 'libNatNet.so' -print -quit)" + source_include="$(find "${temp_dir}/extract" -type d -name include -print -quit)" + + [ -n "${source_lib}" ] || fail "libNatNet.so was not found inside the SDK archive" + [ -n "${source_include}" ] || fail "include/ was not found inside the SDK archive" + + mkdir -p "${sdk_lib_dir}" "${sdk_include_dir}" + + info "Installing library and headers into the module tree" + cp "${source_lib}" "${sdk_lib_dir}/" + find "${source_include}" -maxdepth 1 -type f -name 'NatNet*' -exec cp -f {} "${sdk_include_dir}/" \; + + [ -f "${sdk_lib_dir}/libNatNet.so" ] || fail "NatNet library copy did not complete" + [ -n "$(find "${sdk_include_dir}" -maxdepth 1 -type f -name 'NatNet*' -print -quit)" ] || fail "NatNet headers copy did not complete" +} + +main() { + # Non-interactive / CI mode: set NATNET_ACCEPT_LICENSE=1 or pass --accept-license. + # By using this flag you confirm that you have read and accept the OptiTrack + # Software License Agreement (https://optitrack.com/about/legal/eula). + local auto_accept=false + for arg in "$@"; do + [[ "$arg" == "--accept-license" ]] && auto_accept=true + done + [[ "${NATNET_ACCEPT_LICENSE:-0}" == "1" ]] && auto_accept=true + + if sdk_is_installed; then + info "NatNet SDK already installed" + info "Library: ${sdk_lib_dir}/libNatNet.so" + info "Headers: ${sdk_include_dir}" + exit 0 + fi + + choose_archive + show_license_notice + + if [[ "$auto_accept" == "true" ]]; then + info "NATNET_ACCEPT_LICENSE=1 / --accept-license set — accepting license non-interactively" + else + read -r -p "Accept the OptiTrack NatNet SDK license and download the SDK now? [Y/n] " reply + reply="${reply:-y}" + if ! [[ "${reply}" =~ ^[Yy]$ ]]; then + warn "NatNet SDK installation skipped" + exit 1 + fi + fi + + temp_dir="$(mktemp -d "/tmp/natnet-sdk.XXXXXX")" + download_archive + extract_archive + install_files + + info "NatNet SDK installation completed" + info "Rebuild with: colcon build --packages-select natnet_ros2" +} + +main "$@" diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp new file mode 100644 index 000000000..186f5572c --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp @@ -0,0 +1,176 @@ +// Copyright (c) 2024 Carnegie Mellon University +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in all +// copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +// SOFTWARE. +// +// natnet_client_adapter.cpp — NatNetClientAdapter implementation. +// +// This is the ONLY translation unit that includes NatNet SDK headers. +// All other code (including tests) depends only on INatNetClient. + +#include "natnet_ros2/natnet_client_adapter.hpp" + +// NatNet SDK (bundled: include/natnet/, lib/libNatNet.so) +#include "NatNetClient.h" +#include "NatNetCAPI.h" +#include "NatNetTypes.h" + +#include + +namespace natnet_ros2 +{ + +// --------------------------------------------------------------------------- +// SDK frame callback trampoline — file-scope so it has C linkage compatible +// with the NATNET_CALLCONV calling convention. +// --------------------------------------------------------------------------- +namespace +{ + +void NATNET_CALLCONV sdk_frame_callback(sFrameOfMocapData * data, void * ctx) +{ + auto * frame_cb = static_cast *>(ctx); + if (!data || !frame_cb || !*frame_cb) { return; } + + FrameSample fs; + fs.frame_num = data->iFrame; + fs.timestamp = data->fTimestamp; + fs.params = static_cast(data->params); + + fs.bodies.reserve(static_cast(data->nRigidBodies)); + for (int i = 0; i < data->nRigidBodies; ++i) { + const sRigidBodyData & rb = data->RigidBodies[i]; + RigidBodySample s; + s.id = rb.ID; + s.x = rb.x; s.y = rb.y; s.z = rb.z; + s.qx = rb.qx; s.qy = rb.qy; s.qz = rb.qz; s.qw = rb.qw; + s.params = static_cast(rb.params); + fs.bodies.push_back(s); + } + + (*frame_cb)(fs); +} + +/// Map NatNet SDK ErrorCode to our NatNetResult. +/// Note: ErrorCode_Timeout was added in NatNet SDK >= 4.5 and is absent in 4.4. +/// If upgrading the SDK, add: case ErrorCode_Timeout: return NatNetResult::Timeout; +NatNetResult from_sdk_error(ErrorCode ec) +{ + switch (ec) { + case ErrorCode_OK: return NatNetResult::OK; + case ErrorCode_Network: return NatNetResult::NetworkError; + case ErrorCode_InvalidArgument: return NatNetResult::InvalidAddress; + default: return NatNetResult::InternalError; + } +} + +} // anonymous namespace + + +// --------------------------------------------------------------------------- +NatNetClientAdapter::NatNetClientAdapter() +: client_(std::make_unique()) +{} + +NatNetClientAdapter::~NatNetClientAdapter() +{ + disconnect(); +} + +// --------------------------------------------------------------------------- +NatNetResult NatNetClientAdapter::connect(const ConnectConfig & cfg) +{ + sNatNetClientConnectParams params; + params.serverAddress = cfg.server_ip.c_str(); + params.localAddress = cfg.client_ip.c_str(); + params.serverCommandPort = cfg.command_port; + params.serverDataPort = cfg.data_port; + + if (is_multicast(cfg)) { + params.connectionType = ConnectionType_Multicast; + params.multicastAddress = cfg.multicast_address.c_str(); + } else { + params.connectionType = ConnectionType_Unicast; + params.multicastAddress = nullptr; + } + + return from_sdk_error(client_->Connect(params)); +} + +// --------------------------------------------------------------------------- +bool NatNetClientAdapter::get_server_info(ServerInfo & out) +{ + sServerDescription desc; + std::memset(&desc, 0, sizeof(desc)); + const ErrorCode ec = client_->GetServerDescription(&desc); + if (ec != ErrorCode_OK) { return false; } + + out.host_present = desc.HostPresent; + out.host_app_name = desc.szHostApp; + for (int i = 0; i < 4; ++i) { + out.host_app_version[i] = static_cast(desc.HostAppVersion[i]); + out.natnet_version[i] = static_cast(desc.NatNetVersion[i]); + } + return true; +} + +// --------------------------------------------------------------------------- +std::vector NatNetClientAdapter::get_body_descriptors() +{ + std::vector result; + + sDataDescriptions * desc_list = nullptr; + if (client_->GetDataDescriptionList(&desc_list) != ErrorCode_OK || !desc_list) { + return result; + } + + for (int i = 0; i < desc_list->nDataDescriptions; ++i) { + const sDataDescription & dd = desc_list->arrDataDescriptions[i]; + if (dd.type != Descriptor_RigidBody || !dd.Data.RigidBodyDescription) { continue; } + const sRigidBodyDescription & rb = *dd.Data.RigidBodyDescription; + + BodyDescriptor bd; + bd.id = rb.ID; + bd.name = rb.szName; + bd.parent_id = rb.parentID; + result.push_back(bd); + } + + NatNet_FreeDescriptions(desc_list); + return result; +} + +// --------------------------------------------------------------------------- +void NatNetClientAdapter::set_frame_callback( + std::function cb) +{ + user_cb_ = std::move(cb); + client_->SetFrameReceivedCallback(sdk_frame_callback, &user_cb_); +} + +// --------------------------------------------------------------------------- +void NatNetClientAdapter::disconnect() +{ + if (client_) { + client_->SetFrameReceivedCallback(sdk_frame_callback, nullptr); + client_->Disconnect(); + } + user_cb_ = nullptr; +} + +} // namespace natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp new file mode 100644 index 000000000..65059f659 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp @@ -0,0 +1,325 @@ +// natnet_ros2_node.cpp +// +// ROS 2 NatNet SDK node for OptiTrack Motive integration. +// +// Published topics (per tracked rigid body): +// /{robot_name}/perception/optitrack/{body_name} → PoseStamped +// /{robot_name}/perception/optitrack/{body_name}/pose_cov → PoseWithCovarianceStamped +// +// Parameters (see config/natnet_config.yaml): +// server_ip, client_ip, command_port, data_port, +// body_name, body_id (-1 = all), publish_direct_optitrack, +// frame_id, debug, position_covariance, orientation_covariance +// +// ROBOT_NAME is read from the environment variable set by AirStack's +// robot_name_map resolver at container startup. + +#include +#include +#include +#include +#include +#include +#include + +// ROS 2 +#include "rclcpp/rclcpp.hpp" +#include "geometry_msgs/msg/pose_stamped.hpp" +#include "geometry_msgs/msg/pose_with_covariance_stamped.hpp" + +// Pure logic + interface (no SDK, testable with FakeNatNetClient) +#include "natnet_ros2/natnet_logic.hpp" +#include "natnet_ros2/natnet_client_adapter.hpp" + + +// --------------------------------------------------------------------------- +// NatNetROS2Node +// --------------------------------------------------------------------------- +class NatNetROS2Node : public rclcpp::Node +{ +public: + using PoseStamped = geometry_msgs::msg::PoseStamped; + using PoseWithCovarianceStamped = geometry_msgs::msg::PoseWithCovarianceStamped; + + struct BodyPublishers + { + rclcpp::Publisher::SharedPtr pose_pub; + rclcpp::Publisher::SharedPtr pose_cov_pub; + }; + + // ----------------------------------------------------------------------- + explicit NatNetROS2Node() + : Node("natnet_ros2_node") + { + // ----- Parameters -------------------------------------------------- + this->declare_parameter("server_ip", "192.168.1.1"); + this->declare_parameter("client_ip", "0.0.0.0"); + this->declare_parameter("command_port", 1510); + this->declare_parameter("data_port", 1511); + this->declare_parameter("connection_type", std::string("unicast")); + this->declare_parameter("multicast_address", std::string("239.255.42.99")); + this->declare_parameter("body_name", "robot_1"); + this->declare_parameter("body_id", -1); + this->declare_parameter("publish_direct_optitrack", true); + this->declare_parameter("publish_to_mavros", false); + this->declare_parameter("frame_id", "world"); + this->declare_parameter("debug", false); + this->declare_parameter( + "position_covariance", + std::vector{0.1,0.,0., 0.,0.1,0., 0.,0.,0.1}); + this->declare_parameter( + "orientation_covariance", + std::vector{0.01,0.,0., 0.,0.01,0., 0.,0.,0.01}); + + // ----- Read parameters --------------------------------------------- + const auto connect_cfg = natnet_ros2::make_connect_config( + this->get_parameter("server_ip").as_string(), + this->get_parameter("client_ip").as_string(), + static_cast(this->get_parameter("command_port").as_int()), + static_cast(this->get_parameter("data_port").as_int()), + this->get_parameter("connection_type").as_string(), + this->get_parameter("multicast_address").as_string()); + + if (connect_cfg.connection_type != + this->get_parameter("connection_type").as_string()) + { + RCLCPP_WARN(get_logger(), + "Unknown connection_type '%s' — falling back to 'unicast'.", + this->get_parameter("connection_type").as_string().c_str()); + } + + body_name_ = this->get_parameter("body_name").as_string(); + body_id_ = static_cast(this->get_parameter("body_id").as_int()); + publish_direct_ = this->get_parameter("publish_direct_optitrack").as_bool(); + frame_id_ = this->get_parameter("frame_id").as_string(); + debug_ = this->get_parameter("debug").as_bool(); + + covariance_6x6_ = natnet_ros2::build_covariance_6x6( + this->get_parameter("position_covariance").as_double_array(), + this->get_parameter("orientation_covariance").as_double_array()); + + const char * rn = std::getenv("ROBOT_NAME"); + robot_name_ = rn ? rn : "robot_1"; + + RCLCPP_INFO(get_logger(), "========================================="); + RCLCPP_INFO(get_logger(), "NatNet ROS 2 Node"); + RCLCPP_INFO(get_logger(), " robot_name: %s", robot_name_.c_str()); + RCLCPP_INFO(get_logger(), " server_ip: %s", connect_cfg.server_ip.c_str()); + RCLCPP_INFO(get_logger(), " command_port: %d", static_cast(connect_cfg.command_port)); + RCLCPP_INFO(get_logger(), " connection_type: %s", connect_cfg.connection_type.c_str()); + if (natnet_ros2::is_multicast(connect_cfg)) { + RCLCPP_INFO(get_logger(), " multicast_addr: %s", connect_cfg.multicast_address.c_str()); + } + RCLCPP_INFO(get_logger(), " body_id: %d (%s)", + static_cast(body_id_), + (body_id_ < 0) ? "track all" : "single body"); + RCLCPP_INFO(get_logger(), "========================================="); + + // Production client — NatNetClientAdapter wraps the SDK + client_ = std::make_unique(); + connect_and_setup(connect_cfg); + + refresh_timer_ = this->create_wall_timer( + std::chrono::seconds(1), + std::bind(&NatNetROS2Node::refresh_descriptions_if_needed, this)); + } + + // ----------------------------------------------------------------------- + ~NatNetROS2Node() + { + if (client_) { client_->disconnect(); } + } + + // ----------------------------------------------------------------------- + // Called from the NatNetClientAdapter's frame trampoline. + // publish() and Clock::now() are thread-safe; pub_mutex_ guards map access. + // ----------------------------------------------------------------------- + void on_frame(const natnet_ros2::FrameSample & frame) + { + if (natnet_ros2::model_list_changed(frame.params)) { + needs_description_refresh_.store(true, std::memory_order_relaxed); + } + + if (debug_) { + RCLCPP_DEBUG(get_logger(), "Frame %d: %zu rigid bodies, ts=%.4f s", + frame.frame_num, frame.bodies.size(), static_cast(frame.timestamp)); + } + + const rclcpp::Time stamp = this->get_clock()->now(); + + for (const auto & rb : frame.bodies) { + if (!natnet_ros2::is_tracking_valid(rb.params)) { + if (debug_) { + RCLCPP_DEBUG(get_logger(), " RB id=%d: tracking invalid, skipping", rb.id); + } + continue; + } + if (!natnet_ros2::should_publish_body(body_id_, rb.id)) { continue; } + + std::lock_guard lock(pub_mutex_); + + const auto pub_it = publishers_.find(rb.id); + if (pub_it == publishers_.end()) { + needs_description_refresh_.store(true, std::memory_order_relaxed); + continue; + } + + const natnet_ros2::PoseData pose = natnet_ros2::rb_to_pose(rb); + const BodyPublishers & bp = pub_it->second; + + if (publish_direct_ && bp.pose_pub) { + PoseStamped msg; + msg.header.frame_id = frame_id_; + msg.header.stamp = stamp; + msg.pose.position.x = pose.x; + msg.pose.position.y = pose.y; + msg.pose.position.z = pose.z; + msg.pose.orientation.x = pose.qx; + msg.pose.orientation.y = pose.qy; + msg.pose.orientation.z = pose.qz; + msg.pose.orientation.w = pose.qw; + bp.pose_pub->publish(msg); + } + + if (bp.pose_cov_pub) { + PoseWithCovarianceStamped cov_msg; + cov_msg.header.frame_id = frame_id_; + cov_msg.header.stamp = stamp; + cov_msg.pose.pose.position.x = pose.x; + cov_msg.pose.pose.position.y = pose.y; + cov_msg.pose.pose.position.z = pose.z; + cov_msg.pose.pose.orientation.x = pose.qx; + cov_msg.pose.pose.orientation.y = pose.qy; + cov_msg.pose.pose.orientation.z = pose.qz; + cov_msg.pose.pose.orientation.w = pose.qw; + cov_msg.pose.covariance = covariance_6x6_; + bp.pose_cov_pub->publish(cov_msg); + } + } + } + +private: + // ----------------------------------------------------------------------- + void connect_and_setup(const natnet_ros2::ConnectConfig & cfg) + { + const natnet_ros2::NegotiationResult neg = + natnet_ros2::negotiate(*client_, cfg); + + if (!neg.ok) { + RCLCPP_ERROR(get_logger(), "%s", neg.log_message.c_str()); + return; + } + + if (neg.server_info.host_present) { + RCLCPP_INFO(get_logger(), "%s", neg.log_message.c_str()); + } else { + RCLCPP_WARN(get_logger(), "%s", neg.log_message.c_str()); + } + + refresh_descriptions_locked(); + + client_->set_frame_callback( + [this](const natnet_ros2::FrameSample & f) { on_frame(f); }); + RCLCPP_INFO(get_logger(), "Frame callback registered — receiving mocap data."); + } + + // ----------------------------------------------------------------------- + void refresh_descriptions_if_needed() + { + if (!needs_description_refresh_.exchange(false, std::memory_order_relaxed)) { + return; + } + RCLCPP_INFO(get_logger(), "Model list change detected — refreshing data descriptions."); + std::lock_guard lock(pub_mutex_); + refresh_descriptions_locked(); + } + + // ----------------------------------------------------------------------- + // Must be called with pub_mutex_ held (or from single-threaded init). + // ----------------------------------------------------------------------- + void refresh_descriptions_locked() + { + if (!client_) { return; } + + // Always ensure the statically-configured body has a publisher + if (body_id_ >= 0) { + ensure_publisher_locked(body_id_, body_name_); + } + + const auto bodies = client_->get_body_descriptors(); + int newly_created = 0; + for (const auto & bd : bodies) { + // Store name for every body (including skeleton bones) + body_names_[bd.id] = bd.name; + + // Skip skeleton bones (parent_id >= 0) + if (bd.parent_id >= 0) { continue; } + + // When tracking a single body, skip others + if (!natnet_ros2::should_publish_body(body_id_, bd.id)) { continue; } + + if (ensure_publisher_locked(bd.id, bd.name)) { ++newly_created; } + } + + if (newly_created > 0) { + RCLCPP_INFO(get_logger(), + "Data descriptions refreshed: %d new publisher(s) created.", newly_created); + } else { + RCLCPP_DEBUG(get_logger(), "Data descriptions refreshed: no new publishers."); + } + } + + // ----------------------------------------------------------------------- + bool ensure_publisher_locked(int32_t id, const std::string & name) + { + if (publishers_.count(id)) { return false; } + + const std::string topic_base = + natnet_ros2::optitrack_topic_base(robot_name_, name); + + BodyPublishers bp; + if (publish_direct_) { + bp.pose_pub = this->create_publisher(topic_base, 10); + } + bp.pose_cov_pub = this->create_publisher( + natnet_ros2::optitrack_pose_cov_topic(robot_name_, name), 10); + + publishers_.emplace(id, std::move(bp)); + + RCLCPP_INFO(get_logger(), + "Publisher registered: id=%d name='%s' → %s[/pose_cov]", + static_cast(id), name.c_str(), topic_base.c_str()); + return true; + } + + // ----------------------------------------------------------------------- + // Parameters / state + std::string body_name_; + int32_t body_id_ = -1; + bool publish_direct_ = true; + std::string frame_id_; + bool debug_ = false; + std::string robot_name_; + + std::array covariance_6x6_{}; + + std::unique_ptr client_; + + std::mutex pub_mutex_; + std::unordered_map body_names_; + std::unordered_map publishers_; + + std::atomic needs_description_refresh_{false}; + rclcpp::TimerBase::SharedPtr refresh_timer_; +}; + + +// --------------------------------------------------------------------------- +int main(int argc, char ** argv) +{ + rclcpp::init(argc, argv); + auto node = std::make_shared(); + rclcpp::spin(node); + rclcpp::shutdown(); + return 0; +} diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py b/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py new file mode 100755 index 000000000..9a36f879d --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py @@ -0,0 +1,116 @@ +#!/usr/bin/env python3 + +""" +Vision Pose Converter Node + +Optional converter that bridges NatNet pose data to MAVROS vision_pose format +for PX4 external pose estimation and state fusion. + +Converts from NatNet coordinate frame to a frame suitable for MAVROS. +""" + +import rclpy +from rclpy.node import Node +from geometry_msgs.msg import PoseStamped, PoseWithCovarianceStamped + + +class VisionPoseConverterNode(Node): + """ + Converts NatNet pose to MAVROS vision_pose format. + + Listens to NatNet pose data and publishes to MAVROS for external + pose feedback to PX4 autopilot. + """ + + def __init__(self): + super().__init__('vision_pose_converter') + + self.declare_parameter('frame_id', 'world') + self.declare_parameter('child_frame_id', 'base_link') + self.declare_parameter('canonical_quaternion', True) + + self.frame_id = self.get_parameter('frame_id').value + self.child_frame_id = self.get_parameter('child_frame_id').value + self.canonical_quaternion = self.get_parameter('canonical_quaternion').value + + # Subscribers + self.pose_sub = self.create_subscription( + PoseWithCovarianceStamped, + 'input_pose', + self._on_pose, + 10 + ) + + # Publishers + self.pose_pub = self.create_publisher( + PoseStamped, + 'output_pose', + 10 + ) + self.pose_cov_pub = self.create_publisher( + PoseWithCovarianceStamped, + 'output_pose_cov', + 10 + ) + + self.get_logger().info( + f'Vision pose converter started ' + f'(frame_id={self.frame_id!r}, child_frame_id={self.child_frame_id!r}, ' + f'canonical_quaternion={self.canonical_quaternion})' + ) + + @staticmethod + def _canonical_quaternion(o): + """ + Return the quaternion in canonical form (qw >= 0) by negating all + four components when qw < 0. + + q and -q represent the same 3-D rotation, but some EKF implementations + (ArduPilot EKF3 in particular) are sensitive to sign flips between + consecutive frames. Keeping qw >= 0 guarantees a consistent + representation across the full orientation space. + """ + if o.w < 0.0: + o.x, o.y, o.z, o.w = -o.x, -o.y, -o.z, -o.w + return o + + def _on_pose(self, msg: PoseWithCovarianceStamped): + """ + Callback for incoming NatNet pose. + + Converts and republishes for MAVROS consumption. + Normalises the quaternion to canonical form (qw >= 0) before publishing + so that EKF consumers never see a sign-flip discontinuity. + """ + try: + msg.header.frame_id = self.frame_id + if self.canonical_quaternion: + msg.pose.pose.orientation = self._canonical_quaternion( + msg.pose.pose.orientation + ) + + self.pose_cov_pub.publish(msg) + + pose_msg = PoseStamped() + pose_msg.header = msg.header + pose_msg.pose = msg.pose.pose + self.pose_pub.publish(pose_msg) + + except Exception as e: + self.get_logger().error(f"Error converting pose: {e}") + + +def main(args=None): + """Main entry point""" + rclpy.init(args=args) + try: + node = VisionPoseConverterNode() + rclpy.spin(node) + except KeyboardInterrupt: + pass + finally: + rclpy.shutdown() + + +if __name__ == '__main__': + main() diff --git a/robot/ros_ws/src/perception/natnet_ros2/test/fake_natnet_client.hpp b/robot/ros_ws/src/perception/natnet_ros2/test/fake_natnet_client.hpp new file mode 100644 index 000000000..3ee0dd8b2 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/test/fake_natnet_client.hpp @@ -0,0 +1,163 @@ +// Copyright (c) 2024 Carnegie Mellon University +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in all +// copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +// SOFTWARE. +// +// fake_natnet_client.hpp — in-process test double for INatNetClient. +// +// Used exclusively in unit tests (test_natnet_logic.cpp). +// Never included in production binaries. +// +// Usage: +// FakeNatNetClient fake; +// fake.connect_result = NatNetResult::OK; +// fake.server_info = { .host_present = true, .host_app_name = "Motive" }; +// fake.body_descriptors = {{ .id=1, .name="Drone", .parent_id=-1 }}; +// +// auto result = natnet_ros2::negotiate(fake, cfg); +// EXPECT_TRUE(result.ok); + +#pragma once + +#include "natnet_ros2/natnet_logic.hpp" + +#include +#include +#include + +namespace natnet_ros2 +{ + +class FakeNatNetClient : public INatNetClient +{ +public: + // ----------------------------------------------------------------------- + // Configurable behaviour — set before calling negotiate() / testing + // ----------------------------------------------------------------------- + + /// What connect() should return. + NatNetResult connect_result = NatNetResult::OK; + + /// What get_server_info() should populate and return. + /// Set host_present = true to simulate a fully-identified server. + ServerInfo server_info; + + /// get_server_info() return value (independent of server_info.host_present, + /// so tests can simulate "SDK call failed" vs. "host not present"). + bool server_info_call_succeeds = true; + + /// What get_body_descriptors() should return. + std::vector body_descriptors; + + // ----------------------------------------------------------------------- + // Call-record state — inspect after exercising the fake + // ----------------------------------------------------------------------- + + bool connect_was_called = false; + ConnectConfig last_connect_config; + + bool server_info_was_called = false; + bool descriptors_was_called = false; + bool set_callback_was_called = false; + bool disconnect_was_called = false; + + /// Frames that were injected via inject_frame(). + int frames_injected = 0; + + // ----------------------------------------------------------------------- + // INatNetClient overrides + // ----------------------------------------------------------------------- + + NatNetResult connect(const ConnectConfig & cfg) override + { + connect_was_called = true; + last_connect_config = cfg; + return connect_result; + } + + bool get_server_info(ServerInfo & out) override + { + server_info_was_called = true; + out = server_info; + return server_info_call_succeeds; + } + + std::vector get_body_descriptors() override + { + descriptors_was_called = true; + return body_descriptors; + } + + void set_frame_callback(std::function cb) override + { + set_callback_was_called = true; + frame_cb_ = cb; + } + + void disconnect() override + { + disconnect_was_called = true; + } + + // ----------------------------------------------------------------------- + // Test helper: push a synthetic frame into the registered callback. + // ----------------------------------------------------------------------- + void inject_frame(const FrameSample & frame) + { + if (frame_cb_) { + ++frames_injected; + frame_cb_(frame); + } + } + + /// Convenience: build and inject a single-body tracking frame. + void inject_body(int32_t id, float x, float y, float z, + float qx = 0.f, float qy = 0.f, + float qz = 0.f, float qw = 1.f, + int16_t rb_params = 0x01 /* tracking valid */, + int16_t frame_params = 0x00) + { + FrameSample f; + f.params = frame_params; + RigidBodySample rb; + rb.id = id; rb.x = x; rb.y = y; rb.z = z; + rb.qx = qx; rb.qy = qy; rb.qz = qz; rb.qw = qw; + rb.params = rb_params; + f.bodies.push_back(rb); + inject_frame(f); + } + + // ----------------------------------------------------------------------- + // Reset all recorded state (keep configuration). + // ----------------------------------------------------------------------- + void reset_records() + { + connect_was_called = false; + server_info_was_called = false; + descriptors_was_called = false; + set_callback_was_called = false; + disconnect_was_called = false; + frames_injected = 0; + last_connect_config = {}; + } + +private: + std::function frame_cb_; +}; + +} // namespace natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp new file mode 100644 index 000000000..7a144ef9b --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp @@ -0,0 +1,757 @@ +// Copyright (c) 2024 Carnegie Mellon University +// +// Permission is hereby granted, free of charge, to any person obtaining a copy +// of this software and associated documentation files (the "Software"), to deal +// in the Software without restriction, including without limitation the rights +// to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +// copies of the Software, and to permit persons to whom the Software is +// furnished to do so, subject to the following conditions: +// +// The above copyright notice and this permission notice shall be included in all +// copies or substantial portions of the Software. +// +// THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +// IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +// FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +// AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +// LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +// OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +// SOFTWARE. +// +// Unit tests for natnet_ros2/natnet_logic.hpp. +// +// NO dependency on the NatNet SDK or rclcpp — compiles with gtest only. +// Run via: +// colcon test --packages-select natnet_ros2 --event-handlers console_direct+ +// colcon test-result --test-result-base build/natnet_ros2 --verbose + +#include +#include "natnet_ros2/natnet_logic.hpp" +#include "fake_natnet_client.hpp" + +using namespace natnet_ros2; + + +// =========================================================================== +// Covariance +// =========================================================================== + +TEST(BuildCovariance6x6, DiagonalBlocksLandInCorrectSlots) +{ + const std::vector pos = {0.1, 0.0, 0.0, + 0.0, 0.1, 0.0, + 0.0, 0.0, 0.1}; + const std::vector ori = {0.01, 0.0, 0.0, + 0.0, 0.01, 0.0, + 0.0, 0.0, 0.01}; + + auto cov = build_covariance_6x6(pos, ori); + + ASSERT_EQ(cov.size(), 36u); + EXPECT_DOUBLE_EQ(cov[0 * 6 + 0], 0.1); + EXPECT_DOUBLE_EQ(cov[1 * 6 + 1], 0.1); + EXPECT_DOUBLE_EQ(cov[2 * 6 + 2], 0.1); + EXPECT_DOUBLE_EQ(cov[3 * 6 + 3], 0.01); + EXPECT_DOUBLE_EQ(cov[4 * 6 + 4], 0.01); + EXPECT_DOUBLE_EQ(cov[5 * 6 + 5], 0.01); +} + +TEST(BuildCovariance6x6, CrossBlockEntriesAreZero) +{ + const std::vector ones(9, 1.0); + auto cov = build_covariance_6x6(ones, ones); + + for (int r = 0; r < 6; ++r) { + for (int c = 0; c < 6; ++c) { + const bool in_pos = (r < 3 && c < 3); + const bool in_ori = (r >= 3 && c >= 3); + if (!in_pos && !in_ori) { + EXPECT_DOUBLE_EQ(cov[r * 6 + c], 0.0) + << "Expected 0 at [" << r << "][" << c << "]"; + } + } + } +} + +TEST(BuildCovariance6x6, OffDiagonalEntriesPreserved) +{ + const std::vector pos = {1, 2, 3, 4, 5, 6, 7, 8, 9}; + const std::vector ori = {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}; + + auto cov = build_covariance_6x6(pos, ori); + + EXPECT_DOUBLE_EQ(cov[1 * 6 + 2], 6.0); // pos[1][2] + EXPECT_DOUBLE_EQ(cov[2 * 6 + 0], 7.0); // pos[2][0] + EXPECT_DOUBLE_EQ(cov[3 * 6 + 4], 0.2); // ori[0][1] + EXPECT_DOUBLE_EQ(cov[5 * 6 + 4], 0.8); // ori[2][1] +} + +TEST(BuildCovariance6x6, ShortInputFillsRemainingWithZero) +{ + auto cov = build_covariance_6x6({0.5}, {0.05}); + + EXPECT_DOUBLE_EQ(cov[0 * 6 + 0], 0.5); + EXPECT_DOUBLE_EQ(cov[3 * 6 + 3], 0.05); + EXPECT_DOUBLE_EQ(cov[0 * 6 + 1], 0.0); + EXPECT_DOUBLE_EQ(cov[4 * 6 + 4], 0.0); +} + +TEST(BuildCovariance6x6, EmptyInputsProduceAllZeros) +{ + auto cov = build_covariance_6x6({}, {}); + for (double v : cov) { EXPECT_DOUBLE_EQ(v, 0.0); } +} + +TEST(BuildCovariance6x6, OutputIsExactly36Elements) +{ + EXPECT_EQ(static_cast(build_covariance_6x6({}, {}).size()), 36); +} + + +// =========================================================================== +// Topic names +// =========================================================================== + +TEST(TopicNames, BaseTopicFormat) +{ + EXPECT_EQ(optitrack_topic_base("robot_1", "Drone"), + "/robot_1/perception/optitrack/Drone"); +} + +TEST(TopicNames, PoseCovTopicAppendsSuffix) +{ + const std::string base = optitrack_topic_base("robot_1", "Drone"); + const std::string cov = optitrack_pose_cov_topic("robot_1", "Drone"); + EXPECT_EQ(cov, base + "/pose_cov"); +} + +TEST(TopicNames, DifferentRobotsGetDifferentNamespaces) +{ + EXPECT_NE(optitrack_topic_base("robot_1", "Body"), + optitrack_topic_base("robot_2", "Body")); +} + +TEST(TopicNames, LeadingSlashPresent) +{ + EXPECT_EQ(optitrack_topic_base("robot_1", "Body")[0], '/'); +} + + +// =========================================================================== +// Server negotiation — validate_connection_type +// =========================================================================== + +TEST(ValidateConnectionType, UnicastPassesThrough) +{ + EXPECT_EQ(validate_connection_type("unicast"), "unicast"); +} + +TEST(ValidateConnectionType, MulticastPassesThrough) +{ + EXPECT_EQ(validate_connection_type("multicast"), "multicast"); +} + +TEST(ValidateConnectionType, UnknownFallsBackToUnicast) +{ + EXPECT_EQ(validate_connection_type("broadcast"), "unicast"); + EXPECT_EQ(validate_connection_type(""), "unicast"); + EXPECT_EQ(validate_connection_type("UDP"), "unicast"); +} + +TEST(ValidateConnectionType, CaseSensitiveFallsBack) +{ + EXPECT_EQ(validate_connection_type("Unicast"), "unicast"); + EXPECT_EQ(validate_connection_type("MULTICAST"), "unicast"); +} + + +// =========================================================================== +// Server negotiation — ConnectConfig + make_connect_config +// =========================================================================== + +TEST(ConnectConfig, DefaultsAreUnicast) +{ + const ConnectConfig cfg{}; + EXPECT_EQ(cfg.connection_type, "unicast"); + EXPECT_FALSE(is_multicast(cfg)); +} + +TEST(ConnectConfig, UnicastConfigNotMulticast) +{ + const auto cfg = make_connect_config( + "10.0.0.1", "0.0.0.0", 1510, 1511, "unicast"); + EXPECT_FALSE(is_multicast(cfg)); + EXPECT_FALSE(needs_multicast_address(cfg)); +} + +TEST(ConnectConfig, MulticastConfigIsMulticast) +{ + const auto cfg = make_connect_config( + "10.0.0.1", "0.0.0.0", 1510, 1511, "multicast", "239.255.42.99"); + EXPECT_TRUE(is_multicast(cfg)); + EXPECT_TRUE(needs_multicast_address(cfg)); + EXPECT_EQ(cfg.multicast_address, "239.255.42.99"); +} + +TEST(ConnectConfig, InvalidConnectionTypeFallsBackToUnicast) +{ + const auto cfg = make_connect_config( + "10.0.0.1", "0.0.0.0", 1510, 1511, "broadcast"); + EXPECT_EQ(cfg.connection_type, "unicast"); + EXPECT_FALSE(is_multicast(cfg)); +} + +TEST(ConnectConfig, PortsArePreserved) +{ + const auto cfg = make_connect_config( + "192.168.0.100", "192.168.0.200", 9000u, 9001u, "unicast"); + EXPECT_EQ(cfg.server_ip, "192.168.0.100"); + EXPECT_EQ(cfg.client_ip, "192.168.0.200"); + EXPECT_EQ(cfg.command_port, 9000u); + EXPECT_EQ(cfg.data_port, 9001u); +} + +TEST(ConnectConfig, CustomMulticastAddress) +{ + const auto cfg = make_connect_config( + "10.0.0.1", "0.0.0.0", 1510, 1511, "multicast", "239.0.0.1"); + EXPECT_EQ(cfg.multicast_address, "239.0.0.1"); +} + +TEST(ConnectConfig, UnicastAddressFieldIgnored) +{ + // multicast_address is still stored but should not be passed to the SDK + const auto cfg = make_connect_config( + "10.0.0.1", "0.0.0.0", 1510, 1511, "unicast", "239.255.42.99"); + EXPECT_FALSE(needs_multicast_address(cfg)); +} + + +// =========================================================================== +// Data streaming — is_tracking_valid +// =========================================================================== + +TEST(IsTrackingValid, Bit0SetMeansValid) +{ + EXPECT_TRUE(is_tracking_valid(0x01)); + EXPECT_TRUE(is_tracking_valid(0x03)); // bits 0 and 1 + EXPECT_TRUE(is_tracking_valid(0xFF)); +} + +TEST(IsTrackingValid, Bit0ClearMeansInvalid) +{ + EXPECT_FALSE(is_tracking_valid(0x00)); + EXPECT_FALSE(is_tracking_valid(0x02)); // only bit 1 set + EXPECT_FALSE(is_tracking_valid(0xFE)); // all bits except 0 +} + + +// =========================================================================== +// Data streaming — model_list_changed +// =========================================================================== + +TEST(ModelListChanged, Bit1SetMeansChanged) +{ + EXPECT_TRUE(model_list_changed(0x02)); + EXPECT_TRUE(model_list_changed(0x03)); + EXPECT_TRUE(model_list_changed(0xFF)); +} + +TEST(ModelListChanged, Bit1ClearMeansNotChanged) +{ + EXPECT_FALSE(model_list_changed(0x00)); + EXPECT_FALSE(model_list_changed(0x01)); // only bit 0 + EXPECT_FALSE(model_list_changed(0xFD)); // all bits except 1 +} + + +// =========================================================================== +// Data streaming — should_publish_body +// =========================================================================== + +TEST(ShouldPublishBody, NegativeFilterMeansPublishAll) +{ + EXPECT_TRUE(should_publish_body(-1, 0)); + EXPECT_TRUE(should_publish_body(-1, 1)); + EXPECT_TRUE(should_publish_body(-1, 999)); +} + +TEST(ShouldPublishBody, ZeroFilterAllowsOnlyId0) +{ + EXPECT_TRUE(should_publish_body(0, 0)); + EXPECT_FALSE(should_publish_body(0, 1)); + EXPECT_FALSE(should_publish_body(0, 999)); +} + +TEST(ShouldPublishBody, PositiveFilterMatchesExact) +{ + EXPECT_TRUE(should_publish_body(5, 5)); + EXPECT_FALSE(should_publish_body(5, 4)); + EXPECT_FALSE(should_publish_body(5, 6)); +} + + +// =========================================================================== +// Data streaming — rb_to_pose (sample data conversion) +// =========================================================================== + +TEST(RbToPose, PositionComponentsConvertedToDouble) +{ + RigidBodySample rb; + rb.x = 1.5f; rb.y = -2.25f; rb.z = 0.5f; + rb.qx = 0.f; rb.qy = 0.f; rb.qz = 0.f; rb.qw = 1.f; + + const PoseData p = rb_to_pose(rb); + + EXPECT_DOUBLE_EQ(p.x, static_cast(1.5f)); + EXPECT_DOUBLE_EQ(p.y, static_cast(-2.25f)); + EXPECT_DOUBLE_EQ(p.z, static_cast(0.5f)); +} + +TEST(RbToPose, OrientationComponentsConvertedToDouble) +{ + RigidBodySample rb; + rb.x = 0.f; rb.y = 0.f; rb.z = 0.f; + // 90-degree rotation about Z: qw = cos(45°), qz = sin(45°) + rb.qx = 0.f; + rb.qy = 0.f; + rb.qz = 0.7071068f; + rb.qw = 0.7071068f; + + const PoseData p = rb_to_pose(rb); + + EXPECT_NEAR(p.qz, 0.7071068, 1e-6); + EXPECT_NEAR(p.qw, 0.7071068, 1e-6); + EXPECT_DOUBLE_EQ(p.qx, 0.0); + EXPECT_DOUBLE_EQ(p.qy, 0.0); +} + +TEST(RbToPose, IdentityOrientationPreserved) +{ + RigidBodySample rb; // default: x=y=z=0, qw=1 + const PoseData p = rb_to_pose(rb); + + EXPECT_DOUBLE_EQ(p.x, 0.0); + EXPECT_DOUBLE_EQ(p.y, 0.0); + EXPECT_DOUBLE_EQ(p.z, 0.0); + EXPECT_DOUBLE_EQ(p.qx, 0.0); + EXPECT_DOUBLE_EQ(p.qy, 0.0); + EXPECT_DOUBLE_EQ(p.qz, 0.0); + EXPECT_DOUBLE_EQ(p.qw, 1.0); +} + +TEST(RbToPose, NegativeCoordinates) +{ + RigidBodySample rb; + rb.x = -10.f; rb.y = -20.f; rb.z = -30.f; + rb.qw = 1.f; + + const PoseData p = rb_to_pose(rb); + + EXPECT_DOUBLE_EQ(p.x, static_cast(-10.f)); + EXPECT_DOUBLE_EQ(p.y, static_cast(-20.f)); + EXPECT_DOUBLE_EQ(p.z, static_cast(-30.f)); +} + + +// =========================================================================== +// Data streaming — FrameSample helpers (integration-style scenarios) +// =========================================================================== + +// Simulate a frame where one body is tracking and one is not. +TEST(FrameSample, TrackingFilterApplied) +{ + FrameSample frame; + frame.frame_num = 42; + frame.timestamp = 1.234f; + frame.params = 0x00; + + RigidBodySample tracking, lost; + tracking.id = 1; + tracking.params = 0x01; // valid + lost.id = 2; + lost.params = 0x00; // invalid + + frame.bodies = {tracking, lost}; + + int published = 0; + for (const auto & rb : frame.bodies) { + if (is_tracking_valid(rb.params)) { + ++published; + } + } + EXPECT_EQ(published, 1); +} + +// Simulate a frame that signals model-list changed while also carrying data. +TEST(FrameSample, ModelListChangedFlagDetected) +{ + FrameSample frame; + frame.params = 0x02; // bit 1 set + + EXPECT_TRUE(model_list_changed(frame.params)); +} + +// Simulate single-body tracking filter: only body id=3 should be published. +TEST(FrameSample, SingleBodyFilterSelectsCorrectBody) +{ + FrameSample frame; + frame.params = 0x00; + + for (int id : {1, 2, 3, 4, 5}) { + RigidBodySample rb; + rb.id = id; + rb.params = 0x01; // all tracking valid + frame.bodies.push_back(rb); + } + + constexpr int32_t filter = 3; + std::vector published; + + for (const auto & rb : frame.bodies) { + if (is_tracking_valid(rb.params) && should_publish_body(filter, rb.id)) { + published.push_back(rb.id); + } + } + + ASSERT_EQ(published.size(), 1u); + EXPECT_EQ(published[0], 3); +} + +// Simulate all-body mode: every valid body gets a PoseData. +TEST(FrameSample, AllBodyModePublishesAllTrackedBodies) +{ + FrameSample frame; + for (int id = 1; id <= 4; ++id) { + RigidBodySample rb; + rb.id = id; + rb.x = static_cast(id); + rb.params = (id % 2 == 0) ? int16_t(0x01) : int16_t(0x00); // even = valid + frame.bodies.push_back(rb); + } + + std::vector out; + for (const auto & rb : frame.bodies) { + if (is_tracking_valid(rb.params) && should_publish_body(-1, rb.id)) { + out.push_back(rb_to_pose(rb)); + } + } + + // Bodies 2 and 4 are valid + ASSERT_EQ(out.size(), 2u); + EXPECT_DOUBLE_EQ(out[0].x, static_cast(2.f)); + EXPECT_DOUBLE_EQ(out[1].x, static_cast(4.f)); +} + +// Verify covariance is stamped into the output as expected. +TEST(FrameSample, CovarianceStampedIntoMessage) +{ + const std::vector pos_cov(9, 0.1); + const std::vector ori_cov(9, 0.01); + const auto cov = build_covariance_6x6(pos_cov, ori_cov); + + // Simulate what natnet_ros2_node.cpp does when building PoseWithCovarianceStamped + std::array msg_covariance = cov6x6_to_array(cov); + + // Position diagonal + EXPECT_DOUBLE_EQ(msg_covariance[0 * 6 + 0], 0.1); + EXPECT_DOUBLE_EQ(msg_covariance[1 * 6 + 1], 0.1); + EXPECT_DOUBLE_EQ(msg_covariance[2 * 6 + 2], 0.1); + // Orientation diagonal + EXPECT_DOUBLE_EQ(msg_covariance[3 * 6 + 3], 0.01); + EXPECT_DOUBLE_EQ(msg_covariance[4 * 6 + 4], 0.01); + EXPECT_DOUBLE_EQ(msg_covariance[5 * 6 + 5], 0.01); + // Cross-block zeros + EXPECT_DOUBLE_EQ(msg_covariance[0 * 6 + 3], 0.0); + EXPECT_DOUBLE_EQ(msg_covariance[3 * 6 + 0], 0.0); +} + + +// =========================================================================== +// Server negotiation — negotiate() + FakeNatNetClient +// =========================================================================== + +// --------------- helpers --------------------------------------------------- + +static ConnectConfig make_test_cfg(const std::string & ct = "unicast") +{ + return make_connect_config("192.168.1.100", "0.0.0.0", 1510u, 1511u, ct); +} + +static ServerInfo make_server_info(bool present = true, + const std::string & app = "Motive", + int vmaj = 3, int vmin = 1, + int nnmaj = 4, int nnmin = 1) +{ + ServerInfo si; + si.host_present = present; + si.host_app_name = app; + si.host_app_version[0] = vmaj; + si.host_app_version[1] = vmin; + si.natnet_version[0] = nnmaj; + si.natnet_version[1] = nnmin; + return si; +} + +// ----------- negotiate() success paths ------------------------------------ + +TEST(Negotiate, SuccessWithHostPresent) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::OK; + fake.server_info = make_server_info(true, "Motive", 3, 1, 4, 1); + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_TRUE(result.ok); + EXPECT_TRUE(result.server_info.host_present); + EXPECT_EQ(result.server_info.host_app_name, "Motive"); + EXPECT_EQ(result.server_info.host_app_version[0], 3); + EXPECT_EQ(result.server_info.natnet_version[0], 4); + + EXPECT_TRUE(fake.connect_was_called); + EXPECT_TRUE(fake.server_info_was_called); + // log message should mention the server IP + EXPECT_NE(result.log_message.find("192.168.1.100"), std::string::npos); +} + +TEST(Negotiate, SuccessButHostNotPresent) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::OK; + fake.server_info = make_server_info(false); + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_TRUE(result.ok); // connection itself succeeded + EXPECT_FALSE(result.server_info.host_present); + // log message should flag the missing host info + EXPECT_NE(result.log_message.find("no host info"), std::string::npos); +} + +TEST(Negotiate, SuccessServerInfoCallFails) +{ + // SDK's GetServerDescription returns an error (simulated via call_succeeds=false) + FakeNatNetClient fake; + fake.connect_result = NatNetResult::OK; + fake.server_info_call_succeeds = false; + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_TRUE(result.ok); + EXPECT_FALSE(result.server_info.host_present); + EXPECT_NE(result.log_message.find("no host info"), std::string::npos); +} + +// ----------- negotiate() failure paths ------------------------------------ + +TEST(Negotiate, NetworkErrorReturnsFalse) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::NetworkError; + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_FALSE(result.ok); + EXPECT_FALSE(fake.server_info_was_called); // should not reach GetServerDescription + EXPECT_NE(result.log_message.find("NetworkError"), std::string::npos); + EXPECT_NE(result.log_message.find("192.168.1.100"), std::string::npos); +} + +TEST(Negotiate, InvalidAddressReturnsFalse) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::InvalidAddress; + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_FALSE(result.ok); + EXPECT_NE(result.log_message.find("InvalidAddress"), std::string::npos); +} + +TEST(Negotiate, TimeoutReturnsFalse) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::Timeout; + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_FALSE(result.ok); + EXPECT_NE(result.log_message.find("Timeout"), std::string::npos); +} + +// ----------- negotiate() passes ConnectConfig correctly ------------------- + +TEST(Negotiate, UnicastConfigPassedToClient) +{ + FakeNatNetClient fake; + const auto cfg = make_test_cfg("unicast"); + negotiate(fake, cfg); + + EXPECT_EQ(fake.last_connect_config.connection_type, "unicast"); + EXPECT_EQ(fake.last_connect_config.server_ip, "192.168.1.100"); + EXPECT_EQ(fake.last_connect_config.command_port, 1510u); +} + +TEST(Negotiate, MulticastConfigPassedToClient) +{ + FakeNatNetClient fake; + const auto cfg = make_connect_config( + "10.0.0.1", "0.0.0.0", 1510u, 1511u, "multicast", "239.0.0.1"); + negotiate(fake, cfg); + + EXPECT_EQ(fake.last_connect_config.connection_type, "multicast"); + EXPECT_EQ(fake.last_connect_config.multicast_address, "239.0.0.1"); +} + +// ----------- log message content ------------------------------------------ + +TEST(Negotiate, SuccessLogContainsAppAndVersion) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::OK; + fake.server_info = make_server_info(true, "MotiveBody", 2, 5, 4, 0); + + const auto result = negotiate(fake, make_test_cfg()); + + EXPECT_NE(result.log_message.find("MotiveBody"), std::string::npos); + EXPECT_NE(result.log_message.find("2.5"), std::string::npos); // v2.5 + EXPECT_NE(result.log_message.find("4.0"), std::string::npos); // NatNet 4.0 +} + +TEST(Negotiate, FailureLogContainsPortAndType) +{ + FakeNatNetClient fake; + fake.connect_result = NatNetResult::Timeout; + const auto cfg = make_connect_config( + "10.1.2.3", "0.0.0.0", 9000u, 9001u, "multicast"); + + const auto result = negotiate(fake, cfg); + + EXPECT_NE(result.log_message.find("9000"), std::string::npos); + EXPECT_NE(result.log_message.find("multicast"), std::string::npos); +} + + +// =========================================================================== +// FakeNatNetClient — frame injection +// =========================================================================== + +TEST(FakeNatNetClient, CallbackNotCalledBeforeRegistration) +{ + FakeNatNetClient fake; + // No set_frame_callback called — inject_frame should be a no-op + fake.inject_body(1, 1.f, 2.f, 3.f); + EXPECT_EQ(fake.frames_injected, 0); +} + +TEST(FakeNatNetClient, CallbackInvokedAfterRegistration) +{ + FakeNatNetClient fake; + + std::vector received; + fake.set_frame_callback([&](const FrameSample & f) { received.push_back(f); }); + + fake.inject_body(1, 1.f, 2.f, 3.f); + fake.inject_body(2, 4.f, 5.f, 6.f); + + EXPECT_EQ(fake.frames_injected, 2); + ASSERT_EQ(received.size(), 2u); +} + +TEST(FakeNatNetClient, InjectedBodyDataIsPreserved) +{ + FakeNatNetClient fake; + + FrameSample captured; + fake.set_frame_callback([&](const FrameSample & f) { captured = f; }); + + fake.inject_body(42, 1.5f, -2.5f, 0.75f, + 0.f, 0.f, 0.7071068f, 0.7071068f, + 0x01 /* tracking valid */); + + ASSERT_EQ(captured.bodies.size(), 1u); + const auto & rb = captured.bodies[0]; + EXPECT_EQ(rb.id, 42); + EXPECT_FLOAT_EQ(rb.x, 1.5f); + EXPECT_FLOAT_EQ(rb.y, -2.5f); + EXPECT_FLOAT_EQ(rb.z, 0.75f); + EXPECT_NEAR(rb.qz, 0.7071068f, 1e-6f); + EXPECT_TRUE(is_tracking_valid(rb.params)); +} + +TEST(FakeNatNetClient, ModelListChangedFlagDeliveredInFrame) +{ + FakeNatNetClient fake; + + bool model_changed = false; + fake.set_frame_callback([&](const FrameSample & f) { + model_changed = model_list_changed(f.params); + }); + + FrameSample f; + f.params = 0x02; // bit 1 = model list changed + fake.inject_frame(f); + + EXPECT_TRUE(model_changed); +} + +TEST(FakeNatNetClient, ResetRecordsClearsState) +{ + FakeNatNetClient fake; + fake.set_frame_callback([](const FrameSample &) {}); + fake.inject_body(1, 0.f, 0.f, 0.f); + + fake.reset_records(); + + EXPECT_FALSE(fake.connect_was_called); + EXPECT_FALSE(fake.set_callback_was_called); + EXPECT_EQ(fake.frames_injected, 0); +} + + +// =========================================================================== +// Body descriptors + filtering +// =========================================================================== + +TEST(BodyDescriptor, SkeletonBoneHasPositiveParentId) +{ + BodyDescriptor bone; + bone.id = 10; + bone.name = "Hip"; + bone.parent_id = 5; // part of skeleton with id=5 + + EXPECT_GE(bone.parent_id, 0); // should be skipped in publisher creation +} + +TEST(BodyDescriptor, TopLevelBodyHasNegativeParentId) +{ + BodyDescriptor body; + body.id = 1; + body.name = "Drone"; + body.parent_id = -1; + + EXPECT_LT(body.parent_id, 0); // should be published +} + +TEST(FakeNatNetClient, GetBodyDescriptorsReturnsConfigured) +{ + FakeNatNetClient fake; + fake.body_descriptors = { + {1, "Drone1", -1}, + {2, "Drone2", -1}, + {3, "Hip", 2}, // skeleton bone + }; + + const auto descs = fake.get_body_descriptors(); + + ASSERT_EQ(descs.size(), 3u); + EXPECT_TRUE(fake.descriptors_was_called); + + // Only top-level bodies should be published (parent_id < 0) + int top_level = 0; + for (const auto & d : descs) { + if (d.parent_id < 0) { ++top_level; } + } + EXPECT_EQ(top_level, 2); +} diff --git a/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py new file mode 100644 index 000000000..cba5a3444 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py @@ -0,0 +1,152 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Unit tests for natnet_ros2 Python source code. + +These tests import the actual production source files and stub out ROS at the +import boundary so no ROS installation is required. + +Coverage here: + vision_pose_converter_node.py → VisionPoseConverterNode._canonical_quaternion() + → VisionPoseConverterNode._on_pose() frame_id assignment + +NOT covered here (C++ — requires colcon build + gtest): + natnet_ros2_node.cpp → build_covariance_6x6(), topic name construction, + connection_type validation, SDK frame callback logic. + These live in test_natnet_logic.cpp in the same test/ directory. +""" + +import sys +from pathlib import Path +from types import SimpleNamespace +from unittest.mock import MagicMock + +# --------------------------------------------------------------------------- +# Stub ROS before importing the source. +# +# The key subtlety: VisionPoseConverterNode inherits from rclpy.node.Node. +# If Node is a plain MagicMock() the class body is never executed (Python's +# metaclass machinery returns a Mock for attribute access instead of running +# __init_subclass__ / defining methods). We supply a real dummy base class +# so the actual class body — including _canonical_quaternion — is defined. +# --------------------------------------------------------------------------- + +class _FakeNode: + def __init__(self, name: str): + pass + def get_logger(self): + return MagicMock() + def declare_parameter(self, *args, **kwargs): + pass + def get_parameter(self, name): + m = MagicMock() + m.value = MagicMock() + return m + def create_subscription(self, *args, **kwargs): + return MagicMock() + def create_publisher(self, *args, **kwargs): + return MagicMock() + + +_rclpy_node_mod = MagicMock() +_rclpy_node_mod.Node = _FakeNode +sys.modules.setdefault("rclpy", MagicMock()) +sys.modules["rclpy.node"] = _rclpy_node_mod +sys.modules.setdefault("geometry_msgs", MagicMock()) +sys.modules.setdefault("geometry_msgs.msg", MagicMock()) + +# Add the package's src/ directory (co-located: test/ → package root → src/). +_natnet_src = Path(__file__).resolve().parent.parent / "src" +if str(_natnet_src) not in sys.path: + sys.path.insert(0, str(_natnet_src)) + +from vision_pose_converter_node import VisionPoseConverterNode # noqa: E402 + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +def _quat(x: float, y: float, z: float, w: float) -> SimpleNamespace: + """Minimal quaternion-like object matching the expected interface.""" + return SimpleNamespace(x=x, y=y, z=z, w=w) + + +# --------------------------------------------------------------------------- +# VisionPoseConverterNode._canonical_quaternion +# --------------------------------------------------------------------------- + +import pytest + + +@pytest.mark.unit +def test_canonical_quaternion_positive_w_unchanged(): + """Quaternion with w > 0 must not be altered.""" + q = _quat(0.1, 0.2, 0.3, 0.9) + out = VisionPoseConverterNode._canonical_quaternion(q) + assert out.w == pytest.approx(0.9) + assert out.x == pytest.approx(0.1) + assert out.y == pytest.approx(0.2) + assert out.z == pytest.approx(0.3) + + +@pytest.mark.unit +def test_canonical_quaternion_negative_w_flipped(): + """Quaternion with w < 0 must have all four components negated.""" + q = _quat(0.1, 0.2, 0.3, -0.9) + out = VisionPoseConverterNode._canonical_quaternion(q) + assert out.w == pytest.approx(0.9) + assert out.x == pytest.approx(-0.1) + assert out.y == pytest.approx(-0.2) + assert out.z == pytest.approx(-0.3) + + +@pytest.mark.unit +def test_canonical_quaternion_zero_w_unchanged(): + """w == 0 satisfies w >= 0 so no flip should occur.""" + q = _quat(1.0, 0.0, 0.0, 0.0) + out = VisionPoseConverterNode._canonical_quaternion(q) + assert out.w == pytest.approx(0.0) + assert out.x == pytest.approx(1.0) + + +@pytest.mark.unit +def test_canonical_quaternion_identity(): + q = _quat(0.0, 0.0, 0.0, 1.0) + out = VisionPoseConverterNode._canonical_quaternion(q) + assert out.w == pytest.approx(1.0) + assert out.x == pytest.approx(0.0) + + +@pytest.mark.unit +def test_canonical_quaternion_returns_same_object(): + """The method mutates and returns the same object (not a copy).""" + q = _quat(0.0, 0.0, 0.0, 1.0) + out = VisionPoseConverterNode._canonical_quaternion(q) + assert out is q + + +@pytest.mark.unit +def test_canonical_quaternion_w_stays_non_negative(): + """After canonicalisation w must always be >= 0.""" + cases = [ + _quat(0.0, 0.0, 0.7071, 0.7071), + _quat(0.0, 0.0, -0.7071, -0.7071), + _quat(0.5, -0.5, 0.5, -0.5), + _quat(0.0, 0.0, 1.0, 0.0), + ] + for q in cases: + out = VisionPoseConverterNode._canonical_quaternion(q) + assert out.w >= 0.0, f"w={out.w} after canonicalisation of {q}" + + +@pytest.mark.unit +def test_canonical_quaternion_dual_sign_produces_same_result(): + """q and -q must both canonicalise to the same output.""" + q_pos = _quat(0.1, 0.2, 0.3, 0.9) + q_neg = _quat(-0.1, -0.2, -0.3, -0.9) + out_pos = VisionPoseConverterNode._canonical_quaternion(q_pos) + out_neg = VisionPoseConverterNode._canonical_quaternion(q_neg) + assert out_pos.w == pytest.approx(out_neg.w) + assert out_pos.x == pytest.approx(out_neg.x) + assert out_pos.y == pytest.approx(out_neg.y) + assert out_pos.z == pytest.approx(out_neg.z) diff --git a/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml b/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml index bdc1d7d25..e433e94d4 100644 --- a/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml +++ b/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml @@ -2,6 +2,9 @@ + + + @@ -67,7 +70,11 @@ + + + + diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/README.md b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/README.md index 2822c059f..44449f50f 100644 --- a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/README.md +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/README.md @@ -37,10 +37,10 @@ Included from `sensors_bringup` under the robot and `sensors` namespaces. Defaul ## System tests (`sensors` mark) -Sensor checks (sim + robot topic rates, LiDAR validation) live in repo-root **`tests/test_sensors.py`** (`pytest -m sensors`), which runs **after** **`tests/test_liveliness.py`** in the default collection order. For **Isaac Sim** (`--sim isaacsim`), that suite: +Sensor checks (sim + robot topic rates, LiDAR validation) live in repo-root **`tests/system/test_sensors.py`** (`pytest -m sensors`), which runs **after** **`tests/system/test_liveliness.py`** in the default collection order. **Numpy-only** LiDAR filter rules live in **`lidar_point_cloud_filter/validation_core.py`** (this package's module directory), covered by **`test/test_validation_core.py`** (`pytest -m unit` via proxy in `tests/robot/`) and imported by **`scripts/validate_lidar_filter_clouds.py`** at runtime. For **Isaac Sim** (`--sim isaacsim`), that suite: - Proves the **filtered** topic is alive (`ros2 topic echo --once` on `.../point_cloud` — large clouds are not probed with `ros2 topic hz`). -- Runs `scripts/validate_lidar_filter_clouds.py` inside each robot container: checks the **filtered** cloud against `near_range_m`, optionally compares behavior when **`point_cloud_raw`** has near-field returns. +- Runs `robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py` inside each robot container: checks the **filtered** cloud against `near_range_m`, optionally compares behavior when **`point_cloud_raw`** has near-field returns. **Microsoft AirSim** does not guarantee `sensors/ouster` topics on that profile; those steps are skipped there. diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/lidar_point_cloud_filter/validation_core.py b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/lidar_point_cloud_filter/validation_core.py new file mode 100644 index 000000000..5d3f22cfd --- /dev/null +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/lidar_point_cloud_filter/validation_core.py @@ -0,0 +1,73 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Pure-numeric LiDAR filter validation helpers (no ROS imports). + +Shared by: + scripts/validate_lidar_filter_clouds.py — runtime echo-based cloud check (robot container) + test/test_validation_core.py — pytest unit tests (``pytest -m unit``) +""" + +from __future__ import annotations + +import numpy as np + + +def near_range_tolerance(near_range_m: float) -> float: + """Slack around ``near_range_m`` (same rule as the validate script).""" + return max(0.05, float(near_range_m) * 0.05) + + +def ranges_xyz_from_points_xyz(points: np.ndarray) -> np.ndarray | None: + """Euclidean range per row for ``(N, 3)`` xyz. + + Returns ``None`` if shape is wrong or any coordinate is non-finite. + """ + arr = np.asarray(points, dtype=np.float64) + if arr.ndim != 2 or arr.shape[1] != 3: + return None + if arr.size == 0: + return np.array([], dtype=np.float64) + if not np.isfinite(arr).all(): + return None + return np.linalg.norm(arr, axis=1) + + +def validate_filtered_ranges( + ranges: np.ndarray, + near_range_m: float, + *, + long_range_min_m: float = 2.0, +) -> tuple[bool, str]: + """Check filtered cloud range statistics against ``near_range_m``. + + Returns ``(True, "")`` on success, else ``(False, reason)``. + """ + fr = np.asarray(ranges, dtype=np.float64) + if fr.size == 0: + return False, 'filtered cloud is empty' + mn_f = float(fr.min()) + tol = near_range_tolerance(near_range_m) + if mn_f < float(near_range_m) - tol: + return ( + False, + f'filtered min range {mn_f:.4f}m < near_range_m ({near_range_m}) - tol {tol:.4f}', + ) + if float(fr.max()) < long_range_min_m: + return ( + False, + f'expected long-range returns; filtered max range {float(fr.max()):.4f}m', + ) + return True, '' + + +def raw_filtered_near_range_ok( + mn_raw: float, + mn_filtered: float, + near_range_m: float, + tol: float | None = None, +) -> bool: + """If raw has near-field clutter, filtered minimum must still clear ``near_range_m``.""" + t = near_range_tolerance(near_range_m) if tol is None else tol + if mn_raw < float(near_range_m) - t and mn_filtered < float(near_range_m) - t: + return False + return True diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py index 7f50d45f6..f53460a56 100644 --- a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py @@ -1,6 +1,6 @@ #!/usr/bin/env python3 -# Copyright 2026 AirLab CMU -# SPDX-License-Identifier: Apache-2.0 +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. """One-shot ROS 2 check for liveliness: filtered LiDAR cloud vs raw (Isaac / Pegasus). Run inside the robot container with workspace sourced and ROS_DOMAIN_ID set:: @@ -11,7 +11,7 @@ * Filtered ``.../point_cloud``: all coordinates finite; **minimum range** must be at least ``near_range_m`` from the running filter node (minus tolerance). The tolerance is ``max(0.05 m, 5% of near_range_m)`` — the ``0.05`` is **slack - around the configured near range**, not a standalone “no points within 5 cm” + around the configured near range**, not a standalone "no points within 5 cm" rule. At least one return beyond 2 m so long-range points are not stripped. * Raw ``.../point_cloud_raw`` (optional): if present and contains near-field returns below ``near_range_m``, the filtered cloud must still respect the @@ -28,6 +28,7 @@ import subprocess import sys import time +from pathlib import Path import numpy as np import rclpy @@ -36,6 +37,19 @@ from sensor_msgs.msg import PointCloud2 from sensor_msgs_py import point_cloud2 +# validation_core lives in the package module (installed by colcon, or importable +# by adding the package root to sys.path for development use). +_pkg_root = Path(__file__).resolve().parent.parent +if str(_pkg_root) not in sys.path: + sys.path.insert(0, str(_pkg_root)) + +from lidar_point_cloud_filter.validation_core import ( # noqa: E402 + near_range_tolerance, + ranges_xyz_from_points_xyz, + raw_filtered_near_range_ok, + validate_filtered_ranges, +) + def _read_near_range_m(robot_num: int) -> float: """Query ``near_range_m`` from the running filter node; default 0.75.""" @@ -74,9 +88,7 @@ def _ranges_xyz(msg: PointCloud2) -> np.ndarray | None: if not pts: return np.array([], dtype=np.float64) arr = np.array([(float(p[0]), float(p[1]), float(p[2])) for p in pts], dtype=np.float64) - if not np.isfinite(arr).all(): - return None - return np.linalg.norm(arr, axis=1) + return ranges_xyz_from_points_xyz(arr) def _wait_for_cloud( @@ -120,7 +132,7 @@ def main() -> int: reliable = not args.qos_best_effort near_range_m = _read_near_range_m(n) - tol = max(0.05, near_range_m * 0.05) + tol = near_range_tolerance(near_range_m) node = None try: @@ -139,20 +151,11 @@ def main() -> int: print('ERROR: filtered cloud is empty', file=sys.stderr) return 1 - mn_f = float(fr.min()) - if mn_f < near_range_m - tol: - print( - f'ERROR: filtered min range {mn_f:.4f}m < near_range_m ({near_range_m}) - tol {tol:.4f}', - file=sys.stderr, - ) - return 1 - - if float(fr.max()) < 2.0: - print( - f'ERROR: expected long-range returns; filtered max range {float(fr.max()):.4f}m', - file=sys.stderr, - ) + ok_f, err_f = validate_filtered_ranges(fr, near_range_m) + if not ok_f: + print(f'ERROR: {err_f}', file=sys.stderr) return 1 + mn_f = float(fr.min()) rmsg = _wait_for_cloud(node, raw_topic, min(30.0, args.timeout), reliable) if rmsg is not None: @@ -164,7 +167,7 @@ def main() -> int: f'INFO: raw min range {mn_r:.4f}m (< near_range_m); ' f'filtered min {mn_f:.4f}m (must stay >= near_range_m)', ) - if mn_r < near_range_m - tol and mn_f < near_range_m - tol: + if not raw_filtered_near_range_ok(mn_r, mn_f, near_range_m, tol): print( 'ERROR: raw has near-field clutter but filtered min still below near_range_m', file=sys.stderr, diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg index ba67e395d..55f87f13c 100644 --- a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg @@ -2,3 +2,11 @@ script_dir=$base/lib/lidar_point_cloud_filter [install] install_scripts=$base/lib/lidar_point_cloud_filter + +[tool:pytest] +testpaths = test +python_files = test_*.py +python_classes = Test* +python_functions = test_* +markers = + unit: Hermetic unit tests (no ROS stack required) diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.py b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.py index 3935a7fe3..3c09016db 100644 --- a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.py +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.py @@ -18,7 +18,9 @@ maintainer_email='ajong@andrew.cmu.edu', description='Near-range sphere filter for lidar PointCloud2', license='Apache-2.0', - tests_require=['pytest'], + extras_require={ + 'test': ['pytest'], + }, entry_points={ 'console_scripts': [ 'lidar_point_cloud_filter_node = lidar_point_cloud_filter.lidar_point_cloud_filter_node:main', diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py new file mode 100644 index 000000000..04526c478 --- /dev/null +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py @@ -0,0 +1,75 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Unit tests for ``validation_core`` (numpy-only).""" + +import sys +from pathlib import Path + +import numpy as np +import pytest + +# validation_core lives in the package module directory (importable as a +# package member both here and in the validate_lidar_filter_clouds.py script). +_pkg_root = Path(__file__).resolve().parent.parent +if str(_pkg_root) not in sys.path: + sys.path.insert(0, str(_pkg_root)) + +from lidar_point_cloud_filter.validation_core import ( # noqa: E402 + near_range_tolerance, + ranges_xyz_from_points_xyz, + raw_filtered_near_range_ok, + validate_filtered_ranges, +) + + +@pytest.mark.unit +def test_near_range_tolerance(): + assert near_range_tolerance(0.5) == pytest.approx(0.05) + assert near_range_tolerance(2.0) == pytest.approx(0.1) + + +@pytest.mark.unit +def test_ranges_xyz_from_points_xyz(): + pts = np.array([[3.0, 4.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float64) + r = ranges_xyz_from_points_xyz(pts) + assert r is not None + assert r[0] == pytest.approx(5.0) + assert r[1] == pytest.approx(1.0) + + +@pytest.mark.unit +def test_ranges_xyz_rejects_nonfinite(): + pts = np.array([[1.0, np.nan, 0.0]], dtype=np.float64) + assert ranges_xyz_from_points_xyz(pts) is None + + +@pytest.mark.unit +def test_validate_filtered_ranges_ok(): + # All points between 1 m and 5 m from origin — clears default long_range_min_m=2 + fr = np.array([1.0, 2.0, 5.0], dtype=np.float64) + ok, msg = validate_filtered_ranges(fr, near_range_m=0.75) + assert ok + assert msg == '' + + +@pytest.mark.unit +def test_validate_filtered_ranges_too_close(): + fr = np.array([0.2, 5.0], dtype=np.float64) + ok, msg = validate_filtered_ranges(fr, near_range_m=0.75) + assert not ok + assert 'min range' in msg + + +@pytest.mark.unit +def test_validate_filtered_ranges_no_long_range(): + fr = np.array([1.0, 1.5], dtype=np.float64) + ok, msg = validate_filtered_ranges(fr, near_range_m=0.75) + assert not ok + assert 'long-range' in msg + + +@pytest.mark.unit +def test_raw_filtered_near_range_ok(): + tol = near_range_tolerance(0.75) + assert raw_filtered_near_range_ok(0.1, 0.8, 0.75, tol) + assert not raw_filtered_near_range_ok(0.1, 0.2, 0.75, tol) diff --git a/robot/ros_ws/src/sensors/sensor_interfaces/package.xml b/robot/ros_ws/src/sensors/sensor_interfaces/package.xml index c3df3349b..0fd76afe0 100644 --- a/robot/ros_ws/src/sensors/sensor_interfaces/package.xml +++ b/robot/ros_ws/src/sensors/sensor_interfaces/package.xml @@ -12,11 +12,12 @@ rosidl_default_runtime - rosidl_interface_packages + sensor_msgs + ament_lint_auto ament_lint_common - sensor_msgs + rosidl_interface_packages ament_cmake diff --git a/tests/README.md b/tests/README.md index e4362dd4c..f10942ff7 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,6 +1,12 @@ -# System Testing +# Testing (`tests/`) -AirStack's system tests bring up the full Docker-based stack — simulator, robot containers, and GCS — and verify end-to-end behavior: container health, ROS 2 node presence, sensor publishing rates (in the `sensors` mark), and compute resource usage. Tests are written in Python with pytest and live under `tests/` at the repo root. +AirStack's **pytest** tree under `tests/` has three roles: + +1. **`tests/system/`** — Docker stack tests (sim + robot + GCS): liveliness, sensor Hz, takeoff/hover/land, image/workspace builds. +2. **`tests/robot/`** — Fast **unit** tests that mirror `robot/ros_ws/src/` (`behavior`, `global`, `interface`, `local`, `perception`, `sensors`). Mark: `unit`. +3. **`tests/sim/`** — Unit tests for simulation-side helpers (e.g. Motive / NatNet emulator). Mark: `unit`. + +Shared fixtures live in `tests/conftest.py`. Use `airstack test -m unit -v` for hermetic tests only, or the marks below for the full stack. @@ -8,16 +14,36 @@ AirStack's system tests bring up the full Docker-based stack — simulator, robo ## Test Suite Structure +### System tests (`tests/system/`) + | Module | Mark | What it tests | Hardware required | |--------|------|---------------|-------------------| -| [`test_build_docker.py`](../../../../tests/test_build_docker.py) | `build_docker` | Docker image builds (robot-desktop, gcs, isaac-sim, ms-airsim); records image sizes | Docker daemon | -| [`test_build_packages.py`](../../../../tests/test_build_packages.py) | `build_packages` | `colcon build` inside each container (robot, GCS, ms-airsim ROS workspace) | Docker daemon | -| [`test_liveliness.py`](../../../../tests/test_liveliness.py) | `liveliness` | Stack bring-up: container Running state, ``/clock`` readiness, tmux panes, sentinel ROS 2 nodes, compute snapshot, infra-only ``test_stable`` (tmux + nodes + compute) | Docker daemon, GPU, sim license | -| [`test_sensors.py`](../../../../tests/test_sensors.py) | `sensors` | After liveliness in collection order: sim + robot stereo/depth Hz (**Isaac:** batched ``ros2 topic hz`` to avoid bridge overload; **ms-airsim:** single batch), filtered LiDAR via ``echo --once`` + cloud sanity (isaacsim), sim RTF, ``test_sensor_streams_stable`` | Docker daemon, GPU, sim license | -| [`test_takeoff_hover_land.py`](../../../../tests/test_takeoff_hover_land.py) | `takeoff_hover_land` | End-to-end flight: PX4 readiness gate, takeoff to 10 m, hover stability, land — one chain per (sim, num_robots, iteration, velocity) | Docker daemon, GPU, sim license | +| [`system/test_build_docker.py`](system/test_build_docker.py) | `build_docker` | Docker image builds (robot-desktop, gcs, isaac-sim, ms-airsim); records image sizes | Docker daemon | +| [`system/test_build_packages.py`](system/test_build_packages.py) | `build_packages` | `colcon build` inside each container (robot, GCS, ms-airsim ROS workspace) | Docker daemon | +| [`system/test_liveliness.py`](system/test_liveliness.py) | `liveliness` | Stack bring-up: container Running state, ``/clock`` readiness, tmux panes, sentinel ROS 2 nodes, compute snapshot, infra-only ``test_stable`` (tmux + nodes + compute) | Docker daemon, GPU, sim license | +| [`system/test_sensors.py`](system/test_sensors.py) | `sensors` | After liveliness in collection order: sim + robot stereo/depth Hz (**Isaac:** batched ``ros2 topic hz`` to avoid bridge overload; **ms-airsim:** single batch), filtered LiDAR via ``echo --once`` + cloud sanity (isaacsim), sim RTF, ``test_sensor_streams_stable`` | Docker daemon, GPU, sim license | +| [`system/test_takeoff_hover_land.py`](system/test_takeoff_hover_land.py) | `takeoff_hover_land` | End-to-end flight: PX4 readiness gate, takeoff to 10 m, hover stability, land — one chain per (sim, num_robots, iteration, velocity) | Docker daemon, GPU, sim license | + +### Unit tests (`tests/robot/`, `tests/sim/`) + +Hermetic tests use `@pytest.mark.unit` (see [`pytest.ini`](pytest.ini)). + +**Co-location + proxy pattern:** test source lives alongside its ROS 2 package at +`robot/ros_ws/src///test/test_*.py` (the ROS 2 / colcon convention). +Files in `tests/robot/` are thin proxies that re-export those tests so that +`pytest tests/` discovers them. Both `airstack test -m unit` and +`colcon test --packages-select ` run the same test source. + +Example: `robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py` +tests the numpy-only range validation rules in +`robot/ros_ws/src/sensors/lidar_point_cloud_filter/lidar_point_cloud_filter/validation_core.py` +(also used by `scripts/validate_lidar_filter_clouds.py` inside the robot container). + +See [Unit Testing Guide](../docs/development/intermediate/testing/unit_testing.md) +and the `add-unit-tests` agent skill for full details. Marks can be combined with pytest logic: -`-m "build_docker or build_packages"`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). +`-m unit`, `-m "build_docker or build_packages"`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). ### Bring-up scope (`airstack_env`) @@ -27,7 +53,7 @@ Marks can be combined with pytest logic: ## Test Infrastructure -All shared fixtures, helpers, and configuration live in [`tests/conftest.py`](../../../../tests/conftest.py). +All shared fixtures, helpers, and configuration live in [`conftest.py`](conftest.py). ### `airstack_env` fixture @@ -40,21 +66,21 @@ Parametrized over `(sim, num_robots, iteration)` tuples derived from CLI flags. ### Isaac Sim and the `sensors` mark -**LiDAR in pytest:** [`tests/conftest.py`](../../../../tests/conftest.py) sets +**LiDAR in pytest:** [`conftest.py`](conftest.py) sets `ENABLE_LIDAR=true` in `SIM_CONFIG["isaacsim"]["extra_env"]` so the multi-drone Pegasus script (`example_multi_px4_pegasus_launch_script.py`) attaches RTX LiDAR the same way the single-drone script always does. Without that flag the multi script would not spawn LiDAR OmniGraphs. -**Topic checks** live in [`tests/sensor_probes.py`](../../../../tests/sensor_probes.py) -and are driven by [`tests/test_sensors.py`](../../../../tests/test_sensors.py): +**Topic checks** live in [`sensor_probes.py`](sensor_probes.py) +and are driven by [`system/test_sensors.py`](system/test_sensors.py): | Path | What we measure | How | |------|-----------------|-----| | Sim → `/clock`, stereo images, stereo depth | Publish rate | ``ros2 topic hz`` on the sim container: ``/clock`` alone, then **chunks of two** ``image_rect`` topics, then **chunks of two** depth topics (``ISAACSIM_HZ_CHUNK_SIZE`` in ``sensor_probes.py``). | | Robot → same topic names (bridge) | Publish rate | Same **two-at-a-time** chunking on the robot container for Isaac. ms-airsim: one batch of four topics. | | Robot → filtered ``.../ouster/point_cloud`` | Stream alive | ``ros2 topic echo --once`` per robot (not Hz — large ``PointCloud2``). | -| LiDAR geometry | Near-range vs ``near_range_m`` | ``lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py`` (raw vs filtered). | +| LiDAR geometry | Near-range vs ``near_range_m`` | ``robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/validate_lidar_filter_clouds.py`` (raw vs filtered). | Sim **RTF** (real-time factor from ``/clock``) is also in the `sensors` suite. **`test_sensor_streams_stable`** repeats sim + robot stereo + LiDAR probes every @@ -88,11 +114,11 @@ tests/results/ ├── results.xml # JUnit XML — test durations and pass/fail status ├── metrics.json # Custom metrics (image sizes, Hz, compute, timing) └── logs/ - ├── test_build_docker.TestDockerBuilds.test_build_robot_desktop.log - ├── airstack_env.test_liveliness.TestLiveliness.test_robot_containers_running[msairsim-rob#1-iter0].log - ├── test_liveliness.TestLiveliness.test_robot_containers_running[msairsim-rob#1-iter0].log - ├── test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0].log - ├── test_sensors.TestSensors.test_sensor_streams_stable[msairsim-rob#1-iter0].log + ├── system.test_build_docker.TestDockerBuilds.test_build_robot_desktop.log + ├── airstack_env.system.test_liveliness.TestLiveliness.test_robot_containers_running[msairsim-rob#1-iter0].log + ├── system.test_liveliness.TestLiveliness.test_robot_containers_running[msairsim-rob#1-iter0].log + ├── system.test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0].log + ├── system.test_sensors.TestSensors.test_sensor_streams_stable[msairsim-rob#1-iter0].log └── ... # More per-test logs; another airstack_env.* per class using the fixture ``` @@ -109,6 +135,9 @@ arguments directly to pytest. No local Python environment needed. ```bash # From the repo root (AirStack must be set up: airstack setup): +# Unit tests only — no GPU, no full Docker stack (numpy-only + pure Python) +airstack test -m unit -v + # Build tests only — fast, no GPU needed airstack test -m "build_docker or build_packages" -v @@ -191,7 +220,7 @@ pytest tests/ -m sensors \ --- -## Autonomy Tests (`test_takeoff_hover_land.py`) +## Autonomy Tests (`system/test_takeoff_hover_land.py`) `TestTakeoffHoverLand` runs a **4-phase flight chain** for every combination of `(sim, num_robots, iteration, velocity)`. The drone returns to the ground after @@ -255,7 +284,7 @@ airstack test -m takeoff_hover_land \ ## Metrics Reporting (`parse_metrics.py`) -[`tests/parse_metrics.py`](../../../../tests/parse_metrics.py) reads `results.xml` and `metrics.json` from a run directory and produces a markdown report. It has two modes: +[`parse_metrics.py`](parse_metrics.py) reads `results.xml` and `metrics.json` from a run directory and produces a markdown report. It has two modes: ### Single-run report diff --git a/tests/colcon_unit_test_packages.yaml b/tests/colcon_unit_test_packages.yaml new file mode 100644 index 000000000..5e0bd0ebb --- /dev/null +++ b/tests/colcon_unit_test_packages.yaml @@ -0,0 +1,13 @@ +# Packages run via `colcon test` in system.test_build_packages.test_colcon_test_robot. +# +# Add a package here when it has gtests (ament_add_gtest) and/or pytest tests under +# /test/. Keep in sync with tests/robot/ proxies for Python unit tests. +# +# See: docs/development/intermediate/testing/unit_testing.md + +robot: + packages: + - natnet_ros2 + - lidar_point_cloud_filter + # Skips ament_copyright / flake8 / pep257 on Python packages; run linters separately. + pytest_args: "-m not linter" diff --git a/tests/conftest.py b/tests/conftest.py index ef8b88ec6..31fd29076 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -4,6 +4,7 @@ import re import shlex import subprocess +import sys import threading import time from concurrent.futures import ThreadPoolExecutor, as_completed @@ -12,6 +13,7 @@ from pathlib import Path import pytest +import yaml SIM_CONFIG = { "msairsim": { @@ -44,6 +46,47 @@ } AIRSTACK_ROOT = os.environ.get("AIRSTACK_ROOT", str(Path(__file__).parent.parent)) +COLCON_UNIT_TEST_PACKAGES_YAML = ( + Path(AIRSTACK_ROOT) / "tests" / "colcon_unit_test_packages.yaml" +) + + +def load_colcon_unit_test_config(workspace="robot"): + """Load colcon test package list and pytest args from tests/colcon_unit_test_packages.yaml.""" + if not COLCON_UNIT_TEST_PACKAGES_YAML.is_file(): + raise FileNotFoundError( + f"Missing {COLCON_UNIT_TEST_PACKAGES_YAML} — add packages to gate in colcon test." + ) + with COLCON_UNIT_TEST_PACKAGES_YAML.open(encoding="utf-8") as f: + data = yaml.safe_load(f) or {} + if workspace not in data: + raise KeyError( + f"No '{workspace}' entry in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" + ) + cfg = data[workspace] or {} + packages = cfg.get("packages") or [] + if not packages: + raise ValueError( + f"'{workspace}.packages' is empty in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" + ) + return packages, cfg.get("pytest_args", "") + + +def colcon_test_robot_command(workspace="robot"): + """Shell command for colcon test over unit-test packages (robot workspace).""" + packages, pytest_args = load_colcon_unit_test_config(workspace) + pkg_list = " ".join(packages) + cmd = ( + f"colcon test --packages-select {pkg_list} " + "--event-handlers console_direct+ --return-code-on-test-failure" + ) + if pytest_args: + cmd += f' --pytest-args "{pytest_args}"' + return cmd +# Unit tests live co-located with their ROS 2 packages in robot/ros_ws/src/. +# Thin proxy files under tests/robot/ re-export those tests so that +# `pytest tests/` and `airstack test -m unit` discover them without any +# sys.path manipulation here. Each proxy file sets up its own paths. RUN_DIR = None LOGS_DIR = None ROS_DISTRO_SETUP = "/opt/ros/jazzy/setup.bash" @@ -156,11 +199,16 @@ def pytest_generate_tests(metafunc): # docker image builds → colcon workspace builds → liveliness (infra) → sensors # (ROS topic streams) → autonomy flight tests. _MODULE_ORDER = [ - "test_build_docker", - "test_build_packages", - "test_liveliness", - "test_sensors", - "test_takeoff_hover_land", + # Unit tests first — fast, hermetic, no Docker. Any module whose dotted + # name starts with "robot." or "sim." is a proxy for a package-level unit + # test and sorts into this leading slot via the prefix check below. + "__unit__", + # System tests follow in dependency order. + "system.test_build_docker", + "system.test_build_packages", + "system.test_liveliness", + "system.test_sensors", + "system.test_takeoff_hover_land", ] # Within test_takeoff_hover_land, each (env, velocity) runs phases in this chain order. @@ -177,16 +225,28 @@ def _rank(name, order): return order.index(name) if name in order else len(order) +def _module_key(item): + """Return the ordering key for an item. + + Unit-test proxies live under ``robot/``, ``sim/``, or ``gcs/`` and are + identified by their nodeid prefix. Everything else uses the dotted module + ``__name__`` looked up against ``_MODULE_ORDER``. + """ + if item.nodeid.startswith(("robot/", "sim/", "gcs/")): + return _rank("__unit__", _MODULE_ORDER) + return _rank(getattr(item.module, "__name__", ""), _MODULE_ORDER) + + def pytest_collection_modifyitems(items): # 1. Cross-module: enforce `_MODULE_ORDER`. Stable sort keeps within-module # order intact, so pytest's default file/class order survives. - items.sort(key=lambda it: _rank(getattr(it.module, "__name__", ""), _MODULE_ORDER)) + items.sort(key=_module_key) # 2. Within test_takeoff_hover_land: sort by (airstack_env, velocity, phase) so each # (sim, robots, iter) env brings up the stack once and the drone goes # ground→air→ground per velocity. def phase(item): - if getattr(item.module, "__name__", "") != "test_takeoff_hover_land": + if getattr(item.module, "__name__", "") != "system.test_takeoff_hover_land": return None name = item.originalname or item.name.split("[", 1)[0] return _rank(name, _AUTONOMY_PHASE_ORDER) diff --git a/tests/parse_metrics.py b/tests/parse_metrics.py index 6900e7305..f2267e627 100644 --- a/tests/parse_metrics.py +++ b/tests/parse_metrics.py @@ -51,9 +51,15 @@ def _split_test_name(name): - """`test_liveliness.TestLiveliness.test_foo[id]` → - (module="test_liveliness", display="test_foo[id]"). Drops the Class segment - for display since there's one class per module (same for ``test_sensors``).""" + """Dotted pytest node id → (module_prefix, display_without_class). + + Supports legacy ``test_liveliness.TestLiveliness.test_foo[id]`` and nested + ``system.test_liveliness.TestLiveliness.test_foo[id]``. + """ + m = re.match(r"^(.*)\.(Test\w+)\.(test\w+)(\[.*\])?$", name) + if m: + mod, _cls, test, bracket = m.groups() + return mod, f"{test}{bracket or ''}" parts = name.split(".", 2) if len(parts) == 3: return parts[0], parts[2] diff --git a/tests/pytest.ini b/tests/pytest.ini index 78a916a0c..a664ccbf3 100644 --- a/tests/pytest.ini +++ b/tests/pytest.ini @@ -1,5 +1,6 @@ [pytest] markers = + unit: Fast hermetic tests (no Docker stack; numpy / pure Python) build_docker: Docker image build tests build_packages: Colcon workspace build tests liveliness: Container and process health (Docker, tmux, sentinel ROS 2 nodes) diff --git a/tests/requirements.txt b/tests/requirements.txt index bc2a16d4d..a4b43e6bb 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -1,6 +1,8 @@ pytest pytest-timeout pytest-dependency +pyyaml tabulate psutil pandas +numpy diff --git a/tests/robot/README.md b/tests/robot/README.md new file mode 100644 index 000000000..a4409c4b1 --- /dev/null +++ b/tests/robot/README.md @@ -0,0 +1,37 @@ +# Robot-side unit test proxies + +Layout mirrors [`robot/ros_ws/src/`](../../robot/ros_ws/src/) autonomy layers: + +| Directory | Maps to ROS workspace | +|-----------|----------------------| +| `behavior/` | `robot/ros_ws/src/behavior/` | +| `global/` | `robot/ros_ws/src/global/` | +| `interface/` | `robot/ros_ws/src/interface/` | +| `local/` | `robot/ros_ws/src/local/` | +| `perception/` | `robot/ros_ws/src/perception/` | +| `sensors/` | `robot/ros_ws/src/sensors/` | + +## Design: co-location + proxy + +**Test source** lives co-located with each ROS 2 package (the standard colcon +convention): + +``` +robot/ros_ws/src///test/test_.py ← source of truth +``` + +**This directory** contains thin proxy files that load the real test module via +`importlib` and re-export its `test_*` functions, making them discoverable by +`pytest tests/` and `airstack test -m unit` without any changes to the CI +workflow. Each proxy is ~15 lines. + +``` +tests/robot///test_.py ← proxy (re-exports above) +``` + +Both `airstack test -m unit` (pytest path via proxy) and +`colcon test --packages-select ` (direct path to source) run the same +test functions from the same file. + +All test functions must carry `@pytest.mark.unit`. For adding new tests see the +`add-unit-tests` agent skill. diff --git a/tests/robot/behavior/README.md b/tests/robot/behavior/README.md new file mode 100644 index 000000000..713fd31f2 --- /dev/null +++ b/tests/robot/behavior/README.md @@ -0,0 +1,3 @@ +# Unit tests — behavior layer + +Add `@pytest.mark.unit` tests for `robot/ros_ws/src/behavior/` packages here. diff --git a/tests/robot/global/README.md b/tests/robot/global/README.md new file mode 100644 index 000000000..280c41dec --- /dev/null +++ b/tests/robot/global/README.md @@ -0,0 +1,3 @@ +# Unit tests — global layer + +Add `@pytest.mark.unit` tests for `robot/ros_ws/src/global/` packages here. diff --git a/tests/robot/interface/README.md b/tests/robot/interface/README.md new file mode 100644 index 000000000..ea4ee8b5c --- /dev/null +++ b/tests/robot/interface/README.md @@ -0,0 +1,3 @@ +# Unit tests — interface layer + +Add `@pytest.mark.unit` tests for `robot/ros_ws/src/interface/` packages here. diff --git a/tests/robot/local/README.md b/tests/robot/local/README.md new file mode 100644 index 000000000..118cc2071 --- /dev/null +++ b/tests/robot/local/README.md @@ -0,0 +1,3 @@ +# Unit tests — local layer + +Add `@pytest.mark.unit` tests for `robot/ros_ws/src/local/` packages here. diff --git a/tests/robot/perception/README.md b/tests/robot/perception/README.md new file mode 100644 index 000000000..350ef9fb0 --- /dev/null +++ b/tests/robot/perception/README.md @@ -0,0 +1,3 @@ +# Unit tests — perception layer + +Add `@pytest.mark.unit` tests for `robot/ros_ws/src/perception/` packages here. diff --git a/tests/robot/perception/natnet_ros2/test_natnet_ros2.py b/tests/robot/perception/natnet_ros2/test_natnet_ros2.py new file mode 100644 index 000000000..fc9a78bde --- /dev/null +++ b/tests/robot/perception/natnet_ros2/test_natnet_ros2.py @@ -0,0 +1,32 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Proxy: re-exposes natnet_ros2 unit tests from the package source tree. + +Unit test logic lives co-located with its package (ROS 2 / colcon convention): + robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py + +This file makes those tests discoverable by ``pytest tests/`` (CI) and +``airstack test -m unit`` without any changes to the CI workflow. +Run ``colcon test --packages-select natnet_ros2`` to also execute the C++ +gtests and ament linters. +""" + +import importlib.util +import sys +from pathlib import Path + +_repo_root = Path(__file__).resolve().parents[4] +_pkg_test = _repo_root / "robot/ros_ws/src/perception/natnet_ros2/test" +_real_file = _pkg_test / "test_natnet_ros2.py" + +# Load the real module under a unique name to avoid the circular-import that +# would occur if we used `from test_natnet_ros2 import *` (this file has the +# same name and pytest adds its directory to sys.path at collection time). +_spec = importlib.util.spec_from_file_location("_natnet_ros2_unit_tests", _real_file) +_real = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_real) + +# Re-export every test_* symbol so pytest collects them from this proxy. +for _name in dir(_real): + if _name.startswith("test_"): + globals()[_name] = getattr(_real, _name) diff --git a/tests/robot/sensors/README.md b/tests/robot/sensors/README.md new file mode 100644 index 000000000..8a44129eb --- /dev/null +++ b/tests/robot/sensors/README.md @@ -0,0 +1,4 @@ +# Unit tests — sensors layer + +Package-specific folders (for example `lidar_point_cloud_filter/`) mirror +`robot/ros_ws/src/sensors//`. diff --git a/tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py b/tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py new file mode 100644 index 000000000..e7babf9d4 --- /dev/null +++ b/tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py @@ -0,0 +1,38 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Proxy: re-exposes validation_core unit tests from the package source tree. + +Unit test logic lives co-located with its package (ROS 2 / colcon convention): + robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py + +This file makes those tests discoverable by ``pytest tests/`` (CI) and +``airstack test -m unit`` without any changes to the CI workflow. +Run ``colcon test --packages-select lidar_point_cloud_filter`` to also execute +the ament linters. +""" + +import importlib.util +import sys +from pathlib import Path + +_repo_root = Path(__file__).resolve().parents[4] +_pkg_test = _repo_root / "robot/ros_ws/src/sensors/lidar_point_cloud_filter/test" +_pkg_root = _pkg_test.parent # adds lidar_point_cloud_filter/ package to sys.path +_real_file = _pkg_test / "test_validation_core.py" + +# Make the package module importable so the real test can do +# `from lidar_point_cloud_filter.validation_core import ...` +if str(_pkg_root) not in sys.path: + sys.path.insert(0, str(_pkg_root)) + +# Load the real module under a unique name to avoid the circular-import that +# would occur if we used `from test_validation_core import *` (this file has +# the same name and pytest adds its directory to sys.path at collection time). +_spec = importlib.util.spec_from_file_location("_lidar_validation_unit_tests", _real_file) +_real = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(_real) + +# Re-export every test_* symbol so pytest collects them from this proxy. +for _name in dir(_real): + if _name.startswith("test_"): + globals()[_name] = getattr(_real, _name) diff --git a/tests/sensor_probes.py b/tests/sensor_probes.py index cbb634261..140e10c0a 100644 --- a/tests/sensor_probes.py +++ b/tests/sensor_probes.py @@ -2,7 +2,7 @@ # SPDX-License-Identifier: Apache-2.0 """ROS 2 sensor stream checks (sim + robot) for system tests. -Used by ``test_sensors.py``. Liveliness (``test_liveliness.py``) stays limited to +Used by ``system/test_sensors.py``. Liveliness (``system/test_liveliness.py``) stays limited to containers, tmux, and sentinel nodes; sensor Hz / LiDAR validation lives here. **Isaac Sim (`env["sim"] == "isaacsim"`)** — Pegasus / OmniGraph ROS bridges are @@ -50,8 +50,8 @@ ] LIDAR_CLOUD_VALIDATE_SCRIPT = ( - "/root/AirStack/robot/ros_ws/src/sensors/lidar_point_cloud_filter/scripts/" - "validate_lidar_filter_clouds.py" + "/root/AirStack/robot/ros_ws/src/sensors/lidar_point_cloud_filter/" + "scripts/validate_lidar_filter_clouds.py" ) # Shorter Hz sample during sensor stability polling. diff --git a/tests/sim/README.md b/tests/sim/README.md new file mode 100644 index 000000000..09f45f6a6 --- /dev/null +++ b/tests/sim/README.md @@ -0,0 +1,14 @@ +# Simulation-side unit tests + +Tests for **simulation components** that are not part of the onboard ROS workspace +(for example an OptiTrack Motive / NatNet emulator, Isaac launch helpers, or +AirSim bridge utilities). + +Mark fast, hermetic checks with `@pytest.mark.unit`. Tests that require a GPU, +full sim, or Docker belong in [`tests/system/`](../system/) instead. + +Suggested layout: + +| Directory | Purpose | +|-----------|---------| +| `motive_emulator/` | Motive / NatNet protocol emulation / parsing | diff --git a/tests/sim/motive_emulator/README.md b/tests/sim/motive_emulator/README.md new file mode 100644 index 000000000..0e682c448 --- /dev/null +++ b/tests/sim/motive_emulator/README.md @@ -0,0 +1,61 @@ +# Motive / NatNet Emulator + +This directory is the future home of **integration tests** that drive a real +NatNet wire-protocol mock server against `natnet_ros2_node`. + +## Why here, not in the package test/ dir? + +Unit tests for pure logic live in +`tests/robot/perception/natnet_ros2/test_natnet_logic.cpp` and run via `colcon +test` with no network or SDK required (uses `FakeNatNetClient`). + +The emulator tests here will require an actual UDP server that speaks the NatNet +protocol, so they belong in the `sensors` mark of the system test suite alongside +other topic-streaming tests. + +## Planned implementation + +The mock server should: + +1. Open a UDP socket on the NatNet command port (default 1510). +2. Respond to `NAT_CONNECT` (message type 0) with a `NAT_SERVERINFO` (type 1) + packet containing a canned `sServerDescription`. +3. Respond to `NAT_REQUEST_MODELDEF` (type 4) with a `NAT_MODELDEF` (type 5) + packet describing one or more rigid bodies. +4. Stream `NAT_FRAMEOFDATA` (type 7) packets to the client's data port at a + configurable rate with synthetic pose data. + +### Reference + +The NatNet wire format is documented in the NatNet SDK developer notes and the +`PacketClient` example shipped with the SDK (available inside the robot Docker +container after `airstack setup --natnet`). + +## Relationship to `FakeNatNetClient` + +``` + ┌──────────────────────────────────────┐ + │ Test boundary │ + colcon gtest │ FakeNatNetClient (in-process) │ ← unit tests (no network) + │ test_natnet_logic.cpp │ + └──────────────────────────────────────┘ + + ┌──────────────────────────────────────┐ + │ Network boundary │ + pytest sensors │ MotiveEmulator (UDP server, Python) │ ← integration tests + │ NatNetClientAdapter → NatNetClient │ + │ natnet_ros2_node (full ROS node) │ + └──────────────────────────────────────┘ +``` + +The `FakeNatNetClient` seam (already implemented) lets unit tests verify all +connection-outcome logic paths. The emulator here will verify the full +end-to-end path including the NatNet SDK's own parser. + +## When to add this + +Implement the emulator when: +- The OptiTrack emulator service is placed under `simulation/optitrack-emulator/` + or `tests/sim/motive_emulator/` +- The `sensors` test mark is extended to include `natnet_ros2` topic checks +- CI has access to the robot container with the NatNet SDK installed diff --git a/tests/system/__init__.py b/tests/system/__init__.py new file mode 100644 index 000000000..0169df793 --- /dev/null +++ b/tests/system/__init__.py @@ -0,0 +1 @@ +"""Docker-based system and integration tests (stack bring-up, sensors, flight).""" diff --git a/tests/test_build_docker.py b/tests/system/test_build_docker.py similarity index 100% rename from tests/test_build_docker.py rename to tests/system/test_build_docker.py diff --git a/tests/test_build_packages.py b/tests/system/test_build_packages.py similarity index 64% rename from tests/test_build_packages.py rename to tests/system/test_build_packages.py index 40bcf978b..5c54cfee3 100644 --- a/tests/test_build_packages.py +++ b/tests/system/test_build_packages.py @@ -2,7 +2,8 @@ import pytest -from conftest import (AIRSTACK_ROOT, airstack_cmd, docker_exec, logger, +from conftest import (AIRSTACK_ROOT, airstack_cmd, colcon_test_robot_command, + docker_exec, load_colcon_unit_test_config, logger, read_log_tail, wait_for_container) @@ -41,6 +42,44 @@ def test_colcon_build_robot(self): finally: airstack_cmd("down") + def test_colcon_test_robot(self): + """Build with BUILD_TESTING=ON then run colcon test inside the robot container. + + Kept separate from test_colcon_build_robot so that a test failure does + not block the build-only health check, and so the test step can be + re-run independently without a full rebuild. + + Package list and pytest args come from tests/colcon_unit_test_packages.yaml. + Workspace-wide ament linter tests are not gated here. + """ + packages, _ = load_colcon_unit_test_config("robot") + try: + result = airstack_cmd("up", "robot-desktop", + env_overrides={"AUTOLAUNCH": "false", "DISPLAY": ""}, + timeout=120) + assert result.returncode == 0, f"airstack up failed:\n{read_log_tail()}" + + container = wait_for_container("robot.*desktop", timeout=60) + assert container, "Robot container not found" + + build = docker_exec( + container, + "bash -ic \"bws --cmake-args '-DBUILD_TESTING=ON'\"", + timeout=600, + ) + assert build.returncode == 0, f"colcon build (with testing) failed:\n{read_log_tail()}" + + test = docker_exec( + container, + f"bash -ic '{colcon_test_robot_command('robot')}'", + timeout=300, + ) + assert test.returncode == 0, ( + f"colcon test failed (packages: {', '.join(packages)}):\n{read_log_tail()}" + ) + finally: + airstack_cmd("down") + def test_colcon_build_gcs(self): _warn_if_prebuilt("gcs/ros_ws") try: diff --git a/tests/test_liveliness.py b/tests/system/test_liveliness.py similarity index 99% rename from tests/test_liveliness.py rename to tests/system/test_liveliness.py index cd8b027e0..342e5f49a 100644 --- a/tests/test_liveliness.py +++ b/tests/system/test_liveliness.py @@ -5,7 +5,7 @@ snapshots, and a short stability window (infra only — no camera/LiDAR Hz here). Sensor topic rates, bridge stereo Hz, LiDAR echo/sanity, and sim RTF live in -``test_sensors.py`` (``@pytest.mark.sensors``), ordered after this module. +``system/test_sensors.py`` (``@pytest.mark.sensors``), ordered after this module. """ import time diff --git a/tests/test_sensors.py b/tests/system/test_sensors.py similarity index 97% rename from tests/test_sensors.py rename to tests/system/test_sensors.py index f00facb33..8170286d6 100644 --- a/tests/test_sensors.py +++ b/tests/system/test_sensors.py @@ -1,4 +1,4 @@ -"""Sensor stream and LiDAR validation — runs after ``test_liveliness`` (see ``_MODULE_ORDER``). +"""Sensor stream and LiDAR validation — runs after ``system.test_liveliness`` (see ``_MODULE_ORDER``). Uses the same ``airstack_env`` parametrization as liveliness. With ``class``-scoped fixtures this module performs its **own** stack bring-up when selected; combined @@ -20,7 +20,7 @@ check_robot_stereo_hz, check_sim_publishing, ) -from test_liveliness import _check_sentinel_nodes, _poll_until +from system.test_liveliness import _check_sentinel_nodes, _poll_until @pytest.mark.sensors diff --git a/tests/test_takeoff_hover_land.py b/tests/system/test_takeoff_hover_land.py similarity index 100% rename from tests/test_takeoff_hover_land.py rename to tests/system/test_takeoff_hover_land.py From 6279bea8bea7d3c5e4ea26fb9dde70eda58ff4a5 Mon Sep 17 00:00:00 2001 From: Krrish Jain Date: Mon, 6 Jul 2026 21:37:48 -0700 Subject: [PATCH 05/21] Fix/camera init (#368) * re-ordered initialization of stereo render product node to only initialize after right camera is initialized, ensuring camera is initialized as stereo (left camera is assumed, but right is optional) --------- Co-authored-by: John --- .env | 2 +- .../perception/perception_bringup/launch/perception.launch.xml | 2 ++ simulation/isaac-sim/extensions/PegasusSimulator | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/.env b/.env index 82cc01ccb..0d994bc51 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.3" +VERSION="0.19.0-alpha.4" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml b/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml index e433e94d4..a79ff85a1 100644 --- a/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml +++ b/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml @@ -50,6 +50,7 @@ to="/$(env ROBOT_NAME)/perception/stereo_image_proc/disparity" /> + + diff --git a/simulation/isaac-sim/extensions/PegasusSimulator b/simulation/isaac-sim/extensions/PegasusSimulator index fe8b5a101..8c7a66409 160000 --- a/simulation/isaac-sim/extensions/PegasusSimulator +++ b/simulation/isaac-sim/extensions/PegasusSimulator @@ -1 +1 @@ -Subproject commit fe8b5a101857f2cda290b9b677b3a95c4cca6b09 +Subproject commit 8c7a664095396bc9c4f4609531445d4cf6390fe4 From fa990f4dc03e8bee48d76aa73b1d6285b8bd16f9 Mon Sep 17 00:00:00 2001 From: pvkumara <99618405+pvkumara@users.noreply.github.com> Date: Fri, 10 Jul 2026 16:19:09 -0400 Subject: [PATCH 06/21] Add fixed-trajectory system tests with cross-track error metrics (#365) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add fixed-trajectory evaluation tests New tests/test_fixed_trajectory.py evaluates drone performance on Circle, Figure8, Racetrack, and Line trajectories: takeoff -> execute -> land with cross-track error, path RMSE, execution time, and success metrics recorded to metrics.json for baseline comparison. - Python ideal-path generators mirror fixed_trajectory_task.cpp equations - Cross-track error uses robot pose snapshot at dispatch to transform base_link ideal path to world frame for odom comparison - 5m loose tolerance documents the known circle failure without stranding drone - conftest.py gains --trajectory-types CLI option and generalised phase-order sorting/ID-rewriting for both autonomy test modules - tests/README.md documents the new module, all 11 metrics, and run commands Made-with: Cursor * Remove module docstring from test_fixed_trajectory.py Made-with: Cursor * Aj/GitHub ci cd (#347) * Add link to PAT * Change to new orchestrator instance workflow * Add availability zone * Bump version to 0.18.0-alpha.7 * Add fix for boot volume size blocking orchestrator * Add floating IPs to CI/CD * Bump gh runner_version to latest * Update cicd defaults * Rename integration-tests.yml to system-tests.yml * Add debugging tips and add to mkdocs * Use venv instead of pip3 to fix error: externally-managed-environment * Explicitly fail autonomy test if images not yet built * Enable using docker cache from docker registry to speed up docker image build tests for ci/cd * Fix bug * Update docs and change docker image build/push to also run on self-hosted runner * Enable trigger docker build workflow on via manual dispatch * Increase instance volume size so that space doesn't run out when building docker images * Update to always try build all images * Create dummy file for docker compose push to pass * Add omni_pass.env with guest access to AirLab nucleus * Update ci/cd tests to make sure image is present before running tests * Make sure images for profiles get built * Update system tests to not build images if pull available * Make build/pull quiet * Pin empy version to fix ROS2 jazzy version bug * Switch image to desktop so that tests run successfully * Add docker image signing to workflow * Change pytest mark 'autonomy' to 'takeoff_hover_land' * update comments on workflow * Recurisve checkout of airstack * Log more to GitHub * Better error logging for ci/cd orchestrator * Add check system resources before spawning server; if resources not available, report back and try again later * Make it so that pytest no longer triggers from pushes on PR; make it so we can manually trigger pytest by commenting /pytest * Update PR template * Update AGENTS.md * Fix finding baseline metrics * Update workflow to comment instead of react * Fix bug * Try fix another bug * Update omni_pass_TEMPLATE.env to use 'guest'; update default on system tests to include build_packages * Auto prepend 'build_packages' mark to ensure code is built before tests * Lower default stress-iterations to 1 and single takeoff-velocity to 0.5 * Johnliu/px4 cpu optimization (#348) * added option for physics step frequency * reverted example launch script * patches PX4 simulation startup script and fixes robot DDS version * set default physics Hz for PX4 to be 100Hz which is the minimum. * reverted simulation changes * updated docs * Better error logging for ci/cd orchestrator * Add check system resources before spawning server; if resources not available, report back and try again later * added option for physics step frequency * added option for physics step frequency * removed physics frequency from .env and set working PX4 values in docker-compose defaults. * removed unnecessary benchmarking from AirStack launch scripts. --------- Co-authored-by: Andrew Jong * Add new skills * Revise pull request template for clarity and detail Update pull request template with versioning guidelines Added guidelines for versioning in the pull request template. Update pull request template for media uploads Clarified instructions for adding videos and images in the PR template. * Johnliu/rtx lidar update (#351) * Update PegasusSim lidar to new rtx lidar and optional min_sensor_range parameter to vdb model to avoid self-detection. * removed deprecated ouster lidar. Completely integrated new rtx lidar * renaming frame id back to ouster * Added node to filter near and invalid lidar points * reconciled topic names for lidar point cloud * fixed example scripts to use rtx lidar api * fixed tmux closing and rclpy path issue * uses add_rtx in multi px4 script * bumping version index * docs added * unit testing and documentation updates * cleaning code from copilot suggestions * docs(tests): fix pytest marker example for running liveliness and sensors Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/7ce7609a-a7f3-414d-9d42-0c9999d0459f Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> * docs(tests): fix marker semantics in test_sensors module docstring Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/bdf00f6f-1d9f-4597-bf57-b96f99421646 Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> * addressing github copilot concerns * docs(bridge): remove stale camera topics comment Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/2d5718ac-20e3-4f10-a12e-05d601cf000c Co-authored-by: JohnYanxinLiu <63010779+JohnYanxinLiu@users.noreply.github.com> * addressing copilot concerns * removing debug print statement from reading point cloud Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fix(isaac-sim): align drone1 lidar prim path with spawned prim Agent-Logs-Url: https://github.com/castacks/AirStack/sessions/fbad2b9c-1761-45b1-b464-3e874511255c Co-authored-by: JohnYanxinLiu <63010779+JohnYanxinLiu@users.noreply.github.com> * more succint comment in sim bashrc * resolving discrepant comments in ros bridge yaml * removed bug allocated new copy of point cloud array * logs lidaar test with boolean instead of hz * and --> or for marks --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Andrew Jong * Krrish/coord pr (#350) * Fixed multi-drone global plan * added sep files for fire and retro * added robot2 relative pos; diff rviz files; bridge for rayfronts topics * added sharing of semantic rays * changed rviz for both drones * added target sharing * changed drone start pos * gossip layer w/o relay * added global coords under /{ROBOT_NAME}/interface/mavros/global_position/raw/fix(not my topic, it was already publishing to that) * gossip, threedrone,peerprofile * multi drone vis in foxglove, odom doesn't work in foxglove yet * multi drone vis in foxglove works with odom * global plan added * added image, vdb markers(not transformed yet) * fixed state estimation flickering and vdb transform * added custom foxglove buttons for commands * added modular payloads to peerprofile, foxglove reads the payloads and vizualizes it,currently works for rayfronts * fixing the rotation of payload * syncing devices * fixed gossip + translate * added skill for foxglove/coordination * removed VDB ENV * rebase with main * updated docs * fixed launch files so they have play start on sim. scene_prep utils: added non-world prims to save in flattened manner * created raven_nav package * moved coordination to common * fixed gcs<->robot dds * added hitl functionality * fixes to dds * put dds hitl under gcs * fixes to robot hitl * syncing both computers * mimiced robot-l4t for dataflow * fixed path to ddsrouter_yaml * fixed dds server * fixed two_drone_fire * rayfronts is now a ros package * added feedback, it's sending success too early though * fixed raven behavior * foxglove panel with working executors * random walk fixed * fixed random walk bringup. Added saves and viz for multiple waypoints and polygons * fixed bounds for exploration task, combined waypoint/polygon editor into task panel * made waypoint/polygon gui larger * added 2d map to foxglove * WIP: pre-merge snapshot * added changes from main * WIP: pre-branch-split snapshot * PR for foxglove+multi-robot * merged with main * PR cleanup: revert unrelated changes and drop extra files - Restore main's robot.rviz (drop redundant robot_1/robot_2.rviz) - Restore ms-airsim include in root docker-compose.yaml - Restore airsim sections in docs/simulation/index.md - Restore docs/gcs/docker/index.md (VERSION env name) - Restore robot/docker/{.bashrc, Dockerfile.robot} to main - Restore SIM_IP in robot/docker/docker-compose.yaml - Restore takeoff_landing_planner takeoff_height: 8.0 - Drop docs/action_bridging.md (internal design memo) - Drop personal launch scripts (two_drone_fire*, three_drone_scene_import, two_drone_RetroNeighbourhood) - Trim verbose comments in gps_utils.py and example_multi_drone_scene_import.py * Trim noisy inline comments in PR-added Python files * Pin vdb_mapping_ros2 to public main (was at unpushed 68fe8dde) * fixed launch script * fixed foxglove bugs, added dynamic fg layout, updated docs * fixed bugs found by copilot. Removed rviz by adding a node * fixed comment Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fixed path in skill Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * fixes from copilot * Fix Pegasus submodule pointer after merge Advance to 8e01d013 (main's pointer) which contains spawn_rtx_lidar.py, required by example_one_px4_pegasus_launch_script.py and the multi script after the rtx-lidar update merged from main. Co-Authored-By: Claude Opus 4.7 * fix(coordination): align gossip with steady clock + manifest hygiene - gossip_node: swap startup log + outgoing-stamp clock to STEADY_TIME so the dedup-by-stamp invariant survives /clock pauses; subscribe to /global_position/global to match foxglove_visualizer and action_relay - gossip_node docstring: drop the false "waypoint triggers immediate publish" claim - coordination README: rename peer_registry node block to the actual per-robot registry topic; "wall-clock" -> "steady" - package.xml: add missing exec/depend rules - coordination_bringup -> autonomy_bringup - autonomy_bringup -> coordination_bringup - desktop_bringup -> coordination_bringup, gcs_visualizer - gcs_visualizer -> std_msgs, coordination_msgs, coordination_bringup - task_msgs: replace TODO license with BSD-3-Clause - gcs.launch.xml: comment had `--no-sandbox` (`--` is illegal inside an XML comment and crashed the ROS launch parser) Co-Authored-By: Claude Opus 4.7 * fix(gcs+autonomy): drop dead BT panel, lint payload imports, name-map override - payload_visualizer_node: remove unused PointCloud2 / transform_point_cloud2 imports (F401), collapse Marker/MarkerArray - action_relay launch: ROBOT_RELAY_MAP env override for non-default robot_name -> domain mappings (default behavior unchanged) - desktop_bringup robot.rviz: drop BehaviorTreePanel entry pointing at /behavior/behavior_tree_graphviz (publisher package was removed) - autonomy_bringup domain_bridge: bridge /global_position/global to match the dds_router and the rest of the stack (was /raw/fix) Co-Authored-By: Claude Opus 4.7 * fix(foxglove): clean panel-id stacking, atomic render, drop dead .foxe - render_layout: regex now strips every trailing _r (was: only the last one), fixes _r1_r1_r1... stacking on repeated runs - render_layout: atomic write via tmp + os.replace so a partial json.dump doesn't corrupt the layout file - airstack_default.json: re-render with fixed stripper to commit a clean source template (no stacked _r1 suffixes) - install.sh -> install.py: file is Python, shebang is python3 - install.py: slugify publisher into the on-disk extension dir name so "AirLab CMU" doesn't produce a directory with a space - drop robot-commands/robot-commands.foxe (duplicate; canonical is at foxglove_extensions/robot-commands.foxe) and the .foxe.bak Co-Authored-By: Claude Opus 4.7 * bug fixes * bug fixes * version * reverted env * updated gitignore and docs * updated foxglove viz + consistent spellings across repo * Move layout file to /root/ so it's immediately accessible, also fix template path * Change so that file name reflects NUM_ROBOTS * Add a DEBUG_RVIZ flag to launch robot rviz if needed --------- Co-authored-by: krrishj18 Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 Co-authored-by: Andrew Jong * Scene prep bug fix (#354) * fixes to scene_prep_utils.py * edited docs * clean launch script * updated version * fixed comments inconsistency and typos * formatting fix Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * bug in gossip if payload is empty * fixed omni_pass.env file creation bug from CICD guest default profile * fixed depth topic naming in foxglove gcs * changed gps topic * removed redundant exntentions --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: airlab * Add workflows to (1) enforce correct branch merge convention (2) update develop from main * Update docs on branches * Update workflow to handle develop version increment * Release 0.18.0 * Bump VERSION to after sync from main * feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO (#352) * feat(osmo): VS Code/Cursor dev workflow on NVIDIA OSMO Adds a privileged Docker-in-Docker workspace task that lets a developer run the full AirStack docker-compose stack on OSMO and attach an IDE over SSH, with Isaac Sim WebRTC livestream + Foxglove websocket exposed via osmo port-forward. Components: - osmo/workspace/{Dockerfile,entrypoint.sh,sshd_config}: airstack-osmo-workspace image. Ubuntu 24.04 + sshd (pubkey-only) + Docker CE + Docker Compose + nvidia-container-toolkit + fuse-overlayfs (DinD-on-overlayfs needs it, otherwise dockerd falls back to vfs which bloats AirStack images ~10x). - osmo/workflows/airstack-dev.yaml: single privileged GPU task. Materializes Nucleus + airlab-docker secrets from OSMO credentials, clones AirStack, starts inner dockerd, runs `airstack up` with desktop + isaac-sim-livestream Compose profiles. - simulation/isaac-sim: isaac-sim-livestream Compose service that runs Pegasus standalone with --/app/livestream/enabled=true and exposes WebRTC port ranges 47995-48012 / 49000-49007 / 49100; launch script gates headless+livestream extension on ISAAC_SIM_LIVESTREAM env var. - .airstack/modules/osmo.sh: airstack osmo:{up,ide,foxglove,webrtc,logs,down} CLI wrappers around `osmo workflow submit` / `port-forward` / `cancel`. Persists the active workflow id and validates it's still running before each command (prevents the stale-state 410 error). - airstack.sh: bash 4+ re-exec bootstrap (macOS ships 3.2; the CLI uses `declare -A`). - osmo/README.md + docs/tutorials/airstack_on_osmo.md: admin pool setup (privileged_allowed) + per-user credentials (airlab-docker-login, airlab-nucleus) + student-facing IDE attach + WebRTC/Foxglove flow. Pool requirements: privileged_allowed: true, GPU pool with nvidia-container-toolkit on the host, ample node ephemeral storage (AirStack images extracted are ~50-100Gi via fuse-overlayfs; vfs needs ~500Gi+). Co-authored-by: Cursor * fix(osmo): harden CLI + workspace image against stale-state, port-forward race, and cursor-server install hangs Four bugs that bit the first end-to-end runs (airstack-dev-10 → -13): - _osmo_wf_id: validate saved workflow id against `osmo workflow query` before returning. Without this, the state file at ~/.airstack/osmo-state outlives the workflow it points at and every subsequent osmo:webrtc / osmo:foxglove / osmo:ide call surfaces the same confusing "Workflow airstack-dev-N is not running! (status 410)" instead of the obvious "run airstack osmo:up to launch a fresh workflow". - cmd_osmo_up: `osmo workflow submit --set-env` is variadic. Passing two separate `--set-env A=1 --set-env B=2` silently drops the first one — this is what made airstack-dev-11 fail with "ERROR: SSH_PUB_KEY not set" when --branch was passed alongside the pubkey. Collapse the K=V pairs into a single --set-env. - cmd_osmo_ide: previously launched the IDE before starting the port-forward, so Cursor/VS Code would try to SSH localhost:2200 a few hundred ms before the tunnel listener existed and fail with "connect to host localhost port 2200: Connection refused". Now: detect an existing forward and reuse it (also avoids the "Address already in use" if osmo:foxglove was started in parallel), otherwise spawn the forward in the background, wait up to 30s for it to bind, then launch the IDE. Ctrl+C tears down the spawned forward cleanly via a trap. - workspace image / entrypoint: Cursor Remote-SSH hung indefinitely on airstack-dev-13 because (a) cursor-server's installer fell back to wget when curl timed out and wget was not in the image, and (b) a /tmp/cursor-remote-lock.* file left behind by the first crashed install blocked every silent retry. Add wget to the apt install list and rm -f the stale Cursor / VS Code remote lock files at the very top of entrypoint.sh so each fresh pod starts from a clean slate. Co-authored-by: Cursor * fix(osmo): correct osmo:logs CLI invocation; install Foxglove extensions locally on osmo:foxglove osmo:logs was invoking `osmo workflow logs workspace --follow`, but the real CLI takes the task via `-t TASK` (not positionally) and has no `--follow` flag at all — so the command failed immediately with "unrecognized arguments: workspace --follow". Replace with a polling loop that uses `-t workspace -n ` on a short interval, prints only the suffix that appeared since the previous fetch (find-the-last-seen-line trick; degrades to "reprint tail" with a warning if the cursor outruns -n), and exits cleanly once the workflow reaches a terminal state. Tunables: OSMO_LOGS_TASK / OSMO_LOGS_TAIL / OSMO_LOGS_INTERVAL. osmo:foxglove now installs the AirStack Foxglove extensions (robot-commands / waypoint-editor / polygon-editor) into the laptop's local Foxglove user-extensions directory before opening the port-forward. Without this, custom panels show up as "Unknown panel type: robot-commands.Robot Tasks" in the laptop's Foxglove Desktop because it has no way to discover the extension folders that live inside the GCS container. To avoid duplicating the install logic, the existing gcs/foxglove_extensions/install.py is refactored to read FOXGLOVE_EXT_SRC / FOXGLOVE_EXT_DST env vars (the in-container call already in gcs/docker/gcs-base-docker-compose.yaml keeps working unchanged via defaults). The wrapper sets those vars to ${PROJECT_ROOT}/gcs/foxglove_extensions and ~/.foxglove-studio/extensions respectively, overridable with OSMO_FOXGLOVE_EXT_DIR / skippable with OSMO_FOXGLOVE_SKIP_EXTENSIONS=1. Co-authored-by: Cursor * fix(osmo): pin Kit livestream UDP media port to 49099 so osmo:webrtc actually shows pixels Kit 107's WebRTC livestream picks a UDP media port dynamically. The documented `omni.services.livestream.nvcf` defaults (minHostPort=47998 maxHostPort=48020 fixedHostPort=0) are ignored by the stock standalone Kit binary — on airstack-dev-13 it bound to UDP 49042, outside both the Compose-published range AND the default `osmo:webrtc --udp` forward of `47995-48012,49000-49007`. Result: TCP signaling on 49100 worked, the WebRTC Streaming Client window opened, but every SRTP media packet was dropped → black viewport plus the recurring `NVST_CCE_DISCONNECTED when m_connectionCount 0 != 1` underflow in Kit's log. Pin the media port via three `app.livestream.*` settings set on `SimulationApp` before `omni.kit.livestream.webrtc` is enabled, so whichever code path the carb.livestream-rtc.plugin consults lands on the same port: app.livestream.fixedHostPort = 49099 app.livestream.minHostPort = 49099 app.livestream.maxHostPort = 49099 49099 is a deliberate one-off from the 49100 TCP signaling port — same neighborhood, easy to remember. Verified live on airstack-dev-13 after `docker compose up -d --force-recreate isaac-sim-livestream`: Kit binds UDP 49099 (`/proc/net/udp` hex BFCB on 0.0.0.0) and docker-proxy publishes it from the pod host network. Knock-on cleanups: - `simulation/isaac-sim/docker/docker-compose.yaml` shrinks the isaac-sim-livestream `ports:` from 27 forwarded ports (`47995-48012, 49000-49007 TCP+UDP, 49100 TCP`) to just two: `49100/tcp` + `49099/udp`. - `.airstack/modules/osmo.sh` shrinks `OSMO_WEBRTC_TCP` to `49100` and `OSMO_WEBRTC_UDP` to `49099`, so `airstack osmo:webrtc` spawns two port-forwards instead of thirty. - `.gitignore` ignores `.DS_Store` so working from a Mac doesn't leak Finder metadata. After pulling this commit into a running pod: `docker compose up -d --force-recreate isaac-sim-livestream` to apply the new port mapping; then re-run `airstack osmo:webrtc` on the laptop to pick up the new forward ranges. The standalone WebRTC Streaming Client connects to `localhost` (same address as before) and now actually receives frames. Co-authored-by: Cursor * fix(osmo): render Kit GUI in WebRTC stream; document SSH agent forward for in-pod git push Two paper-cuts that bit airstack-dev-13 after the WebRTC media port pin landed (commit 2d9b1611): (1) The WebRTC stream showed only the bare 3D viewport — no menu bar, no toolbar, no panels, no console. Cause: SimulationApp's default when `headless=True` is to also hide the UI (`hide_ui=True`). The NVIDIA reference at `simulation/isaac-sim/standalone_examples/api/isaacsim.simulation_app/livestream.py` explicitly opts back into UI rendering plus picks explicit window sizing and `display_options=3286` to keep the default grid/axes visible. Mirror that config in `example_one_px4_pegasus_launch_script.py` when `ISAAC_SIM_LIVESTREAM=true` (local desktop dev keeps the minimal `headless=False` path unchanged). (2) The pod has no SSH private key, only an `authorized_keys` for inbound connections from the user's laptop. As a result, `git push` from inside the Cursor / VS Code Remote-SSH session inside the pod fails with "Permission denied (publickey)". sshd inside the workspace image already has `AllowAgentForwarding yes` baked in via `osmo/workspace/sshd_config`; the missing piece is purely on the Mac side. Update the `~/.ssh/config` block in the tutorial to include `ForwardAgent yes` (so the local agent's keys are exposed in the pod), `AddKeysToAgent yes` (auto-load on first push), and `UseKeychain yes` (macOS-only Keychain unlock without passphrase prompts; ignored on Linux). Adds an `ssh-add -l` smoke-test note. Co-authored-by: Cursor * fix(osmo): make osmo:setup idempotent + paste-safe; document Nucleus auth-debug path osmo:setup hit two failure modes that wasted a debug session each: - `osmo credential set` is not an upsert for GENERIC creds — re-running setup (e.g. to rotate a Nucleus API token) failed with `400 duplicate key value violates unique constraint "credential_pkey"` and bailed before reaching the airlab-nucleus credential. Delete-then-set each credential so re-running is idempotent. - Bracket-paste mode and cross-OS clipboards routinely smuggle invisible bytes around long pastes. Nucleus's auth endpoint silently DENIES a token with one extra trailing byte, with no actionable error from the client side. _osmo_prompt now strips leading/trailing whitespace and CR/NUL bytes via a new _osmo_trim helper, and warns when bytes were stripped. cmd_osmo_setup additionally JWT-shape-checks the Nucleus token (must be eyJ...) before submitting it, so a wrong paste fails at setup time instead of silently DENIED at pod boot. Also documents how to debug the "Login Required: Unable to connect server omniverse://airlab-nucleus..." popup: SSH the Nucleus host and tail base_stack-nucleus-auth-1 for InternalCredentials.auth status: DENIED. Adds a "Nucleus connectivity from OSMO" section to the admin README clarifying that Nucleus over HTTPS uses a single 443 (no need to open the native 3009-3180 range from the OSMO cluster), per NVIDIA's TLS docs. Co-authored-by: Cursor * fix(osmo): use Nucleus API-token auth, with double-dollar to survive compose parser The OSMO entrypoint was writing OMNI_USER= alongside an API token JWT in OMNI_PASS, which routes the JWT through the password- verification path. Nucleus silently DENIES — visible only in base_stack-nucleus-auth-1 as `InternalCredentials.auth … 'username': '' … status: DENIED` (no Tokens.auth_with_api_token call). Kit then pops "Login Required: Unable to connect server omniverse://...". omniclient expects the literal sentinel username `$omni-api-token` paired with the JWT as the password. The entrypoint now detects a JWT-shaped OMNI_PASS (header starts with `eyJ`) and emits OMNI_USER=$$omni-api-token into omni_pass.env. The `$$` is intentional: docker-compose v2 interpolates env_file values, and a single `$` would be eaten by the parser (`OMNI_USER=$omni-api-token` becomes `OMNI_USER=-api-token` after ${omni}- expansion to empty). The container ultimately sees OMNI_USER=$omni-api-token, which is the correct sentinel. Also note for the next debugger: `docker compose restart` does NOT re-read env_file. Use `docker compose up -d ` to recreate the container after editing omni_pass.env. Updates omni_pass_TEMPLATE.env header to document the API-token pattern explicitly (with the $$ caveat), and adds a troubleshooting row that distinguishes "wrong auth path" (DENIED with no Tokens.auth_with_api_token call) from "bad/expired token" (Tokens.auth_with_api_token: DENIED). Co-authored-by: Cursor * docs(osmo): make OSMO the recommended dev path, single clone-the-repo flow Reposition the OSMO tutorial as AirStack's recommended day-to-day development path (not just a fallback for laptops without GPUs) and collapse it onto a single recipe: clone the repo, then drive everything through the airstack osmo:* wrappers in .airstack/modules/osmo.sh. - docs/tutorials/airstack_on_osmo.md - Retitle + rewrite the intro to lead with five concrete advantages (pooled GPUs, no local CUDA/Docker/driver maintenance, same image as CI + field robots, one-command onboarding, hardware bigger than your laptop). Demote the Linux+GPU-desktop path to an escape hatch. - Drop the Mac/Windows/no-GPU framing in 'Who is this for?' and the mermaid laptop subgraph label. - Add 'a local clone of AirStack' to Prerequisites; remove it from the 'do not need' list. - Replace Option A/B credential split with a single ./airstack.sh osmo:setup recipe; move the three raw osmo credential set calls into a collapsible 'Under the hood' footnote. - Replace each step's raw osmo workflow ... command with the corresponding airstack osmo:up/logs/ide/webrtc/foxglove/down wrapper; preserve the raw form in 'Under the hood' footnotes that cross-link cmd_osmo_* in .airstack/modules/osmo.sh. - Drop the export WF=... paragraph — the wrappers read the id from ~/.airstack/osmo-state automatically; AIRSTACK_OSMO_WF overrides per-invocation. \$WF now only appears inside the raw-form footnotes. - Sweep Troubleshooting + What-survives tables: redirect raw port-forward fixes to the airstack osmo:* equivalents and rename the section to 'What survives airstack osmo:down?'. - Fix WebRTC edge label (49100/tcp + 49099/udp) to match the pinned ports the workflow actually uses today. Companion cleanups now that the privileged_allowed flip is automatic on the OSMO autosync side (synchronize_osmo_team_pools.py forces privileged_allowed: true on every platform of every pool, so students never see the 'platform does not have privileged flag enabled' error): - osmo/README.md: drop the 'Most common blocker' privileged warning, the privileged_allowed row from the pool-requirements table, and the 'privileged GPU pod' / '(privileged, GPU)' descriptors in the architecture summary. Simplify the validation-stage SSH-failure hint. - osmo/workflows/airstack-dev.yaml: trim the long DinD-requires-privileged comment to a one-liner (the privileged: true directive itself stays). - .airstack/modules/osmo.sh: remove the special-case 'privileged flag enabled' error branch in cmd_osmo_up — it should never fire now. Co-authored-by: Cursor * fix(osmo): make osmo:logs actually stream + survive pod host-key churn osmo:logs was silent because cmd_osmo_logs wrapped osmo workflow logs in $( ... ) on the assumption that -n LAST_N_LINES exits after dumping the tail. Empirically the CLI keeps the stream open as new lines arrive (it already behaves like tail -f, despite --help advertising only -n), so command substitution waited forever and printed nothing. Drop the polling loop and just exec the command directly. Each fresh OSMO pod also ships a new sshd host key, so every osmo:up trips StrictHostKeyChecking against the previous workflow's fingerprint and SSH/Cursor abort with "Host key for [localhost]:2200 has changed". Switch the recommended ~/.ssh/config block (and osmo/README.md) to the ephemeral-host pattern (StrictHostKeyChecking no + UserKnownHostsFile /dev/null + LogLevel ERROR), and have cmd_osmo_ide ssh-keygen -R the stale loopback entry on every run so users on the old config get unblocked automatically. Co-authored-by: Cursor * fix(osmo): auto-pin --branch to local checkout + clean error UX when workflow dies The pod's entrypoint clones AirStack fresh from GitHub on every workflow start (the pod fs is ephemeral). It defaulted to `main`, so any developer testing branch-only OSMO changes silently ran their pod against stale `main` code — most visibly: COMPOSE_PROFILES=desktop,isaac-sim-livestream resolved to "desktop" alone on `main` because the isaac-sim-livestream service only exists on the feature branch, so isaac-sim never came up and `airstack osmo:webrtc` showed a blank stream. - cmd_osmo_up now defaults --branch to the local repo's current branch (git rev-parse --abbrev-ref HEAD). Detached HEAD or non-git checkouts fall back to `main` cleanly. Pass --branch explicitly to override. - New _osmo_check_branch_pushed warns up-front when the about-to- submit branch has no upstream, is ahead of origin, or has an uncommitted working tree. The pod doesn't see your laptop's edits. Separately, when an OSMO workflow gets canceled mid-flight (osmo:down in another shell, or OSMO timing it out), the in-flight port-forward and logs streams raise OSMOUserError("Workflow X is not running!") from inside an asyncio Task. The CLI prints "Task exception was never retrieved" + a multi-line Traceback that buries the actual one-line cause. New _osmo_pf_filter awk script collapses that into a single [ERROR] line pointing at `airstack osmo:up`. Wired into webrtc, foxglove, and logs. webrtc also gains a cleanup trap that kills the backgrounded UDP port-forward on EXIT/INT/TERM so we don't leak it against a dead workflow. Tutorial Step 2 documents the new --branch default and the "pod-clones-from-GitHub-not-your-laptop" gotcha. Co-authored-by: Cursor * perf(osmo): bump inner dockerd concurrency to saturate 10 GbE pulls dockerd's defaults of --max-concurrent-downloads=3 / --max-concurrent -uploads=5 cap a fresh airstack-dev pod's image-pull at ~300 MiB/s against the airlab-backup-10g registry — single-stream TLS tops out around 300-500 MiB/s per core, and three parallel streams of unevenly sized blobs serialize down to that ceiling. Ceph (1014 TiB, 92 OSDs, SSD pools) and 10 GbE both have far more headroom than that. Bump to 10/10 to overlap enough blob downloads to saturate the pipe. Threaded through the DOCKERD_MAX_DOWNLOADS / DOCKERD_MAX_UPLOADS env vars so a pool can be tuned at submit time without rebuilding the workspace image. Workspace image needs a rebuild + push for this to take effect: cd osmo/workspace docker build -t airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest . docker push airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest Co-authored-by: Cursor * docs(osmo): require buildx --platform linux/amd64 for workspace image A plain `docker build && docker push` on an Apple Silicon Mac silently produces a linux/arm64-only `latest` manifest. OSMO workers are amd64, so every subsequent workflow fails at the outer pod-image pull with "no match for platform in manifest" before the entrypoint even runs — a confusing failure mode whose root cause lives entirely in the push, not in the workflow yaml or the entrypoint. Switch the README and the Dockerfile docstring to the buildx form, explain the why, and document the post-push manifest check. Co-authored-by: Cursor * perf(osmo): move dockerd data-root to /osmo/run for native overlay2 The OSMO pod's `/` is itself a containerd overlay snapshot, and Linux refuses to stack a second overlayfs on top of an overlay rootfs — which is why the inner dockerd was falling through to fuse-overlayfs. That costs a kernel↔userspace FUSE round-trip on every `creat()` during layer extraction, which murders throughput on apt/pip/ROS layers (measured: 32-50 MB/s for small-file-heavy layers vs 480 MB/s for big-file layers in the same pull). Pointing dockerd at /osmo/run/docker (the kubelet emptyDir backed by ext4 on /dev/vda3) lets the existing overlay2-first fallback chain actually succeed on its first try, restoring kernel-overlay extraction performance. emptyDir lifetime matches the workflow lifetime, so the docker layer cache gets the right scope automatically. Falls back to /var/lib/docker if /osmo/run isn't present so the image still works in non-OSMO test contexts. Co-authored-by: Cursor * updated version * added virtual display for GL context * added virtual display for droan_gl * droan_gl patch * run Xvfb in its own tmux session * updated dockerfile + version * typo in docs Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in comments Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in osmo logs, renamed airstack-isaac-sim to just isaac-sim Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * typo in container name for isaac-sim-livestream Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * airstack-dev version overwrite removed Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Cursor Co-authored-by: krrishj18 Co-authored-by: Andrew Jong Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * Add fixed-trajectory evaluation tests New tests/test_fixed_trajectory.py evaluates drone performance on Circle, Figure8, Racetrack, and Line trajectories: takeoff -> execute -> land with cross-track error, path RMSE, execution time, and success metrics recorded to metrics.json for baseline comparison. - Python ideal-path generators mirror fixed_trajectory_task.cpp equations - Cross-track error uses robot pose snapshot at dispatch to transform base_link ideal path to world frame for odom comparison - 5m loose tolerance documents the known circle failure without stranding drone - conftest.py gains --trajectory-types CLI option and generalised phase-order sorting/ID-rewriting for both autonomy test modules - tests/README.md documents the new module, all 11 metrics, and run commands Made-with: Cursor * Spherical lookahead bug that fixed the circle test and caused the circle test to pass * Added in code that consolidated all the results code so the user can easily see their results in one file without having to wade through a ton of log files to get what they need * Results for 10 tries headless summary statistics * Fixed the logging files so now it only outputs one summary file and it doesn't inundate the user with a ton of log files for no reason * deleted cleanup_old_results.sh which was a local tool for cleaning up everything * Added preliminary docs to explain changes made * Changed .env to say 0.19.0-alpha.4 * Resolved all the merge conflicts that are in this file * Revert sphere_radius to 1.0; velocity_sphere_radius_multiplier=1.0 makes the fixed value inert Co-authored-by: Cursor * Remove internal branch reference from baseline; note AirStation hardware Co-authored-by: Cursor * Remove parameter tuning bullet from docs after reverting sphere_radius Co-authored-by: Cursor * Move system-test prerequisites to index.md and reference it from fixed-trajectory doc Co-authored-by: Cursor * Remove path tracker bug fixes section from docs (covered in PR description) Co-authored-by: Cursor * Trim duplicated stack bring-up from manual usage; link to Getting Started Co-authored-by: Cursor * Reframe fixed-trajectory doc as end-to-end testing guide Rename fixed_trajectory_testing.md to end_to_end_testing.md (history preserved), add e2e intro and future-work note, fix stale test path to tests/system, and update mkdocs nav, testing index, and tests/README references. Co-authored-by: Cursor * removed stale test_sensors file * incremented version tag * Fixed the summary.txt file after it broke after a ton of commits were completed. * resyncing Pegasus module to fixed camera initialization fix --------- Co-authored-by: pvkumara Co-authored-by: Andrew Jong Co-authored-by: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: andrewjong <8121216+andrewjong@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Krrish Jain Co-authored-by: krrishj18 Co-authored-by: Claude Opus 4.7 Co-authored-by: airlab Co-authored-by: Andrew Jong Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Sebastian Scherer Co-authored-by: Cursor --- --gui | 0 --num-robots | 0 --sim | 0 --stress-iterations | 0 --trajectory-types | 0 -v | 11 + .agents/skills/run-system-tests/SKILL.md | 31 +- .env | 2 +- AGENTS.md | 6 +- .../testing/end_to_end_testing.md | 460 ++++++++++++ .../development/intermediate/testing/index.md | 43 +- ...docker-image-tag_BACKUP_3660135.pre-commit | 150 ---- ...e-docker-image-tag_BASE_3660135.pre-commit | 42 -- ...-docker-image-tag_LOCAL_3660135.pre-commit | 114 --- ...docker-image-tag_REMOTE_3660135.pre-commit | 81 --- mkdocs.yml | 1 + .../src/trajectory_controller.cpp | 32 +- .../src/trajectory_library.cpp | 9 +- tests/README.md | 120 +++- tests/conftest.py | 158 ++-- tests/pytest.ini | 1 + tests/run_summary.py | 398 ++++++++++ tests/system/test_fixed_trajectory.py | 678 ++++++++++++++++++ 23 files changed, 1813 insertions(+), 524 deletions(-) create mode 100644 --gui create mode 100644 --num-robots create mode 100644 --sim create mode 100644 --stress-iterations create mode 100644 --trajectory-types create mode 100644 -v create mode 100644 docs/development/intermediate/testing/end_to_end_testing.md delete mode 100755 git-hooks/docker-versioning/update-docker-image-tag_BACKUP_3660135.pre-commit delete mode 100644 git-hooks/docker-versioning/update-docker-image-tag_BASE_3660135.pre-commit delete mode 100644 git-hooks/docker-versioning/update-docker-image-tag_LOCAL_3660135.pre-commit delete mode 100644 git-hooks/docker-versioning/update-docker-image-tag_REMOTE_3660135.pre-commit create mode 100644 tests/run_summary.py create mode 100644 tests/system/test_fixed_trajectory.py diff --git a/--gui b/--gui new file mode 100644 index 000000000..e69de29bb diff --git a/--num-robots b/--num-robots new file mode 100644 index 000000000..e69de29bb diff --git a/--sim b/--sim new file mode 100644 index 000000000..e69de29bb diff --git a/--stress-iterations b/--stress-iterations new file mode 100644 index 000000000..e69de29bb diff --git a/--trajectory-types b/--trajectory-types new file mode 100644 index 000000000..e69de29bb diff --git a/-v b/-v new file mode 100644 index 000000000..fa52f4143 --- /dev/null +++ b/-v @@ -0,0 +1,11 @@ +access control disabled, clients can connect from any host +============================= test session starts ============================== +platform linux -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 -- /usr/local/bin/python3.12 +cachedir: /tmp/.pytest_cache +rootdir: /home/pranavkumara/Desktop/AirStack/tests +configfile: pytest.ini +plugins: dependency-0.6.1, timeout-2.4.0 +collecting ... collected 0 items + +- generated xml file: /home/pranavkumara/Desktop/AirStack/tests/results/2026-05-28_14-14-04/results.xml - +============================ no tests ran in 0.00s ============================= diff --git a/.agents/skills/run-system-tests/SKILL.md b/.agents/skills/run-system-tests/SKILL.md index 453bbc953..f9b41b727 100644 --- a/.agents/skills/run-system-tests/SKILL.md +++ b/.agents/skills/run-system-tests/SKILL.md @@ -1,6 +1,6 @@ --- name: run-system-tests -description: Run, interpret, and extend AirStack's pytest system test suite (build_packages, build_docker, liveliness, sensors, takeoff_hover_land), trigger runs via /pytest PR comments, and read metrics.json regression reports. Use for invoking tests, debugging failures from results.xml/metrics.json, or adding a new system test. +description: Run, interpret, and extend AirStack's pytest system test suite (build_packages, build_docker, liveliness, sensors, takeoff_hover_land, autonomy), trigger runs via /pytest PR comments, and read metrics.json regression reports. Use for invoking tests, debugging failures from results.xml/metrics.json, or adding a new system test. license: Apache-2.0 metadata: author: AirLab CMU @@ -14,7 +14,7 @@ metadata: Use this skill when you need to: - Invoke the pytest system tests locally (via `airstack test`) or on CI (via `/pytest` PR comment or `workflow_dispatch`) -- Diagnose a failing system test — interpret `results.xml`, per-test logs, and `metrics.json` from `tests/results//` +- Diagnose a failing system test — interpret `summary.txt`, `results.xml`, and `metrics.json` from `tests/results//` - Compare metrics against a baseline run (`parse_metrics.py --baseline`) to confirm a regression or improvement - Add a new system test to `tests/`: pick the right mark, wire up `airstack_env` parametrization, and record metrics with `MetricsRecorder` @@ -24,7 +24,7 @@ This skill is about the **test harness itself** — pytest marks, fixtures, the The suite lives at `tests/` (repo root) and is fully pytest-based. Configuration is in `tests/pytest.ini` and shared infrastructure in `tests/conftest.py`. -- **`tests/system/`** — Docker stack integration tests. Marks: `build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`. +- **`tests/system/`** — Docker stack integration tests. Marks: `build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`, `autonomy`. - **`tests/robot/`** and **`tests/sim/`** — Hermetic **unit** tests (`@pytest.mark.unit`). These are **thin proxy files** that re-export tests from each ROS 2 package's own `test/` directory (co-located with the source, the ROS 2 / colcon convention). The proxy pattern keeps test source next to the code it tests while making tests discoverable by `pytest tests/`. ### Unit tests vs system tests @@ -55,6 +55,7 @@ For details on the proxy pattern and adding new unit tests, see the | `tests/system/test_liveliness.py` | `liveliness` | Stack bring-up: containers Running, `/clock` readiness, tmux panes, sentinel ROS 2 nodes, compute, infra-only `test_stable` | Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, sim license / Omniverse creds | | `tests/system/test_sensors.py` | `sensors` | Topic Hz (Isaac: batched on sim + robot; LiDAR `echo-once` + cloud sanity), RTF, `test_sensor_streams_stable` | Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, sim license / Omniverse creds | | `tests/system/test_takeoff_hover_land.py` | `takeoff_hover_land` | 4-phase flight chain per `(sim, num_robots, iteration, velocity)`: `test_px4_ready` → `test_takeoff` → `test_hover` → `test_landing`. Records altitude error, overshoot, hover stability, landing accuracy, odometry drift | Docker daemon, NVIDIA GPU, sim license | +| `tests/system/test_fixed_trajectory.py` | `autonomy` | 4-phase flight chain per `(sim, num_robots, iteration, trajectory_type)`: `test_px4_ready` → `test_takeoff` → `test_fixed_trajectory` → `test_landing`. Records cross-track error, path RMSE, trajectory success/time for Circle/Figure8/Racetrack/Line | Docker daemon, NVIDIA GPU, sim license | The marks are declared in `tests/pytest.ini`. **Do not invent new marks ad-hoc** — register any new mark there or pytest will warn about unknown marks. @@ -63,10 +64,10 @@ The marks are declared in `tests/pytest.ini`. **Do not invent new marks ad-hoc** `conftest.py` enforces a deterministic global order so cheap-and-fast-failing tests surface first: ``` -system.test_build_docker → system.test_build_packages → system.test_liveliness → system.test_sensors → system.test_takeoff_hover_land +system.test_build_docker → system.test_build_packages → system.test_liveliness → system.test_sensors → system.test_takeoff_hover_land → system.test_fixed_trajectory ``` -Within `system.test_takeoff_hover_land`, items are re-sorted to `(airstack_env, velocity, phase)` so each `(sim, robots, iter)` env brings the stack up once and the drone goes ground → air → ground per velocity before pytest moves to the next velocity. +Within `system.test_takeoff_hover_land`, items are re-sorted to `(airstack_env, velocity, phase)` so each `(sim, robots, iter)` env brings the stack up once and the drone goes ground → air → ground per velocity before pytest moves to the next velocity. `system.test_fixed_trajectory` is re-sorted the same way by `(airstack_env, trajectory_type, phase)`. ### Isaac Sim (`sensors`): why Hz is batched and LiDAR uses `echo --once` @@ -219,17 +220,17 @@ Every run (local or CI) produces a fresh timestamped directory under `tests/resu ``` tests/results/2025-04-21_14-30-00/ +├── summary.txt # Human-readable key metrics — open this first ├── results.xml # JUnit XML — durations + pass/fail per test -├── metrics.json # Custom metrics keyed by test_node_id → metric_key -└── logs/ - ├── system.test_build_docker.TestDockerBuilds.test_build_robot_desktop.log - ├── system.test_sensors.TestSensors.test_sensor_streams_stable[msairsim-rob#1-iter0].log - ├── system.test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0].log - ├── airstack_env.system.test_liveliness.TestLiveliness.test_robot_containers_running[...].log - └── ... +└── metrics.json # Custom metrics keyed by test_node_id → metric_key ``` -**One log file per test execution**, plus separate `airstack_env.*.log` files for fixture narration (the `up`/`down` of each parametrize tuple). The fixture log file is named to track the rewritten test ID so it lands next to the triggering test. +There is **no `logs/` subdirectory**. Live output streams to the terminal during +the run (pytest `log_cli`), and each subprocess's combined stdout/stderr is held +in memory so a failed assertion can include the tail of the last command's output +inline. `summary.txt` is written once at session end by +`run_summary.write_summary()`, so the key metrics land in one place without +digging through raw output. ### `metrics.json` structure @@ -345,7 +346,7 @@ Conventions: ### 6. Fixture extension -If multiple tests need the same setup, add a fixture in `conftest.py` (not in your test file) so it's available repo-wide. Mirror the `airstack_env` pattern: yield a dict, narrate via `logger_to(log)`, record any setup/teardown timing as metrics. +If multiple tests need the same setup, add a fixture in `conftest.py` (not in your test file) so it's available repo-wide. Mirror the `airstack_env` pattern: yield a dict, log progress via the shared `logger` (output streams to the terminal via `log_cli`), record any setup/teardown timing as metrics. ## Common Pitfalls @@ -356,7 +357,7 @@ If multiple tests need the same setup, add a fixture in `conftest.py` (not in yo - **Not capturing metrics in a new test**. If a test fails silently (no metric recorded) the regression report has nothing to compare. Always record at least one scalar via `MetricsRecorder` so the test shows up in `metrics.json`. - **Letting parametrize cardinality explode**. Defaults `--sim msairsim,isaacsim --num-robots 1,3` with `--stress-iterations 3` multiply stack bring-ups for each selected mark (`liveliness`, `sensors`, `takeoff_hover_land`, …) — expensive. Override locally to a single tuple while iterating. - **Hardcoded container names**. Always use `find_container`, `get_robot_containers`, or `wait_for_container` — replica suffixes (`-1`, `-2`, `-3`) and compose project prefixes change. -- **Asserting on stdout instead of using `read_log_tail`**. The conftest tees subprocess output to per-test log files; assertions should reference those logs (`f"airstack up failed:\n{read_log_tail()}"`) so failures attach the relevant context to the JUnit XML. +- **Asserting on stdout instead of using `read_log_tail`**. The conftest captures each subprocess's combined stdout/stderr in memory; assertions should reference it via `read_log_tail()` (`f"airstack up failed:\n{read_log_tail()}"`) so failures attach the relevant context to the JUnit XML. - **Trying to SSH into a CI runner mid-job**. Workers are ephemeral OpenStack VMs destroyed within ~30s of job completion. Re-running the job creates a fresh VM. For genuine debugging on the runner, see `.github/orchestrator/README.md` (also exposed at `tests/ci-cd-orchestrator.md`) — but in 99% of cases, reproduce locally with `airstack test`. - **Forgetting to register a new mark**. Adding `@pytest.mark.my_new_mark` without updating `tests/pytest.ini` produces "PytestUnknownMarkWarning" and makes `-m my_new_mark` fail to filter as expected. diff --git a/.env b/.env index 0d994bc51..77ed88ef4 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.4" +VERSION="0.19.0-alpha.5" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/AGENTS.md b/AGENTS.md index b53069e75..0a6b86015 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -196,7 +196,7 @@ docker exec airstack-robot-desktop-1 bash -c "ros2 topic echo --onc - Verify module behavior in isolation - Test with synthetic data - Located in module's `test/` directory - - **Run in the robot container** with `colcon test` (after `bws`), not via `airstack test -m unit`. The root [`tests/`](tests/) suite does **not** register a `unit` pytest mark; `airstack test -m ` only selects marks declared in [`tests/pytest.ini`](tests/pytest.ini) (`build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`). + - **Run in the robot container** with `colcon test` (after `bws`) for the full ROS 2 build + test. The same co-located test source is re-exported to the root [`tests/`](tests/) suite via thin proxies (see Unit tests below), so `airstack test -m unit` runs it too. Marks are declared in [`tests/pytest.ini`](tests/pytest.ini) (`unit`, `build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`, `autonomy`). ```bash docker exec airstack-robot-desktop-1 bash -c "sws && colcon test --packages-select natnet_ros2 --event-handlers console_direct+" @@ -221,8 +221,9 @@ Pytest-based system tests live under [`tests/system/`](tests/system/). They brin | [`tests/system/test_liveliness.py`](tests/system/test_liveliness.py) | `liveliness` | Stack bring-up: containers, ``/clock`` readiness, tmux, sentinel ROS 2 nodes, compute, infra-only stability poll | Docker, GPU, sim license | | [`tests/system/test_sensors.py`](tests/system/test_sensors.py) | `sensors` | Topic Hz (Isaac: batched sim + robot ``ros2 topic hz``; filtered LiDAR ``echo-once`` + validation script), RTF, sensor stability time-series | Docker, GPU, sim license | | [`tests/system/test_takeoff_hover_land.py`](tests/system/test_takeoff_hover_land.py) | `takeoff_hover_land` | 4-phase flight chain (PX4 ready → takeoff → hover → land) per (sim, num_robots, iter, velocity) | Docker, GPU, sim license | +| [`tests/system/test_fixed_trajectory.py`](tests/system/test_fixed_trajectory.py) | `autonomy` | 4-phase flight chain (PX4 ready → takeoff → execute Circle/Figure8/Racetrack/Line trajectory → land) per (sim, num_robots, iter, trajectory_type); records cross-track error and path RMSE | Docker, GPU, sim license | -Shared fixtures, the `airstack_env` parametrized fixture, and `MetricsRecorder` live in [`tests/conftest.py`](tests/conftest.py). Each run produces a timestamped directory under `tests/results//` with `results.xml`, `metrics.json`, and per-test logs. [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). +Shared fixtures, the `airstack_env` parametrized fixture, and `MetricsRecorder` live in [`tests/conftest.py`](tests/conftest.py). Each run produces a timestamped directory under `tests/results//` with `summary.txt`, `results.xml`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). **Run via the CLI** (containerized runner — no local Python needed): @@ -232,6 +233,7 @@ airstack test -m "build_docker or build_packages" -v airstack test -m liveliness --sim msairsim --num-robots 1 --stress-iterations 1 -v airstack test -m sensors --sim isaacsim --num-robots 1 --stress-iterations 1 -v airstack test -m takeoff_hover_land --sim msairsim --takeoff-velocities 0.5,1,2 -v +airstack test -m autonomy --sim msairsim --trajectory-types Circle,Figure8,Racetrack,Line -v ``` Full reference: [`tests/README.md`](tests/README.md) — including **liveliness vs diff --git a/docs/development/intermediate/testing/end_to_end_testing.md b/docs/development/intermediate/testing/end_to_end_testing.md new file mode 100644 index 000000000..4863d7964 --- /dev/null +++ b/docs/development/intermediate/testing/end_to_end_testing.md @@ -0,0 +1,460 @@ +# End-to-End Testing + +End-to-end (e2e) tests exercise the **full autonomy stack in simulation** — the drone takes off, performs an action, and lands — so one run validates the whole pipeline (interface → perception → planning → control → PX4) rather than a single module. + +AirStack currently has two e2e suites: + +- **Takeoff / hover / land** (`takeoff_hover_land` mark) — the basic flight chain, documented in [`tests/README.md`](../../../../tests/README.md). +- **Fixed-trajectory path-tracker benchmark** (`autonomy` mark) — takeoff → execute a fixed pattern (Circle / Figure8 / Racetrack / Line) → land, measuring cross-track error. Documented below. + +!!! note "Future work: unify the e2e marks" + `takeoff_hover_land` and `autonomy` are separate marks today. As the suite grows they can be consolidated into a single general **e2e** mark/pipeline. Tracked as follow-up — not part of this change. + +--- + +## Fixed-Trajectory Path-Tracker Benchmark + +This section documents the **fixed-trajectory evaluation test suite** (`tests/system/test_fixed_trajectory.py`): why it exists, how it is implemented, how to run it, how to interpret results, and how to use it to **compare path trackers** without rewriting tests. + +--- + +## Purpose + +AirStack's local controls stack separates **reference-path generation**, **path tracking**, and **low-level control**: + +```mermaid +flowchart LR + FT[FixedTrajectoryTask] --> TL[trajectory_library] + TL --> TC[trajectory_controller
path tracker] + TC -->|~/tracking_point| PID[pid_controller] + PID --> FC[Flight computer / PX4] + FC --> ODOM[local_position/odom] + ODOM --> TC +``` + +The benchmark harness holds the **reference trajectory** and **flight procedure** constant so maintainers can: + +- **Swap or retune path trackers** and measure the same metrics every time. +- **Compare execution time** — how long does a standard pattern take in sim-time? +- **Compare tracking error** — mean/max cross-track error and path RMSE against a known ideal path. +- **Detect regressions** — action timeouts, stalls, or catastrophic drift via `trajectory_success` and assertion thresholds. + +Today the default tracker is the **sphere-intersection pure-pursuit** implementation in `trajectory_controller` + `trajectory_library`. The downstream `pid_controller` is held fixed so changes isolate tracker behavior. A different tracker can replace the `trajectory_controller` node (or its parameters) in launch; the pytest module does not need to change as long as `FixedTrajectoryTask` and odom topics remain the same. + +--- + +## What gets tested + +### Test module + +| Item | Value | +| ---- | ----- | +| File | [`tests/system/test_fixed_trajectory.py`](../../../../tests/system/test_fixed_trajectory.py) | +| Pytest mark | `autonomy` | +| Class | `TestFixedTrajectory` | +| Timeout | 2400 s per test class invocation | + +### Parametrization + +Each run sweeps: + +``` +(sim, num_robots, iteration, trajectory_type) +``` + +| Parameter | CLI flag | Default | +| --------- | -------- | ------- | +| Simulator | `--sim` | `msairsim,isaacsim` | +| Robot count | `--num-robots` | `1,3` | +| Repeat count | `--stress-iterations` | `1` | +| Trajectory type | `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | + +!!! tip "Pin your sweep for local runs" + Defaults multiply configs and run for hours. For development, always set explicit values: + + ```bash + airstack test -m autonomy \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + -v + ``` + +### Four-phase flight chain + +For every `(sim, num_robots, iteration, trajectory_type)` tuple the drone runs: + +| Phase | Test | Action | Pass criteria | +| ----- | ---- | ------ | ------------- | +| 1 | `test_px4_ready` | Wait for MAVROS + odom | All robots connected and publishing within 300 s wall-clock | +| 2 | `test_takeoff` | `TakeoffTask` to 10 m @ 1 m/s | Steady-state altitude within ±10% of target | +| 3 | `test_fixed_trajectory` | `FixedTrajectoryTask` | Cross-track mean < 5 m; records success + timing | +| 4 | `test_landing` | `LandTask` @ 1 m/s | Final altitude < 0.5 m | + +```mermaid +stateDiagram-v2 + [*] --> PX4Ready + PX4Ready --> Takeoff + Takeoff --> ExecuteTrajectory + ExecuteTrajectory --> Land : always + ExecuteTrajectory --> Land : even on trajectory failure + Land --> [*] + Takeoff --> Poisoned : takeoff fails + Land --> Poisoned : landing fails + Poisoned --> [*] : skip remaining types in env +``` + +**Chain guard:** a failure in phase 3 (`test_fixed_trajectory`) does **not** poison the environment — landing always runs so the drone returns to the ground before the next trajectory type. Failures in takeoff or landing **do** poison the env and skip subsequent trajectory types for that `(sim, num_robots, iteration)`. + +Phase 1 (`test_px4_ready`) runs once per env regardless of how many trajectory types are swept. + +--- + +## Reference trajectories + +The test uses the same patterns as the `FixedTrajectoryTask` action server in `trajectory_controller` (`fixed_trajectory_task.cpp`). Default parameters are defined in `TRAJECTORY_CONFIGS` inside `test_fixed_trajectory.py` and must stay in sync with the C++ generators. + +| Type | Parameters | Approx. path length | Expected sim-time* | +| ---- | ---------- | ------------------- | ------------------ | +| **Circle** | radius=10 m, velocity=2 m/s | ~63 m loop + return segments | **~45–50 s** | +| **Figure8** | length=15 m, width=8 m, v=2 m/s, max_accel=1 m/s² | ~100+ m | **~50–70 s** | +| **Racetrack** | length=30 m, width=10 m, v=3 m/s, turn_v=1.5 m/s | ~80+ m | **~30–50 s** | +| **Line** | length=20 m, v=2 m/s, max_accel=1 m/s² | 20 m | **~12–15 s** | + +\*Sim-time from odom timestamps; wall-clock varies with sim real-time factor (RTF). + +### Circle geometry (ideal path) + +Python `_ideal_circle()` mirrors `generate_circle()` in C++: + +- Start at origin, move to `(radius, 0, 0)`. +- Trace the circle in 10° steps. +- Return to `(radius, 0, 0)` then origin. + +The trajectory is defined in **`base_link`** at dispatch; the test transforms it to **world frame** using the robot pose snapshot (see below). + +--- + +## Metrics + +All metrics are recorded per robot as `robot_N.` in `tests/results//metrics.json` and rolled up into `summary.txt`. + +### Flight metrics + +| Key | Unit | Better | Description | +| --- | ---- | ------ | ----------- | +| `ready_duration_sys_s` | s | lower | Wall-clock time until PX4/MAVROS ready | +| `takeoff_duration_sim_s` | s | lower | Sim-time from first motion to 95% of 10 m target | +| `altitude_error_m` | m | lower | Signed steady-state altitude error after takeoff | +| `overshoot_m` | m | lower | Unsigned overshoot above 10 m | +| `trajectory_success` | — | **higher** | `1.0` if action returned `success: true`, else `0.0` | +| `trajectory_execution_time_sim_s` | s | lower | Sim-time from action dispatch to completion | +| `cross_track_error_mean_m` | m | lower | Mean 2-D lateral distance to nearest ideal point | +| `cross_track_error_max_m` | m | lower | Worst 2-D lateral deviation | +| `path_rmse_m` | m | lower | 2-D RMSE against ideal polyline | +| `land_duration_sim_s` | s | lower | Sim-time from 80% peak descent to < 0.5 m | +| `final_altitude_m` | m | lower | Altitude when landing action completes | + +### How to read metrics when comparing trackers + +| Observation | Likely meaning | +| ----------- | -------------- | +| High `cross_track_error_max_m`, moderate mean | Turn/corner lag (common on Circle) | +| High mean and max | Tracker not keeping up or wrong frame | +| Long `trajectory_execution_time_sim_s` at same velocity | Virtual time stalling behind the robot | +| `trajectory_success = 0` | Action timed out or aborted — fix before interpreting error | +| Good mean, bad max | Occasional spikes — check sphere intersection on curves | + +### Observed baseline (Circle, Isaac Sim, 10 headless runs) + +Measured on the **AirStations** (Linux workstations with GPU support): + +| Metric | Typical value | +| ------ | ------------- | +| Tests | 40 passed / 0 failed (10 iter × 4 phases) | +| `trajectory_success` | yes (every run) | +| `trajectory_execution_time_sim_s` | ~46 s | +| `cross_track_error_mean_m` | ~0.98 m | +| `cross_track_error_max_m` | ~5.0 m | +| `path_rmse_m` | ~1.55 m | +| `final_altitude_m` | < 0.05 m | + +The assertion tolerance is **`CROSS_TRACK_TOLERANCE_M = 5.0`** in `test_fixed_trajectory.py` — intentionally loose while the default tracker matures. Tighten this constant as tracking improves. + +--- + +## Cross-track error algorithm + +The test measures **end-to-end** tracking (tracker + PID + sim physics), not the tracker in isolation. + +### Steps + +1. **Snapshot pose** — immediately before sending `FixedTrajectoryTask`, read one odom sample: `(x₀, y₀, z₀, yaw₀)`. +2. **Build ideal path** — generate waypoints in `base_link` using the same equations as C++ (`_ideal_circle`, `_ideal_figure8`, etc.). +3. **Transform to world** — rotate by `yaw₀` and translate by `(x₀, y₀, z₀)`. +4. **Capture odom** — background `ros2 topic echo --csv` on `/robot_N/interface/mavros/local_position/odom` for the action duration (timeout 180 s). +5. **Compute error** — for each odom sample, find the nearest ideal waypoint in **XY**; record distance statistics. + +Altitude is not part of cross-track error (these patterns are flat; altitude is checked at takeoff). + +### Why world-frame alignment matters + +`FixedTrajectoryTask` publishes the path in `base_link` relative to the robot at dispatch. Without transforming the ideal path to world frame, odom (world-fixed) would be compared against the wrong reference and error would be meaningless. + +--- + +## Results pipeline + +Every `airstack test` run writes: + +``` +tests/results// +├── summary.txt ← open this first (human-readable) +├── results.xml ← JUnit pass/fail + durations +└── metrics.json ← structured metrics for diff tools +``` + +| Artifact | Producer | Use | +| -------- | -------- | --- | +| `summary.txt` | `tests/run_summary.py` (auto at session end via `conftest.py`) | Quick pass/fail + key numbers per trajectory type | +| `results.xml` | pytest `--junitxml` | CI, phase wall times | +| `metrics.json` | `MetricsRecorder` in `conftest.py` | Regression diffs | + +### Regenerate or inspect + +```bash +# Latest run +LATEST=$(ls -1t tests/results/ | head -1) + +# Human summary +cat "tests/results/$LATEST/summary.txt" + +# Regenerate summary manually +python3 tests/run_summary.py "tests/results/$LATEST/" + +# Markdown table of all metrics +python3 tests/parse_metrics.py --current "tests/results/$LATEST/" + +# Compare two tracker configs +python3 tests/parse_metrics.py \ + --current "tests/results/$NEW/" \ + --baseline "tests/results/$OLD/" \ + --threshold 20 \ + --output report.md +``` + +`parse_metrics.py` exits **1** when any metric regresses beyond the threshold percentage. + +--- + +## Running tests (complete CLI reference) + +### Prerequisites + +See **[Testing → Prerequisites](index.md#prerequisites)** for the shared setup (Docker daemon, NVIDIA GPU + `nvidia-container-toolkit`, and Isaac Sim `omni_pass.env`). + +### Primary interface + +```bash +airstack test [pytest options] +``` + +All arguments are forwarded to pytest inside the containerized test runner (`tests/docker/`). + +### Rebuild after C++ changes + +```bash +airstack test -m build_packages -v +``` + +Always run this after modifying `trajectory_controller`, `trajectory_library`, or launch params before flight tests. + +### Fixed-trajectory commands + +```bash +# Quick Circle regression (recommended smoke test) +airstack test -m "build_packages or autonomy" \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + -v + +# All four trajectory types, ms-airsim +airstack test -m autonomy \ + --sim msairsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle,Figure8,Racetrack,Line \ + -v + +# Stress: 10 iterations (statistical stability) +airstack test -m autonomy \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 10 \ + --trajectory-types Circle \ + -v + +# Visual debug (sim GUI) +airstack test -m autonomy \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + --gui \ + -v + +# Run only the trajectory phase (debugging) +airstack test -m autonomy \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + -k test_fixed_trajectory \ + -v +``` + +### Global CLI options + +| Option | Default | Description | +| ------ | ------- | ----------- | +| `--sim` | `msairsim,isaacsim` | Comma-separated sim targets | +| `--num-robots` | `1,3` | Comma-separated robot counts | +| `--stress-iterations` | `1` | Repeat count per `(sim, num_robots)` | +| `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | Trajectory sweep | +| `--gui` | off | Show simulator windows | +| `-v` | — | Verbose pytest | +| `-k EXPR` | — | Filter test names | + +### Direct pytest (local Python env) + +For faster iteration when editing test code: + +```bash +export AIRSTACK_ROOT=$(pwd) +pip install -r tests/requirements.txt + +pytest tests/ -m autonomy \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + -v +``` + +### CI: `/pytest` PR comment + +Core contributors can trigger runs by commenting on the PR: + +``` +/pytest -m "build_packages or autonomy" --sim isaacsim --num-robots 1 --stress-iterations 1 --trajectory-types Circle -v +``` + +The workflow auto-prepends `build_packages` when not already specified. + +--- + +## Comparing path trackers + +### What to change + +| Layer | Location | Examples | +| ----- | -------- | -------- | +| Tracker params | `robot/ros_ws/src/local/local_bringup/launch/local.launch.xml` (or `local_droan_cpu.launch.xml`) | `sphere_radius`, `look_ahead_time`, `search_ahead_factor`, `min_virtual_tracking_velocity` | +| Tracker implementation | Replace or fork `trajectory_controller` node | Alternative pure-pursuit, different intersection logic | +| Low-level control | Swap `pid_controller` for `attitude_controller` in launch | Changes end-to-end error, not tracker-only | + +Key `trajectory_controller` parameters today: + +| Param | Current value | Role | +| ----- | ------------- | ---- | +| `sphere_radius` | `2.0` | Lookahead sphere radius (m) | +| `look_ahead_time` | `1.0` | Look-ahead horizon for local planner feed | +| `virtual_tracking_ahead_time` | `0.5` | Virtual tracking search window | +| `min_virtual_tracking_velocity` | `0.5` | Below this, time-advance mode instead of sphere mode | +| `search_ahead_factor` | `1.5` | Multiplier on sphere radius when searching intersection | + +### Recommended A/B workflow + +```bash +# 1. Baseline run +airstack test -m "build_packages or autonomy" \ + --sim isaacsim --num-robots 1 --stress-iterations 5 \ + --trajectory-types Circle -v +BASELINE=$(ls -1t tests/results/ | head -1) + +# 2. Edit tracker params in local.launch.xml, rebuild +airstack test -m build_packages -v + +# 3. Candidate run +airstack test -m autonomy \ + --sim isaacsim --num-robots 1 --stress-iterations 5 \ + --trajectory-types Circle -v +CURRENT=$(ls -1t tests/results/ | head -1) + +# 4. Diff +python3 tests/parse_metrics.py \ + --current "tests/results/$CURRENT/" \ + --baseline "tests/results/$BASELINE/" \ + --threshold 20 +``` + +Focus on: `cross_track_error_mean_m`, `cross_track_error_max_m`, `path_rmse_m`, `trajectory_execution_time_sim_s`, `trajectory_success`. + +--- + +## Manual stack usage (without pytest) + +Bring up the stack and take off as described in **[Getting Started](../../../getting_started/index.md)** (`airstack up`, then use the RViz task panel). Once the drone is hovering, dispatch a fixed trajectory directly: + +```bash +docker exec -it airstack-robot-desktop-1 bash -c ' + source /opt/ros/jazzy/setup.bash && + source /root/AirStack/robot/ros_ws/install/setup.bash && + ros2 action send_goal --feedback /robot_1/tasks/fixed_trajectory \ + task_msgs/action/FixedTrajectoryTask \ + "{trajectory_spec: {type: Circle, attributes: [{key: radius, value: \"10.0\"}, {key: velocity, value: \"2.0\"}]}, loop: false}" +' +``` + +Action server: `/{robot_name}/tasks/fixed_trajectory` — see also [Tasks and Task Executors](../../../robot/autonomy/tasks.md). + +--- + +## Troubleshooting + +| Symptom | Likely cause | Fix | +| ------- | ------------ | --- | +| Sentinel nodes missing | Workspace not built in container | `-m "build_packages or autonomy"` | +| PX4 ready timeout | Sim not running, GPU issue | Check `nvidia-smi`, Isaac `omni_pass.env` | +| `trajectory_success = 0` | Tracker stall or timeout | Check trajectory_controller logs; rebuild the workspace (`-m build_packages`) | +| Cross-track error >> 5 m | Wrong tracker params or frame bug | Compare launch params; check world-frame transform | +| Tests run for hours | Default `--sim` and `--num-robots` sweep | Pin `--sim isaacsim --num-robots 1 --stress-iterations 1` | +| Unknown mark warning `autonomy` | Mark not in `pytest.ini` | Harmless; filter still works | + +--- + +## Source file reference + +| File | Role | +| ---- | ---- | +| [`tests/system/test_fixed_trajectory.py`](../../../../tests/system/test_fixed_trajectory.py) | Test module, ideal paths, metrics | +| [`tests/conftest.py`](../../../../tests/conftest.py) | Fixtures, `--trajectory-types`, summary hook, collection order | +| [`tests/run_summary.py`](../../../../tests/run_summary.py) | `summary.txt` generator | +| [`tests/parse_metrics.py`](../../../../tests/parse_metrics.py) | Markdown reports + regression diff | +| [`tests/pytest.ini`](../../../../tests/pytest.ini) | Registered marks | +| [`robot/.../fixed_trajectory_task.cpp`](../../../../robot/ros_ws/src/local/controls/trajectory_controller/src/fixed_trajectory_task.cpp) | C++ reference path generators | +| [`robot/.../trajectory_controller.cpp`](../../../../robot/ros_ws/src/local/controls/trajectory_controller/src/trajectory_controller.cpp) | Pure-pursuit path tracker | +| [`robot/.../trajectory_library.cpp`](../../../../robot/ros_ws/src/local/planners/trajectory_library/src/trajectory_library.cpp) | Trajectory math, sphere intersection | +| [`robot/.../local.launch.xml`](../../../../robot/ros_ws/src/local/local_bringup/launch/local.launch.xml) | Tracker + PID params | + +--- + +## Related documentation + +- [System tests overview (`tests/README.md`)](../../../../tests/README.md) +- [Trajectory Controller README](../../../../robot/ros_ws/src/local/controls/trajectory_controller/README.md) +- [Tasks and Task Executors](../../../robot/autonomy/tasks.md) +- [CI/CD orchestrator](../../../../tests/ci-cd-orchestrator.md) diff --git a/docs/development/intermediate/testing/index.md b/docs/development/intermediate/testing/index.md index 0769e2b09..e6ce09c0b 100644 --- a/docs/development/intermediate/testing/index.md +++ b/docs/development/intermediate/testing/index.md @@ -1,6 +1,6 @@ # Testing -AirStack uses three complementary test layers, each with a distinct scope and +AirStack uses several complementary test layers, each with a distinct scope and hardware requirement: | Layer | Where | Mark / Tool | Hardware | @@ -34,13 +34,17 @@ Full Docker-stack integration tests. The canonical reference is | Mark | Module | Role | |---|---|---| +| `build_docker` | `system/test_build_docker.py` | Docker image builds | +| `build_packages` | `system/test_build_packages.py` | `colcon build` inside containers | | `liveliness` | `system/test_liveliness.py` | Containers, `/clock` readiness, tmux, sentinel ROS 2 nodes, compute, infra-only stability poll | | `sensors` | `system/test_sensors.py` | Sim + robot stereo/depth Hz, filtered LiDAR (`echo --once` + validation script on Isaac), sim RTF, sensor stability time-series | -| `takeoff_hover_land` | `system/test_takeoff_hover_land.py` | Four-phase flight chain per configuration | +| `takeoff_hover_land` | `system/test_takeoff_hover_land.py` | Four-phase flight chain per configuration (takeoff → hover → land) | +| `autonomy` | `system/test_fixed_trajectory.py` | Fixed-pattern path-tracker benchmark (takeoff → trajectory → land) | -Collection order is defined in `tests/conftest.py` (`liveliness` before `sensors` -before `takeoff_hover_land`). Each mark's test class uses **class-scoped** -`airstack_env`, so combining marks with `and` runs multiple full stack bring-ups +Collection order is defined in `tests/conftest.py` (unit tests first, then +`build_docker` → `build_packages` → `liveliness` → `sensors` → `takeoff_hover_land` +→ `test_fixed_trajectory`). Each mark's test **class** uses **class-scoped** +`airstack_env`, so combining marks with **`or`** runs multiple full stack bring-ups per `(sim, num_robots, iteration)` — see *Bring-up scope* in `tests/README.md`. **Isaac Sim:** the `sensors` implementation batches `ros2 topic hz` on sim and @@ -48,6 +52,35 @@ robot paths and avoids `hz` on filtered `PointCloud2`; pytest enables `ENABLE_LI for the multi-drone Pegasus script. Details: **`tests/README.md`** → *Isaac Sim and the sensors mark*. +### Prerequisites + +All system tests share the same setup: + +```bash +cd /path/to/AirStack +airstack setup +``` + +- Docker daemon (user in `docker` group) +- NVIDIA GPU + `nvidia-container-toolkit` for sim tests +- Isaac Sim: `simulation/isaac-sim/docker/omni_pass.env` configured + +### End-to-end testing + +For the full guide — e2e overview, the fixed-trajectory benchmark, metrics, CLI, +comparing trackers, and baselines — see **[End-to-End Testing](end_to_end_testing.md)**. + +Quick smoke test: + +```bash +airstack test -m "build_packages or autonomy" \ + --sim isaacsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + -v +``` + ## Other testing docs - [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, proxy pattern, CI workflow diff --git a/git-hooks/docker-versioning/update-docker-image-tag_BACKUP_3660135.pre-commit b/git-hooks/docker-versioning/update-docker-image-tag_BACKUP_3660135.pre-commit deleted file mode 100755 index 43c8e1d00..000000000 --- a/git-hooks/docker-versioning/update-docker-image-tag_BACKUP_3660135.pre-commit +++ /dev/null @@ -1,150 +0,0 @@ -<<<<<<< HEAD -<<<<<<< HEAD - -======= ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key -======= ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key -#!/bin/bash -# Pre-commit hook to update VERSION in .env file with git commit hash -# when Dockerfile is modified or docker-compose.yaml has changes under build: key - -# Check if any Dockerfile files are staged for commit -DOCKERFILE_CHANGED=$(git diff --cached --name-only | grep -E 'Dockerfile$') -<<<<<<< HEAD -<<<<<<< HEAD -======= ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - -# Check if docker-compose.yaml has changes under build: key -COMPOSE_BUILD_CHANGED=false -COMPOSE_FILES=$(git diff --cached --name-only | grep -E 'docker-compose\.yaml$') - -if [ -n "$COMPOSE_FILES" ]; then - for file in $COMPOSE_FILES; do - # Get the diff for the docker-compose.yaml file - DIFF_OUTPUT=$(git diff --cached "$file") -<<<<<<< HEAD - -======= - -# Check if docker-compose.yaml has changes under build: key -COMPOSE_BUILD_CHANGED=false -COMPOSE_FILES=$(git diff --cached --name-only | grep -E 'docker-compose\.yaml$') - -if [ -n "$COMPOSE_FILES" ]; then - for file in $COMPOSE_FILES; do - # Get the diff for the docker-compose.yaml file - DIFF_OUTPUT=$(git diff --cached "$file") - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - # Check if any lines with changes (+ or -) contain build: or are indented under a build: section - # This regex looks for lines that: - # 1. Start with + or - (indicating changes) - # 2. Either contain "build:" directly, or - # 3. Are indented lines that could be under a build: section - if echo "$DIFF_OUTPUT" | grep -E '^[+-].*build:' > /dev/null; then - COMPOSE_BUILD_CHANGED=true - break - fi -<<<<<<< HEAD - -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - # More sophisticated check: look for changes in build context - # Extract the full diff and check for build-related changes - BUILD_SECTION_CHANGED=$(echo "$DIFF_OUTPUT" | awk ' - /^[+-].*build:/ { in_build=1; print; next } - /^[+-]/ && in_build && /^[+-][[:space:]]+/ { print; next } - /^[+-]/ && !/^[+-][[:space:]]/ { in_build=0 } - /^[+-].*build:/ { print } - ') -<<<<<<< HEAD - -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key -======= - - # Check if any lines with changes (+ or -) contain build: or are indented under a build: section - # This regex looks for lines that: - # 1. Start with + or - (indicating changes) - # 2. Either contain "build:" directly, or - # 3. Are indented lines that could be under a build: section - if echo "$DIFF_OUTPUT" | grep -E '^[+-].*build:' > /dev/null; then - COMPOSE_BUILD_CHANGED=true - break - fi - - # More sophisticated check: look for changes in build context - # Extract the full diff and check for build-related changes - BUILD_SECTION_CHANGED=$(echo "$DIFF_OUTPUT" | awk ' - /^[+-].*build:/ { in_build=1; print; next } - /^[+-]/ && in_build && /^[+-][[:space:]]+/ { print; next } - /^[+-]/ && !/^[+-][[:space:]]/ { in_build=0 } - /^[+-].*build:/ { print } - ') - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - if [ -n "$BUILD_SECTION_CHANGED" ]; then - COMPOSE_BUILD_CHANGED=true - break - fi - done -fi - -on_gh_pages=$([ "$(git rev-parse --abbrev-ref HEAD)" = "gh-pages" ] && echo true || echo false) - -# Trigger update if we're not on gh-pages and either Dockerfile changed or build section in docker-compose.yaml changed -if [ "$on_gh_pages" = false ] && ([ -n "$DOCKERFILE_CHANGED" ] || [ "$COMPOSE_BUILD_CHANGED" = true ]); then - if [ -n "$DOCKERFILE_CHANGED" ]; then - echo "Dockerfile changed. Updating VERSION in .env file..." - fi - if [ "$COMPOSE_BUILD_CHANGED" = true ]; then - echo "docker-compose.yaml build configuration changed. Updating VERSION in .env file..." - fi -<<<<<<< HEAD -<<<<<<< HEAD - -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - # Get the current commit hash (short version) - COMMIT_HASH=$(git rev-parse --short HEAD) - - # Update the VERSION in .env file - if [ -f ".env" ]; then - # Check if VERSION line exists - if grep -q "^VERSION=" .env; then - # Replace the existing VERSION line and ensure comment is above it - # First, remove any existing auto-generated comment - sed -i '/^# auto-generated from git commit hash$/d' .env - # Add the comment above the VERSION line - sed -i '/^VERSION=/i\# auto-generated from git commit hash' .env - # Update the VERSION value - sed -i "s/^VERSION=.*$/VERSION=\"$COMMIT_HASH\"/" .env - echo "Updated VERSION to $COMMIT_HASH in .env file" - - # Stage the modified .env file for commit - git add .env - else - echo "Error: VERSION line not found in .env file" - exit 1 - fi - else - echo "Error: .env file not found" - exit 1 - fi -fi -<<<<<<< HEAD -<<<<<<< HEAD -exit 0 -======= -======= ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key -exit 0 ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key diff --git a/git-hooks/docker-versioning/update-docker-image-tag_BASE_3660135.pre-commit b/git-hooks/docker-versioning/update-docker-image-tag_BASE_3660135.pre-commit deleted file mode 100644 index a7bd36a2b..000000000 --- a/git-hooks/docker-versioning/update-docker-image-tag_BASE_3660135.pre-commit +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash - -# Pre-commit hook to update VERSION in .env file with git commit hash -# when Dockerfile or docker-compose.yaml files are modified - -# Check if any Dockerfile or docker-compose.yaml files are staged for commit -DOCKER_FILES_CHANGED=$(git diff --cached --name-only | grep -E 'Dockerfile|docker-compose\.yaml$') - -on_gh_pages=$([ "$(git rev-parse --abbrev-ref HEAD)" = "gh-pages" ] && echo true || echo false) - -if [ "$on_gh_pages" = false ] && [ -n "$DOCKER_FILES_CHANGED" ]; then - echo "Docker-related files changed. Updating VERSION in .env file..." - - # Get the current commit hash (short version) - COMMIT_HASH=$(git rev-parse --short HEAD) - - # Update the VERSION in .env file - if [ -f ".env" ]; then - # Check if VERSION line exists - if grep -q "^VERSION=" .env; then - # Replace the existing VERSION line and ensure comment is above it - # First, remove any existing auto-generated comment - sed -i '/^# auto-generated from git commit hash$/d' .env - # Add the comment above the VERSION line - sed -i '/^VERSION=/i\# auto-generated from git commit hash' .env - # Update the VERSION value - sed -i "s/^VERSION=.*$/VERSION=\"$COMMIT_HASH\"/" .env - echo "Updated VERSION to $COMMIT_HASH in .env file" - - # Stage the modified .env file for commit - git add .env - else - echo "Error: VERSION line not found in .env file" - exit 1 - fi - else - echo "Error: .env file not found" - exit 1 - fi -fi - -exit 0 diff --git a/git-hooks/docker-versioning/update-docker-image-tag_LOCAL_3660135.pre-commit b/git-hooks/docker-versioning/update-docker-image-tag_LOCAL_3660135.pre-commit deleted file mode 100644 index 8cddc2b65..000000000 --- a/git-hooks/docker-versioning/update-docker-image-tag_LOCAL_3660135.pre-commit +++ /dev/null @@ -1,114 +0,0 @@ -<<<<<<< HEAD - -======= ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key -#!/bin/bash -# Pre-commit hook to update VERSION in .env file with git commit hash -# when Dockerfile is modified or docker-compose.yaml has changes under build: key - -# Check if any Dockerfile files are staged for commit -DOCKERFILE_CHANGED=$(git diff --cached --name-only | grep -E 'Dockerfile$') -<<<<<<< HEAD - -# Check if docker-compose.yaml has changes under build: key -COMPOSE_BUILD_CHANGED=false -COMPOSE_FILES=$(git diff --cached --name-only | grep -E 'docker-compose\.yaml$') - -if [ -n "$COMPOSE_FILES" ]; then - for file in $COMPOSE_FILES; do - # Get the diff for the docker-compose.yaml file - DIFF_OUTPUT=$(git diff --cached "$file") - -======= - -# Check if docker-compose.yaml has changes under build: key -COMPOSE_BUILD_CHANGED=false -COMPOSE_FILES=$(git diff --cached --name-only | grep -E 'docker-compose\.yaml$') - -if [ -n "$COMPOSE_FILES" ]; then - for file in $COMPOSE_FILES; do - # Get the diff for the docker-compose.yaml file - DIFF_OUTPUT=$(git diff --cached "$file") - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - # Check if any lines with changes (+ or -) contain build: or are indented under a build: section - # This regex looks for lines that: - # 1. Start with + or - (indicating changes) - # 2. Either contain "build:" directly, or - # 3. Are indented lines that could be under a build: section - if echo "$DIFF_OUTPUT" | grep -E '^[+-].*build:' > /dev/null; then - COMPOSE_BUILD_CHANGED=true - break - fi -<<<<<<< HEAD - -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - # More sophisticated check: look for changes in build context - # Extract the full diff and check for build-related changes - BUILD_SECTION_CHANGED=$(echo "$DIFF_OUTPUT" | awk ' - /^[+-].*build:/ { in_build=1; print; next } - /^[+-]/ && in_build && /^[+-][[:space:]]+/ { print; next } - /^[+-]/ && !/^[+-][[:space:]]/ { in_build=0 } - /^[+-].*build:/ { print } - ') -<<<<<<< HEAD - -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - if [ -n "$BUILD_SECTION_CHANGED" ]; then - COMPOSE_BUILD_CHANGED=true - break - fi - done -fi - -on_gh_pages=$([ "$(git rev-parse --abbrev-ref HEAD)" = "gh-pages" ] && echo true || echo false) - -# Trigger update if we're not on gh-pages and either Dockerfile changed or build section in docker-compose.yaml changed -if [ "$on_gh_pages" = false ] && ([ -n "$DOCKERFILE_CHANGED" ] || [ "$COMPOSE_BUILD_CHANGED" = true ]); then - if [ -n "$DOCKERFILE_CHANGED" ]; then - echo "Dockerfile changed. Updating VERSION in .env file..." - fi - if [ "$COMPOSE_BUILD_CHANGED" = true ]; then - echo "docker-compose.yaml build configuration changed. Updating VERSION in .env file..." - fi -<<<<<<< HEAD - -======= - ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key - # Get the current commit hash (short version) - COMMIT_HASH=$(git rev-parse --short HEAD) - - # Update the VERSION in .env file - if [ -f ".env" ]; then - # Check if VERSION line exists - if grep -q "^VERSION=" .env; then - # Replace the existing VERSION line and ensure comment is above it - # First, remove any existing auto-generated comment - sed -i '/^# auto-generated from git commit hash$/d' .env - # Add the comment above the VERSION line - sed -i '/^VERSION=/i\# auto-generated from git commit hash' .env - # Update the VERSION value - sed -i "s/^VERSION=.*$/VERSION=\"$COMMIT_HASH\"/" .env - echo "Updated VERSION to $COMMIT_HASH in .env file" - - # Stage the modified .env file for commit - git add .env - else - echo "Error: VERSION line not found in .env file" - exit 1 - fi - else - echo "Error: .env file not found" - exit 1 - fi -fi -<<<<<<< HEAD -exit 0 -======= -exit 0 ->>>>>>> 9181923d... Only update image tag if docker-compose.yaml has a change to the 'build:' key diff --git a/git-hooks/docker-versioning/update-docker-image-tag_REMOTE_3660135.pre-commit b/git-hooks/docker-versioning/update-docker-image-tag_REMOTE_3660135.pre-commit deleted file mode 100644 index 656e19706..000000000 --- a/git-hooks/docker-versioning/update-docker-image-tag_REMOTE_3660135.pre-commit +++ /dev/null @@ -1,81 +0,0 @@ -#!/bin/bash -# Pre-commit hook to update VERSION in .env file with git commit hash -# when Dockerfile is modified or docker-compose.yaml has changes under build: key - -# Check if any Dockerfile files are staged for commit -DOCKERFILE_CHANGED=$(git diff --cached --name-only | grep -E 'Dockerfile$') - -# Check if docker-compose.yaml has changes under build: key -COMPOSE_BUILD_CHANGED=false -COMPOSE_FILES=$(git diff --cached --name-only | grep -E 'docker-compose\.yaml$') - -if [ -n "$COMPOSE_FILES" ]; then - for file in $COMPOSE_FILES; do - # Get the diff for the docker-compose.yaml file - DIFF_OUTPUT=$(git diff --cached "$file") - - # Check if any lines with changes (+ or -) contain build: or are indented under a build: section - # This regex looks for lines that: - # 1. Start with + or - (indicating changes) - # 2. Either contain "build:" directly, or - # 3. Are indented lines that could be under a build: section - if echo "$DIFF_OUTPUT" | grep -E '^[+-].*build:' > /dev/null; then - COMPOSE_BUILD_CHANGED=true - break - fi - - # More sophisticated check: look for changes in build context - # Extract the full diff and check for build-related changes - BUILD_SECTION_CHANGED=$(echo "$DIFF_OUTPUT" | awk ' - /^[+-].*build:/ { in_build=1; print; next } - /^[+-]/ && in_build && /^[+-][[:space:]]+/ { print; next } - /^[+-]/ && !/^[+-][[:space:]]/ { in_build=0 } - /^[+-].*build:/ { print } - ') - - if [ -n "$BUILD_SECTION_CHANGED" ]; then - COMPOSE_BUILD_CHANGED=true - break - fi - done -fi - -on_gh_pages=$([ "$(git rev-parse --abbrev-ref HEAD)" = "gh-pages" ] && echo true || echo false) - -# Trigger update if we're not on gh-pages and either Dockerfile changed or build section in docker-compose.yaml changed -if [ "$on_gh_pages" = false ] && ([ -n "$DOCKERFILE_CHANGED" ] || [ "$COMPOSE_BUILD_CHANGED" = true ]); then - if [ -n "$DOCKERFILE_CHANGED" ]; then - echo "Dockerfile changed. Updating VERSION in .env file..." - fi - if [ "$COMPOSE_BUILD_CHANGED" = true ]; then - echo "docker-compose.yaml build configuration changed. Updating VERSION in .env file..." - fi - - # Get the current commit hash (short version) - COMMIT_HASH=$(git rev-parse --short HEAD) - - # Update the VERSION in .env file - if [ -f ".env" ]; then - # Check if VERSION line exists - if grep -q "^VERSION=" .env; then - # Replace the existing VERSION line and ensure comment is above it - # First, remove any existing auto-generated comment - sed -i '/^# auto-generated from git commit hash$/d' .env - # Add the comment above the VERSION line - sed -i '/^VERSION=/i\# auto-generated from git commit hash' .env - # Update the VERSION value - sed -i "s/^VERSION=.*$/VERSION=\"$COMMIT_HASH\"/" .env - echo "Updated VERSION to $COMMIT_HASH in .env file" - - # Stage the modified .env file for commit - git add .env - else - echo "Error: VERSION line not found in .env file" - exit 1 - fi - else - echo "Error: .env file not found" - exit 1 - fi -fi -exit 0 diff --git a/mkdocs.yml b/mkdocs.yml index 4dc57b961..e75d85bc7 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -69,6 +69,7 @@ nav: - Overview: docs/development/intermediate/testing/index.md - Unit Testing: docs/development/intermediate/testing/unit_testing.md - System Tests: tests/README.md + - End-to-End Testing: docs/development/intermediate/testing/end_to_end_testing.md - CI/CD Orchestrator: tests/ci-cd-orchestrator.md - Frame Conventions: docs/development/intermediate/frame_conventions.md - Docker Build Profiles: docs/development/intermediate/docker-build-profiles.md diff --git a/robot/ros_ws/src/local/controls/trajectory_controller/src/trajectory_controller.cpp b/robot/ros_ws/src/local/controls/trajectory_controller/src/trajectory_controller.cpp index ffc3fac2b..1dd389fe3 100644 --- a/robot/ros_ws/src/local/controls/trajectory_controller/src/trajectory_controller.cpp +++ b/robot/ros_ws/src/local/controls/trajectory_controller/src/trajectory_controller.cpp @@ -302,7 +302,18 @@ void TrajectoryControlNode::timer_callback() { virtual_time, search_ahead_factor * get_sphere_radius(closest_ahead_wp.velocity().length()), prev_vtp_time + look_ahead_time, robot_point, get_sphere_radius(closest_ahead_wp.velocity().length()), min_virtual_tracking_velocity, &vtp_wp, &end_wp); - if (vtp_valid) current_virtual_ahead_time = vtp_wp.get_time() - virtual_time; + if (vtp_valid) { + current_virtual_ahead_time = vtp_wp.get_time() - virtual_time; + } else { + // Keep the tracking point ahead when sphere intersection fails so the + // controller does not collapse onto the robot projection and stall. + Waypoint ahead_wp(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0); + if (trajectory->get_waypoint_distance_ahead( + virtual_time, get_sphere_radius(closest_ahead_wp.velocity().length()), + &ahead_wp)) { + current_virtual_ahead_time = ahead_wp.get_time() - virtual_time; + } + } // visualization if (vtp_valid) @@ -320,14 +331,17 @@ void TrajectoryControlNode::timer_callback() { .add_sphere(target_frame, now, end_wp.get_x(), end_wp.get_y(), end_wp.get_z(), 0.025f) .set_color(0.f, 0.f, 1.f); - } else{ - RCLCPP_INFO(this->get_logger(), "AHEAD NOT VALID"); - - markers - .add_sphere(target_frame, now, robot_point.x(), - robot_point.y(), robot_point.z(), 1.) - .set_color(1.f, 0.f, 0.f, 0.7f); - } + } else { + RCLCPP_WARN_THROTTLE(this->get_logger(), *this->get_clock(), 5000, + "AHEAD NOT VALID — advancing virtual_time by elapsed sim time"); + virtual_time = std::min(trajectory->get_duration(), + virtual_time + time_multiplier * execute_elapsed); + + markers + .add_sphere(target_frame, now, robot_point.x(), robot_point.y(), robot_point.z(), + 1.) + .set_color(1.f, 0.f, 0.f, 0.7f); + } } else { if (new_rewind) { float before = virtual_time; diff --git a/robot/ros_ws/src/local/planners/trajectory_library/src/trajectory_library.cpp b/robot/ros_ws/src/local/planners/trajectory_library/src/trajectory_library.cpp index 5f35a379a..f3ac7bd60 100644 --- a/robot/ros_ws/src/local/planners/trajectory_library/src/trajectory_library.cpp +++ b/robot/ros_ws/src/local/planners/trajectory_library/src/trajectory_library.cpp @@ -390,6 +390,7 @@ bool Trajectory::merge(Trajectory traj, double min_time) { if (waypoints.size() == 0) { waypoints.insert(waypoints.end(), transformed_traj.waypoints.begin(), transformed_traj.waypoints.end()); + generate_waypoint_times(); return true; } @@ -501,15 +502,15 @@ bool Trajectory::get_waypoint_sphere_intersection(double initial_time, double ah Waypoint wp_end = waypoints[i]; last_waypoint_index = i; - // if the very first waypoint we check isn't within the sphere, then return not found - if(i == 1 && wp_start.position().distance(sphere_center) > sphere_radius) - return false; - // handle the case that the initial_time is between waypoint i-1 and waypoint i if (wp_start.get_time() < initial_time) wp_start = wp_start.interpolate(wp_end, (initial_time - wp_start.get_time()) / (wp_end.get_time() - wp_start.get_time())); + // if the first segment we check starts outside the sphere, there is no intersection + if (i == 1 && wp_start.position().distance(sphere_center) > sphere_radius) + return false; + // sphere line intersection equations: // http://www.ambrsoft.com/TrigoCalc/Sphere/SpherLineIntersection_.htm double x1 = wp_start.get_x(); diff --git a/tests/README.md b/tests/README.md index f10942ff7..6aec42e1b 100644 --- a/tests/README.md +++ b/tests/README.md @@ -23,6 +23,7 @@ Shared fixtures live in `tests/conftest.py`. Use `airstack test -m unit -v` for | [`system/test_liveliness.py`](system/test_liveliness.py) | `liveliness` | Stack bring-up: container Running state, ``/clock`` readiness, tmux panes, sentinel ROS 2 nodes, compute snapshot, infra-only ``test_stable`` (tmux + nodes + compute) | Docker daemon, GPU, sim license | | [`system/test_sensors.py`](system/test_sensors.py) | `sensors` | After liveliness in collection order: sim + robot stereo/depth Hz (**Isaac:** batched ``ros2 topic hz`` to avoid bridge overload; **ms-airsim:** single batch), filtered LiDAR via ``echo --once`` + cloud sanity (isaacsim), sim RTF, ``test_sensor_streams_stable`` | Docker daemon, GPU, sim license | | [`system/test_takeoff_hover_land.py`](system/test_takeoff_hover_land.py) | `takeoff_hover_land` | End-to-end flight: PX4 readiness gate, takeoff to 10 m, hover stability, land — one chain per (sim, num_robots, iteration, velocity) | Docker daemon, GPU, sim license | +| [`system/test_fixed_trajectory.py`](system/test_fixed_trajectory.py) | `autonomy` | Fixed-pattern trajectory evaluation: takeoff, execute a trajectory (Circle, Figure8, Racetrack, Line), record path deviation metrics, land — one chain per (sim, num_robots, iteration, trajectory_type) | Docker daemon, GPU, sim license | ### Unit tests (`tests/robot/`, `tests/sim/`) @@ -43,11 +44,11 @@ See [Unit Testing Guide](../docs/development/intermediate/testing/unit_testing.m and the `add-unit-tests` agent skill for full details. Marks can be combined with pytest logic: -`-m unit`, `-m "build_docker or build_packages"`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). +`-m unit`, `-m "build_docker or build_packages"`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, `-m autonomy`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). ### Bring-up scope (`airstack_env`) -`airstack_env` is **class-scoped** and parametrized per `(sim, num_robots, iteration)`. Each test **class** that uses it (`TestLiveliness`, `TestSensors`, `TestTakeoffHoverLand`, …) performs its **own** ``airstack up`` / ``airstack down`` for that parametrization. Selecting both classes (for example, ``-m "liveliness or sensors"``) runs **two** full stack cycles per tuple (liveliness class, then sensors class). Collection order (see ``conftest.py``) runs **liveliness before sensors** when both are selected. To save wall time, run ``-m liveliness`` or ``-m sensors`` alone when one suite is enough. +`airstack_env` is **class-scoped** and parametrized per `(sim, num_robots, iteration)`. Each test **class** that uses it (`TestLiveliness`, `TestSensors`, `TestTakeoffHoverLand`, `TestFixedTrajectory`, …) performs its **own** ``airstack up`` / ``airstack down`` for that parametrization. Selecting both classes (for example, ``-m "liveliness or sensors"``) runs **two** full stack cycles per tuple (liveliness class, then sensors class). Collection order (see ``conftest.py``) runs **liveliness before sensors** when both are selected. To save wall time, run ``-m liveliness`` or ``-m sensors`` alone when one suite is enough. --- @@ -93,35 +94,23 @@ Writes custom metrics to `tests/results//metrics.json` after each `re ### Output files -Every test run produces a timestamped directory. **Per-test logs** — for each -pytest function, `pytest_runtest_setup` in `conftest.py` attaches the shared -logger to `logs/test_..[].log` (param ids are -rewritten for readability, e.g. `msairsim-rob#1-iter0`; see -`pytest_collection_modifyitems`). - -**`airstack_env.<…>.log`** — the class-scoped `airstack_env` fixture wraps -`airstack up` / `airstack down` in `logger_to("airstack_env." + )` -(see `conftest.py`). So you get an extra file whose name is the word -`airstack_env.` plus the **node id of whichever test was running when the -fixture first ran** for that class. For `TestLiveliness` that is almost always -`test_robot_containers_running` (first test in the class), not `test_stable`. -That file holds compose / `airstack` subprocess output; each test still has its -own log for assertions and `docker exec` / `ros2` lines. +Every test run produces a timestamped directory containing only `summary.txt`, +`results.xml`, and `metrics.json` — there is **no** `logs/` subdirectory and no +per-test log files are written under the run directory. ``` tests/results/ └── 2025-04-21_14-30-00/ + ├── summary.txt # Human-readable key metrics — open this first ├── results.xml # JUnit XML — test durations and pass/fail status - ├── metrics.json # Custom metrics (image sizes, Hz, compute, timing) - └── logs/ - ├── system.test_build_docker.TestDockerBuilds.test_build_robot_desktop.log - ├── airstack_env.system.test_liveliness.TestLiveliness.test_robot_containers_running[msairsim-rob#1-iter0].log - ├── system.test_liveliness.TestLiveliness.test_robot_containers_running[msairsim-rob#1-iter0].log - ├── system.test_liveliness.TestLiveliness.test_stable[msairsim-rob#1-iter0].log - ├── system.test_sensors.TestSensors.test_sensor_streams_stable[msairsim-rob#1-iter0].log - └── ... # More per-test logs; another airstack_env.* per class using the fixture + └── metrics.json # Custom metrics (image sizes, Hz, compute, timing) ``` +Live test output goes to the terminal (pytest `log_cli`). On failure, assertion +messages include the tail of the last subprocess output (the in-memory +`read_log_tail` of the relevant `docker` / `ros2` subprocess) — no per-test log +files are written under the run directory. + --- ## Running Tests @@ -175,7 +164,7 @@ can reach the host X server; it is a no-op when `DISPLAY` is not set. ### Prerequisites - Docker daemon running with your user in the `docker` group -- NVIDIA drivers + `nvidia-container-toolkit` for liveliness, sensors, and takeoff_hover_land tests +- NVIDIA drivers + `nvidia-container-toolkit` for liveliness, sensors, takeoff_hover_land, and autonomy tests - `airstack setup` completed (adds `airstack` to `PATH`) ### Direct pytest (for development / debugging) @@ -282,6 +271,87 @@ airstack test -m takeoff_hover_land \ --- +## Fixed Trajectory Tests (`system/test_fixed_trajectory.py`) + +!!! note "Detailed guide" + For the full end-to-end testing guide — architecture, the fixed-trajectory benchmark, metrics, CLI reference, comparing trackers, and baselines — see **[End-to-End Testing](../docs/development/intermediate/testing/end_to_end_testing.md)**. + +`TestFixedTrajectory` runs a **4-phase flight chain** for every combination of +`(sim, num_robots, iteration, trajectory_type)`. For each trajectory type the drone +takes off, executes the pattern, then lands — regardless of whether the trajectory +phase passes or fails (a trajectory failure does not skip landing). + +Supported trajectory types: `Circle`, `Figure8`, `Racetrack`, `Line` (same patterns as +the `fixed_trajectory_task` ROS 2 action server in `trajectory_controller`). + +### Phase order + +| Phase | Test | What happens | +| ----- | ---- | ------------ | +| 1 | `test_px4_ready` | Waits for MAVROS + PX4 EKF ready; once per env | +| 2 | `test_takeoff` | Takeoff to 10 m at 1 m/s; asserts altitude within 10 % | +| 3 | `test_fixed_trajectory` | Sends `FixedTrajectoryTask`; captures odom; asserts cross-track error | +| 4 | `test_landing` | Sends `LandTask`; asserts final altitude < 0.5 m | + +A failure in `test_fixed_trajectory` does **not** poison the chain — `test_landing` always +runs so the drone returns to the ground before the next trajectory type starts. + +### Recorded metrics + +| Metric key | Unit | Description | +| ---------- | ---- | ----------- | +| `ready_duration_sys_s` | s | Wall-clock time from test start until PX4 ready | +| `takeoff_duration_sim_s` | s | Sim-time from first motion to 95 % of target altitude | +| `altitude_error_m` | m | Signed steady-state altitude error after takeoff | +| `overshoot_m` | m | Unsigned transient overshoot above target | +| `trajectory_success` | — | 1.0 if action returned `success: true`, 0.0 otherwise (`higher_is_better`) | +| `trajectory_execution_time_sim_s` | s | Sim-time elapsed from action dispatch to completion | +| `cross_track_error_mean_m` | m | Mean 2-D lateral distance from nearest ideal-path point | +| `cross_track_error_max_m` | m | Worst-case lateral deviation | +| `path_rmse_m` | m | 2-D RMSE against the ideal path | +| `final_altitude_m` | m | Altitude at landing action completion | +| `land_duration_sim_s` | s | Sim-time from 80 % peak descent to < 0.5 m | + + +Metrics reported in one .txt file called summary.txt which automatically populates once your run completes + +### Default trajectory parameters + +| Type | Parameters | +| ---- | ---------- | +| Circle | radius=10 m, velocity=2 m/s | +| Figure8 | length=15 m, width=8 m, height=0 m, velocity=2 m/s, max_acceleration=1 m/s² | +| Racetrack | length=30 m, width=10 m, height=0 m, velocity=3 m/s, turn_velocity=1.5 m/s, max_acceleration=1 m/s² | +| Line | length=20 m, height=0 m, velocity=2 m/s, max_acceleration=1 m/s² | + +### Running fixed trajectory tests + +```bash +# All four trajectory types; ms-airsim; 1 robot +airstack test -m autonomy \ + --sim msairsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle,Figure8,Racetrack,Line \ + -v + +# Circle only (quick check of the known failure case) +airstack test -m autonomy \ + --sim msairsim \ + --num-robots 1 \ + --stress-iterations 1 \ + --trajectory-types Circle \ + -v +``` + +### CLI option reference (trajectory-specific) + +| Option | Default | Description | +|--------|---------|-------------| +| `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | Comma-separated trajectory types to sweep | + +--- + ## Metrics Reporting (`parse_metrics.py`) [`parse_metrics.py`](parse_metrics.py) reads `results.xml` and `metrics.json` from a run directory and produces a markdown report. It has two modes: diff --git a/tests/conftest.py b/tests/conftest.py index 31fd29076..2a51c569a 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -88,8 +88,9 @@ def colcon_test_robot_command(workspace="robot"): # `pytest tests/` and `airstack test -m unit` discover them without any # sys.path manipulation here. Each proxy file sets up its own paths. RUN_DIR = None -LOGS_DIR = None ROS_DISTRO_SETUP = "/opt/ros/jazzy/setup.bash" +_LAST_CMD_OUTPUT: dict[str, str] = {} +_DEFAULT_LOG_KEY = "_last" # Track the currently-running pytest item so current_log() and current_test_id() # can pick up the parametrize id without tests having to pass `request` around. @@ -99,8 +100,6 @@ def colcon_test_robot_command(workspace="robot"): logger = logging.getLogger("airstack") logger.setLevel(logging.INFO) -_LOG_FORMAT = logging.Formatter("[%(asctime)s] %(levelname)s %(message)s", "%H:%M:%S") -_test_log_handler = None # ── pytest config / hooks ────────────────────────────────────────────────── @@ -122,36 +121,42 @@ def pytest_addoption(parser): parser.addoption("--takeoff-velocities", default="0.5", help="Comma-separated takeoff/land velocities (m/s) to " "sweep in test_takeoff_hover_land. Default: 0.5,1,2") + parser.addoption("--trajectory-types", default="Circle,Figure8,Racetrack,Line", + help="Comma-separated fixed trajectory types to sweep in " + "test_fixed_trajectory. Default: Circle,Figure8,Racetrack,Line") def pytest_configure(config): - global RUN_DIR, LOGS_DIR + global RUN_DIR timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") results_root = Path(AIRSTACK_ROOT) / "tests" / "results" RUN_DIR = results_root / timestamp - LOGS_DIR = RUN_DIR / "logs" - LOGS_DIR.mkdir(parents=True, exist_ok=True) + RUN_DIR.mkdir(parents=True, exist_ok=True) config.option.xmlpath = str(RUN_DIR / "results.xml") def pytest_runtest_setup(item): - global _CURRENT_ITEM, _test_log_handler + global _CURRENT_ITEM _CURRENT_ITEM = item - log_path = LOGS_DIR / f"{current_log()}.log" - _test_log_handler = logging.FileHandler(log_path) - _test_log_handler.setFormatter(_LOG_FORMAT) - logger.addHandler(_test_log_handler) def pytest_runtest_teardown(item): - global _CURRENT_ITEM, _test_log_handler - if _test_log_handler is not None: - logger.removeHandler(_test_log_handler) - _test_log_handler.close() - _test_log_handler = None + global _CURRENT_ITEM _CURRENT_ITEM = None +def pytest_sessionfinish(session, exitstatus): + """Write summary.txt with key metrics so users don't need to dig through logs.""" + if RUN_DIR is None: + return + try: + from run_summary import write_summary + summary_path = write_summary(RUN_DIR) + logger.info("Wrote run summary to %s", summary_path) + except Exception as exc: + logger.warning("Failed to write run summary: %s", exc) + + @pytest.hookimpl(hookwrapper=True) def pytest_runtest_makereport(item, call): """Attach phase reports to the item so fixtures can inspect pass/fail.""" @@ -162,21 +167,8 @@ def pytest_runtest_makereport(item, call): @contextmanager def logger_to(log_name): - """Temporarily route `logger` to a different file. Suspends any handlers - already attached so narration isn't duplicated across files.""" - existing = list(logger.handlers) - for h in existing: - logger.removeHandler(h) - fh = logging.FileHandler(LOGS_DIR / f"{log_name}.log") - fh.setFormatter(_LOG_FORMAT) - logger.addHandler(fh) - try: - yield - finally: - logger.removeHandler(fh) - fh.close() - for h in existing: - logger.addHandler(h) + """No-op kept for fixture call sites; output goes to pytest log_cli only.""" + yield def pytest_generate_tests(metafunc): @@ -209,6 +201,7 @@ def pytest_generate_tests(metafunc): "system.test_liveliness", "system.test_sensors", "system.test_takeoff_hover_land", + "system.test_fixed_trajectory", ] # Within test_takeoff_hover_land, each (env, velocity) runs phases in this chain order. @@ -219,6 +212,20 @@ def pytest_generate_tests(metafunc): "test_landing", ] +# Within test_fixed_trajectory, each (env, trajectory_type) runs phases in this order. +_FIXED_TRAJ_PHASE_ORDER = [ + "test_px4_ready", + "test_takeoff", + "test_fixed_trajectory", + "test_landing", +] + +# Maps module name → phase order list for per-module chain sorting. +_MODULE_PHASE_ORDERS = { + "system.test_takeoff_hover_land": _AUTONOMY_PHASE_ORDER, + "system.test_fixed_trajectory": _FIXED_TRAJ_PHASE_ORDER, +} + def _rank(name, order): """Index of `name` in `order`; `len(order)` if unknown (i.e., sort last).""" @@ -242,32 +249,34 @@ def pytest_collection_modifyitems(items): # order intact, so pytest's default file/class order survives. items.sort(key=_module_key) - # 2. Within test_takeoff_hover_land: sort by (airstack_env, velocity, phase) so each - # (sim, robots, iter) env brings up the stack once and the drone goes - # ground→air→ground per velocity. - def phase(item): - if getattr(item.module, "__name__", "") != "system.test_takeoff_hover_land": - return None - name = item.originalname or item.name.split("[", 1)[0] - return _rank(name, _AUTONOMY_PHASE_ORDER) + # 2. Within each parametrized autonomy-style module, sort by + # (airstack_env, secondary_param, phase) so each env brings up the stack + # once and the drone goes ground→air→ground per secondary parameter. + for mod_name, phase_order in _MODULE_PHASE_ORDERS.items(): + def _phase(item, _order=phase_order, _mod=mod_name): + if getattr(item.module, "__name__", "") != _mod: + return None + name = item.originalname or item.name.split("[", 1)[0] + return _rank(name, _order) + + def _sort_key(item, _mod=mod_name): + cs = getattr(item, "callspec", None) + env = cs.params.get("airstack_env", ()) if cs else () + # test_takeoff_hover_land sweeps velocity; test_fixed_trajectory sweeps type + secondary = ( + float(cs.params["velocity"]) if cs and "velocity" in cs.params + else (cs.params.get("trajectory_type", "") if cs else "") + ) + return (env, secondary, _phase(item)) - def sort_key(item): - cs = getattr(item, "callspec", None) - env = cs.params.get("airstack_env", ()) if cs else () - vel = float(cs.params.get("velocity", 0.0)) if cs else 0.0 - return (env, vel, phase(item)) - - slots = [(i, it) for i, it in enumerate(items) if phase(it) is not None] - if slots: - sorted_items = sorted((it for _, it in slots), key=sort_key) - for (i, _), new_item in zip(slots, sorted_items): - items[i] = new_item - - # 3. Rewrite bracketed test IDs into a consistent hierarchy: sim > robots > - # velocity > iteration. Bypasses pytest's own concatenation (which would - # otherwise order by reverse-parametrize-call order). Keeps pytest console, - # JUnit XML, and metrics.json all in the same natural order without - # refactoring the parametrize structure. + slots = [(i, it) for i, it in enumerate(items) if _phase(it) is not None] + if slots: + sorted_items = sorted((it for _, it in slots), key=_sort_key) + for (i, _), new_item in zip(slots, sorted_items): + items[i] = new_item + + # 3. Rewrite bracketed test IDs into a consistent hierarchy: + # sim > robots > secondary param > iteration. for item in items: cs = getattr(item, "callspec", None) if cs is None: @@ -279,6 +288,8 @@ def sort_key(item): parts.append(f"{sim}-rob#{n}") if "velocity" in cs.params: parts.append(f"v{cs.params['velocity']}") + if "trajectory_type" in cs.params: + parts.append(f"traj{cs.params['trajectory_type']}") if env: parts.append(f"iter{i}") if not parts: @@ -310,31 +321,26 @@ def current_log(): def read_log_tail(log_name=None, lines=50): - log_name = log_name or current_log() - if not log_name: + """Return the tail of the most recent subprocess output for this context.""" + key = log_name or _DEFAULT_LOG_KEY + text = _LAST_CMD_OUTPUT.get(key) or _LAST_CMD_OUTPUT.get(_DEFAULT_LOG_KEY, "") + if not text: return "" - log_path = LOGS_DIR / f"{log_name}.log" - if log_path.exists(): - all_lines = log_path.read_text().splitlines() - return "\n".join(all_lines[-lines:]) - return "" + return "\n".join(text.splitlines()[-lines:]) def _run_teed(cmd_list, timeout, log_name=None, env=None, cwd=None): - """Run a subprocess, teeing stdout+stderr live to the log file and - capturing them for parsing.""" - log_name = log_name or current_log() - if not log_name: - return subprocess.run(cmd_list, capture_output=True, text=True, - timeout=timeout, env=env, cwd=cwd) - log_path = LOGS_DIR / f"{log_name}.log" + """Run a subprocess and capture stdout+stderr for parsing and failure messages.""" quoted = " ".join(shlex.quote(a) for a in cmd_list) - with open(log_path, "a") as f: - f.write(f"\n$ {quoted}\n") - shell_cmd = f"set -o pipefail; {quoted} 2>&1 | tee -a {shlex.quote(str(log_path))}" - return subprocess.run(["bash", "-c", shell_cmd], - capture_output=True, text=True, - timeout=timeout, env=env, cwd=cwd) + logger.info("$ %s", quoted) + result = subprocess.run( + cmd_list, capture_output=True, text=True, timeout=timeout, env=env, cwd=cwd, + ) + combined = (result.stdout or "") + (result.stderr or "") + key = log_name or _DEFAULT_LOG_KEY + _LAST_CMD_OUTPUT[key] = combined + _LAST_CMD_OUTPUT[_DEFAULT_LOG_KEY] = combined + return result def docker_exec(container, cmd, timeout=60, log_name=None): diff --git a/tests/pytest.ini b/tests/pytest.ini index a664ccbf3..03fee8c32 100644 --- a/tests/pytest.ini +++ b/tests/pytest.ini @@ -6,6 +6,7 @@ markers = liveliness: Container and process health (Docker, tmux, sentinel ROS 2 nodes) sensors: Sim and robot sensor topic rates, LiDAR validation, sim RTF takeoff_hover_land: End-to-end takeoff / hover / land action tests + autonomy: Fixed-pattern trajectory path-tracker benchmark (test_fixed_trajectory.py) testpaths = . addopts = -v --durations=0 cache_dir = /tmp/.pytest_cache diff --git a/tests/run_summary.py b/tests/run_summary.py new file mode 100644 index 000000000..c8ebaac48 --- /dev/null +++ b/tests/run_summary.py @@ -0,0 +1,398 @@ +"""Write a human-readable summary.txt for each test run. + +Called automatically at pytest session end (see conftest.py). Users can also +regenerate manually: + + python3 tests/run_summary.py tests/results// +""" +from __future__ import annotations + +import argparse +import json +import re +import statistics +import xml.etree.ElementTree as ET +from pathlib import Path + +PARAM_RE = re.compile(r"\[(.+)\]$") +ITER_RE = re.compile(r"-iter\d+$") +ROBOT_METRIC_RE = re.compile(r"^robot_\d+\.(.+)$") +# pytest nodeid path prefix (metrics.json) vs JUnit classname (results.xml) +MODULE_RE = re.compile(r"(?:^|\.)(test_\w+)\.Test") +PHASE_RE = re.compile(r"\.Test[A-Za-z0-9_]+\.(test_\w+)(?:\[|$)") + +# Ordered (metric_key, label) groups per test module. Only scalar metrics with +# a numeric "value" field are emitted. +FLIGHT_METRICS = [ + ("ready_duration_sys_s", "PX4 ready time"), + ("takeoff_duration_sim_s", "Takeoff duration"), + ("altitude_error_m", "Altitude error after takeoff"), + ("overshoot_m", "Takeoff overshoot"), + ("trajectory_success", "Trajectory success"), + ("trajectory_execution_time_sim_s", "Trajectory duration"), + ("cross_track_error_mean_m", "Cross-track error (mean)"), + ("cross_track_error_max_m", "Cross-track error (max)"), + ("path_rmse_m", "Path RMSE"), + ("hover_duration_sim_s", "Hover duration"), + ("hover_altitude_error_m", "Hover altitude error"), + ("land_duration_sim_s", "Landing duration"), + ("final_altitude_m", "Final altitude"), +] + +LIVELINESS_METRICS = [ + ("sim_ready_duration_s", "Sim ready time"), + ("sensors_sim_ready_duration_s", "Sensors sim ready time"), +] + +# Some metrics were recorded with wrong units before METRIC_UNITS was updated. +UNIT_OVERRIDES = { + "ready_duration_sys_s": "s", + "airstack_up_duration_s": "s", + "airstack_down_duration_s": "s", +} + +PHASE_ORDER = { + "test_px4_ready": 0, + "test_takeoff": 1, + "test_fixed_trajectory": 2, + "test_hover": 2, + "test_landing": 3, + "test_land": 3, +} + + +def _canonical_test_id(name: str) -> str: + """Unify metrics.json path slashes with JUnit classname dots. + + metrics.json keys look like ``system/test_fixed_trajectory.Class.test_x[...]`` + (pytest nodeid). results.xml uses ``system.test_fixed_trajectory.Class.test_x[...]``. + """ + head, dot, rest = name.partition(".") + if "/" in head: + head = head.replace("/", ".") + return head + dot + rest if dot else head + return name + + +def _normalize_keyed_map(raw: dict) -> dict: + """Merge entries that differ only by path-slash vs dot classname form.""" + out: dict = {} + for key, value in raw.items(): + out[_canonical_test_id(key)] = value + return out + + +def _parse_results_xml(path: Path) -> tuple[dict[str, str], dict[str, float]]: + """Return ({full_test_name: status}, {full_test_name: wall_time_s}).""" + if not path.exists(): + return {}, {} + statuses: dict[str, str] = {} + durations: dict[str, float] = {} + for tc in ET.parse(path).iter("testcase"): + full = _canonical_test_id(f"{tc.get('classname')}.{tc.get('name')}") + if tc.find("failure") is not None or tc.find("error") is not None: + statuses[full] = "FAILED" + elif tc.find("skipped") is not None: + statuses[full] = "SKIPPED" + else: + statuses[full] = "PASSED" + if tc.get("time"): + try: + durations[full] = float(tc.get("time")) + except ValueError: + pass + return statuses, durations + + +def _load_metrics(path: Path) -> dict: + if not path.exists(): + return {} + return _normalize_keyed_map(json.loads(path.read_text())) + + +def _param_id(test_name: str) -> str: + m = PARAM_RE.search(test_name) + return m.group(1) if m else test_name + + +def _module_name(test_name: str) -> str: + canonical = _canonical_test_id(test_name) + match = MODULE_RE.search(canonical) + if match: + return match.group(1) + return canonical.split(".", 1)[0] + + +def _phase_name(test_name: str) -> str: + """test_fixed_trajectory.TestFixedTrajectory.test_takeoff[...] -> test_takeoff""" + canonical = _canonical_test_id(test_name) + match = PHASE_RE.search(canonical) + if match: + return match.group(1) + return canonical.split("[", 1)[0] + + +def _base_param_id(param: str) -> str: + """isaacsim-rob#1-trajCircle-iter3 -> isaacsim-rob#1-trajCircle""" + return ITER_RE.sub("", param) + + +def _format_scalar(key: str, value: float | int, unit: str) -> str: + if key == "trajectory_success": + if value == 1.0: + return "yes" + if value == 0.0: + return "no" + text = f"{value:g}" + return f"{text} {unit}".strip() if unit else text + + +def _format_value(key: str, entry: dict) -> str: + value = entry.get("value") + unit = UNIT_OVERRIDES.get(key, entry.get("unit", "")) + if isinstance(value, (int, float)): + return _format_scalar(key, value, unit) + if value is None: + return "n/a" + return str(value) + + +def _format_aggregated(key: str, values: list[float], unit: str) -> str: + if not values: + return "n/a" + if key == "trajectory_success": + passed = sum(1 for v in values if v >= 1.0) + return f"{passed}/{len(values)} passed" + mean = statistics.mean(values) + if len(values) == 1: + return _format_scalar(key, round(mean, 3), unit) + std = statistics.pstdev(values) + base = _format_scalar(key, round(mean, 3), unit) + return f"{base} ± {std:.3g} {unit}".strip() if unit else f"{base} ± {std:.3g} (n={len(values)})" + + +def _collect_scalar_metrics(metrics_blob: dict) -> dict[str, list[dict]]: + """Flatten per-test metrics.json into {metric_key: [entries]}. + + Preserves one entry per robot for multi-robot runs (robot_N.metric_key). + """ + out: dict[str, list[dict]] = {} + for key, entry in metrics_blob.items(): + if not isinstance(entry, dict) or "value" not in entry: + continue + match = ROBOT_METRIC_RE.match(key) + metric_key = match.group(1) if match else key + out.setdefault(metric_key, []).append(entry) + return out + + +def _metrics_blob(metrics: dict, test_name: str) -> dict: + canonical = _canonical_test_id(test_name) + return metrics.get(canonical, {}) + + +def _aggregate_metrics( + test_names: list[str], + metrics: dict, + schema: list[tuple[str, str]], +) -> dict[str, list[float]]: + """Collect numeric metric values across all test phases / iterations.""" + buckets: dict[str, list[float]] = {key: [] for key, _ in schema} + for name in test_names: + for metric_key, entries in _collect_scalar_metrics(_metrics_blob(metrics, name)).items(): + if metric_key not in buckets: + continue + for entry in entries: + value = entry.get("value") + if isinstance(value, (int, float)): + buckets[metric_key].append(float(value)) + return buckets + + +def _chain_title(module: str, param: str) -> str: + if module == "test_fixed_trajectory": + traj = re.search(r"traj(\w+)", param) + traj_label = traj.group(1) if traj else "trajectory" + sim = param.split("-", 1)[0] + robots = re.search(r"rob#(\d+)", param) + n_robots = robots.group(1) if robots else "?" + return f"{traj_label} | {sim} | {n_robots} robot(s)" + if module == "test_takeoff_hover_land": + vel = re.search(r"v([\d.]+)", param) + vel_label = f"{vel.group(1)} m/s" if vel else param + sim = param.split("-", 1)[0] + return f"takeoff-hover-land @ {vel_label} | {sim}" + return param + + +def _metric_schema(module: str) -> list[tuple[str, str]]: + if module in ("test_fixed_trajectory", "test_takeoff_hover_land"): + if module == "test_takeoff_hover_land": + return [m for m in FLIGHT_METRICS if m[0] != "trajectory_success" + and not m[0].startswith("cross_track") + and m[0] != "path_rmse_m" + and m[0] != "trajectory_execution_time_sim_s"] + return FLIGHT_METRICS + if module in ("test_liveliness", "test_sensors"): + return LIVELINESS_METRICS + return [] + + +def _group_tests( + metrics: dict, + statuses: dict[str, str], + durations: dict[str, float], +) -> dict[tuple[str, str], list[str]]: + """Group full test names by (module, base_param_id) across stress iterations.""" + groups: dict[tuple[str, str], list[str]] = {} + all_names = {_canonical_test_id(name) for name in set(metrics) | set(statuses)} + for name in sorted(all_names): + module = _module_name(name) + param = _base_param_id(_param_id(name)) + groups.setdefault((module, param), []).append(name) + for names in groups.values(): + names.sort(key=lambda n: ( + int(ITER_RE.search(_param_id(n)).group(0).replace("-iter", "")) + if ITER_RE.search(_param_id(n)) else 0, + PHASE_ORDER.get(_phase_name(n), 99), + )) + return groups + + +def _iteration_count(test_names: list[str]) -> int: + iters = set() + for name in test_names: + m = ITER_RE.search(_param_id(name)) + if m: + iters.add(m.group(0)) + return len(iters) or 1 + + +def _chain_status(test_names: list[str], statuses: dict[str, str]) -> str: + n_iter = _iteration_count(test_names) + if n_iter > 1: + landing_phases = [n for n in test_names if _phase_name(n) in ("test_landing", "test_land")] + check = landing_phases or test_names + passed = sum(1 for n in check if statuses.get(n) == "PASSED") + total = len(check) + return f"{passed}/{total} flight cycles passed ({n_iter} iterations)" + if any(statuses.get(n) == "FAILED" for n in test_names): + return "FAILED" + if test_names and all(statuses.get(n) == "PASSED" for n in test_names): + return "PASSED" + if any(statuses.get(n) == "SKIPPED" for n in test_names): + return "SKIPPED" + return "UNKNOWN" + + +def build_summary_lines(run_dir: Path) -> list[str]: + metrics_path = run_dir / "metrics.json" + results_path = run_dir / "results.xml" + statuses, durations = _parse_results_xml(results_path) + metrics = _load_metrics(metrics_path) + + passed = sum(1 for s in statuses.values() if s == "PASSED") + failed = sum(1 for s in statuses.values() if s == "FAILED") + skipped = sum(1 for s in statuses.values() if s == "SKIPPED") + total = len(statuses) + + lines = [ + "AirStack Test Run Summary", + f"Run directory: {run_dir.name}", + f"Overall: {passed} passed, {failed} failed, {skipped} skipped ({total} tests)", + "", + ] + + groups = _group_tests(metrics, statuses, durations) + if not groups: + lines.append("No metrics or test results recorded for this run.") + return lines + + for (module, param), test_names in sorted(groups.items()): + title = _chain_title(module, param) + chain_status = _chain_status(test_names, statuses) + lines.append(f"── {title} ──") + lines.append(f"Result: {chain_status}") + lines.append("") + + schema = _metric_schema(module) + aggregated = _aggregate_metrics(test_names, metrics, schema) + n_iter = _iteration_count(test_names) + emitted = False + for metric_key, label in schema: + values = aggregated.get(metric_key, []) + if not values: + continue + unit = UNIT_OVERRIDES.get( + metric_key, + next( + (entry.get("unit", "") + for name in test_names + for entry in _collect_scalar_metrics(_metrics_blob(metrics, name)).get( + metric_key, [])), + "", + ), + ) + if n_iter > 1 or len(values) > 1: + lines.append(f"{label}: {_format_aggregated(metric_key, values, unit)}") + else: + entry = {"value": values[-1], "unit": unit} + lines.append(f"{label}: {_format_value(metric_key, entry)}") + emitted = True + + if not emitted: + lines.append("(no key metrics recorded)") + + if n_iter > 1: + lines.append("") + lines.append(f"Aggregated over {n_iter} stress iterations (mean ± stddev).") + + # Phase wall times help debugging without opening results.xml. + phase_wall: dict[str, list[float]] = {} + for name in test_names: + phase = _phase_name(name) + wall = durations.get(name) + if wall is not None: + phase_wall.setdefault(phase, []).append(wall) + if phase_wall: + lines.append("") + lines.append("Phase wall times:") + for phase, walls in sorted(phase_wall.items(), key=lambda x: PHASE_ORDER.get(x[0], 99)): + if n_iter > 1 and len(walls) > 1: + mean = statistics.mean(walls) + std = statistics.pstdev(walls) + lines.append(f" {phase}: {mean:.1f}s ± {std:.1f}s (n={len(walls)})") + else: + status = statuses.get( + next((n for n in test_names if _phase_name(n) == phase), ""), + "?", + ) + lines.append(f" {phase}: {walls[0]:.1f}s ({status})") + + lines.append("") + + # Trim trailing blank line + if lines and lines[-1] == "": + lines.pop() + return lines + + +def write_summary(run_dir: Path) -> Path: + run_dir = Path(run_dir) + out_path = run_dir / "summary.txt" + lines = build_summary_lines(run_dir) + out_path.write_text("\n".join(lines) + "\n") + return out_path + + +def main() -> int: + parser = argparse.ArgumentParser(description="Generate summary.txt for a test run") + parser.add_argument("run_dir", type=Path, help="Path to tests/results//") + args = parser.parse_args() + out = write_summary(args.run_dir) + print(out.read_text()) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/system/test_fixed_trajectory.py b/tests/system/test_fixed_trajectory.py new file mode 100644 index 000000000..169574cf2 --- /dev/null +++ b/tests/system/test_fixed_trajectory.py @@ -0,0 +1,678 @@ +"""Fixed-trajectory performance tests. + +Per (sim, num_robots, iter, trajectory_type): ready → takeoff → execute trajectory → land. + +The drone takes off to TARGET_ALTITUDE_M, executes one fixed-pattern trajectory +(Circle, Figure8, Racetrack, or Line), then lands. Odometry is captured throughout +the trajectory phase and compared against an ideal reference path (generated in Python +from the same equations as fixed_trajectory_task.cpp) to measure cross-track error. + +Each trajectory type is an independent full-cycle test so failures in one type do not +prevent the remaining types from running — the drone always returns to the ground at +the end of each cycle via the landing phase. +""" + +import math +import statistics +import subprocess +import time +from concurrent.futures import ThreadPoolExecutor +from io import StringIO +from pathlib import Path + +import pandas as pd +import pytest + +from conftest import ( + ROS_DISTRO_SETUP, + current_test_id, + get_metrics, + get_robot_containers, + logger, + ros2_exec, +) + +# ── constants ───────────────────────────────────────────────────────────── + +TARGET_ALTITUDE_M = 10.0 +PX4_READY_TIMEOUT_S = 300.0 +PX4_POLL_INTERVAL_S = 2.0 +TAKEOFF_MOTION_THRESHOLD_M = 0.3 # z rise above starting z to count as "moving" +SETTLING_WINDOW_S = 1.0 # trailing window for steady-state altitude check +MAX_GT_MATCH_AGE_S = 0.1 + +# Cross-track tolerance is intentionally loose: we know the circle trajectory +# currently fails, so the assertion documents the failure without blocking landing. +CROSS_TRACK_TOLERANCE_M = 5.0 + +# Generous timeout covers the full trajectory execution at low velocity in slow sims. +TRAJ_EXEC_TIMEOUT_S = 180.0 + +# Odom CSV schema (ros2 topic echo --csv flattens all primitives in declaration order). +ODOM_SCHEMA = ( + ["header.stamp.sec", "header.stamp.nanosec", + "header.frame_id", "child_frame_id", + "pose.pose.position.x", "pose.pose.position.y", "pose.pose.position.z", + "pose.pose.orientation.x", "pose.pose.orientation.y", + "pose.pose.orientation.z", "pose.pose.orientation.w"] + + [f"pose.covariance[{i}]" for i in range(36)] + + ["twist.twist.linear.x", "twist.twist.linear.y", "twist.twist.linear.z", + "twist.twist.angular.x", "twist.twist.angular.y", "twist.twist.angular.z"] + + [f"twist.covariance[{i}]" for i in range(36)] +) + +METRIC_UNITS = { + "ready_duration_sys_s": "s", + "trajectory_execution_time_sim_s": "s", + "takeoff_duration_sim_s": "s", + "land_duration_sim_s": "s", + "trajectory_success": "", + # Everything else defaults to "m". +} + +# ── default trajectory parameters ──────────────────────────────────────── + +# These mirror the attributes consumed by fixed_trajectory_task.cpp. +# frame_id is omitted — the action server defaults to "base_link". +TRAJECTORY_CONFIGS: dict[str, dict[str, str]] = { + "Circle": { + "radius": "10.0", + "velocity": "2.0", + }, + "Figure8": { + "length": "15.0", + "width": "8.0", + "height": "0.0", + "velocity": "2.0", + "max_acceleration": "1.0", + }, + "Racetrack": { + "length": "30.0", + "width": "10.0", + "height": "0.0", + "velocity": "3.0", + "turn_velocity": "1.5", + "max_acceleration": "1.0", + }, + "Line": { + "length": "20.0", + "height": "0.0", + "velocity": "2.0", + "max_acceleration": "1.0", + }, +} + + +# ── pytest hooks ────────────────────────────────────────────────────────── + +def pytest_generate_tests(metafunc): + """Parametrize tests that request `trajectory_type` from --trajectory-types.""" + if "trajectory_type" in metafunc.fixturenames: + raw = metafunc.config.getoption("--trajectory-types") + types = [t.strip() for t in raw.split(",") if t.strip()] + metafunc.parametrize("trajectory_type", types, ids=types) + + +# ── ideal-path generators (Python mirrors of fixed_trajectory_task.cpp) ── + +def _ideal_circle(radius: float) -> list[tuple[float, float, float]]: + """Circle waypoints in base_link frame matching generate_circle() in C++.""" + pts: list[tuple[float, float, float]] = [] + pts.append((0.0, 0.0, 0.0)) + pts.append((radius, 0.0, 0.0)) + angle = 0.0 + step = 10.0 * math.pi / 180.0 + while angle < 2.0 * math.pi: + pts.append((radius * math.cos(angle), radius * math.sin(angle), 0.0)) + angle += step + pts.append((radius, 0.0, 0.0)) + pts.append((0.0, 0.0, 0.0)) + return pts + + +def _ideal_figure8(length: float, width: float, height: float) -> list[tuple[float, float, float]]: + """Figure-8 waypoints in base_link frame matching generate_figure8() in C++.""" + n = 600 + pts: list[tuple[float, float, float]] = [] + for i in range(n - 1): + t = 2.0 * math.pi * i / n + x = math.cos(t) * length - length + y = math.cos(t) * math.sin(t) * 2.0 * width + pts.append((x, y, height)) + return pts + + +def _ideal_racetrack(length: float, width: float, height: float) -> list[tuple[float, float, float]]: + """Racetrack waypoints in base_link frame matching generate_racetrack() in C++.""" + sl = length - width + pts: list[tuple[float, float, float]] = [] + + for i in range(80): + x = sl * i / 79.0 + pts.append((x, 0.0, height)) + + turn_n = 48 + for i in range(1, turn_n + 1): + t = -math.pi / 2.0 + math.pi * i / (turn_n + 1) + x = width / 2.0 * math.cos(t) + sl + y = width / 2.0 * math.sin(t) + width / 2.0 + pts.append((x, y, height)) + + for i in range(80): + x = sl * (1.0 - i / 79.0) + pts.append((x, width, height)) + + for i in range(1, turn_n + 1): + t = math.pi / 2.0 + math.pi * i / (turn_n + 1) + x = width / 2.0 * math.cos(t) + y = width / 2.0 * math.sin(t) + width / 2.0 + pts.append((x, y, height)) + + return pts + + +def _ideal_line(length: float, height: float) -> list[tuple[float, float, float]]: + """Line waypoints in base_link frame matching generate_line() in C++. + + C++ iterates `y` from 0 down to -length in steps of 0.5 and sets x = -y, + so the drone moves along +x from 0 to length. + """ + pts: list[tuple[float, float, float]] = [] + y = 0.0 + while y > -length: + pts.append((-y, 0.0, height)) + y -= 0.5 + return pts + + +def _generate_ideal_path(traj_type: str, config: dict[str, str]) -> list[tuple[float, float, float]]: + """Dispatch to the correct ideal-path generator.""" + if traj_type == "Circle": + return _ideal_circle(float(config["radius"])) + if traj_type == "Figure8": + return _ideal_figure8(float(config["length"]), float(config["width"]), + float(config.get("height", "0"))) + if traj_type == "Racetrack": + return _ideal_racetrack(float(config["length"]), float(config["width"]), + float(config.get("height", "0"))) + if traj_type == "Line": + return _ideal_line(float(config["length"]), float(config.get("height", "0"))) + return [] + + +# ── geometry helpers ─────────────────────────────────────────────────────── + +def _quat_to_yaw(qx: float, qy: float, qz: float, qw: float) -> float: + """Extract yaw (heading) from a unit quaternion.""" + return math.atan2(2.0 * (qw * qz + qx * qy), + 1.0 - 2.0 * (qy * qy + qz * qz)) + + +def _transform_to_world( + base_link_pts: list[tuple[float, float, float]], + x0: float, y0: float, z0: float, yaw0: float, +) -> list[tuple[float, float, float]]: + """Rotate+translate base_link-frame points into the world frame. + + The trajectory controller publishes the trajectory in base_link at the + moment of dispatch, so the reference frame origin is (x0, y0, z0) with + heading yaw0. + """ + cos_y = math.cos(yaw0) + sin_y = math.sin(yaw0) + world_pts: list[tuple[float, float, float]] = [] + for lx, ly, lz in base_link_pts: + wx = x0 + lx * cos_y - ly * sin_y + wy = y0 + lx * sin_y + ly * cos_y + wz = z0 + lz + world_pts.append((wx, wy, wz)) + return world_pts + + +# ── metric computations ─────────────────────────────────────────────────── + +def _cross_track_metrics( + odom_rows: list[dict], + ideal_world_pts: list[tuple[float, float, float]], +) -> dict: + """Cross-track error statistics: mean, max, and RMSE against ideal path. + + Error is measured in the XY plane (these trajectories are flat; altitude + hold is evaluated separately by the takeoff/hover tests). + """ + if not odom_rows or not ideal_world_pts: + return {} + + ideal_xy = [(px, py) for px, py, _ in ideal_world_pts] + sq_dists: list[float] = [] + for row in odom_rows: + ox = row["pose.pose.position.x"] + oy = row["pose.pose.position.y"] + sq_dists.append(min((ox - px) ** 2 + (oy - py) ** 2 for px, py in ideal_xy)) + + dists = [math.sqrt(d) for d in sq_dists] + return { + "cross_track_error_mean_m": round(statistics.mean(dists), 3), + "cross_track_error_max_m": round(max(dists), 3), + "path_rmse_m": round(math.sqrt(statistics.mean(sq_dists)), 3), + } + + +def _takeoff_metrics(odom: list[dict], target: float, velocity: float) -> dict: + """Altitude error and duration from takeoff odom samples.""" + zs = [r["pose.pose.position.z"] for r in odom] + ts = [_stamp(r) for r in odom] + peak = max(zs) + cutoff = ts[-1] - SETTLING_WINDOW_S + settled = [z for z, t in zip(zs, ts) if t >= cutoff] + out: dict = { + "altitude_error_m": round(statistics.mean(settled) - target, 3), + "overshoot_m": round(max(0.0, peak - target), 3), + } + z0 = zs[0] + first_motion = next((i for i, z in enumerate(zs) if z > z0 + TAKEOFF_MOTION_THRESHOLD_M), None) + first_at_target = next((i for i, z in enumerate(zs) if z >= target * 0.95), None) + if first_motion is not None and first_at_target is not None and first_at_target > first_motion: + out["takeoff_duration_sim_s"] = round(ts[first_at_target] - ts[first_motion], 3) + return out + + +def _landing_metrics(odom: list[dict]) -> dict: + """Final altitude and landing duration from landing odom samples.""" + zs = [r["pose.pose.position.z"] for r in odom] + ts = [_stamp(r) for r in odom] + out: dict = {"final_altitude_m": round(zs[-1], 3)} + peak = max(zs) + first_descent = next((i for i, z in enumerate(zs) if z < peak * 0.8), None) + first_at_ground = next((i for i, z in enumerate(zs) if z < 0.5), None) + if first_descent is not None and first_at_ground is not None and first_at_ground > first_descent: + out["land_duration_sim_s"] = round(ts[first_at_ground] - ts[first_descent], 3) + return out + + +def _record(robot_n: int, metrics_dict: dict) -> None: + """Record per-robot scalar metrics; unit inferred from the METRIC_UNITS table.""" + m = get_metrics() + tid = current_test_id() + for key, value in metrics_dict.items(): + if value is None: + continue + unit = METRIC_UNITS.get(key, "m") + direction = "higher_is_better" if key == "trajectory_success" else "lower_is_better" + m.record(tid, f"robot_{robot_n}.{key}", value, unit=unit, direction=direction) + + +# ── CSV / subprocess helpers ─────────────────────────────────────────────── + +def _stamp(row: dict) -> float: + """Sim-time seconds from a parsed odometry CSV row.""" + return row["header.stamp.sec"] + row["header.stamp.nanosec"] * 1e-9 + + +def _start_csv_stream( + container: str, topic: str, domain: int, setup_bash: str, + duration_s: float, out_path: str, +) -> tuple: + """Background `ros2 topic echo --csv` stream to out_path. + + Returns (popen, file_handle, err_file_handle). Caller must close both + handles after the process terminates (see _finish_captures). + """ + cmd = ( + f"source {ROS_DISTRO_SETUP} && source {setup_bash} && " + f"export ROS_DOMAIN_ID={domain} && " + f"timeout {int(duration_s)} ros2 topic echo --csv {topic}" + ) + f = open(out_path, "w") + ef = open(out_path + ".err", "w") + try: + proc = subprocess.Popen( + ["docker", "exec", container, "bash", "-c", cmd], + stdout=f, stderr=ef, + ) + except BaseException: + f.close() + ef.close() + raise + return proc, f, ef + + +def _parse_csv(path: str, schema: list[str]) -> list[dict]: + """Read ros2 `--csv` output, filtering non-CSV lines ros2 emits to stdout.""" + with open(path) as fh: + good = [line for line in fh if line.count(",") >= len(schema) - 1] + if not good: + return [] + df = pd.read_csv(StringIO("".join(good)), header=None, names=schema) + return df.to_dict("records") + + +def _start_captures( + robot_container: str, setup_bash: str, domain: int, duration_s: float, tag: str, +) -> dict: + """Start odom CSV stream for one robot. Returns a handle for _finish_captures.""" + odom_path = f"/tmp/traj_r{domain}_{tag}_odom.csv" + odom_proc, odom_fh, odom_ef = _start_csv_stream( + robot_container, + f"/robot_{domain}/interface/mavros/local_position/odom", + domain, setup_bash, duration_s, odom_path, + ) + return {"duration_s": duration_s, "odom": (odom_proc, odom_fh, odom_ef, odom_path)} + + +def _finish_captures(streams: dict) -> list[dict]: + """Terminate capture and return parsed odom rows.""" + odom_proc, odom_fh, odom_ef, odom_path = streams["odom"] + try: + odom_proc.terminate() + try: + odom_proc.wait(timeout=5) + except subprocess.TimeoutExpired: + odom_proc.kill() + odom_proc.wait(timeout=5) + finally: + odom_fh.close() + odom_ef.close() + odom = _parse_csv(odom_path, ODOM_SCHEMA) + if not odom: + logger.warning( + "odom capture empty. stdout=%r stderr=%r", + Path(odom_path).read_text()[:500], + Path(odom_path + ".err").read_text()[:500], + ) + return odom + + +def _action_ok(stdout: str) -> bool: + return "success: true" in stdout + + +def _action_message(stdout: str) -> str: + for line in stdout.splitlines(): + s = line.strip() + if s.startswith("message:"): + return s[len("message:"):].strip().strip("'\"") + return "\n".join(stdout.strip().splitlines()[-5:]) + + +def _run_parallel(num_robots: int, fn) -> None: + """Run fn(n) for n=1..num_robots concurrently.""" + if num_robots == 1: + fn(1) + return + with ThreadPoolExecutor(max_workers=num_robots) as ex: + list(ex.map(fn, range(1, num_robots + 1))) + + +def _build_traj_goal(traj_type: str, config: dict[str, str]) -> str: + """Build the YAML goal string for a FixedTrajectoryTask action send_goal call.""" + attrs = ", ".join(f"{{key: {k}, value: '{v}'}}" for k, v in config.items()) + return f"{{trajectory_spec: {{type: {traj_type}, attributes: [{attrs}]}}, loop: false}}" + + +# ── per-robot workers ────────────────────────────────────────────────────── + +def _takeoff_one_robot(n: int, robot_container: str, cfg: dict, target: float) -> None: + velocity = 1.0 # fixed takeoff velocity for trajectory tests + timeout = max(30.0, target / velocity + 15.0) + streams = _start_captures(robot_container, cfg["robot_setup_bash"], + n, timeout + 5, "traj_takeoff") + goal = f"{{target_altitude_m: {target}, velocity_m_s: {velocity}}}" + result = ros2_exec( + robot_container, + f'ros2 action send_goal --feedback /robot_{n}/tasks/takeoff ' + f'task_msgs/action/TakeoffTask "{goal}"', + domain_id=n, setup_bash=cfg["robot_setup_bash"], + timeout=int(timeout + 10), + ) + odom = _finish_captures(streams) + if not _action_ok(result.stdout): + pytest.fail(f"robot_{n} takeoff failed: {_action_message(result.stdout)}") + if not odom: + pytest.fail(f"robot_{n} takeoff: no odom samples captured") + metrics = _takeoff_metrics(odom, target, velocity) + _record(n, metrics) + err = metrics.get("altitude_error_m", 0.0) + assert abs(err) <= target * 0.1, ( + f"robot_{n} settled altitude {target + err:.2f}m differs from " + f"target {target:.1f}m by more than 10%" + ) + + +def _trajectory_one_robot( + n: int, robot_container: str, cfg: dict, traj_type: str, +) -> None: + config = TRAJECTORY_CONFIGS[traj_type] + streams = _start_captures(robot_container, cfg["robot_setup_bash"], + n, TRAJ_EXEC_TIMEOUT_S + 10, f"traj_{traj_type.lower()}") + goal = _build_traj_goal(traj_type, config) + + # Snapshot the robot's world-frame pose immediately before dispatch so we can + # transform the base_link ideal path to world frame for metric computation. + odom_snap = ros2_exec( + robot_container, + f"timeout 5 ros2 topic echo --once --csv " + f"/robot_{n}/interface/mavros/local_position/odom", + domain_id=n, setup_bash=cfg["robot_setup_bash"], timeout=10, + ) + x0, y0, z0, yaw0 = 0.0, 0.0, TARGET_ALTITUDE_M, 0.0 + for line in odom_snap.stdout.splitlines(): + parts = line.strip().split(",") + if len(parts) >= len(ODOM_SCHEMA): + try: + row = dict(zip(ODOM_SCHEMA, parts)) + x0 = float(row["pose.pose.position.x"]) + y0 = float(row["pose.pose.position.y"]) + z0 = float(row["pose.pose.position.z"]) + yaw0 = _quat_to_yaw( + float(row["pose.pose.orientation.x"]), + float(row["pose.pose.orientation.y"]), + float(row["pose.pose.orientation.z"]), + float(row["pose.pose.orientation.w"]), + ) + break + except (ValueError, KeyError): + pass + + t_start = time.monotonic() + result = ros2_exec( + robot_container, + f'ros2 action send_goal --feedback /robot_{n}/tasks/fixed_trajectory ' + f'task_msgs/action/FixedTrajectoryTask "{goal}"', + domain_id=n, setup_bash=cfg["robot_setup_bash"], + timeout=int(TRAJ_EXEC_TIMEOUT_S + 15), + ) + exec_time_s = round(time.monotonic() - t_start, 3) + + odom = _finish_captures(streams) + + success = _action_ok(result.stdout) + _record(n, {"trajectory_success": 1.0 if success else 0.0}) + + if not success: + logger.warning("robot_%d %s trajectory did not succeed: %s", + n, traj_type, _action_message(result.stdout)) + + if odom: + ts = [_stamp(r) for r in odom] + exec_sim_s = round(ts[-1] - ts[0], 3) if len(ts) > 1 else exec_time_s + _record(n, {"trajectory_execution_time_sim_s": exec_sim_s}) + + ideal_base = _generate_ideal_path(traj_type, config) + if ideal_base: + ideal_world = _transform_to_world(ideal_base, x0, y0, z0, yaw0) + ct_metrics = _cross_track_metrics(odom, ideal_world) + _record(n, ct_metrics) + + mean_err = ct_metrics.get("cross_track_error_mean_m") + if mean_err is not None: + assert mean_err < CROSS_TRACK_TOLERANCE_M, ( + f"robot_{n} {traj_type}: mean cross-track error {mean_err:.2f}m " + f"exceeds tolerance {CROSS_TRACK_TOLERANCE_M:.1f}m" + ) + else: + logger.warning("robot_%d %s: no odom samples captured", n, traj_type) + + +def _landing_one_robot(n: int, robot_container: str, cfg: dict) -> None: + velocity = 1.0 + timeout = max(30.0, TARGET_ALTITUDE_M / velocity + 15.0) + streams = _start_captures(robot_container, cfg["robot_setup_bash"], + n, timeout + 5, "traj_land") + goal = f"{{velocity_m_s: {velocity}}}" + result = ros2_exec( + robot_container, + f'ros2 action send_goal --feedback /robot_{n}/tasks/land ' + f'task_msgs/action/LandTask "{goal}"', + domain_id=n, setup_bash=cfg["robot_setup_bash"], + timeout=int(timeout + 10), + ) + odom = _finish_captures(streams) + if not _action_ok(result.stdout): + pytest.fail(f"robot_{n} landing failed: {_action_message(result.stdout)}") + if not odom: + pytest.fail(f"robot_{n} landing: no odom samples captured") + metrics = _landing_metrics(odom) + _record(n, metrics) + final = metrics.get("final_altitude_m", 1.0) + assert final < 0.5, f"robot_{n} final altitude {final:.2f}m > 0.5m" + + +# ── test class ───────────────────────────────────────────────────────────── + +@pytest.mark.autonomy +@pytest.mark.timeout(2400) +class TestFixedTrajectory: + """Full takeoff → fixed trajectory → land evaluation suite. + + Parametrized over trajectory_type (from --trajectory-types). + Each trajectory type runs as an independent flight cycle so a failure on + one type does not prevent other types from being evaluated. + """ + + @pytest.fixture(scope="session") + def _failed_envs(self): + return set() + + @pytest.fixture(scope="session") + def _ready_envs(self): + return set() + + @pytest.fixture(autouse=True) + def _chain_guard(self, request, airstack_env, _failed_envs): + """Skip tests whose env was poisoned by an earlier failure. + + Trajectory execution failures do NOT poison the env — landing always + runs after a successful takeoff, keeping the drone from being stranded. + Takeoff or landing failures do poison subsequent tests in the same env. + """ + env_id = (airstack_env["sim"], airstack_env["num_robots"], + airstack_env["iteration"]) + if env_id in _failed_envs: + pytest.skip(f"earlier fixed-trajectory test failed in {env_id}") + yield + rep = getattr(request.node, "_rep_call", None) + if rep is not None and rep.failed: + if "test_fixed_trajectory" not in request.node.name: + _failed_envs.add(env_id) + + @pytest.mark.dependency(name="ftraj_ready") + def test_px4_ready(self, airstack_env, trajectory_type, _ready_envs): + """Wait until MAVROS is connected and local_position/odom is publishing. + + Runs only once per (sim, num_robots, iteration) env regardless of how + many trajectory types are being tested. + """ + env_id = (airstack_env["sim"], airstack_env["num_robots"], + airstack_env["iteration"]) + if env_id in _ready_envs: + logger.info("px4_ready already confirmed for %s; skipping", env_id) + return + + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + num_robots = airstack_env["num_robots"] + + started = time.time() + connected: set[int] = set() + pending = list(range(1, num_robots + 1)) + ready_at: dict[int, float] = {} + deadline = started + PX4_READY_TIMEOUT_S + + while pending and time.time() < deadline: + for n in list(pending): + if n not in connected: + r = ros2_exec( + robot_container, + f"timeout 5 ros2 topic echo --once --csv " + f"--field connected /robot_{n}/interface/mavros/state", + domain_id=n, setup_bash=cfg["robot_setup_bash"], timeout=10, + ) + if any(line.strip() == "True" for line in r.stdout.splitlines()): + connected.add(n) + else: + continue + + r = ros2_exec( + robot_container, + f"timeout 5 ros2 topic echo --once " + f"/robot_{n}/interface/mavros/local_position/odom", + domain_id=n, setup_bash=cfg["robot_setup_bash"], timeout=10, + ) + if r.returncode == 0 and "pose:" in r.stdout: + ready_at[n] = round(time.time() - started, 2) + pending.remove(n) + + if pending: + logger.info("px4_ready: connected=%s pending=%s elapsed=%.0fs", + sorted(connected), pending, time.time() - started) + time.sleep(PX4_POLL_INTERVAL_S) + + if pending: + not_connected = [n for n in pending if n not in connected] + if not_connected: + pytest.fail( + f"robots {sorted(not_connected)} never reported MAVROS connected=True " + f"within {PX4_READY_TIMEOUT_S:.0f}s" + ) + pytest.fail( + f"robots {sorted(pending)} connected but never published " + f"local_position/odom within {PX4_READY_TIMEOUT_S:.0f}s" + ) + + for n, dur in ready_at.items(): + _record(n, {"ready_duration_sys_s": dur}) + _ready_envs.add(env_id) + + @pytest.mark.dependency(name="ftraj_takeoff", depends=["ftraj_ready"]) + def test_takeoff(self, airstack_env, trajectory_type): + """Take off to TARGET_ALTITUDE_M at a fixed velocity of 1 m/s.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + num_robots = airstack_env["num_robots"] + _run_parallel( + num_robots, + lambda n: _takeoff_one_robot(n, robot_container, cfg, TARGET_ALTITUDE_M), + ) + + @pytest.mark.dependency(name="ftraj_execute", depends=["ftraj_takeoff"]) + def test_fixed_trajectory(self, airstack_env, trajectory_type): + """Send FixedTrajectoryTask, capture odom, compute and record path deviation.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + num_robots = airstack_env["num_robots"] + _run_parallel( + num_robots, + lambda n: _trajectory_one_robot(n, robot_container, cfg, trajectory_type), + ) + + @pytest.mark.dependency(name="ftraj_land", depends=["ftraj_takeoff"]) + def test_landing(self, airstack_env, trajectory_type): + """Land the drone; runs even when test_fixed_trajectory fails.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + num_robots = airstack_env["num_robots"] + _run_parallel( + num_robots, + lambda n: _landing_one_robot(n, robot_container, cfg), + ) From c476db32a452962d0200a99cda3881aa5ff2783e Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Tue, 21 Jul 2026 11:11:01 -0400 Subject: [PATCH 07/21] General robot deployment infra: aarch64 build args + robot-name resolution fixes (#370) Foundational real-robot deployment fixes extracted from the OptiTrack emulation PR (#367) so they can be reviewed and merged first; #367 will be rebased on top afterward, shrinking its diff. Docker / ARM build: - Add TARGET_ARCH build arg (default x86_64) to Dockerfile.robot and use it to parametrize LD_LIBRARY_PATH, so the aarch64 (Jetson/l4t, voxl) images link against the correct arch triplet. - docker-compose.yaml passes TARGET_ARCH: aarch64 to the voxl and l4t image builds. - Install ros-${ROS_DISTRO}-mavros-extras (generic dep; also provides the vision_pose plugin used by external-pose deployments). Robot name resolution: - .bashrc now follows a pre-set ROBOT_NAME (e.g. injected by docker compose) instead of always overriding it from the container/hostname mapping. The bws() flock build lock is retained. - default_robot_name_map.yaml catch-all fallback maps to unknown_robot (valid ROS namespace token) instead of unknown-robot. Version bumped 0.19.0-alpha.5 -> 0.19.0-alpha.6 for the version-increment gate. Note: the trajectory_controller/trajectory_library robustness fixes originally listed for extraction are already present on develop (PR #365), so they are not included here. Co-authored-by: Claude Opus 4.8 --- .env | 2 +- CHANGELOG.md | 7 +++ robot/docker/.bashrc | 56 ++++++++++--------- robot/docker/Dockerfile.robot | 8 ++- robot/docker/docker-compose.yaml | 2 + .../default_robot_name_map.yaml | 2 +- 6 files changed, 47 insertions(+), 30 deletions(-) diff --git a/.env b/.env index 77ed88ef4..bdd6f251e 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.5" +VERSION="0.19.0-alpha.6" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/CHANGELOG.md b/CHANGELOG.md index a87adb205..747f670b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added - Battery and telemetry display in GCS RQT control panel (voltage and percentage per robot when MAVROS battery topic is bridged) +- `TARGET_ARCH` build arg (default `x86_64`) in `Dockerfile.robot` to arch-parametrize `LD_LIBRARY_PATH`; `docker-compose.yaml` passes `TARGET_ARCH: aarch64` to the `voxl` and `l4t` real-robot image builds +- `ros-${ROS_DISTRO}-mavros-extras` in the robot image (provides the vision_pose plugin used for external-pose deployments) + +### Fixed + +- Robot name resolution now honors a pre-set `ROBOT_NAME` (e.g. injected via docker compose) instead of always overriding it from the container/hostname mapping (`robot/docker/.bashrc`) +- Robot name-map catch-all fallback now maps to `unknown_robot` (valid ROS namespace token) instead of `unknown-robot` (`default_robot_name_map.yaml`) ## [1.0.0] - 2024-12-19 diff --git a/robot/docker/.bashrc b/robot/docker/.bashrc index d2c3c877d..b3903a144 100755 --- a/robot/docker/.bashrc +++ b/robot/docker/.bashrc @@ -69,35 +69,39 @@ function cws(){ source /opt/ros/jazzy/setup.bash sws # source the ROS2 workspace by default -# Only extract robot name and ROS domain ID iff they are not already set in the environment (e.g. by docker compose) -if [ "$ROBOT_NAME_SOURCE" == "container_name" ]; then - # https://wiki.psuter.ch/doku.php?id=get_docker_container_name_from_within_the_container - # WARNING: this technique ONLY works with docker version 29 and up. - name_to_map=$(host $(host $(hostname) | awk '{print $NF}') | awk '{print $NF}' | awk -F . '{print $1}') - CONTAINER_NAME=":$name_to_map" -elif [ "$ROBOT_NAME_SOURCE" == "hostname" ]; then - name_to_map=$(hostname) -else - echo "Warning: ROBOT_NAME_SOURCE=$ROBOT_NAME_SOURCE not set to a valid value. Defaulting to 'unknown_robot'." - name_to_map="" - export ROBOT_NAME="unknown_robot" - export ROS_DOMAIN_ID=0 -fi -# set ROBOT_NAME and ROS_DOMAIN_ID from the mapping script if NAME_TO_MAP is not empty -if [ -n "$name_to_map" ]; then - script_path="$HOME/AirStack/robot/docker/robot_name_map/resolve_robot_name.py" - script_dir=$(dirname "$script_path") +# If ROBOT_NAME is pre-set (e.g. via docker compose), keep it. +# Otherwise extract robot name and ROS domain ID from the container/hostname mapping. +if [ -z "${ROBOT_NAME:-}" ]; then + if [ "$ROBOT_NAME_SOURCE" == "container_name" ]; then + # https://wiki.psuter.ch/doku.php?id=get_docker_container_name_from_within_the_container + # WARNING: this technique ONLY works with docker version 29 and up. + name_to_map=$(host $(host $(hostname) | awk '{print $NF}') | awk '{print $NF}' | awk -F . '{print $1}') + CONTAINER_NAME=":$name_to_map" + elif [ "$ROBOT_NAME_SOURCE" == "hostname" ]; then + name_to_map=$(hostname) + else + echo "Warning: ROBOT_NAME_SOURCE=$ROBOT_NAME_SOURCE not set to a valid value. Defaulting to 'unknown_robot'." + name_to_map="" + export ROBOT_NAME="unknown_robot" + export ROS_DOMAIN_ID=0 + fi - existing_robot_domain_id=${ROS_DOMAIN_ID:-} + # set ROBOT_NAME and ROS_DOMAIN_ID from the mapping script if NAME_TO_MAP is not empty + if [ -n "$name_to_map" ]; then + script_path="$HOME/AirStack/robot/docker/robot_name_map/resolve_robot_name.py" + script_dir=$(dirname "$script_path") - eval "$($script_path $name_to_map $script_dir/$ROBOT_NAME_MAP_CONFIG_FILE)" - export ROBOT_NAME + existing_robot_domain_id=${ROS_DOMAIN_ID:-} - # if ROS_DOMAIN_ID was already set in the environment, use that instead of the mapped value - if [ -z "$existing_robot_domain_id" ]; then - export ROS_DOMAIN_ID - else - export ROS_DOMAIN_ID=$existing_robot_domain_id + eval "$($script_path $name_to_map $script_dir/$ROBOT_NAME_MAP_CONFIG_FILE)" + export ROBOT_NAME + + # if ROS_DOMAIN_ID was already set in the environment, use that instead of the mapped value + if [ -z "$existing_robot_domain_id" ]; then + export ROS_DOMAIN_ID + else + export ROS_DOMAIN_ID=$existing_robot_domain_id + fi fi fi diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index 894874961..92fd116fe 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -12,6 +12,7 @@ ARG UPDATE_FLAGS="-o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDo ARG INSTALL_FLAGS="-o APT::Get::AllowUnauthenticated=true" ARG SKIP_MACVO=false ARG SKIP_TENSORRT=false +ARG TARGET_ARCH=x86_64 ARG PIP_VERSION=24.0 ARG PYTHON_VERSION=3.12 @@ -65,7 +66,7 @@ RUN sudo add-apt-repository universe \ ENV AMENT_PREFIX_PATH=/opt/ros/${ROS_DISTRO} ENV COLCON_PREFIX_PATH=/opt/ros/${ROS_DISTRO} -ENV LD_LIBRARY_PATH=/opt/ros/${ROS_DISTRO}/lib/x86_64-linux-gnu:/opt/ros/${ROS_DISTRO}/lib +ENV LD_LIBRARY_PATH=/opt/ros/${ROS_DISTRO}/lib/${TARGET_ARCH}-linux-gnu:/opt/ros/${ROS_DISTRO}/lib ENV PATH=/opt/ros/${ROS_DISTRO}/bin:$PATH ENV PYTHONPATH=/opt/ros/${ROS_DISTRO}/local/lib/python${PYTHON_VERSION}/dist-packages:/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION}/site-packages ENV ROS_PYTHON_VERSION=3 @@ -96,6 +97,7 @@ RUN python3 -m pip install --no-cache-dir --break-system-packages --ignore-insta RUN apt update -y && apt install -y --no-install-recommends \ ros-dev-tools \ ros-${ROS_DISTRO}-mavros \ + ros-${ROS_DISTRO}-mavros-extras \ ros-${ROS_DISTRO}-tf2* \ ros-${ROS_DISTRO}-stereo-image-proc \ ros-${ROS_DISTRO}-image-view \ @@ -245,6 +247,7 @@ ARG UPDATE_FLAGS="-o Acquire::AllowInsecureRepositories=true -o Acquire::AllowDo ARG INSTALL_FLAGS="-o APT::Get::AllowUnauthenticated=true" ARG SKIP_MACVO=false ARG SKIP_TENSORRT=false +ARG TARGET_ARCH=x86_64 ARG PIP_VERSION=24.0 ARG PYTHON_VERSION=3.12 @@ -297,7 +300,7 @@ RUN sudo add-apt-repository universe \ ENV AMENT_PREFIX_PATH=/opt/ros/${ROS_DISTRO} ENV COLCON_PREFIX_PATH=/opt/ros/${ROS_DISTRO} -ENV LD_LIBRARY_PATH=/opt/ros/${ROS_DISTRO}/lib/x86_64-linux-gnu:/opt/ros/${ROS_DISTRO}/lib +ENV LD_LIBRARY_PATH=/opt/ros/${ROS_DISTRO}/lib/${TARGET_ARCH}-linux-gnu:/opt/ros/${ROS_DISTRO}/lib ENV PATH=/opt/ros/${ROS_DISTRO}/bin:$PATH ENV PYTHONPATH=/opt/ros/${ROS_DISTRO}/local/lib/python${PYTHON_VERSION}/dist-packages:/opt/ros/${ROS_DISTRO}/lib/python${PYTHON_VERSION}/site-packages ENV ROS_PYTHON_VERSION=3 @@ -328,6 +331,7 @@ RUN python3 -m pip install --no-cache-dir --break-system-packages --ignore-insta RUN apt update -y && apt install -y --no-install-recommends \ ros-dev-tools \ ros-${ROS_DISTRO}-mavros \ + ros-${ROS_DISTRO}-mavros-extras \ ros-${ROS_DISTRO}-tf2* \ ros-${ROS_DISTRO}-stereo-image-proc \ ros-${ROS_DISTRO}-image-view \ diff --git a/robot/docker/docker-compose.yaml b/robot/docker/docker-compose.yaml index f1799f48d..71b71e425 100644 --- a/robot/docker/docker-compose.yaml +++ b/robot/docker/docker-compose.yaml @@ -122,6 +122,7 @@ services: REAL_ROBOT: true SKIP_MACVO: true SKIP_TENSORRT: true + TARGET_ARCH: aarch64 ROS_DISTRO: jazzy tags: - *voxl_image @@ -179,6 +180,7 @@ services: REAL_ROBOT: true SKIP_MACVO: true SKIP_TENSORRT: true + TARGET_ARCH: aarch64 ROS_DISTRO: jazzy tags: - *l4t_image diff --git a/robot/docker/robot_name_map/default_robot_name_map.yaml b/robot/docker/robot_name_map/default_robot_name_map.yaml index d01cda199..5d638b6ed 100644 --- a/robot/docker/robot_name_map/default_robot_name_map.yaml +++ b/robot/docker/robot_name_map/default_robot_name_map.yaml @@ -12,5 +12,5 @@ mappings: # catch all fall-back - pattern: '.*' - robot: 'unknown-robot' + robot: 'unknown_robot' domain_id: '0' \ No newline at end of file From 1a25d60b439e4e9cc98d2f8b1f1116f5fab51a54 Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Wed, 22 Jul 2026 12:39:13 -0400 Subject: [PATCH 08/21] l4t deployment fixes: make the Jetson profile build + boot on real hardware (#371) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(l4t): make robot-l4t deployment knobs overridable + document name resolution Parametrize the robot-l4t compose service so a single service covers real deployments without editing compose: - AUTONOMY_ROLE and FCU_URL are now ${VAR:-default} overridable (and FCU_URL is unquoted so the literal serial path reaches mavros). - Rosbag output path is BAG_STORAGE_PATH-overridable. Update the configure-multi-robot skill to reflect the honor-pre-set-ROBOT_NAME guard (#370): document pinning ROBOT_NAME in an override for a single real robot, the never-on-the-shared-service caveat, and the unknown_robot fallback fixes by topology. Co-Authored-By: Claude Opus 4.8 * feat(l4t): add site-agnostic l4t-px4-realrobot override template Deployment override for a single real PX4 robot on a Jetson (aarch64/l4t). Surfaces the common knobs at the top with sensible defaults: ROBOT_NAME pinned directly (single-robot shortcut honored by .bashrc), FCU_URL, AUTONOMY_ROLE, BAG_STORAGE_PATH, and RECORD_BAGS. Mocap-agnostic — NatNet/external-vision settings are added by a separate optitrack override. Co-Authored-By: Claude Opus 4.8 * fix(l4t): entrypoint passthrough + ZED SDK 5.2; document build gotchas Two real-hardware build fixes for the Jetson profile: - Dockerfile.l4t-stack-base: overwrite dustynv's /ros_entrypoint.sh with an `exec "$@"` passthrough. Its prebuilt source-ROS libs (fastcdr 2.2.5) were shadowing the apt Jazzy (2.2.7) that Dockerfile.robot layers on, crashing apt-built nodes like mavros with symbol-lookup errors under tmux autolaunch. - zed/Dockerfile.zed-l4t: bump ZED SDK 4.2 -> 5.2 and move the coupled ROS deps together (zed_msgs 5.2.1, point_cloud_transport(_plugins) 4.x, add backward_ros). Document both gotchas in the docker-build-profiles skill, and correct the stale unknown-robot -> unknown_robot in the robot_identity reference doc. Co-Authored-By: Claude Opus 4.8 * chore: bump version to 0.19.0-alpha.7 Version-increment gate: bump above develop's 0.19.0-alpha.6 and record the l4t deployment changes in the changelog. Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- .agents/skills/configure-multi-robot/SKILL.md | 36 ++++++++++++++++--- .agents/skills/docker-build-profiles/SKILL.md | 2 ++ .env | 2 +- CHANGELOG.md | 7 ++++ docs/robot/docker/robot_identity.md | 2 +- overrides/l4t-px4-realrobot.env | 34 ++++++++++++++++++ robot/docker/Dockerfile.l4t-stack-base | 4 +++ robot/docker/docker-compose.yaml | 6 ++-- robot/docker/zed/Dockerfile.zed-l4t | 10 +++--- 9 files changed, 89 insertions(+), 14 deletions(-) create mode 100644 overrides/l4t-px4-realrobot.env diff --git a/.agents/skills/configure-multi-robot/SKILL.md b/.agents/skills/configure-multi-robot/SKILL.md index d147fdcf2..c70ba8dde 100644 --- a/.agents/skills/configure-multi-robot/SKILL.md +++ b/.agents/skills/configure-multi-robot/SKILL.md @@ -42,6 +42,9 @@ docker-compose.yaml (ROBOT_NAME_SOURCE=container_name | hostname, │ ▼ robot/docker/.bashrc (runs on container shell start) + │ + ├─ ROBOT_NAME already set in env? → KEEP IT, skip resolution entirely + │ (guard: `if [ -z "${ROBOT_NAME:-}" ]`; lets an override/compose pin the name) │ ├─ ROBOT_NAME_SOURCE=container_name → resolve `hostname` back to docker container name │ (e.g. `airstack-robot-desktop-1`) @@ -67,7 +70,7 @@ The default mapping rule in [`robot/docker/robot_name_map/default_robot_name_map robot: 'robot_{1}' domain_id: '{1}' - pattern: '.*' # catch-all - robot: 'unknown-robot' + robot: 'unknown_robot' # must be a valid ROS token (no hyphen) or launch fails domain_id: '0' ``` @@ -97,7 +100,18 @@ docker exec airstack-robot-desktop-1 bash -c 'echo $ROBOT_NAME $ROS_DOMAIN_ID' # robot_1 1 ``` -If you need a non-default name (custom hostname scheme on a physical robot, or you want `drone_alpha` instead of `robot_1`), write a new mapping YAML in `robot/docker/robot_name_map/` and point `ROBOT_NAME_MAP_CONFIG_FILE` at it. Do **not** hardcode `ROBOT_NAME=...` in compose unless you know what you are doing — it bypasses the resolver and you lose `ROS_DOMAIN_ID` co-assignment. +If you need a non-default name (custom hostname scheme on a physical robot, or you want `drone_alpha` instead of `robot_1`), you have two options: + +1. **Write a mapping YAML** in `robot/docker/robot_name_map/` and point `ROBOT_NAME_MAP_CONFIG_FILE` at it. Preferred when the name should be derived from the machine (hostname/container) — keeps the resolver in charge of `ROS_DOMAIN_ID` co-assignment. +2. **Pin `ROBOT_NAME` directly** in a per-deployment override env file. `.bashrc` honors a pre-set `ROBOT_NAME` (guard: `if [ -z "${ROBOT_NAME:-}" ]`) and skips the map lookup. This is the clean shortcut for a **single real robot** whose hostname doesn't match `robot-` (see [Real robots and the `unknown_robot` fallback](#real-robots-and-the-unknown_robot-fallback) below): + + ```bash + # overrides/.env — single robot, named directly + ROBOT_NAME=robot_1 + ROS_DOMAIN_ID=1 # set alongside — pinning ROBOT_NAME skips the map's domain co-assignment + ``` + +**Only pin `ROBOT_NAME` in an *override env file*, never on the shared `robot-desktop`/`robot-l4t` *service* in compose.** The service is reused for every replica; a hardcoded `ROBOT_NAME` there collapses all robots onto one name/domain and silently breaks multi-robot. And when you pin it, set `ROS_DOMAIN_ID` too — the resolver is what normally co-assigns the domain, and skipping it leaves the domain at whatever the environment defaults to. For a one-off override (e.g. ad hoc debugging): @@ -286,7 +300,7 @@ Without `allow_substs="true"`, the substitution string is loaded literally and t If two robots share a domain, every topic collides — both `/robot_1/odometry` publishers will be visible to both subscribers, and DDS will sometimes deliver crossed data. The default `robot_name_map` derives the domain from the robot index, so this only happens if you: - Hardcode `ROS_DOMAIN_ID` in compose to the same value for two replicas -- Use a hostname that doesn't match any rule and falls through to the catch-all (both robots get `unknown-robot`, domain `0`) +- Use a hostname that doesn't match any rule and falls through to the catch-all (both robots get `unknown_robot`, domain `0`) Always verify after starting: @@ -329,9 +343,21 @@ This is a common foot-gun: Either keep the remap relative (`to="odometry"`) so it joins the namespace, or write the full path explicitly (`to="/$(env ROBOT_NAME)/odometry"`). -### 9. Hostname doesn't match any rule on real robots +### 9. Real robots and the `unknown_robot` fallback + +On VOXL/Jetson the service uses `ROBOT_NAME_SOURCE=hostname`, so the **OS hostname** is what gets mapped — not a compose replica index. The stock `default_robot_name_map.yaml` only matches `robot-`, so a device named `airlab-jetson-42` falls through to the catch-all and comes up as **`ROBOT_NAME=unknown_robot`, domain `0`** (with a map that has *no* catch-all, the resolver instead exits non-zero and `ROBOT_NAME` is left unset — same confusing "empty namespace" symptom). This is the usual "why is my real robot `unknown_robot`?" report. -On VOXL/Jetson with `ROBOT_NAME_SOURCE=hostname`, the device hostname must match a rule in the mapping YAML. If `hostname` returns `airlab-jetson-42` and your config only matches `robot-N`, the resolver exits non-zero and `ROBOT_NAME` is unset — the autonomy stack will then launch with empty namespaces and break in confusing ways. Either rename the device or extend the mapping config. +Pick whichever fix matches your topology (see [Configuring a Single Robot](#configuring-a-single-robot)): + +- **One robot, quickest:** pin `ROBOT_NAME=robot_1` + `ROS_DOMAIN_ID=1` in the deployment's override env file. The `.bashrc` guard honors it and skips the lookup — no hostname change, no map file. +- **One robot, machine-derived:** rename the device hostname to `robot-1` so the default map resolves it automatically. +- **A fleet:** name each machine `robot-` (default map handles it) **or** ship a mapping YAML that matches your hostnames and point `ROBOT_NAME_MAP_CONFIG_FILE` at it. Do **not** pin a single `ROBOT_NAME` on the shared service — every robot would collide on it. + +Verify on the device: + +```bash +docker exec bash -c 'echo "$(hostname) -> ROBOT_NAME=$ROBOT_NAME ROS_DOMAIN_ID=$ROS_DOMAIN_ID"' +``` ## Pre-Merge Checklist diff --git a/.agents/skills/docker-build-profiles/SKILL.md b/.agents/skills/docker-build-profiles/SKILL.md index 64b579a01..c57f85f3e 100644 --- a/.agents/skills/docker-build-profiles/SKILL.md +++ b/.agents/skills/docker-build-profiles/SKILL.md @@ -49,6 +49,8 @@ Troubleshooting notes - YAML quirk: unquoted `3.10` may be parsed as float `3.1` — this changes path strings and breaks imports (e.g., `python3.1` instead of `python3.10`). - Jetson/L4T builds may require `network: host` during the build to avoid kernel iptables/raw table missing-module errors. - Jetson **`robot-l4t`** builds from **`robot-l4t-stack-base`** (`robot/docker/Dockerfile.l4t-stack-base`), not raw dustynv, so **`Dockerfile.robot` stays Ubuntu-shaped.** `airstack image-build --profile l4t robot-l4t` triggers **`robot-l4t-stack-base`** first (`airstack.sh`); bare `compose build robot-l4t` can still parallelize badly, so list stack-base explicitly if not using AirStack CLI. +- **dustynv `/ros_entrypoint.sh` shadows the apt Jazzy runtime (mavros symbol-lookup crash).** The dustynv base sources a prebuilt *source* ROS at `$ROS_ROOT/install` from PID 1, prepending its older libs (e.g. `fastcdr` 2.2.5) ahead of the apt Jazzy (2.2.7) that `Dockerfile.robot` layers on top — apt-built nodes like mavros then die with symbol-lookup errors under tmux autolaunch. `Dockerfile.l4t-stack-base` neutralizes it by overwriting `/ros_entrypoint.sh` with a `exec "$@"` passthrough; shells get ROS from `/opt/ros/jazzy/setup.bash` via `.bashrc`. If a Jetson node suddenly can't resolve symbols after a base-image bump, check whether the entrypoint passthrough is still in place. +- **ZED SDK version is pinned across `zed/Dockerfile.zed-l4t`** — the `ZED_SDK_URL` (e.g. `.../zedsdk/5.2/...`) and the ROS dep args (`ZED_MSGS_VERSION`, `POINTCLOUD_TRANSPORT*_VERSION`, `BACKWARD_ROS_VERSION`) must move together; a mismatched `zed_msgs` vs SDK breaks the driver build. Bumping the SDK is camera-firmware-coupled, so confirm the target camera runs that SDK line before merging. Examples of agent prompts - "Check `robot/docker/docker-compose.yaml` for `PYTHON_VERSION` entries and quote any unquoted numeric values; open a PR with the fixes and include a test log from a builder-stage build." diff --git a/.env b/.env index bdd6f251e..85280be19 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.6" +VERSION="0.19.0-alpha.7" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/CHANGELOG.md b/CHANGELOG.md index 747f670b4..2bde19e49 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -12,11 +12,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Battery and telemetry display in GCS RQT control panel (voltage and percentage per robot when MAVROS battery topic is bridged) - `TARGET_ARCH` build arg (default `x86_64`) in `Dockerfile.robot` to arch-parametrize `LD_LIBRARY_PATH`; `docker-compose.yaml` passes `TARGET_ARCH: aarch64` to the `voxl` and `l4t` real-robot image builds - `ros-${ROS_DISTRO}-mavros-extras` in the robot image (provides the vision_pose plugin used for external-pose deployments) +- `overrides/l4t-px4-realrobot.env` — site-agnostic deployment override for a single real PX4 robot on a Jetson (aarch64/l4t) + +### Changed + +- `robot-l4t` compose service knobs are now env-overridable (`AUTONOMY_ROLE`, `FCU_URL`, and the rosbag path via `BAG_STORAGE_PATH`); `FCU_URL` unquoted so the literal serial path reaches MAVROS +- `zed-l4t` image: ZED SDK 4.2 → 5.2 with the coupled ROS deps (`zed_msgs` 5.2.1, `point_cloud_transport(_plugins)` 4.x, add `backward_ros`) ### Fixed - Robot name resolution now honors a pre-set `ROBOT_NAME` (e.g. injected via docker compose) instead of always overriding it from the container/hostname mapping (`robot/docker/.bashrc`) - Robot name-map catch-all fallback now maps to `unknown_robot` (valid ROS namespace token) instead of `unknown-robot` (`default_robot_name_map.yaml`) +- l4t robot image: replace dustynv's `/ros_entrypoint.sh` with a passthrough so its prebuilt source-ROS libs (older `fastcdr`) no longer shadow the apt Jazzy runtime and crash apt-built nodes like MAVROS ## [1.0.0] - 2024-12-19 diff --git a/docs/robot/docker/robot_identity.md b/docs/robot/docker/robot_identity.md index 676f20896..494b5daae 100644 --- a/docs/robot/docker/robot_identity.md +++ b/docs/robot/docker/robot_identity.md @@ -51,7 +51,7 @@ mappings: domain_id: '{1}' - pattern: '.*' - robot: 'unknown-robot' + robot: 'unknown_robot' # must be a valid ROS token (no hyphen) or launch fails domain_id: '0' ``` diff --git a/overrides/l4t-px4-realrobot.env b/overrides/l4t-px4-realrobot.env new file mode 100644 index 000000000..271726801 --- /dev/null +++ b/overrides/l4t-px4-realrobot.env @@ -0,0 +1,34 @@ +# Real-robot deployment on an NVIDIA Jetson (aarch64 / l4t) with a PX4 flight +# controller (e.g. Cube Orange) over serial. + +# Build (first time / after image changes): +# airstack image-build --profile l4t robot-l4t +# Run: +# airstack up --env-file overrides/l4t-px4-realrobot.env robot-l4t + +# Only bring up the Jetson stack (robot-l4t + zed-l4t). +COMPOSE_PROFILES="l4t" + +# Launch the autonomy stack automatically on container start. +AUTOLAUNCH="true" +NUM_ROBOTS="1" + +# --- Robot identity ----------------------------------------------------------- +# Shortcut to name only a single agent. +ROBOT_NAME="robot_1" +ROS_DOMAIN_ID="1" + +# Launches entire robot autonomy stack +AUTONOMY_ROLE="full" + +# --- Flight controller (MAVROS) ---------------------------------------------- +# Default is the Jetson UART (ttyTHS4). +FCU_URL="/dev/ttyTHS4:115200" + +# --- Robot description (PX4 iris w/ sensors; override for your airframe) ------ +URDF_FILE="robot_descriptions/iris/urdf/iris_with_sensors.pegasus.robot.urdf" + +# --- Flight-data recording ---------------------------------------------------- +# Where rosbags land on the host (bind-mounted to /bags in-container). +BAG_STORAGE_PATH="/media/airlab/Storage/airstack_collection" +RECORD_BAGS="false" diff --git a/robot/docker/Dockerfile.l4t-stack-base b/robot/docker/Dockerfile.l4t-stack-base index 0ebb8cd14..b0ef6aaf7 100644 --- a/robot/docker/Dockerfile.l4t-stack-base +++ b/robot/docker/Dockerfile.l4t-stack-base @@ -48,3 +48,7 @@ ENV PYTHON_EXECUTABLE=/usr/bin/python3 # Prefer system/cuda tooling over any removed venv prefix (CUDA symlink is usually /usr/local/cuda). ENV PATH="/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" + +# Neutralize dustynv's /ros_entrypoint.sh: it prepends a prebuilt source-ROS lib dir +# that shadows the apt Jazzy (older fastcdr) and crashes mavros. See docker-build-profiles skill. +RUN printf '#!/bin/bash\nexec "$@"\n' > /ros_entrypoint.sh && chmod +x /ros_entrypoint.sh diff --git a/robot/docker/docker-compose.yaml b/robot/docker/docker-compose.yaml index 71b71e425..31b3e368a 100644 --- a/robot/docker/docker-compose.yaml +++ b/robot/docker/docker-compose.yaml @@ -209,15 +209,15 @@ services: - ROBOT_NAME_SOURCE=hostname - AUTOLAUNCH=${AUTOLAUNCH:-true} - LAUNCH_PACKAGE=autonomy_bringup - - AUTONOMY_ROLE=full # l4t profile: Jetson runs everything onboard + - AUTONOMY_ROLE=${AUTONOMY_ROLE:-full} - LAUNCH_NATNET=${LAUNCH_NATNET:-false} # mavros mavlink settings - - FCU_URL="/dev/ttyTHS4:115200" + - FCU_URL=${FCU_URL:-/dev/ttyTHS4:115200} - TGT_SYSTEM=1 # assumes network isolation via a physical router, so uses network_mode=host network_mode: host volumes: - - /media/airlab/Storage/airstack_collection:/bags:rw + - ${BAG_STORAGE_PATH:-/media/airlab/Storage/airstack_collection}:/bags:rw # ----------------------- # Jetson in lite mode: only local/perception/interface modules run onboard. diff --git a/robot/docker/zed/Dockerfile.zed-l4t b/robot/docker/zed/Dockerfile.zed-l4t index b39ec2bc7..a59733e31 100644 --- a/robot/docker/zed/Dockerfile.zed-l4t +++ b/robot/docker/zed/Dockerfile.zed-l4t @@ -15,7 +15,7 @@ ARG ROS2_DIST=jazzy ENV DEBIAN_FRONTEND noninteractive # ZED SDK link -ENV ZED_SDK_URL="https://download.stereolabs.com/zedsdk/4.2/l4t$L4T_MAJOR.$L4T_MINOR/jetsons" +ENV ZED_SDK_URL="https://download.stereolabs.com/zedsdk/5.2/l4t$L4T_MAJOR.$L4T_MINOR/jetsons" RUN mkdir -p /tmp && chmod 1777 /tmp @@ -53,12 +53,13 @@ ARG XACRO_VERSION=2.0.8 ARG DIAGNOSTICS_VERSION=4.0.0 ARG AMENT_LINT_VERSION=0.12.11 ARG ROBOT_LOCALIZATION_VERSION=3.5.3 -ARG ZED_MSGS_VERSION=4.2.2 +ARG ZED_MSGS_VERSION=5.2.1 ARG NMEA_MSGS_VERSION=2.0.0 ARG ANGLES_VERSION=1.15.0 ARG GEOGRAPHIC_INFO_VERSION=1.0.6 -ARG POINTCLOUD_TRANSPORT_VERSION=1.0.18 -ARG POINTCLOUD_TRANSPORT_PLUGINS_VERSION=1.0.11 +ARG POINTCLOUD_TRANSPORT_VERSION=4.0.9 +ARG POINTCLOUD_TRANSPORT_PLUGINS_VERSION=4.0.4 +ARG BACKWARD_ROS_VERSION=1.0.8 RUN wget https://github.com/ros/xacro/archive/refs/tags/${XACRO_VERSION}.tar.gz -O - | tar -xvz && mv xacro-${XACRO_VERSION} xacro && \ wget https://github.com/ros/diagnostics/archive/refs/tags/${DIAGNOSTICS_VERSION}.tar.gz -O - | tar -xvz && mv diagnostics-${DIAGNOSTICS_VERSION} diagnostics && \ @@ -69,6 +70,7 @@ RUN wget https://github.com/ros/xacro/archive/refs/tags/${XACRO_VERSION}.tar.gz wget https://github.com/ros/angles/archive/refs/tags/${ANGLES_VERSION}.tar.gz -O - | tar -xvz && mv angles-${ANGLES_VERSION} angles && \ wget https://github.com/ros-perception/point_cloud_transport/archive/refs/tags/${POINTCLOUD_TRANSPORT_VERSION}.tar.gz -O - | tar -xvz && mv point_cloud_transport-${POINTCLOUD_TRANSPORT_VERSION} point_cloud_transport && \ wget https://github.com/ros-perception/point_cloud_transport_plugins/archive/refs/tags/${POINTCLOUD_TRANSPORT_PLUGINS_VERSION}.tar.gz -O - | tar -xvz && mv point_cloud_transport_plugins-${POINTCLOUD_TRANSPORT_PLUGINS_VERSION} point_cloud_transport_plugins && \ + wget https://github.com/pal-robotics/backward_ros/archive/refs/tags/${BACKWARD_ROS_VERSION}.tar.gz -O - | tar -xvz && mv backward_ros-${BACKWARD_ROS_VERSION} backward_ros && \ wget https://github.com/ros-geographic-info/geographic_info/archive/refs/tags/${GEOGRAPHIC_INFO_VERSION}.tar.gz -O - | tar -xvz && mv geographic_info-${GEOGRAPHIC_INFO_VERSION} geographic-info && \ cp -r geographic-info/geographic_msgs/ . && \ rm -rf geographic-info From 47f8c798d591b99bc79c152ede75755b582bdbcb Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Fri, 31 Jul 2026 17:16:11 -0400 Subject: [PATCH 09/21] Test infra rework: YAML-driven unit-test collection + integration tier (#372) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(l4t): make robot-l4t deployment knobs overridable + document name resolution Parametrize the robot-l4t compose service so a single service covers real deployments without editing compose: - AUTONOMY_ROLE and FCU_URL are now ${VAR:-default} overridable (and FCU_URL is unquoted so the literal serial path reaches mavros). - Rosbag output path is BAG_STORAGE_PATH-overridable. Update the configure-multi-robot skill to reflect the honor-pre-set-ROBOT_NAME guard (#370): document pinning ROBOT_NAME in an override for a single real robot, the never-on-the-shared-service caveat, and the unknown_robot fallback fixes by topology. Co-Authored-By: Claude Opus 4.8 * feat(l4t): add site-agnostic l4t-px4-realrobot override template Deployment override for a single real PX4 robot on a Jetson (aarch64/l4t). Surfaces the common knobs at the top with sensible defaults: ROBOT_NAME pinned directly (single-robot shortcut honored by .bashrc), FCU_URL, AUTONOMY_ROLE, BAG_STORAGE_PATH, and RECORD_BAGS. Mocap-agnostic — NatNet/external-vision settings are added by a separate optitrack override. Co-Authored-By: Claude Opus 4.8 * fix(l4t): entrypoint passthrough + ZED SDK 5.2; document build gotchas Two real-hardware build fixes for the Jetson profile: - Dockerfile.l4t-stack-base: overwrite dustynv's /ros_entrypoint.sh with an `exec "$@"` passthrough. Its prebuilt source-ROS libs (fastcdr 2.2.5) were shadowing the apt Jazzy (2.2.7) that Dockerfile.robot layers on, crashing apt-built nodes like mavros with symbol-lookup errors under tmux autolaunch. - zed/Dockerfile.zed-l4t: bump ZED SDK 4.2 -> 5.2 and move the coupled ROS deps together (zed_msgs 5.2.1, point_cloud_transport(_plugins) 4.x, add backward_ros). Document both gotchas in the docker-build-profiles skill, and correct the stale unknown-robot -> unknown_robot in the robot_identity reference doc. Co-Authored-By: Claude Opus 4.8 * chore: bump version to 0.19.0-alpha.7 Version-increment gate: bump above develop's 0.19.0-alpha.6 and record the l4t deployment changes in the changelog. Co-Authored-By: Claude Opus 4.8 * test(infra): collect co-located unit tests via the package list + integration tier Unit tests are defined by tests/colcon_unit_test_packages.yaml: conftest.py resolves each listed package to its /test dir and collects the non-linter test_*.py files under --import-mode=importlib (set in pytest.ini), marking each `unit` by path. ament lint files are skipped (they run under colcon test). Removes two now-unnecessary files under tests/robot/; the package test/ dirs are collected directly. Also add an integration test tier: tests/integration/ + `integration` mark + a shared robot_autonomy_stack fixture (robot-desktop container, no sim/GPU), slotted into _MODULE_ORDER between build_packages and the sim tiers. Co-Authored-By: Claude Opus 4.8 * docs(testing): describe unit tests as co-located and listed in the package YAML Update the add-unit-tests and run-system-tests skills, AGENTS.md, and the unit-testing docs: adding a unit test is "list the package in colcon_unit_test_packages.yaml", and the source lives in the package's own test/ dir. Document the `integration` mark/tier. Co-Authored-By: Claude Opus 4.8 * chore: bump version to 0.19.0-alpha.8 Version-increment gate: bump above develop (0.19.0-alpha.6); alpha.7 is taken by the l4t-deployment-fix PR. Record the test-infra changes in the changelog. Co-Authored-By: Claude Opus 4.8 * refactor(tests): split unit-test discovery + session state into tests/harness/ Begin modularizing conftest.py (959 lines) by concern. Extract two self-contained pieces into a new tests/harness/ package: - harness/session.py: session-scoped mutable state (results dir, current pytest item, last subprocess output, logger) with setter/getter accessors. Hooks write it; helpers read it, so helper modules no longer reach into conftest globals. - harness/discovery.py: unit-test discovery driven by colcon_unit_test_packages.yaml (repo_path, load_colcon_unit_test_config, colcon_test_robot_command, unit_test_dirs, unit_test_files, _is_unit_item). conftest.py imports from harness and its hooks delegate to the session accessors; it re-exports AIRSTACK_ROOT / colcon_test_robot_command / load_colcon_unit_test_config / logger so existing `from conftest import ...` in the system tests keeps working unchanged. Behavior-preserving (host-validated): `-m unit` still 14 passed / 152 deselected, 166 collected, same order. Follow-on: the commands/containers/metrics/sim helpers and collection ordering move out the same way. Co-Authored-By: Claude Opus 4.8 * refactor(tests): extract commands/containers/metrics/sim helpers into tests/harness/ Continue modularizing conftest.py. Move the subprocess/ros2 command helpers (harness/commands.py), docker container + compute-usage + image helpers (harness/containers.py), MetricsRecorder + get_metrics/current_test_id (harness/metrics.py), and the sim target configs + ros2 topic sampling (harness/sim.py) out of conftest.py. conftest.py drops from 836 to 360 lines and re-exports the harness helper API (`from harness import *`) so `from conftest import ` in the system tests + sensor_probes keeps working unchanged. Behavior-preserving: -m unit still 14 passed / 152 deselected, 166 collected, same order. Remaining in conftest: pytest hooks, collection ordering, and the airstack_env / robot_autonomy_stack fixtures. Co-Authored-By: Claude Opus 4.8 * refactor(tests): extract collection ordering into tests/harness/collection.py Final step of the conftest.py modularization: move test ordering — _MODULE_ORDER, the per-module phase chains, _module_key, and the parametrize-id rewrite — into harness/collection.py. conftest's pytest_collection_modifyitems hook now delegates to collection.modify_items(items). conftest.py is now 246 lines (from 959): pytest hooks + the airstack_env / robot_autonomy_stack fixtures. All helpers live in tests/harness/ by concern (session, discovery, commands, containers, metrics, sim, collection). Behavior-preserving: -m unit still 14 passed / 152 deselected, 166 collected, unit → build → integration → sim order unchanged. Co-Authored-By: Claude Opus 4.8 Also sync docs/skills to the tests/harness/ layout (AGENTS.md, tests/README.md, tests/integration/README.md, run-system-tests + add-unit-tests skills, unit_testing + end_to_end_testing docs): helpers, MetricsRecorder, the workspace globs, and _MODULE_ORDER now point at tests/harness/ instead of conftest.py (still re-exported via conftest). Co-authored-by: Cursor * fix(robot): pin pytest to 7.4.* so apt launch_pytest stays compatible The builder-stage pip block pulled pytest >=8 transitively into /usr/local (copied into the runtime image), shadowing Jazzy's apt python3-pytest 7.4. pytest 8 removed the `path` argument from pytest_pycollect_makemodule, which apt's launch_pytest plugin still declares — so every pytest invocation in the robot container aborted at plugin registration. This broke `colcon test` for ament_python packages (e.g. lidar_point_cloud_filter in test_colcon_test_robot), while ament_cmake gtest packages were unaffected. Pin pytest to Jazzy's version so the container is internally consistent and launch_testing / launch_pytest remain usable for future launch-based tests. The test runner (tests/docker) is a separate interpreter and keeps its newer pytest. Co-Authored-By: Claude Opus 4.8 * fix(isaac-sim): clear LD_LIBRARY_PATH for PX4 ubuntu.sh so ca-certificates configures The global ENV LD_LIBRARY_PATH puts isaac-sim's bundled libs (.../isaacsim.ros2.bridge/jazzy/lib) on the linker path. Its older libcrypto.so.3 shadows the system one, so when the updated ca-certificates (20240203 → 20260601~24.04.1) runs its postinst `openssl`, it fails with `version 'OPENSSL_3.0.9' not found`, aborting the apt transaction and failing the isaac-sim image build (PX4 Tools/setup/ubuntu.sh, exit 100). Clear LD_LIBRARY_PATH for that RUN only so apt/openssl use the system libcrypto; the global ENV still applies to every other layer. Environmental break (new ca-certificates × isaac-sim's stale bundled openssl) — not a code regression. Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 Co-authored-by: Cursor --- .agents/skills/add-unit-tests/SKILL.md | 139 ++-- .agents/skills/docker-build-profiles/SKILL.md | 1 + .agents/skills/run-system-tests/SKILL.md | 16 +- .env | 2 +- AGENTS.md | 8 +- CHANGELOG.md | 2 + .../testing/end_to_end_testing.md | 2 +- .../development/intermediate/testing/index.md | 9 +- .../intermediate/testing/unit_testing.md | 68 +- robot/docker/Dockerfile.robot | 1 + .../isaac-sim/docker/Dockerfile.isaac-ros | 4 +- tests/README.md | 43 +- tests/colcon_unit_test_packages.yaml | 6 +- tests/conftest.py | 739 +++--------------- tests/harness/__init__.py | 63 ++ tests/harness/collection.py | 125 +++ tests/harness/commands.py | 92 +++ tests/harness/containers.py | 164 ++++ tests/harness/discovery.py | 125 +++ tests/harness/metrics.py | 55 ++ tests/harness/session.py | 59 ++ tests/harness/sim.py | 189 +++++ tests/integration/README.md | 36 + tests/pytest.ini | 3 +- tests/robot/README.md | 40 +- .../natnet_ros2/test_natnet_ros2.py | 32 - .../test_validation_core.py | 38 - 27 files changed, 1145 insertions(+), 916 deletions(-) create mode 100644 tests/harness/__init__.py create mode 100644 tests/harness/collection.py create mode 100644 tests/harness/commands.py create mode 100644 tests/harness/containers.py create mode 100644 tests/harness/discovery.py create mode 100644 tests/harness/metrics.py create mode 100644 tests/harness/session.py create mode 100644 tests/harness/sim.py create mode 100644 tests/integration/README.md delete mode 100644 tests/robot/perception/natnet_ros2/test_natnet_ros2.py delete mode 100644 tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py diff --git a/.agents/skills/add-unit-tests/SKILL.md b/.agents/skills/add-unit-tests/SKILL.md index 7d1d3b582..d49a36d73 100644 --- a/.agents/skills/add-unit-tests/SKILL.md +++ b/.agents/skills/add-unit-tests/SKILL.md @@ -1,6 +1,6 @@ --- name: add-unit-tests -description: Add Python or C++ unit tests to an AirStack ROS 2 package. Covers the co-location pattern (test source in package/test/), the thin proxy that makes tests discoverable by pytest tests/ and airstack test -m unit, and how to extend the pattern to sim and GCS modules. +description: Add Python or C++ unit tests to an AirStack ROS 2 package. Covers the co-location pattern (test source in package/test/), registering the package in colcon_unit_test_packages.yaml so pytest tests/ and airstack test -m unit collect it, and how to extend to sim and GCS modules. license: MIT metadata: author: AirLab CMU @@ -23,30 +23,34 @@ For system tests (full Docker stack, sim, sensors, takeoff/hover/land) see the ## Architecture Overview -Unit tests follow a **co-location + proxy** pattern: +Unit test **source lives co-located with its package** (ROS 2 / colcon convention). +`tests/colcon_unit_test_packages.yaml` lists which packages have unit tests, and +`pytest tests/` collects them from there — you only edit files under the package itself. ``` robot/ros_ws/src/// ├── src/ # production source (Python or C++) ├── test/ -│ ├── test_.py # ← unit test SOURCE (canonical location) +│ ├── test_.py # ← unit test SOURCE (collected directly) │ ├── test_.cpp # ← C++ gtest SOURCE (optional) │ └── fake_.hpp # ← C++ test doubles (optional) └── CMakeLists.txt # wires ament_add_gtest under BUILD_TESTING -tests/robot/// -└── test_.py # ← thin PROXY (re-exports tests from above) +tests/colcon_unit_test_packages.yaml # ← list the package here (single source of truth) ``` -The **proxy** is a one-file shim that loads the real test module with `importlib` -and re-exports every `test_*` function. This means: +`tests/conftest.py` reads the YAML, resolves each listed package to its `test/` dir, +and injects the non-linter `test_*.py` files into collection under +`--import-mode=importlib` (set in `tests/pytest.ini`). Each collected item is +auto-tagged `@pytest.mark.unit` by path, so `-m unit` selects it. ament lint files +(`test_copyright.py`, etc.) are excluded — they run under `colcon test`. This means: | Invocation | What runs | |---|---| -| `pytest tests/ -m unit` | Proxy in `tests/robot/` → loads real test from package | +| `pytest tests/ -m unit` | Package `test/test_*.py`, collected directly from source | | `airstack test -m unit` | Same path | | CI `system-tests.yml` (PR open / approved) | Same path via `pytest tests/` | -| `colcon test --packages-select ` | Real test in `package/test/` directly | +| `colcon test --packages-select ` | Real test in `package/test/` (incl. linters + C++) | ## Step-by-Step: Adding a Python Unit Test @@ -112,62 +116,25 @@ For `rclpy.node.Node` subclasses use a real dummy base class instead of a `MagicMock()` to ensure `__init_subclass__` fires and method bodies are defined (see `test_natnet_ros2.py` for the full pattern). -### 3. Write the thin proxy in tests/robot/ +### 3. Register the package in colcon_unit_test_packages.yaml -Create `tests/robot///test_.py`: +If the package isn't already listed, add it under the `robot` workspace in +[`tests/colcon_unit_test_packages.yaml`](../../../tests/colcon_unit_test_packages.yaml): -```python -# Copyright (c) 2024 Carnegie Mellon University -# MIT License - see LICENSE in the repository root for full text. -"""Proxy: re-exposes unit tests from the package source tree. - -Unit test logic lives co-located with the package source (ROS 2 / colcon convention): - robot/ros_ws/src///test/test_.py - -This file makes those tests discoverable by ``pytest tests/`` (CI) and -``airstack test -m unit`` without any changes to the CI workflow. -""" -import importlib.util -import sys -from pathlib import Path - -_repo_root = Path(__file__).resolve().parents[N] # adjust N so this resolves to repo root -_pkg_test = _repo_root / "robot/ros_ws/src///test" -_real_file = _pkg_test / "test_.py" - -# If the test imports from a package module, ensure the package root is on sys.path. -# Example: _pkg_root = _pkg_test.parent; sys.path.insert(0, str(_pkg_root)) - -# Load the real module under a unique name to avoid the circular import that -# would occur if we used `from test_ import *` (this file has the same -# name, and pytest adds its directory to sys.path at collection time). -_spec = importlib.util.spec_from_file_location("__unit_tests", _real_file) -_real = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(_real) - -# Re-export every test_* symbol so pytest collects them from this proxy. -for _name in dir(_real): - if _name.startswith("test_"): - globals()[_name] = getattr(_real, _name) +```yaml +robot: + packages: + - natnet_ros2 + - lidar_point_cloud_filter + - # ← add here + pytest_args: "-m not linter" ``` -**Counting `parents[N]` to reach the repo root:** - -| Proxy location | `parents[N]` for repo root | -|---|---| -| `tests/robot///` | `parents[4]` | -| `tests/sim//` | `parents[3]` | -| `tests/gcs//` | `parents[3]` | - -### 4. Ensure the tests/ directory structure exists - -```bash -mkdir -p tests/robot// -touch tests/robot///__init__.py # only if needed for conftest path discovery -``` - -The READMEs in `tests/robot/behavior/`, `tests/robot/global/`, etc. describe the -purpose of each layer mirror. Update the layer README when you add a new package. +That's the whole registration. `conftest.py` globs +`robot/ros_ws/src/**//test`, collects its non-linter `test_*.py`, and marks +them `unit`. The test file must be self-contained: if it imports package code, set up +`sys.path` at the top of the test file (see `test_validation_core.py`, which inserts its +package root). Same YAML, different workspace key (`sim:`), for Isaac-extension unit tests. ### 5. Run locally to verify @@ -179,10 +146,10 @@ pytest -m unit -v airstack test -m unit -v ``` -All 14+ existing tests plus your new ones should pass. The proxy output shows: +All 14+ existing tests plus your new ones should pass. Collected items point straight +at the co-located source: ``` -robot///test_.py::test_my_function_basic - <- ../robot/ros_ws/src///test/test_.py PASSED +../robot/ros_ws/src///test/test_.py::test_my_function_basic PASSED ``` ### 6. CI picks it up automatically @@ -194,8 +161,7 @@ Unit tests are discovered by `pytest tests/` and run as part of `system-tests.ym ## Step-by-Step: Adding a C++ gtest -C++ tests don't use the proxy pattern — they live entirely within the package and -run exclusively via `colcon test`. +C++ tests live entirely within the package and run exclusively via `colcon test`. ### 1. Write the test in `package/test/` @@ -255,22 +221,17 @@ there are listed in [`tests/colcon_unit_test_packages.yaml`](../../../tests/colc ## Extending to sim and GCS -The same proxy pattern applies verbatim: +The same mechanism applies — add the package under a workspace key in the YAML. The +workspace→source glob is defined in `tests/harness/discovery.py` (`_WORKSPACE_PKG_TEST_GLOBS`): `robot` → +`robot/ros_ws/src/**//test`, `sim` → `simulation/**//test`. Add a new workspace +key there (e.g. `gcs`) if you extend to a new tree. -**Sim-side Python** (e.g. motive emulator protocol logic): +```yaml +# tests/colcon_unit_test_packages.yaml +sim: + packages: + - # → simulation/**//test collected directly ``` -simulation/...//test/test_.py ← source -tests/sim//test_.py ← proxy (parents[3] = repo root) -``` - -**GCS modules**: -``` -gcs/...//test/test_.py ← source -tests/gcs//test_.py ← proxy (parents[3] = repo root) -``` - -`pytest tests/ -m unit` discovers them through the proxy without any -pytest.ini or CI changes needed. --- @@ -279,14 +240,14 @@ pytest.ini or CI changes needed. | Concern | Answer | |---|---| | Where does test source live? | `/…//test/` (co-located with the package) | -| Where does pytest discover tests? | `tests/robot/` (or `tests/sim/`, `tests/gcs/`) via thin proxy | -| How does the proxy avoid circular import? | `importlib.util.spec_from_file_location` with a unique module name | -| What mark do all unit tests use? | `@pytest.mark.unit` | +| Where does pytest discover tests? | From the package `test/` dir listed in `colcon_unit_test_packages.yaml` | +| How are duplicate basenames handled? | `--import-mode=importlib` (set in `pytest.ini`) | +| What mark do all unit tests use? | `@pytest.mark.unit` (auto-applied by path in `conftest.py`) | | What CI workflow runs them? | `system-tests.yml` — runs `pytest tests/` which includes unit tests | | When does that workflow trigger? | PR opened, `/pytest` comment, `workflow_dispatch` | | Do system tests (`liveliness`, etc.) run too? | No — `-m unit` filters to hermetic tests only | | Does `colcon test` also run these? | Yes — Python tests in `package/test/` are discovered by colcon's pytest runner | -| Can I add pure C++ gtests? | Yes — `ament_add_gtest` in CMakeLists.txt, no proxy needed | +| Can I add pure C++ gtests? | Yes — `ament_add_gtest` in CMakeLists.txt | ## Reference Implementations @@ -296,14 +257,12 @@ pytest.ini or CI changes needed. | `natnet_ros2` (C++) | `robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp` | `build_covariance_6x6`, `negotiate()`, `INatNetClient` seam | | `lidar_point_cloud_filter` | `robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py` | Pure-numpy range validation rules | -Corresponding proxies: `tests/robot/perception/natnet_ros2/test_natnet_ros2.py`, -`tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py`. +Both are collected from their package `test/` dir. ## Files to Know - `.github/workflows/system-tests.yml` — CI workflow (runs `pytest tests/` including unit tests) -- `tests/pytest.ini` — mark registration (`unit`, `build_docker`, etc.) -- `tests/robot/` — proxy layer mirroring `robot/ros_ws/src/` -- `tests/sim/` — proxy layer for sim-side code (future) -- `tests/gcs/` — proxy layer for GCS code (future) +- `tests/pytest.ini` — mark registration + `--import-mode=importlib` +- `tests/colcon_unit_test_packages.yaml` — the package list driving unit-test collection +- `tests/conftest.py` — `unit_test_files()` / `pytest_configure` inject package tests; `pytest_itemcollected` auto-marks `unit` - `tests/README.md` — full test harness reference diff --git a/.agents/skills/docker-build-profiles/SKILL.md b/.agents/skills/docker-build-profiles/SKILL.md index c57f85f3e..3cff9e38c 100644 --- a/.agents/skills/docker-build-profiles/SKILL.md +++ b/.agents/skills/docker-build-profiles/SKILL.md @@ -51,6 +51,7 @@ Troubleshooting notes - Jetson **`robot-l4t`** builds from **`robot-l4t-stack-base`** (`robot/docker/Dockerfile.l4t-stack-base`), not raw dustynv, so **`Dockerfile.robot` stays Ubuntu-shaped.** `airstack image-build --profile l4t robot-l4t` triggers **`robot-l4t-stack-base`** first (`airstack.sh`); bare `compose build robot-l4t` can still parallelize badly, so list stack-base explicitly if not using AirStack CLI. - **dustynv `/ros_entrypoint.sh` shadows the apt Jazzy runtime (mavros symbol-lookup crash).** The dustynv base sources a prebuilt *source* ROS at `$ROS_ROOT/install` from PID 1, prepending its older libs (e.g. `fastcdr` 2.2.5) ahead of the apt Jazzy (2.2.7) that `Dockerfile.robot` layers on top — apt-built nodes like mavros then die with symbol-lookup errors under tmux autolaunch. `Dockerfile.l4t-stack-base` neutralizes it by overwriting `/ros_entrypoint.sh` with a `exec "$@"` passthrough; shells get ROS from `/opt/ros/jazzy/setup.bash` via `.bashrc`. If a Jetson node suddenly can't resolve symbols after a base-image bump, check whether the entrypoint passthrough is still in place. - **ZED SDK version is pinned across `zed/Dockerfile.zed-l4t`** — the `ZED_SDK_URL` (e.g. `.../zedsdk/5.2/...`) and the ROS dep args (`ZED_MSGS_VERSION`, `POINTCLOUD_TRANSPORT*_VERSION`, `BACKWARD_ROS_VERSION`) must move together; a mismatched `zed_msgs` vs SDK breaks the driver build. Bumping the SDK is camera-firmware-coupled, so confirm the target camera runs that SDK line before merging. +- **`pytest` is pinned to `7.4.*` in `Dockerfile.robot` — do not remove or bump it.** The builder-stage `pip3 install` pulls `pytest` transitively into `/usr/local` (copied into the runtime image), which shadows Jazzy's apt `python3-pytest` 7.4. `pytest` 8 removed the `path` argument from the `pytest_pycollect_makemodule` hook, which apt's `launch_pytest` plugin still declares — so an unpinned (>=8) pytest aborts **every** pytest run in the container at plugin registration. That breaks `colcon test` for `ament_python` packages (e.g. `lidar_point_cloud_filter` in `test_colcon_test_robot`), while `ament_cmake` gtest packages are unaffected. Keeping the pin at Jazzy's version keeps `launch_testing` / `launch_pytest` usable for launch-based tests. The `tests/docker` runner is a separate interpreter and is free to use a newer pytest. Examples of agent prompts - "Check `robot/docker/docker-compose.yaml` for `PYTHON_VERSION` entries and quote any unquoted numeric values; open a PR with the fixes and include a test log from a builder-stage build." diff --git a/.agents/skills/run-system-tests/SKILL.md b/.agents/skills/run-system-tests/SKILL.md index f9b41b727..3923b6842 100644 --- a/.agents/skills/run-system-tests/SKILL.md +++ b/.agents/skills/run-system-tests/SKILL.md @@ -25,7 +25,8 @@ This skill is about the **test harness itself** — pytest marks, fixtures, the The suite lives at `tests/` (repo root) and is fully pytest-based. Configuration is in `tests/pytest.ini` and shared infrastructure in `tests/conftest.py`. - **`tests/system/`** — Docker stack integration tests. Marks: `build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`, `autonomy`. -- **`tests/robot/`** and **`tests/sim/`** — Hermetic **unit** tests (`@pytest.mark.unit`). These are **thin proxy files** that re-export tests from each ROS 2 package's own `test/` directory (co-located with the source, the ROS 2 / colcon convention). The proxy pattern keeps test source next to the code it tests while making tests discoverable by `pytest tests/`. +- **`tests/integration/`** — Cross-component tests (`integration` mark): robot container + a host-side component, no sim/GPU. +- **Unit tests** (`@pytest.mark.unit`) — Hermetic. Source is **co-located** with each ROS 2 package in its own `test/` dir (ROS 2 / colcon convention). `tests/colcon_unit_test_packages.yaml` lists which packages have unit tests; `conftest.py` resolves each to its `test/` dir and collects the non-linter `test_*.py` under `--import-mode=importlib`. ### Unit tests vs system tests @@ -34,7 +35,7 @@ The suite lives at `tests/` (repo root) and is fully pytest-based. Configuration | Hardware required | None — pure Python | Docker daemon, NVIDIA GPU, sim license | | CI workflow | `system-tests.yml` (included in `pytest tests/`) | `system-tests.yml` (GPU OpenStack VM) | | Trigger | Every push + PR (automatic) | PR opened, `/pytest` comment, `workflow_dispatch` | -| Source location | `/test/test_*.py` (proxied via `tests/robot/`) | `tests/system/` | +| Source location | `/test/test_*.py` (collected directly, listed in `colcon_unit_test_packages.yaml`) | `tests/system/` | | How to add | See `add-unit-tests` skill | See *Adding a New System Test* below | Run unit tests without any Docker stack: @@ -45,7 +46,7 @@ airstack test -m unit -v pytest tests/ -m unit -v # AIRSTACK_ROOT=$(pwd) for direct pytest ``` -For details on the proxy pattern and adding new unit tests, see the +For details on the co-located layout and adding new unit tests, see the `add-unit-tests` skill. | File | Mark | What it tests | Hardware required | @@ -80,7 +81,7 @@ rates if too many `ros2 topic hz` processes run concurrently. - **Robot-side (Isaac):** two passes — both stereo images, then both depths. **ms-airsim** keeps a single four-topic parallel batch on the robot container. - **Filtered LiDAR** (`PointCloud2`): uses `ros2 topic echo --once` per robot - (see `parallel_echo_once_robot_topics` in `conftest.py`), not `topic hz`. + (see `parallel_echo_once_robot_topics` in `tests/harness/sim.py`), not `topic hz`. - **Multi-drone Pegasus script:** pytest sets `ENABLE_LIDAR=true` in `conftest.py` `SIM_CONFIG["isaacsim"]["extra_env"]` so LiDAR matches the single-drone example (which always enables RTX LiDAR). @@ -294,7 +295,7 @@ If your test... - File: `tests/system/test_.py` — matches pytest's default test discovery (`test_*.py`) under the system suite - Class: `Test` with the mark applied at the class level: `@pytest.mark.` - Add a class-level `@pytest.mark.timeout()` — long-running sim tests need it -- Imports: pull helpers from `conftest` directly (`from conftest import ...`); `tests/` is on `sys.path` because `testpaths = .` in pytest.ini +- Imports: pull helpers from `conftest` directly (`from conftest import ...`); they physically live in the `tests/harness/` package but are re-exported through `conftest`, so either `from conftest import ...` or `from harness import ...` works. `tests/` is on `sys.path` because `testpaths = .` in pytest.ini ### 3. Decide if you need `airstack_env` @@ -304,7 +305,7 @@ If your test... ### 4. Use the existing helpers -`conftest.py` exports a deliberate API. Prefer these over rolling your own: +The `tests/harness/` package exports a deliberate API (re-exported through `conftest`). Prefer these over rolling your own: | Helper | Purpose | |--------|---------| @@ -421,7 +422,8 @@ python tests/parse_metrics.py \ ### Files to know -- `tests/conftest.py` — fixtures, helpers, `MetricsRecorder`, ordering hooks +- `tests/conftest.py` — pytest hooks + the `airstack_env` / `robot_autonomy_stack` fixtures (re-exports the harness API) +- `tests/harness/` — helpers split by concern: `session`, `discovery`, `commands`, `containers`, `metrics` (`MetricsRecorder`), `sim`, `collection` (ordering) - `tests/pytest.ini` — mark registration, log format - `tests/parse_metrics.py` — markdown reporter, regression diff - `tests/README.md` — user-facing docs (CLI options, output layout, CI/CD orchestrator) diff --git a/.env b/.env index 85280be19..00cd4c639 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.7" +VERSION="0.19.0-alpha.8" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/AGENTS.md b/AGENTS.md index 0a6b86015..884fc1e0a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -93,7 +93,7 @@ For detailed step-by-step instructions, refer to the **`.agents/skills/`** direc | [debug-module](.agents/skills/debug-module) | Autonomous debugging of ROS 2 modules | | [update-documentation](.agents/skills/update-documentation) | Documenting new modules and updating mkdocs | | [test-in-simulation](.agents/skills/test-in-simulation) | End-to-end simulation testing of a module | -| [add-unit-tests](.agents/skills/add-unit-tests) | Adding Python or C++ unit tests to a ROS 2 package (co-location + proxy pattern, CI workflow, extending to sim/GCS) | +| [add-unit-tests](.agents/skills/add-unit-tests) | Adding Python or C++ unit tests to a ROS 2 package (co-located test/ dir listed in colcon_unit_test_packages.yaml, CI workflow, extending to sim/GCS) | | [run-system-tests](.agents/skills/run-system-tests) | Running the pytest system test harness (marks, MetricsRecorder, /pytest PR trigger) | | [add-behavior-tree-node](.agents/skills/add-behavior-tree-node) | Creating behavior tree nodes | | [use-airstack-cli](.agents/skills/use-airstack-cli) | Using the `airstack` CLI and the non-interactive `docker exec` pattern | @@ -196,13 +196,13 @@ docker exec airstack-robot-desktop-1 bash -c "ros2 topic echo --onc - Verify module behavior in isolation - Test with synthetic data - Located in module's `test/` directory - - **Run in the robot container** with `colcon test` (after `bws`) for the full ROS 2 build + test. The same co-located test source is re-exported to the root [`tests/`](tests/) suite via thin proxies (see Unit tests below), so `airstack test -m unit` runs it too. Marks are declared in [`tests/pytest.ini`](tests/pytest.ini) (`unit`, `build_docker`, `build_packages`, `liveliness`, `sensors`, `takeoff_hover_land`, `autonomy`). + - **Run in the robot container** with `colcon test` (after `bws`) for the full ROS 2 build + test. The same co-located test source is collected by the root [`tests/`](tests/) suite (the packages with unit tests are listed in [`tests/colcon_unit_test_packages.yaml`](tests/colcon_unit_test_packages.yaml)), so `airstack test -m unit` runs it too. Marks are declared in [`tests/pytest.ini`](tests/pytest.ini) (`unit`, `build_docker`, `build_packages`, `integration`, `liveliness`, `sensors`, `takeoff_hover_land`, `autonomy`). ```bash docker exec airstack-robot-desktop-1 bash -c "sws && colcon test --packages-select natnet_ros2 --event-handlers console_direct+" ``` -2. **Unit tests (`pytest`, `unit` mark):** Fast, hermetic checks. Test **source** lives co-located with each ROS 2 package in `/test/` (standard colcon convention). Thin **proxy** files in [`tests/robot/`](tests/robot/) and [`tests/sim/`](tests/sim/) re-export those tests so `pytest tests/` discovers them. Unit tests run as part of the `system-tests.yml` suite. Example: `airstack test -m unit -v`. See `add-unit-tests` skill. +2. **Unit tests (`pytest`, `unit` mark):** Fast, hermetic checks. Test **source** lives co-located with each ROS 2 package in `/test/` (standard colcon convention). [`tests/colcon_unit_test_packages.yaml`](tests/colcon_unit_test_packages.yaml) lists which packages have unit tests, and `tests/conftest.py` collects them from there under `--import-mode=importlib`. To add a package's unit tests, list it in that YAML. Unit tests run as part of the `system-tests.yml` suite. Example: `airstack test -m unit -v`. See `add-unit-tests` skill. 3. **System Level (`tests/system/`):** Full simulation tests (Isaac Sim or Microsoft AirSim legacy) - End-to-end autonomy stack testing @@ -223,7 +223,7 @@ Pytest-based system tests live under [`tests/system/`](tests/system/). They brin | [`tests/system/test_takeoff_hover_land.py`](tests/system/test_takeoff_hover_land.py) | `takeoff_hover_land` | 4-phase flight chain (PX4 ready → takeoff → hover → land) per (sim, num_robots, iter, velocity) | Docker, GPU, sim license | | [`tests/system/test_fixed_trajectory.py`](tests/system/test_fixed_trajectory.py) | `autonomy` | 4-phase flight chain (PX4 ready → takeoff → execute Circle/Figure8/Racetrack/Line trajectory → land) per (sim, num_robots, iter, trajectory_type); records cross-track error and path RMSE | Docker, GPU, sim license | -Shared fixtures, the `airstack_env` parametrized fixture, and `MetricsRecorder` live in [`tests/conftest.py`](tests/conftest.py). Each run produces a timestamped directory under `tests/results//` with `summary.txt`, `results.xml`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). +The pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures live in [`tests/conftest.py`](tests/conftest.py); the shared helpers are split by concern into the [`tests/harness/`](tests/harness/) package (`session`, `discovery`, `commands`, `containers`, `metrics` (with `MetricsRecorder`), `sim`, `collection`) and re-exported through `conftest`, so `from conftest import ` still resolves. Each run produces a timestamped directory under `tests/results//` with `summary.txt`, `results.xml`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). **Run via the CLI** (containerized runner — no local Python needed): diff --git a/CHANGELOG.md b/CHANGELOG.md index 2bde19e49..a5ea32f38 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,11 +13,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `TARGET_ARCH` build arg (default `x86_64`) in `Dockerfile.robot` to arch-parametrize `LD_LIBRARY_PATH`; `docker-compose.yaml` passes `TARGET_ARCH: aarch64` to the `voxl` and `l4t` real-robot image builds - `ros-${ROS_DISTRO}-mavros-extras` in the robot image (provides the vision_pose plugin used for external-pose deployments) - `overrides/l4t-px4-realrobot.env` — site-agnostic deployment override for a single real PX4 robot on a Jetson (aarch64/l4t) +- `integration` test tier (`tests/integration/`, `integration` mark) with a shared `robot_autonomy_stack` fixture (robot container, no sim/GPU) ### Changed - `robot-l4t` compose service knobs are now env-overridable (`AUTONOMY_ROLE`, `FCU_URL`, and the rosbag path via `BAG_STORAGE_PATH`); `FCU_URL` unquoted so the literal serial path reaches MAVROS - `zed-l4t` image: ZED SDK 4.2 → 5.2 with the coupled ROS deps (`zed_msgs` 5.2.1, `point_cloud_transport(_plugins)` 4.x, add `backward_ros`) +- Unit tests are defined by `tests/colcon_unit_test_packages.yaml`: `conftest.py` collects each listed package's co-located `test/` dir under `--import-mode=importlib` and marks it `unit` (ament lint files are skipped and run under `colcon test`) ### Fixed diff --git a/docs/development/intermediate/testing/end_to_end_testing.md b/docs/development/intermediate/testing/end_to_end_testing.md index 4863d7964..6cddf945b 100644 --- a/docs/development/intermediate/testing/end_to_end_testing.md +++ b/docs/development/intermediate/testing/end_to_end_testing.md @@ -219,7 +219,7 @@ tests/results// | -------- | -------- | --- | | `summary.txt` | `tests/run_summary.py` (auto at session end via `conftest.py`) | Quick pass/fail + key numbers per trajectory type | | `results.xml` | pytest `--junitxml` | CI, phase wall times | -| `metrics.json` | `MetricsRecorder` in `conftest.py` | Regression diffs | +| `metrics.json` | `MetricsRecorder` in `tests/harness/metrics.py` | Regression diffs | ### Regenerate or inspect diff --git a/docs/development/intermediate/testing/index.md b/docs/development/intermediate/testing/index.md index e6ce09c0b..c23195477 100644 --- a/docs/development/intermediate/testing/index.md +++ b/docs/development/intermediate/testing/index.md @@ -12,8 +12,9 @@ hardware requirement: ## Unit tests (`pytest -m unit`) Fast, hermetic Python tests that run in seconds with no Docker or GPU. Test source -lives **co-located with its ROS 2 package** (`/test/`) and is re-exported -through thin proxy files in `tests/robot/` for centralized discovery. +lives **co-located with its ROS 2 package** (`/test/`); the packages with unit +tests are listed in `tests/colcon_unit_test_packages.yaml`, and `pytest tests/` collects +them from there. ```bash airstack test -m unit -v @@ -24,7 +25,7 @@ pytest tests/ -m unit -v Unit tests run as part of `system-tests.yml` via `pytest tests/` and can also be run locally with no Docker or GPU needed. -→ **[Unit Testing Guide](unit_testing.md)** — patterns, proxy layout, CI workflow, +→ **[Unit Testing Guide](unit_testing.md)** — patterns, CI workflow, how to add tests for new packages (Python and C++ gtest). ## System tests (`tests/system/`) @@ -83,7 +84,7 @@ airstack test -m "build_packages or autonomy" \ ## Other testing docs -- [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, proxy pattern, CI workflow +- [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, co-located tests, CI workflow - [Testing frameworks](testing_frameworks.md) — `colcon test`, rostest patterns - [Integration testing](integration_testing.md) - [CI/CD](ci_cd.md) — pipeline overview diff --git a/docs/development/intermediate/testing/unit_testing.md b/docs/development/intermediate/testing/unit_testing.md index f69056008..dd2f72cf3 100644 --- a/docs/development/intermediate/testing/unit_testing.md +++ b/docs/development/intermediate/testing/unit_testing.md @@ -5,8 +5,8 @@ AirStack unit tests are **fast, hermetic, and purely Python** — no Docker stac ## Design principles - **Co-located with source.** Test files live in `/test/` alongside the code they test. This is the standard ROS 2 / colcon convention and ensures tests are discovered by both `colcon test` and `pytest`. -- **Proxy for centralized discovery.** A thin shim in `tests/robot///` re-exports the test functions so `pytest tests/` (the CI command) and `airstack test -m unit` discover them without any changes to the CI workflow. -- **`@pytest.mark.unit` on every test.** The `unit` mark is the filter that keeps unit tests isolated from system tests that need Docker, GPUs, and sim licenses. +- **Listed in one place.** `tests/colcon_unit_test_packages.yaml` lists which packages have unit tests. `tests/conftest.py` resolves each to its `test/` dir and collects the non-linter `test_*.py` files under `--import-mode=importlib`. To add a package's unit tests, list it in that YAML. +- **`@pytest.mark.unit` on every test.** Auto-applied by path in `conftest.py` (source files may also declare it). The `unit` mark keeps unit tests isolated from system tests that need Docker, GPUs, and sim licenses. ## Repository layout @@ -15,24 +15,18 @@ robot/ros_ws/src/ └── // ├── src/ # production source └── test/ - ├── test_.py # unit test source ← canonical location + ├── test_.py # unit test source ← canonical location (collected directly) ├── test_.cpp # C++ gtest source (optional) └── fake_.hpp # C++ test doubles (optional) tests/ -├── robot/ -│ └── // -│ └── test_.py # thin proxy → package test/ -├── sim/ # future: sim-side unit tests -└── gcs/ # future: GCS unit tests +└── colcon_unit_test_packages.yaml # lists the packages whose test/ dirs are collected ``` -When pytest collects `tests/robot/…/test_.py`, the `<-` annotation in the -output shows the actual source location: +Collected items point straight at the co-located source: ``` -robot/perception/natnet_ros2/test_natnet_ros2.py::test_canonical_quaternion_identity - <- ../robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py PASSED +../robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py::test_canonical_quaternion_identity PASSED ``` ## Running unit tests @@ -117,29 +111,19 @@ sys.modules["rclpy.node"] = _rclpy_node_mod # ... then import your module ``` -**2. Write the thin proxy in `tests/robot/`:** +**2. Register the package in `tests/colcon_unit_test_packages.yaml`:** -```python -# tests/robot///test_my_module.py -"""Proxy: re-exposes unit tests from the package source tree.""" -import importlib.util -from pathlib import Path - -_repo_root = Path(__file__).resolve().parents[4] # adjust depth if needed -_real_file = _repo_root / "robot/ros_ws/src///test/test_my_module.py" - -_spec = importlib.util.spec_from_file_location("__unit_tests", _real_file) -_real = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(_real) - -for _name in dir(_real): - if _name.startswith("test_"): - globals()[_name] = getattr(_real, _name) +```yaml +robot: + packages: + - # ← add here; conftest.py collects /test/test_*.py + pytest_args: "-m not linter" ``` -The unique module name (e.g. `"__unit_tests"`) prevents a circular import: -pytest adds the proxy's directory to `sys.path` at collection time, which would -cause `from test_my_module import *` to import the proxy itself. +That's the whole registration. If the test imports package code, set up `sys.path` at the +top of the test file — see `test_validation_core.py`, which inserts its package root. +`--import-mode=importlib` (set in `pytest.ini`) means duplicate `test_*.py` basenames +across packages don't collide. **3. Verify:** @@ -149,7 +133,7 @@ pytest tests/ -m unit -v ### C++ (gtest) -C++ tests live entirely in the package and run via `colcon test` — no proxy needed. +C++ tests live entirely in the package and run via `colcon test`. **`CMakeLists.txt`:** @@ -183,16 +167,16 @@ The `build_packages` CI job (`tests/system/test_build_packages.py`) also runs ## Extending to sim and GCS -The proxy pattern extends to other components. As sim-side Python logic (e.g. the -[Motive emulator](../../../../tests/sim/motive_emulator/README.md)) and GCS modules -acquire unit-testable code, follow the same layout: - -``` -simulation/...//test/test_.py ← source -tests/sim//test_.py ← proxy (parents[3] to reach repo root) +The mechanism extends to other components via the same YAML. `tests/harness/discovery.py` +(`_WORKSPACE_PKG_TEST_GLOBS`) maps each workspace key to a source glob — `robot` → +`robot/ros_ws/src/**//test`, `sim` → `simulation/**//test`. Add a `sim:` (or a +new `gcs:`) workspace to the YAML, adding the glob for a new tree in `tests/harness/discovery.py`: -gcs/...//test/test_.py ← source -tests/gcs//test_.py ← proxy +```yaml +# tests/colcon_unit_test_packages.yaml +sim: + packages: + - # → simulation/**//test collected directly ``` `pytest tests/ -m unit` discovers them automatically — no changes to `pytest.ini` diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index 92fd116fe..3ff968750 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -131,6 +131,7 @@ RUN if echo "$BASE_IMAGE" | grep -qE "(nvidia|l4t)" && [ "${SKIP_TENSORRT}" != " # Note: numpy>=1.26 required for Python 3.12 compatibility # Using --ignore-installed to avoid conflicts with system packages RUN pip3 install --break-system-packages --ignore-installed \ + "pytest==7.4.*" \ empy==3.3.4 \ future \ lxml \ diff --git a/simulation/isaac-sim/docker/Dockerfile.isaac-ros b/simulation/isaac-sim/docker/Dockerfile.isaac-ros index 0dca11fb7..92ac63fdf 100644 --- a/simulation/isaac-sim/docker/Dockerfile.isaac-ros +++ b/simulation/isaac-sim/docker/Dockerfile.isaac-ros @@ -155,8 +155,10 @@ RUN sed -i \ /isaac-sim/PX4-Autopilot/ROMFS/px4fmu_common/init.d-posix/px4-rc.simulator # install px4 dependencies and build +# LD_LIBRARY_PATH= so apt/openssl use the system libcrypto, not isaac-sim's older +# bundled one (which breaks the ca-certificates postinst). Cleared for this step only. RUN cd PX4-Autopilot && \ - DEBIAN_FRONTEND=noninteractive ./Tools/setup/ubuntu.sh + LD_LIBRARY_PATH= DEBIAN_FRONTEND=noninteractive ./Tools/setup/ubuntu.sh # build px4 RUN cd PX4-Autopilot && \ make px4_sitl diff --git a/tests/README.md b/tests/README.md index 6aec42e1b..93a07dda8 100644 --- a/tests/README.md +++ b/tests/README.md @@ -1,12 +1,12 @@ # Testing (`tests/`) -AirStack's **pytest** tree under `tests/` has three roles: +AirStack's **pytest** tree under `tests/` has these roles: 1. **`tests/system/`** — Docker stack tests (sim + robot + GCS): liveliness, sensor Hz, takeoff/hover/land, image/workspace builds. -2. **`tests/robot/`** — Fast **unit** tests that mirror `robot/ros_ws/src/` (`behavior`, `global`, `interface`, `local`, `perception`, `sensors`). Mark: `unit`. -3. **`tests/sim/`** — Unit tests for simulation-side helpers (e.g. Motive / NatNet emulator). Mark: `unit`. +2. **Unit tests** — Fast hermetic tests (`unit` mark) whose **source is co-located** with each ROS 2 package at `/test/`. [`colcon_unit_test_packages.yaml`](colcon_unit_test_packages.yaml) lists which packages have unit tests, and `pytest tests/` collects them from there. +3. **`tests/integration/`** — Cross-component tests (`integration` mark) that wire the robot container to a host-side component, without a sim or GPU. -Shared fixtures live in `tests/conftest.py`. Use `airstack test -m unit -v` for hermetic tests only, or the marks below for the full stack. +Pytest hooks and the shared fixtures live in `tests/conftest.py`; reusable helpers are split by concern into the [`tests/harness/`](harness/) package (re-exported through `conftest`). Use `airstack test -m unit -v` for hermetic tests only, or the marks below for the full stack. @@ -25,15 +25,18 @@ Shared fixtures live in `tests/conftest.py`. Use `airstack test -m unit -v` for | [`system/test_takeoff_hover_land.py`](system/test_takeoff_hover_land.py) | `takeoff_hover_land` | End-to-end flight: PX4 readiness gate, takeoff to 10 m, hover stability, land — one chain per (sim, num_robots, iteration, velocity) | Docker daemon, GPU, sim license | | [`system/test_fixed_trajectory.py`](system/test_fixed_trajectory.py) | `autonomy` | Fixed-pattern trajectory evaluation: takeoff, execute a trajectory (Circle, Figure8, Racetrack, Line), record path deviation metrics, land — one chain per (sim, num_robots, iteration, trajectory_type) | Docker daemon, GPU, sim license | -### Unit tests (`tests/robot/`, `tests/sim/`) +### Unit tests (co-located) Hermetic tests use `@pytest.mark.unit` (see [`pytest.ini`](pytest.ini)). -**Co-location + proxy pattern:** test source lives alongside its ROS 2 package at +Test source lives alongside its ROS 2 package at `robot/ros_ws/src///test/test_*.py` (the ROS 2 / colcon convention). -Files in `tests/robot/` are thin proxies that re-export those tests so that -`pytest tests/` discovers them. Both `airstack test -m unit` and -`colcon test --packages-select ` run the same test source. +[`colcon_unit_test_packages.yaml`](colcon_unit_test_packages.yaml) lists which packages +have unit tests; `conftest.py` resolves each to its `test/` dir and collects the +non-linter `test_*.py` files under `--import-mode=importlib`, tagging each `unit`. To add +a package's unit tests, list it in that YAML. Both `airstack test -m unit` and +`colcon test --packages-select ` run the same source (colcon also runs the ament +linters + C++ gtests). Example: `robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py` tests the numpy-only range validation rules in @@ -43,8 +46,16 @@ tests the numpy-only range validation rules in See [Unit Testing Guide](../docs/development/intermediate/testing/unit_testing.md) and the `add-unit-tests` agent skill for full details. +### Integration tests (`tests/integration/`) + +Cross-component tests (`integration` mark) that wire a few real components together — the +robot autonomy container plus a host-side component — **without** a sim or GPU. The +shared `robot_autonomy_stack` fixture (in `conftest.py`) reuses a running `robot-desktop` +container or brings one up automatically (like `build_packages`), then tears it down. +Collection order runs integration after `build_packages` and before the sim tiers. + Marks can be combined with pytest logic: -`-m unit`, `-m "build_docker or build_packages"`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, `-m autonomy`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). +`-m unit`, `-m "build_docker or build_packages"`, `-m integration`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, `-m autonomy`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). ### Bring-up scope (`airstack_env`) @@ -54,7 +65,17 @@ Marks can be combined with pytest logic: ## Test Infrastructure -All shared fixtures, helpers, and configuration live in [`conftest.py`](conftest.py). +[`conftest.py`](conftest.py) holds the pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures. The reusable helpers are split by concern into the [`tests/harness/`](harness/) package and re-exported through `conftest`, so `from conftest import ` (and `from harness import `) both work: + +| Module | Contents | +|--------|----------| +| [`harness/session.py`](harness/session.py) | Session-scoped state (results dir, current item, last cmd output) + shared `logger` | +| [`harness/discovery.py`](harness/discovery.py) | Unit-test discovery driven by `colcon_unit_test_packages.yaml` (`AIRSTACK_ROOT`, `repo_path`, `unit_test_files`, …) | +| [`harness/commands.py`](harness/commands.py) | Subprocess / `docker exec` / `ros2` helpers with per-test output capture (`airstack_cmd`, `docker_exec`, `ros2_exec`, `read_log_tail`) | +| [`harness/containers.py`](harness/containers.py) | Container discovery, compute-usage sampling, image checks (`find_container`, `wait_for_container`, `sample_compute_usage`, `missing_images`) | +| [`harness/metrics.py`](harness/metrics.py) | `MetricsRecorder`, `get_metrics`, `current_test_id` (writes `metrics.json`) | +| [`harness/sim.py`](harness/sim.py) | `SIM_CONFIG` sim targets + ros2 topic sampling (`sample_hz`, `parallel_sample_hz`, `wait_for_first_message`) | +| [`harness/collection.py`](harness/collection.py) | Cross-module test ordering + parametrize-id rewrite (`modify_items`) | ### `airstack_env` fixture diff --git a/tests/colcon_unit_test_packages.yaml b/tests/colcon_unit_test_packages.yaml index 5e0bd0ebb..65cdd38cc 100644 --- a/tests/colcon_unit_test_packages.yaml +++ b/tests/colcon_unit_test_packages.yaml @@ -1,7 +1,9 @@ -# Packages run via `colcon test` in system.test_build_packages.test_colcon_test_robot. +# Defines where the unit tests live: each listed package's /test/ dir is +# collected by `pytest tests/` (via conftest.py) and run by `colcon test` in +# system.test_build_packages.test_colcon_test_robot. # # Add a package here when it has gtests (ament_add_gtest) and/or pytest tests under -# /test/. Keep in sync with tests/robot/ proxies for Python unit tests. +# /test/. # # See: docs/development/intermediate/testing/unit_testing.md diff --git a/tests/conftest.py b/tests/conftest.py index 2a51c569a..47a668e65 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -1,106 +1,22 @@ -import json -import logging -import os -import re -import shlex -import subprocess import sys -import threading import time -from concurrent.futures import ThreadPoolExecutor, as_completed from contextlib import contextmanager -from datetime import datetime from pathlib import Path import pytest -import yaml - -SIM_CONFIG = { - "msairsim": { - "profile": "ms-airsim", - "sim_container": "ms-airsim", - "sim_setup_bash": "/root/ros_ws/install/setup.bash", - "robot_setup_bash": "/root/AirStack/robot/ros_ws/install/setup.bash", - "extra_env": { - "URDF_FILE": "robot_descriptions/iris/urdf/iris_stereo.ms-airsim.urdf", - # Clear any user-set paths in .env so entrypoint auto-fetches Blocks. - # Shell env wins over --env-file in docker compose substitution. - "MS_AIRSIM_ENV_DIR": "", - "MS_AIRSIM_BINARY_PATH": "", - }, - }, - "isaacsim": { - "profile": "isaac-sim", - "sim_container": "isaac-sim", - "sim_setup_bash": "/opt/ros/jazzy/setup.bash", - "robot_setup_bash": "/root/AirStack/robot/ros_ws/install/setup.bash", - "extra_env": { - "ISAAC_SIM_USE_STANDALONE": "true", - "ISAAC_SIM_SCRIPT_NAME": "example_multi_px4_pegasus_launch_script.py", - "PLAY_SIM_ON_START": "true", - # Multi script gates RTX LiDAR on this flag; example_one always spawns it. - # `sensors` tests expect ouster topics + lidar_point_cloud_filter path. - "ENABLE_LIDAR": "true", - }, - }, -} -AIRSTACK_ROOT = os.environ.get("AIRSTACK_ROOT", str(Path(__file__).parent.parent)) -COLCON_UNIT_TEST_PACKAGES_YAML = ( - Path(AIRSTACK_ROOT) / "tests" / "colcon_unit_test_packages.yaml" -) - - -def load_colcon_unit_test_config(workspace="robot"): - """Load colcon test package list and pytest args from tests/colcon_unit_test_packages.yaml.""" - if not COLCON_UNIT_TEST_PACKAGES_YAML.is_file(): - raise FileNotFoundError( - f"Missing {COLCON_UNIT_TEST_PACKAGES_YAML} — add packages to gate in colcon test." - ) - with COLCON_UNIT_TEST_PACKAGES_YAML.open(encoding="utf-8") as f: - data = yaml.safe_load(f) or {} - if workspace not in data: - raise KeyError( - f"No '{workspace}' entry in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" - ) - cfg = data[workspace] or {} - packages = cfg.get("packages") or [] - if not packages: - raise ValueError( - f"'{workspace}.packages' is empty in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" - ) - return packages, cfg.get("pytest_args", "") - - -def colcon_test_robot_command(workspace="robot"): - """Shell command for colcon test over unit-test packages (robot workspace).""" - packages, pytest_args = load_colcon_unit_test_config(workspace) - pkg_list = " ".join(packages) - cmd = ( - f"colcon test --packages-select {pkg_list} " - "--event-handlers console_direct+ --return-code-on-test-failure" - ) - if pytest_args: - cmd += f' --pytest-args "{pytest_args}"' - return cmd -# Unit tests live co-located with their ROS 2 packages in robot/ros_ws/src/. -# Thin proxy files under tests/robot/ re-export those tests so that -# `pytest tests/` and `airstack test -m unit` discover them without any -# sys.path manipulation here. Each proxy file sets up its own paths. -RUN_DIR = None -ROS_DISTRO_SETUP = "/opt/ros/jazzy/setup.bash" -_LAST_CMD_OUTPUT: dict[str, str] = {} -_DEFAULT_LOG_KEY = "_last" - -# Track the currently-running pytest item so current_log() and current_test_id() -# can pick up the parametrize id without tests having to pass `request` around. -_CURRENT_ITEM = None -METRICS = None - - -logger = logging.getLogger("airstack") -logger.setLevel(logging.INFO) +# Make tests/ importable so `from harness import ...` (and `from run_summary import ...`) +# resolve regardless of pytest's import mode. +_TESTS_DIR = str(Path(__file__).resolve().parent) +if _TESTS_DIR not in sys.path: + sys.path.insert(0, _TESTS_DIR) +from harness import collection, session +# Re-export the harness helper API so existing `from conftest import ` in the +# system tests + sensor_probes keeps working unchanged. +from harness import * # noqa: F401,F403 +from harness.commands import _nodeid_dotted +from harness.discovery import _is_unit_item # ── pytest config / hooks ────────────────────────────────────────────────── @@ -127,31 +43,46 @@ def pytest_addoption(parser): def pytest_configure(config): - global RUN_DIR - timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") - results_root = Path(AIRSTACK_ROOT) / "tests" / "results" - RUN_DIR = results_root / timestamp - RUN_DIR.mkdir(parents=True, exist_ok=True) - config.option.xmlpath = str(RUN_DIR / "results.xml") + run_dir = session.init_run_dir(AIRSTACK_ROOT) + config.option.xmlpath = str(run_dir / "results.xml") + + # Collect co-located unit tests: their files live outside tests/, so add the + # explicit non-linter test files to the collection args. Skip when an explicit + # path was given on the CLI (args_source == ARGS) so `pytest tests/system/foo.py` + # still narrows as expected. + src_name = getattr(getattr(config, "args_source", None), "name", "TESTPATHS") + if src_name != "ARGS": + for f in unit_test_files(): + entry = str(f) + if entry not in config.args: + config.args.append(entry) + + +def pytest_itemcollected(item): + """Auto-mark co-located unit tests `unit` (before -m filtering runs). + + Idempotent when the source already declares the mark. + """ + if _is_unit_item(item): + item.add_marker(pytest.mark.unit) def pytest_runtest_setup(item): - global _CURRENT_ITEM - _CURRENT_ITEM = item + session.set_current_item(item) def pytest_runtest_teardown(item): - global _CURRENT_ITEM - _CURRENT_ITEM = None + session.set_current_item(None) -def pytest_sessionfinish(session, exitstatus): +def pytest_sessionfinish(exitstatus): """Write summary.txt with key metrics so users don't need to dig through logs.""" - if RUN_DIR is None: + run_dir = session.run_dir() + if run_dir is None: return try: from run_summary import write_summary - summary_path = write_summary(RUN_DIR) + summary_path = write_summary(run_dir) logger.info("Wrote run summary to %s", summary_path) except Exception as exc: logger.warning("Failed to write run summary: %s", exc) @@ -187,549 +118,8 @@ def pytest_generate_tests(metafunc): metafunc.parametrize("airstack_env", params, ids=ids, indirect=True, scope="class") -# Run cheap/fast-fail tests first so real problems surface early: -# docker image builds → colcon workspace builds → liveliness (infra) → sensors -# (ROS topic streams) → autonomy flight tests. -_MODULE_ORDER = [ - # Unit tests first — fast, hermetic, no Docker. Any module whose dotted - # name starts with "robot." or "sim." is a proxy for a package-level unit - # test and sorts into this leading slot via the prefix check below. - "__unit__", - # System tests follow in dependency order. - "system.test_build_docker", - "system.test_build_packages", - "system.test_liveliness", - "system.test_sensors", - "system.test_takeoff_hover_land", - "system.test_fixed_trajectory", -] - -# Within test_takeoff_hover_land, each (env, velocity) runs phases in this chain order. -_AUTONOMY_PHASE_ORDER = [ - "test_px4_ready", - "test_takeoff", - "test_hover", - "test_landing", -] - -# Within test_fixed_trajectory, each (env, trajectory_type) runs phases in this order. -_FIXED_TRAJ_PHASE_ORDER = [ - "test_px4_ready", - "test_takeoff", - "test_fixed_trajectory", - "test_landing", -] - -# Maps module name → phase order list for per-module chain sorting. -_MODULE_PHASE_ORDERS = { - "system.test_takeoff_hover_land": _AUTONOMY_PHASE_ORDER, - "system.test_fixed_trajectory": _FIXED_TRAJ_PHASE_ORDER, -} - - -def _rank(name, order): - """Index of `name` in `order`; `len(order)` if unknown (i.e., sort last).""" - return order.index(name) if name in order else len(order) - - -def _module_key(item): - """Return the ordering key for an item. - - Unit-test proxies live under ``robot/``, ``sim/``, or ``gcs/`` and are - identified by their nodeid prefix. Everything else uses the dotted module - ``__name__`` looked up against ``_MODULE_ORDER``. - """ - if item.nodeid.startswith(("robot/", "sim/", "gcs/")): - return _rank("__unit__", _MODULE_ORDER) - return _rank(getattr(item.module, "__name__", ""), _MODULE_ORDER) - - def pytest_collection_modifyitems(items): - # 1. Cross-module: enforce `_MODULE_ORDER`. Stable sort keeps within-module - # order intact, so pytest's default file/class order survives. - items.sort(key=_module_key) - - # 2. Within each parametrized autonomy-style module, sort by - # (airstack_env, secondary_param, phase) so each env brings up the stack - # once and the drone goes ground→air→ground per secondary parameter. - for mod_name, phase_order in _MODULE_PHASE_ORDERS.items(): - def _phase(item, _order=phase_order, _mod=mod_name): - if getattr(item.module, "__name__", "") != _mod: - return None - name = item.originalname or item.name.split("[", 1)[0] - return _rank(name, _order) - - def _sort_key(item, _mod=mod_name): - cs = getattr(item, "callspec", None) - env = cs.params.get("airstack_env", ()) if cs else () - # test_takeoff_hover_land sweeps velocity; test_fixed_trajectory sweeps type - secondary = ( - float(cs.params["velocity"]) if cs and "velocity" in cs.params - else (cs.params.get("trajectory_type", "") if cs else "") - ) - return (env, secondary, _phase(item)) - - slots = [(i, it) for i, it in enumerate(items) if _phase(it) is not None] - if slots: - sorted_items = sorted((it for _, it in slots), key=_sort_key) - for (i, _), new_item in zip(slots, sorted_items): - items[i] = new_item - - # 3. Rewrite bracketed test IDs into a consistent hierarchy: - # sim > robots > secondary param > iteration. - for item in items: - cs = getattr(item, "callspec", None) - if cs is None: - continue - env = cs.params.get("airstack_env") - parts = [] - if env: - sim, n, i = env - parts.append(f"{sim}-rob#{n}") - if "velocity" in cs.params: - parts.append(f"v{cs.params['velocity']}") - if "trajectory_type" in cs.params: - parts.append(f"traj{cs.params['trajectory_type']}") - if env: - parts.append(f"iter{i}") - if not parts: - continue - new_id = "-".join(parts) - if cs.id == new_id: - continue - item.name = item.name.replace(f"[{cs.id}]", f"[{new_id}]") - item._nodeid = item._nodeid.replace(f"[{cs.id}]", f"[{new_id}]") - - -# ── logging / subprocess helpers ─────────────────────────────────────────── - -def _nodeid_dotted(nodeid, with_path_sep=False): - """pytest nodeid → `module.Class.test_name[params]` form. When - `with_path_sep=True`, also flattens `/` in path prefixes (for log filenames).""" - out = nodeid.replace(".py::", ".").replace("::", ".") - return out.replace("/", ".") if with_path_sep else out - - -def current_log(): - """Log name for the currently-running pytest item, or None outside a test. - - Subprocess helpers default to this so every call fired from a test auto-logs - to the right file without plumbing log_name through every layer.""" - if _CURRENT_ITEM is None: - return None - return _nodeid_dotted(_CURRENT_ITEM.nodeid, with_path_sep=True) - - -def read_log_tail(log_name=None, lines=50): - """Return the tail of the most recent subprocess output for this context.""" - key = log_name or _DEFAULT_LOG_KEY - text = _LAST_CMD_OUTPUT.get(key) or _LAST_CMD_OUTPUT.get(_DEFAULT_LOG_KEY, "") - if not text: - return "" - return "\n".join(text.splitlines()[-lines:]) - - -def _run_teed(cmd_list, timeout, log_name=None, env=None, cwd=None): - """Run a subprocess and capture stdout+stderr for parsing and failure messages.""" - quoted = " ".join(shlex.quote(a) for a in cmd_list) - logger.info("$ %s", quoted) - result = subprocess.run( - cmd_list, capture_output=True, text=True, timeout=timeout, env=env, cwd=cwd, - ) - combined = (result.stdout or "") + (result.stderr or "") - key = log_name or _DEFAULT_LOG_KEY - _LAST_CMD_OUTPUT[key] = combined - _LAST_CMD_OUTPUT[_DEFAULT_LOG_KEY] = combined - return result - - -def docker_exec(container, cmd, timeout=60, log_name=None): - full_cmd = ["docker", "exec", container, "bash", "-c", cmd] - return _run_teed(full_cmd, timeout=timeout, log_name=log_name) - - -def airstack_cmd(*args, env_overrides=None, timeout=1800, log_name=None): - env = os.environ.copy() - if env_overrides: - env.update(env_overrides) - cmd = [str(Path(AIRSTACK_ROOT) / "airstack.sh")] + list(args) - return _run_teed(cmd, timeout=timeout, log_name=log_name, - env=env, cwd=AIRSTACK_ROOT) - - -def ros2_env(setup_bash, domain_id): - """Shell prefix that makes `ros2` available on the requested domain.""" - return ( - f"source {ROS_DISTRO_SETUP} && source {setup_bash} " - f"&& export ROS_DOMAIN_ID={domain_id}" - ) - - -def ros2_exec(container, ros2_cmd, domain_id=0, setup_bash=None, timeout=15, log_name=None): - """Run `ros2 ...` inside a container with the right workspace sourced.""" - setup = setup_bash or "/root/AirStack/robot/ros_ws/install/setup.bash" - inner = f"{ros2_env(setup, domain_id)} && {ros2_cmd}" - return docker_exec(container, inner, timeout=timeout, log_name=log_name) - - -_HZ_RE = re.compile(r"average rate:\s+([\d.]+)") - - -def _parse_hz(text): - m = _HZ_RE.search(text or "") - return float(m.group(1)) if m else None - - -# ── container helpers ────────────────────────────────────────────────────── - -def find_all_containers(name_pattern): - result = _run_teed( - ["docker", "ps", "--filter", f"name={name_pattern}", "--format", "{{.Names}}"], - timeout=10, - ) - return [n for n in result.stdout.strip().splitlines() if n] - - -def find_container(name_pattern): - names = find_all_containers(name_pattern) - return names[0] if names else None - - -def get_robot_containers(pattern="robot.*desktop"): - """Return running robot containers sorted by their replica index""" - def _index(name): - tail = name.rsplit("-", 1)[-1] - return int(tail) if tail.isdigit() else 0 - return sorted(find_all_containers(pattern), key=_index) - - -def container_running(name): - """True if the named container is currently Running.""" - result = _run_teed( - ["docker", "inspect", "-f", "{{.State.Running}}", name], - timeout=10, - ) - return "true" in result.stdout - - -def wait_for_container(name_pattern, timeout=120): - deadline = time.time() + timeout - while time.time() < deadline: - name = find_container(name_pattern) - if name and container_running(name): - return name - time.sleep(5) - raise TimeoutError(f"Container matching '{name_pattern}' not running after {timeout}s") - - -# ── compute-usage sampling ───────────────────────────────────────────────── - -_BYTES_RE = re.compile(r"([\d.]+)\s*([kKMGT]?i?B)$") -_BYTES_TO_MB = { - "B": 1 / (1024 * 1024), - "KiB": 1 / 1024, "KB": 1 / 1000, "kB": 1 / 1000, - "MiB": 1, "MB": 1, - "GiB": 1024, "GB": 1000, - "TiB": 1024 * 1024, "TB": 1_000_000, -} - - -def _parse_docker_bytes(s): - """Parse a docker-stats byte string (e.g. '123.4MiB', '0B') to MB.""" - m = _BYTES_RE.match((s or "").strip()) - if not m: - return 0.0 - return float(m.group(1)) * _BYTES_TO_MB.get(m.group(2), 1) - - -def sample_compute_usage(sim_container): - """Snapshot of compute resources: per-container CPU/mem/disk-IO/net-IO plus - global host CPU/mem and GPU util/VRAM/temp/power. Returns {key: value}, - keys shaped `{entity}.{metric}` where entity is the full container name or - 'host'. Per-robot replicas (e.g. airstack-robot-desktop-1/2/3) are kept - distinct so raw metrics.json preserves per-robot data; parse_metrics - pools them at report time. Silently omits metrics that fail to sample.""" - import psutil - - out = {} - - stats = _run_teed( - ["docker", "stats", "--no-stream", "--format", "{{json .}}"], - timeout=20, - ) - for line in stats.stdout.strip().splitlines(): - try: - d = json.loads(line) - except json.JSONDecodeError: - continue - name = d.get("Name", "") - if not name or name.startswith("docker-test-run"): - continue - out[f"{name}.cpu_pct"] = float(d.get("CPUPerc", "0%").rstrip("%") or 0) - mem_raw = d.get("MemUsage", "").split("/")[0].strip() - out[f"{name}.mem_mb"] = _parse_docker_bytes(mem_raw) - for io_field, metric in (("BlockIO", "disk_io_mb"), ("NetIO", "net_io_mb")): - parts = (d.get(io_field, "") or "").split("/") - total = sum(_parse_docker_bytes(p.strip()) for p in parts) - out[f"{name}.{metric}"] = total - - out["host.cpu_pct"] = psutil.cpu_percent(interval=0.5) - out["host.mem_mb"] = psutil.virtual_memory().used / (1024 * 1024) - - gpu = _run_teed( - ["docker", "exec", sim_container, "nvidia-smi", - "--query-gpu=utilization.gpu,memory.used,temperature.gpu,power.draw", - "--format=csv,noheader,nounits"], - timeout=10, - ) - if gpu.returncode == 0 and gpu.stdout.strip(): - fields = [f.strip() for f in gpu.stdout.strip().splitlines()[0].split(",")] - if len(fields) >= 4: - try: - out["host.gpu_pct"] = float(fields[0]) - out["host.vram_mb"] = float(fields[1]) - out["host.gpu_temp_c"] = float(fields[2]) - out["host.gpu_power_w"] = float(fields[3]) - except ValueError: - pass - - return out - - -def _compose_images(env=None): - """Resolved image refs that `docker compose up` would use under `env`.""" - compose_env = os.environ.copy() - if env: - compose_env.update(env) - result = _run_teed( - ["docker", "compose", "-f", str(Path(AIRSTACK_ROOT) / "docker-compose.yaml"), - "config", "--images"], - timeout=30, env=compose_env, cwd=AIRSTACK_ROOT, - ) - return [l.strip() for l in result.stdout.strip().splitlines() if l.strip()] - - -def missing_images(env=None): - """Images required by the current compose config but not present locally. - Used by airstack_env to fail fast instead of letting `airstack up` hang - pulling/building when images haven't been prebuilt.""" - missing = [] - for image in _compose_images(env=env): - result = _run_teed( - ["docker", "image", "inspect", image, "--format", "{{.Id}}"], - timeout=10, - ) - if result.returncode != 0: - missing.append(image) - return missing - - -def docker_image_size_mb(service, env=None): - image = next((i for i in _compose_images(env=env) if service in i), None) - if not image: - return None - result = _run_teed( - ["docker", "image", "inspect", image, "--format", "{{.Size}}"], - timeout=10, - ) - if result.returncode == 0 and result.stdout.strip(): - return round(int(result.stdout.strip()) / 1_000_000, 1) - return None - - -# ── metrics ──────────────────────────────────────────────────────────────── - -class MetricsRecorder: - def __init__(self, path): - self._path = path - self._data = json.loads(path.read_text()) if path.exists() else {} - self._lock = threading.Lock() - - def _flush(self): - tmp = self._path.with_suffix(self._path.suffix + ".tmp") - tmp.write_text(json.dumps(self._data, indent=2)) - os.replace(tmp, self._path) - - def record(self, test_name, key, value, unit="", direction="lower_is_better", **extra): - with self._lock: - if test_name not in self._data: - self._data[test_name] = {} - entry = {"value": value, "unit": unit, "direction": direction} - entry.update(extra) - self._data[test_name][key] = entry - self._flush() - - def record_list(self, test_name, key, values): - """Store a raw list (time series) — not scored by parse_metrics.""" - with self._lock: - if test_name not in self._data: - self._data[test_name] = {} - self._data[test_name][key] = {"samples": values} - self._flush() - -def get_metrics(): - global METRICS - if METRICS is None: - METRICS = MetricsRecorder(RUN_DIR / "metrics.json") - return METRICS - - -def current_test_id(): - """Test id used as the metrics.json key. Matches JUnit XML's classname.name - format so parse_metrics.py can merge results.xml and metrics.json entries.""" - if _CURRENT_ITEM is None: - return "unknown" - return _nodeid_dotted(_CURRENT_ITEM.nodeid) - - -# ── shared sim test infrastructure (liveliness, sensors, comms, takeoff reuse) ── - -def wait_for_first_message(container, topic, domain_id, setup_bash, timeout=60): - """Wait up to `timeout` seconds for one message on `topic`. Returns seconds - elapsed on success, None on timeout. Each attempt sources the workspace - and runs `ros2 topic echo --once`; if the workspace isn't built yet or the - topic has no publisher, the attempt fails fast and we retry. - """ - start = time.time() - deadline = start + timeout - logger.info("Probing %s on domain %d in %s (timeout=%ds)", - topic, domain_id, container, timeout) - attempt = 0 - while time.time() < deadline: - attempt += 1 - per_attempt = min(max(1, int(deadline - time.time())), 10) - try: - result = ros2_exec( - container, - f"timeout {per_attempt} ros2 topic echo --once {topic}", - domain_id=domain_id, setup_bash=setup_bash, timeout=per_attempt + 5, - ) - except subprocess.TimeoutExpired: - logger.warning("Attempt %d subprocess timeout for %s, retrying", attempt, topic) - time.sleep(2) - continue - # ros2 prints "---" on its own line after a real message. - if result.stdout.rstrip().endswith("---"): - elapsed = round(time.time() - start, 2) - logger.info("Got first message on %s after %.2fs (attempt %d)", - topic, elapsed, attempt) - return elapsed - logger.warning("Attempt %d failed for %s, retrying", attempt, topic) - time.sleep(2) - logger.error("Timed out waiting for first message on %s after %ds", - topic, timeout) - return None - - -def sample_hz(container, topic, domain_id, setup_bash, duration=5, window=10): - """Sample publish rate on `topic` for `duration` seconds. Returns float or None.""" - result = ros2_exec( - container, - f"timeout {duration} ros2 topic hz --window {window} {topic} 2>&1", - domain_id=domain_id, setup_bash=setup_bash, timeout=duration + 15, - ) - return _parse_hz(result.stdout + result.stderr) - - -def parallel_sample_hz(container, topic_domain_pairs, setup_bash, duration=5, window=10): - """Sample Hz for multiple topics concurrently; return {topic: hz_or_None}. - - One `docker exec` that backgrounds each `ros2 topic hz` probe, waits for all, - then cats each probe's temp file. - """ - probes = [] - temp_files = {} - for i, (topic, domain) in enumerate(topic_domain_pairs): - fname = f"/tmp/hz_{i}.out" - temp_files[topic] = fname - probes.append( - f"(ROS_DOMAIN_ID={domain} timeout {duration} " - f"ros2 topic hz --window {window} {topic} > {fname} 2>&1) &" - ) - # Newlines, not `&& ... &`: bash precedence makes `A && B && C & D &` only - # apply the && chain to C, so later backgrounded probes would miss the - # sourced PATH. One statement per line sidesteps this entirely. - lines = [f"source {ROS_DISTRO_SETUP}", f"source {setup_bash}"] + probes + ["wait"] - for fname in temp_files.values(): - lines.append(f"echo '===FILE {fname}==='") - lines.append(f"cat {fname} 2>/dev/null || true") - script = "\n".join(lines) - result = _run_teed( - ["docker", "exec", container, "bash", "-c", script], - timeout=duration + 30, - ) - rates = {} - if result.returncode == 0 or result.stdout: - chunks = result.stdout.split("===FILE ") - for chunk in chunks[1:]: - header, _, content = chunk.partition("===") - fname = header.strip() - topic = next((t for t, f in temp_files.items() if f == fname), None) - if topic: - rates[topic] = _parse_hz(content) - for topic, _ in topic_domain_pairs: - rates.setdefault(topic, None) - return rates - - -def _echo_once_received_message(result): - """True if ``ros2 topic echo --once`` printed a full message (trailing ``---``).""" - out = (result.stdout or "").rstrip() - return out.endswith("---") - - -def parallel_echo_once_robot_topics( - probes, setup_bash, per_topic_timeout, -): - """Liveliness for heavy topics (e.g. PointCloud2): ``echo --once`` per probe in parallel. - - ``ros2 topic hz`` often never reports a rate on large point clouds (decode backlog). - - Parameters - ---------- - probes : list[tuple[str, str, int]] - ``(container_name, topic, ros_domain_id)`` — use the **robot container** that - hosts that domain's graph (replica ``n`` for ``robot_n``). - setup_bash : str - Workspace ``setup.bash`` path inside the container. - per_topic_timeout : int - Wall seconds per ``timeout … ros2 topic echo --once``. - - Returns - ------- - dict[str, float | None] - ``{topic: 1.0}`` if a message arrived, else ``{topic: None}`` (metrics use 1.0 - as a nonzero "alive" placeholder, not a measured Hz). - """ - rates = {} - - def _one(container, topic, domain_id): - cmd = f"timeout {per_topic_timeout} ros2 topic echo --once {topic}" - return topic, ros2_exec( - container, - cmd, - domain_id=domain_id, - setup_bash=setup_bash, - timeout=per_topic_timeout + 15, - ) - - with ThreadPoolExecutor(max_workers=max(1, len(probes))) as pool: - futures = { - pool.submit(_one, container, topic, domain_id): topic - for container, topic, domain_id in probes - } - for fut in as_completed(futures): - topic = futures[fut] - try: - _, result = fut.result() - except Exception as e: - logger.warning("echo-once probe failed for %s: %s", topic, e) - rates[topic] = None - continue - rates[topic] = 1.0 if _echo_once_received_message(result) else None - for _, topic, _ in probes: - rates.setdefault(topic, None) - return rates + collection.modify_items(items) @pytest.fixture @@ -748,7 +138,7 @@ def airstack_env(request): # test id (see pytest_collection_modifyitems), so airstack up/down output # lands next to the triggering test's own log instead of under pytest's # stale callspec.id. - log = f"airstack_env.{_nodeid_dotted(_CURRENT_ITEM.nodeid, with_path_sep=True)}" + log = f"airstack_env.{_nodeid_dotted(session.current_item().nodeid, with_path_sep=True)}" headless = not request.config.getoption("--gui") env_overrides = { @@ -808,4 +198,49 @@ def airstack_env(request): airstack_cmd("down", timeout=120, log_name=log) down_duration_s = round(time.time() - t3, 2) logger.info("Teardown finished in %.2fs", down_duration_s) - m.record(tid, "airstack_down_duration_s", down_duration_s, unit="s") \ No newline at end of file + m.record(tid, "airstack_down_duration_s", down_duration_s, unit="s") + +# ── integration tier (tests/integration/) ───────────────────────────────── + +_INTEGRATION_ROBOT_PATTERN = "robot.*desktop" +# Robot-only bring-up: autonomy stack on, no sim profile, single robot. +_INTEGRATION_ENV = { + "AUTOLAUNCH": "true", + "NUM_ROBOTS": "1", + "COMPOSE_PROFILES": "desktop", +} + + +@pytest.fixture(scope="module") +def robot_autonomy_stack(request): + """Robot-desktop container for integration tests (no sim, no GPU). + + Yields ``{"container": , "brought_up": bool}``. Reuses an already + running container (fast local iteration, left running afterward); otherwise + runs ``airstack up robot-desktop`` and tears it down after the module. + Behaves like the ``build_packages`` fixture — always brings up Docker when + no container is found. + """ + existing = find_container(_INTEGRATION_ROBOT_PATTERN) + if existing and container_running(existing): + yield {"container": existing, "brought_up": False} + return + + log = "robot_autonomy_stack" + with logger_to(log): + missing = missing_images(env=_INTEGRATION_ENV) + if missing: + pytest.skip("robot-desktop image not built locally: " + ", ".join(missing)) + airstack_cmd("down", timeout=120, log_name=log) + result = airstack_cmd("up", "robot-desktop", + env_overrides=_INTEGRATION_ENV, timeout=180, log_name=log) + if result.returncode != 0: + pytest.fail(f"`airstack up robot-desktop` failed:\n{read_log_tail(log)}") + + container = wait_for_container(_INTEGRATION_ROBOT_PATTERN, timeout=120) + assert container, "robot-desktop container not Running after 120s" + try: + yield {"container": container, "brought_up": True} + finally: + with logger_to(log): + airstack_cmd("down", timeout=120, log_name=log) diff --git a/tests/harness/__init__.py b/tests/harness/__init__.py new file mode 100644 index 000000000..05efe2144 --- /dev/null +++ b/tests/harness/__init__.py @@ -0,0 +1,63 @@ +"""AirStack test harness — helpers split out of conftest.py by concern. + +conftest.py holds the pytest hooks and fixtures and imports what it needs from here. +Public helpers are re-exported so tests and conftest can ``from harness import ``. +""" +from harness.commands import ( + ROS_DISTRO_SETUP, + airstack_cmd, + current_log, + docker_exec, + read_log_tail, + ros2_env, + ros2_exec, +) +from harness.containers import ( + container_running, + docker_image_size_mb, + find_all_containers, + find_container, + get_robot_containers, + missing_images, + sample_compute_usage, + wait_for_container, +) +from harness.discovery import ( + AIRSTACK_ROOT, + COLCON_UNIT_TEST_PACKAGES_YAML, + colcon_test_robot_command, + load_colcon_unit_test_config, + repo_path, + unit_test_dirs, + unit_test_files, +) +from harness.metrics import MetricsRecorder, current_test_id, get_metrics +from harness.session import logger +from harness.sim import ( + SIM_CONFIG, + parallel_echo_once_robot_topics, + parallel_sample_hz, + sample_hz, + wait_for_first_message, +) + +__all__ = [ + # discovery + "AIRSTACK_ROOT", "COLCON_UNIT_TEST_PACKAGES_YAML", "repo_path", + "colcon_test_robot_command", "load_colcon_unit_test_config", + "unit_test_dirs", "unit_test_files", + # session + "logger", + # commands + "ROS_DISTRO_SETUP", "airstack_cmd", "current_log", "docker_exec", + "read_log_tail", "ros2_env", "ros2_exec", + # containers + "find_all_containers", "find_container", "get_robot_containers", + "container_running", "wait_for_container", "missing_images", + "sample_compute_usage", "docker_image_size_mb", + # metrics + "MetricsRecorder", "get_metrics", "current_test_id", + # sim + "SIM_CONFIG", "wait_for_first_message", "sample_hz", "parallel_sample_hz", + "parallel_echo_once_robot_topics", +] diff --git a/tests/harness/collection.py b/tests/harness/collection.py new file mode 100644 index 000000000..aa20c9f1e --- /dev/null +++ b/tests/harness/collection.py @@ -0,0 +1,125 @@ +"""Test collection ordering. + +Cross-module order (unit → docker/package builds → integration → sim tiers), the +per-module phase chains for the autonomy flight tests, and a readable rewrite of the +parametrize ids. conftest's ``pytest_collection_modifyitems`` hook delegates to +``modify_items``. +""" +from harness.discovery import _is_unit_item + +# Run cheap/fast-fail tests first so real problems surface early: +# docker image builds → colcon workspace builds → liveliness (infra) → sensors +# (ROS topic streams) → autonomy flight tests. +_MODULE_ORDER = [ + # Unit tests first — fast, hermetic, no Docker. Co-located package unit tests + # (see unit_test_files) sort into this leading slot via the path check below. + "__unit__", + # System tests follow in dependency order. + "system.test_build_docker", + "system.test_build_packages", + # Integration tests (tests/integration/) need the robot-desktop image + colcon + # build, so they run after build_packages and before the sim tiers. + "__integration__", + "system.test_liveliness", + "system.test_sensors", + "system.test_takeoff_hover_land", + "system.test_fixed_trajectory", +] + +# Within test_takeoff_hover_land, each (env, velocity) runs phases in this chain order. +_AUTONOMY_PHASE_ORDER = [ + "test_px4_ready", + "test_takeoff", + "test_hover", + "test_landing", +] + +# Within test_fixed_trajectory, each (env, trajectory_type) runs phases in this order. +_FIXED_TRAJ_PHASE_ORDER = [ + "test_px4_ready", + "test_takeoff", + "test_fixed_trajectory", + "test_landing", +] + +# Maps module name → phase order list for per-module chain sorting. +_MODULE_PHASE_ORDERS = { + "system.test_takeoff_hover_land": _AUTONOMY_PHASE_ORDER, + "system.test_fixed_trajectory": _FIXED_TRAJ_PHASE_ORDER, +} + + +def _rank(name, order): + """Index of `name` in `order`; `len(order)` if unknown (i.e., sort last).""" + return order.index(name) if name in order else len(order) + + +def _module_key(item): + """Return the ordering key for an item. + + Co-located unit tests (collected from package ``test/`` dirs outside ``tests/``) + are identified by path. Integration tests live under ``integration/``. + Everything else uses the dotted module ``__name__`` against ``_MODULE_ORDER``. + """ + if _is_unit_item(item): + return _rank("__unit__", _MODULE_ORDER) + if item.nodeid.startswith("integration/"): + return _rank("__integration__", _MODULE_ORDER) + return _rank(getattr(item.module, "__name__", ""), _MODULE_ORDER) + + +def modify_items(items): + # 1. Cross-module: enforce `_MODULE_ORDER`. Stable sort keeps within-module + # order intact, so pytest's default file/class order survives. + items.sort(key=_module_key) + + # 2. Within each parametrized autonomy-style module, sort by + # (airstack_env, secondary_param, phase) so each env brings up the stack + # once and the drone goes ground→air→ground per secondary parameter. + for mod_name, phase_order in _MODULE_PHASE_ORDERS.items(): + def _phase(item, _order=phase_order, _mod=mod_name): + if getattr(item.module, "__name__", "") != _mod: + return None + name = item.originalname or item.name.split("[", 1)[0] + return _rank(name, _order) + + def _sort_key(item, _mod=mod_name): + cs = getattr(item, "callspec", None) + env = cs.params.get("airstack_env", ()) if cs else () + # test_takeoff_hover_land sweeps velocity; test_fixed_trajectory sweeps type + secondary = ( + float(cs.params["velocity"]) if cs and "velocity" in cs.params + else (cs.params.get("trajectory_type", "") if cs else "") + ) + return (env, secondary, _phase(item)) + + slots = [(i, it) for i, it in enumerate(items) if _phase(it) is not None] + if slots: + sorted_items = sorted((it for _, it in slots), key=_sort_key) + for (i, _), new_item in zip(slots, sorted_items): + items[i] = new_item + + # 3. Rewrite bracketed test IDs into a consistent hierarchy: + # sim > robots > secondary param > iteration. + for item in items: + cs = getattr(item, "callspec", None) + if cs is None: + continue + env = cs.params.get("airstack_env") + parts = [] + if env: + sim, n, i = env + parts.append(f"{sim}-rob#{n}") + if "velocity" in cs.params: + parts.append(f"v{cs.params['velocity']}") + if "trajectory_type" in cs.params: + parts.append(f"traj{cs.params['trajectory_type']}") + if env: + parts.append(f"iter{i}") + if not parts: + continue + new_id = "-".join(parts) + if cs.id == new_id: + continue + item.name = item.name.replace(f"[{cs.id}]", f"[{new_id}]") + item._nodeid = item._nodeid.replace(f"[{cs.id}]", f"[{new_id}]") diff --git a/tests/harness/commands.py b/tests/harness/commands.py new file mode 100644 index 000000000..9b6bc2fb9 --- /dev/null +++ b/tests/harness/commands.py @@ -0,0 +1,92 @@ +"""Subprocess / docker-exec / ros2 command helpers with per-test output capture. + +Every subprocess runs through ``_run_teed``, which records combined stdout+stderr in the +session so ``read_log_tail`` can surface it in failure messages. ``current_log`` ties +output to the running test's id so callers don't have to plumb a log name through every +layer. +""" +import os +import re +import shlex +import subprocess +from pathlib import Path + +from harness.discovery import AIRSTACK_ROOT +from harness.session import current_item, last_cmd_output, logger, record_cmd_output + +ROS_DISTRO_SETUP = "/opt/ros/jazzy/setup.bash" + + +def _nodeid_dotted(nodeid, with_path_sep=False): + """pytest nodeid → `module.Class.test_name[params]` form. When + `with_path_sep=True`, also flattens `/` in path prefixes (for log filenames).""" + out = nodeid.replace(".py::", ".").replace("::", ".") + return out.replace("/", ".") if with_path_sep else out + + +def current_log(): + """Log name for the currently-running pytest item, or None outside a test. + + Subprocess helpers default to this so every call fired from a test auto-logs + to the right file without plumbing log_name through every layer.""" + item = current_item() + if item is None: + return None + return _nodeid_dotted(item.nodeid, with_path_sep=True) + + +def read_log_tail(log_name=None, lines=50): + """Return the tail of the most recent subprocess output for this context.""" + text = last_cmd_output(log_name) + if not text: + return "" + return "\n".join(text.splitlines()[-lines:]) + + +def _run_teed(cmd_list, timeout, log_name=None, env=None, cwd=None): + """Run a subprocess and capture stdout+stderr for parsing and failure messages.""" + quoted = " ".join(shlex.quote(a) for a in cmd_list) + logger.info("$ %s", quoted) + result = subprocess.run( + cmd_list, capture_output=True, text=True, timeout=timeout, env=env, cwd=cwd, + ) + combined = (result.stdout or "") + (result.stderr or "") + record_cmd_output(combined, log_name) + return result + + +def docker_exec(container, cmd, timeout=60, log_name=None): + full_cmd = ["docker", "exec", container, "bash", "-c", cmd] + return _run_teed(full_cmd, timeout=timeout, log_name=log_name) + + +def airstack_cmd(*args, env_overrides=None, timeout=1800, log_name=None): + env = os.environ.copy() + if env_overrides: + env.update(env_overrides) + cmd = [str(Path(AIRSTACK_ROOT) / "airstack.sh")] + list(args) + return _run_teed(cmd, timeout=timeout, log_name=log_name, + env=env, cwd=AIRSTACK_ROOT) + + +def ros2_env(setup_bash, domain_id): + """Shell prefix that makes `ros2` available on the requested domain.""" + return ( + f"source {ROS_DISTRO_SETUP} && source {setup_bash} " + f"&& export ROS_DOMAIN_ID={domain_id}" + ) + + +def ros2_exec(container, ros2_cmd, domain_id=0, setup_bash=None, timeout=15, log_name=None): + """Run `ros2 ...` inside a container with the right workspace sourced.""" + setup = setup_bash or "/root/AirStack/robot/ros_ws/install/setup.bash" + inner = f"{ros2_env(setup, domain_id)} && {ros2_cmd}" + return docker_exec(container, inner, timeout=timeout, log_name=log_name) + + +_HZ_RE = re.compile(r"average rate:\s+([\d.]+)") + + +def _parse_hz(text): + m = _HZ_RE.search(text or "") + return float(m.group(1)) if m else None diff --git a/tests/harness/containers.py b/tests/harness/containers.py new file mode 100644 index 000000000..71a64d480 --- /dev/null +++ b/tests/harness/containers.py @@ -0,0 +1,164 @@ +"""Docker container discovery, compute-usage sampling, and image helpers.""" +import json +import os +import re +import time +from pathlib import Path + +from harness.commands import _run_teed +from harness.discovery import AIRSTACK_ROOT + + +def find_all_containers(name_pattern): + result = _run_teed( + ["docker", "ps", "--filter", f"name={name_pattern}", "--format", "{{.Names}}"], + timeout=10, + ) + return [n for n in result.stdout.strip().splitlines() if n] + + +def find_container(name_pattern): + names = find_all_containers(name_pattern) + return names[0] if names else None + + +def get_robot_containers(pattern="robot.*desktop"): + """Return running robot containers sorted by their replica index""" + def _index(name): + tail = name.rsplit("-", 1)[-1] + return int(tail) if tail.isdigit() else 0 + return sorted(find_all_containers(pattern), key=_index) + + +def container_running(name): + """True if the named container is currently Running.""" + result = _run_teed( + ["docker", "inspect", "-f", "{{.State.Running}}", name], + timeout=10, + ) + return "true" in result.stdout + + +def wait_for_container(name_pattern, timeout=120): + deadline = time.time() + timeout + while time.time() < deadline: + name = find_container(name_pattern) + if name and container_running(name): + return name + time.sleep(5) + raise TimeoutError(f"Container matching '{name_pattern}' not running after {timeout}s") + + +# ── compute-usage sampling ───────────────────────────────────────────────── + +_BYTES_RE = re.compile(r"([\d.]+)\s*([kKMGT]?i?B)$") +_BYTES_TO_MB = { + "B": 1 / (1024 * 1024), + "KiB": 1 / 1024, "KB": 1 / 1000, "kB": 1 / 1000, + "MiB": 1, "MB": 1, + "GiB": 1024, "GB": 1000, + "TiB": 1024 * 1024, "TB": 1_000_000, +} + + +def _parse_docker_bytes(s): + """Parse a docker-stats byte string (e.g. '123.4MiB', '0B') to MB.""" + m = _BYTES_RE.match((s or "").strip()) + if not m: + return 0.0 + return float(m.group(1)) * _BYTES_TO_MB.get(m.group(2), 1) + + +def sample_compute_usage(sim_container): + """Snapshot of compute resources: per-container CPU/mem/disk-IO/net-IO plus + global host CPU/mem and GPU util/VRAM/temp/power. Returns {key: value}, + keys shaped `{entity}.{metric}` where entity is the full container name or + 'host'. Per-robot replicas (e.g. airstack-robot-desktop-1/2/3) are kept + distinct so raw metrics.json preserves per-robot data; parse_metrics + pools them at report time. Silently omits metrics that fail to sample.""" + import psutil + + out = {} + + stats = _run_teed( + ["docker", "stats", "--no-stream", "--format", "{{json .}}"], + timeout=20, + ) + for line in stats.stdout.strip().splitlines(): + try: + d = json.loads(line) + except json.JSONDecodeError: + continue + name = d.get("Name", "") + if not name or name.startswith("docker-test-run"): + continue + out[f"{name}.cpu_pct"] = float(d.get("CPUPerc", "0%").rstrip("%") or 0) + mem_raw = d.get("MemUsage", "").split("/")[0].strip() + out[f"{name}.mem_mb"] = _parse_docker_bytes(mem_raw) + for io_field, metric in (("BlockIO", "disk_io_mb"), ("NetIO", "net_io_mb")): + parts = (d.get(io_field, "") or "").split("/") + total = sum(_parse_docker_bytes(p.strip()) for p in parts) + out[f"{name}.{metric}"] = total + + out["host.cpu_pct"] = psutil.cpu_percent(interval=0.5) + out["host.mem_mb"] = psutil.virtual_memory().used / (1024 * 1024) + + gpu = _run_teed( + ["docker", "exec", sim_container, "nvidia-smi", + "--query-gpu=utilization.gpu,memory.used,temperature.gpu,power.draw", + "--format=csv,noheader,nounits"], + timeout=10, + ) + if gpu.returncode == 0 and gpu.stdout.strip(): + fields = [f.strip() for f in gpu.stdout.strip().splitlines()[0].split(",")] + if len(fields) >= 4: + try: + out["host.gpu_pct"] = float(fields[0]) + out["host.vram_mb"] = float(fields[1]) + out["host.gpu_temp_c"] = float(fields[2]) + out["host.gpu_power_w"] = float(fields[3]) + except ValueError: + pass + + return out + + +def _compose_images(env=None): + """Resolved image refs that `docker compose up` would use under `env`.""" + compose_env = os.environ.copy() + if env: + compose_env.update(env) + result = _run_teed( + ["docker", "compose", "-f", str(Path(AIRSTACK_ROOT) / "docker-compose.yaml"), + "config", "--images"], + timeout=30, env=compose_env, cwd=AIRSTACK_ROOT, + ) + return [l.strip() for l in result.stdout.strip().splitlines() if l.strip()] + + +def missing_images(env=None): + """Images required by the current compose config but not present locally. + Used by airstack_env to fail fast instead of letting `airstack up` hang + pulling/building when images haven't been prebuilt.""" + missing = [] + for image in _compose_images(env=env): + result = _run_teed( + ["docker", "image", "inspect", image, "--format", "{{.Id}}"], + timeout=10, + ) + if result.returncode != 0: + missing.append(image) + return missing + + +def docker_image_size_mb(service, env=None): + image = next((i for i in _compose_images(env=env) if service in i), None) + if not image: + return None + result = _run_teed( + ["docker", "image", "inspect", image, "--format", "{{.Size}}"], + timeout=10, + ) + if result.returncode == 0 and result.stdout.strip(): + return round(int(result.stdout.strip()) / 1_000_000, 1) + return None diff --git a/tests/harness/discovery.py b/tests/harness/discovery.py new file mode 100644 index 000000000..c0178a92d --- /dev/null +++ b/tests/harness/discovery.py @@ -0,0 +1,125 @@ +"""Unit-test discovery: which packages have unit tests and where their files live. + +Driven by ``tests/colcon_unit_test_packages.yaml``. ``conftest.pytest_configure`` adds +``unit_test_files()`` to the pytest run, and ``pytest_itemcollected`` marks each of those +items ``unit`` via ``_is_unit_item``. ament lint tests are excluded here — they run under +``colcon test`` (see ``colcon_test_robot_command``'s ``-m not linter``). +""" +import os +from pathlib import Path + +import yaml + +AIRSTACK_ROOT = os.environ.get("AIRSTACK_ROOT", str(Path(__file__).resolve().parents[2])) +COLCON_UNIT_TEST_PACKAGES_YAML = ( + Path(AIRSTACK_ROOT) / "tests" / "colcon_unit_test_packages.yaml" +) + + +def repo_path(*parts: str) -> Path: + """Resolve a path relative to the repo root (``AIRSTACK_ROOT``). + + Single source of truth for cross-tree paths — no test or hook should hardcode + ``Path(__file__).parents[N]`` walks. + """ + return Path(AIRSTACK_ROOT).joinpath(*parts) + + +def load_colcon_unit_test_config(workspace="robot"): + """Load colcon test package list and pytest args from tests/colcon_unit_test_packages.yaml.""" + if not COLCON_UNIT_TEST_PACKAGES_YAML.is_file(): + raise FileNotFoundError( + f"Missing {COLCON_UNIT_TEST_PACKAGES_YAML} — add packages to gate in colcon test." + ) + with COLCON_UNIT_TEST_PACKAGES_YAML.open(encoding="utf-8") as f: + data = yaml.safe_load(f) or {} + if workspace not in data: + raise KeyError( + f"No '{workspace}' entry in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" + ) + cfg = data[workspace] or {} + packages = cfg.get("packages") or [] + if not packages: + raise ValueError( + f"'{workspace}.packages' is empty in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" + ) + return packages, cfg.get("pytest_args", "") + + +def colcon_test_robot_command(workspace="robot"): + """Shell command for colcon test over unit-test packages (robot workspace).""" + packages, pytest_args = load_colcon_unit_test_config(workspace) + pkg_list = " ".join(packages) + cmd = ( + f"colcon test --packages-select {pkg_list} " + "--event-handlers console_direct+ --return-code-on-test-failure" + ) + if pytest_args: + cmd += f' --pytest-args "{pytest_args}"' + return cmd + + +# Each listed package resolves to its /test dir via these per-workspace globs. +_WORKSPACE_PKG_TEST_GLOBS = { + "robot": "robot/ros_ws/src/**/{pkg}/test", + "sim": "simulation/**/{pkg}/test", +} + +# ament lint tests ship in every ROS package's test/ dir and import ament_* at +# module load (unavailable outside the built workspace). Skip them here — they run +# under `colcon test` instead (see colcon_test_robot_command's `-m not linter`). +_LINTER_TEST_FILENAMES = { + "test_copyright.py", "test_flake8.py", "test_pep257.py", + "test_pep8.py", "test_xmllint.py", "test_lint_cmake.py", +} + + +def unit_test_dirs(): + """Every co-located unit-test dir resolved from colcon_unit_test_packages.yaml.""" + if not COLCON_UNIT_TEST_PACKAGES_YAML.is_file(): + return [] + with COLCON_UNIT_TEST_PACKAGES_YAML.open(encoding="utf-8") as f: + data = yaml.safe_load(f) or {} + dirs = [] + for workspace, glob_tmpl in _WORKSPACE_PKG_TEST_GLOBS.items(): + cfg = data.get(workspace) or {} + for pkg in cfg.get("packages") or []: + for match in repo_path().glob(glob_tmpl.format(pkg=pkg)): + if match.is_dir(): + dirs.append(match.resolve()) + return dirs + + +_UNIT_TEST_DIRS = None + + +def _unit_test_dirs_cached(): + global _UNIT_TEST_DIRS + if _UNIT_TEST_DIRS is None: + _UNIT_TEST_DIRS = unit_test_dirs() + return _UNIT_TEST_DIRS + + +def _is_unit_item(item): + """True when a collected item's file lives under a co-located unit-test dir.""" + try: + p = Path(str(item.path)).resolve() + except Exception: + return False + return any(p.is_relative_to(d) for d in _unit_test_dirs_cached()) + + +def unit_test_files(): + """Co-located unit-test files to collect: every ``test_*.py`` under a package + ``test/`` dir, minus the ament lint files. + + We collect explicit files (not the dirs) because pytest does not apply ignore + rules to files it recurses into from an explicitly-passed directory — passing + the exact files is the only deterministic way to keep ament lint tests out. + """ + files = [] + for d in _unit_test_dirs_cached(): + for f in sorted(d.glob("test_*.py")): + if f.name not in _LINTER_TEST_FILENAMES: + files.append(f) + return files diff --git a/tests/harness/metrics.py b/tests/harness/metrics.py new file mode 100644 index 000000000..3d6e5e518 --- /dev/null +++ b/tests/harness/metrics.py @@ -0,0 +1,55 @@ +"""Per-run metrics recording (``metrics.json``).""" +import json +import os +import threading + +from harness.commands import _nodeid_dotted +from harness.session import current_item, run_dir + + +class MetricsRecorder: + def __init__(self, path): + self._path = path + self._data = json.loads(path.read_text()) if path.exists() else {} + self._lock = threading.Lock() + + def _flush(self): + tmp = self._path.with_suffix(self._path.suffix + ".tmp") + tmp.write_text(json.dumps(self._data, indent=2)) + os.replace(tmp, self._path) + + def record(self, test_name, key, value, unit="", direction="lower_is_better", **extra): + with self._lock: + if test_name not in self._data: + self._data[test_name] = {} + entry = {"value": value, "unit": unit, "direction": direction} + entry.update(extra) + self._data[test_name][key] = entry + self._flush() + + def record_list(self, test_name, key, values): + """Store a raw list (time series) — not scored by parse_metrics.""" + with self._lock: + if test_name not in self._data: + self._data[test_name] = {} + self._data[test_name][key] = {"samples": values} + self._flush() + + +_METRICS = None + + +def get_metrics(): + global _METRICS + if _METRICS is None: + _METRICS = MetricsRecorder(run_dir() / "metrics.json") + return _METRICS + + +def current_test_id(): + """Test id used as the metrics.json key. Matches JUnit XML's classname.name + format so parse_metrics.py can merge results.xml and metrics.json entries.""" + item = current_item() + if item is None: + return "unknown" + return _nodeid_dotted(item.nodeid) diff --git a/tests/harness/session.py b/tests/harness/session.py new file mode 100644 index 000000000..54277dacc --- /dev/null +++ b/tests/harness/session.py @@ -0,0 +1,59 @@ +"""Session-scoped mutable state shared between conftest hooks and harness helpers. + +pytest hooks (in conftest.py) *write* this state — ``init_run_dir`` in +``pytest_configure``, ``set_current_item`` in ``pytest_runtest_setup``/``teardown``, +``record_cmd_output`` from the subprocess helpers. Helpers *read* it — ``run_dir``, +``current_item``, ``last_cmd_output``. Keeping it here means helper modules never reach +back into conftest globals. +""" +import logging +from datetime import datetime +from pathlib import Path + +# Shared logger used across the harness and the tests. +logger = logging.getLogger("airstack") +logger.setLevel(logging.INFO) + +_DEFAULT_LOG_KEY = "_last" + +_run_dir = None +_current_item = None +_last_cmd_output: dict[str, str] = {} + + +def init_run_dir(airstack_root) -> Path: + """Create and record this session's timestamped results dir; return it.""" + global _run_dir + timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M-%S") + _run_dir = Path(airstack_root) / "tests" / "results" / timestamp + _run_dir.mkdir(parents=True, exist_ok=True) + return _run_dir + + +def run_dir(): + """This session's results dir (``None`` before ``pytest_configure`` runs).""" + return _run_dir + + +def set_current_item(item): + """Record the currently-running pytest item (``None`` between tests).""" + global _current_item + _current_item = item + + +def current_item(): + """The currently-running pytest item, or ``None`` outside a test.""" + return _current_item + + +def record_cmd_output(text, log_name=None): + """Store the latest subprocess output, keyed by ``log_name`` and as the default.""" + key = log_name or _DEFAULT_LOG_KEY + _last_cmd_output[key] = text + _last_cmd_output[_DEFAULT_LOG_KEY] = text + + +def last_cmd_output(log_name=None) -> str: + """The most recent subprocess output for ``log_name`` (or the default).""" + key = log_name or _DEFAULT_LOG_KEY + return _last_cmd_output.get(key) or _last_cmd_output.get(_DEFAULT_LOG_KEY, "") diff --git a/tests/harness/sim.py b/tests/harness/sim.py new file mode 100644 index 000000000..ec4c8a159 --- /dev/null +++ b/tests/harness/sim.py @@ -0,0 +1,189 @@ +"""Shared sim-topic test infrastructure: sim target configs + ros2 topic sampling. + +Used by the liveliness / sensors / takeoff system tests to probe topic liveness and +publish rates on the robot/sim containers. +""" +import subprocess +import time +from concurrent.futures import ThreadPoolExecutor, as_completed + +from harness.commands import ROS_DISTRO_SETUP, _parse_hz, _run_teed, ros2_exec +from harness.session import logger + +SIM_CONFIG = { + "msairsim": { + "profile": "ms-airsim", + "sim_container": "ms-airsim", + "sim_setup_bash": "/root/ros_ws/install/setup.bash", + "robot_setup_bash": "/root/AirStack/robot/ros_ws/install/setup.bash", + "extra_env": { + "URDF_FILE": "robot_descriptions/iris/urdf/iris_stereo.ms-airsim.urdf", + # Clear any user-set paths in .env so entrypoint auto-fetches Blocks. + # Shell env wins over --env-file in docker compose substitution. + "MS_AIRSIM_ENV_DIR": "", + "MS_AIRSIM_BINARY_PATH": "", + }, + }, + "isaacsim": { + "profile": "isaac-sim", + "sim_container": "isaac-sim", + "sim_setup_bash": "/opt/ros/jazzy/setup.bash", + "robot_setup_bash": "/root/AirStack/robot/ros_ws/install/setup.bash", + "extra_env": { + "ISAAC_SIM_USE_STANDALONE": "true", + "ISAAC_SIM_SCRIPT_NAME": "example_multi_px4_pegasus_launch_script.py", + "PLAY_SIM_ON_START": "true", + # Multi script gates RTX LiDAR on this flag; example_one always spawns it. + # `sensors` tests expect ouster topics + lidar_point_cloud_filter path. + "ENABLE_LIDAR": "true", + }, + }, +} + + +def wait_for_first_message(container, topic, domain_id, setup_bash, timeout=60): + """Wait up to `timeout` seconds for one message on `topic`. Returns seconds + elapsed on success, None on timeout. Each attempt sources the workspace + and runs `ros2 topic echo --once`; if the workspace isn't built yet or the + topic has no publisher, the attempt fails fast and we retry. + """ + start = time.time() + deadline = start + timeout + logger.info("Probing %s on domain %d in %s (timeout=%ds)", + topic, domain_id, container, timeout) + attempt = 0 + while time.time() < deadline: + attempt += 1 + per_attempt = min(max(1, int(deadline - time.time())), 10) + try: + result = ros2_exec( + container, + f"timeout {per_attempt} ros2 topic echo --once {topic}", + domain_id=domain_id, setup_bash=setup_bash, timeout=per_attempt + 5, + ) + except subprocess.TimeoutExpired: + logger.warning("Attempt %d subprocess timeout for %s, retrying", attempt, topic) + time.sleep(2) + continue + # ros2 prints "---" on its own line after a real message. + if result.stdout.rstrip().endswith("---"): + elapsed = round(time.time() - start, 2) + logger.info("Got first message on %s after %.2fs (attempt %d)", + topic, elapsed, attempt) + return elapsed + logger.warning("Attempt %d failed for %s, retrying", attempt, topic) + time.sleep(2) + logger.error("Timed out waiting for first message on %s after %ds", + topic, timeout) + return None + + +def sample_hz(container, topic, domain_id, setup_bash, duration=5, window=10): + """Sample publish rate on `topic` for `duration` seconds. Returns float or None.""" + result = ros2_exec( + container, + f"timeout {duration} ros2 topic hz --window {window} {topic} 2>&1", + domain_id=domain_id, setup_bash=setup_bash, timeout=duration + 15, + ) + return _parse_hz(result.stdout + result.stderr) + + +def parallel_sample_hz(container, topic_domain_pairs, setup_bash, duration=5, window=10): + """Sample Hz for multiple topics concurrently; return {topic: hz_or_None}. + + One `docker exec` that backgrounds each `ros2 topic hz` probe, waits for all, + then cats each probe's temp file. + """ + probes = [] + temp_files = {} + for i, (topic, domain) in enumerate(topic_domain_pairs): + fname = f"/tmp/hz_{i}.out" + temp_files[topic] = fname + probes.append( + f"(ROS_DOMAIN_ID={domain} timeout {duration} " + f"ros2 topic hz --window {window} {topic} > {fname} 2>&1) &" + ) + # Newlines, not `&& ... &`: bash precedence makes `A && B && C & D &` only + # apply the && chain to C, so later backgrounded probes would miss the + # sourced PATH. One statement per line sidesteps this entirely. + lines = [f"source {ROS_DISTRO_SETUP}", f"source {setup_bash}"] + probes + ["wait"] + for fname in temp_files.values(): + lines.append(f"echo '===FILE {fname}==='") + lines.append(f"cat {fname} 2>/dev/null || true") + script = "\n".join(lines) + result = _run_teed( + ["docker", "exec", container, "bash", "-c", script], + timeout=duration + 30, + ) + rates = {} + if result.returncode == 0 or result.stdout: + chunks = result.stdout.split("===FILE ") + for chunk in chunks[1:]: + header, _, content = chunk.partition("===") + fname = header.strip() + topic = next((t for t, f in temp_files.items() if f == fname), None) + if topic: + rates[topic] = _parse_hz(content) + for topic, _ in topic_domain_pairs: + rates.setdefault(topic, None) + return rates + + +def _echo_once_received_message(result): + """True if ``ros2 topic echo --once`` printed a full message (trailing ``---``).""" + out = (result.stdout or "").rstrip() + return out.endswith("---") + + +def parallel_echo_once_robot_topics( + probes, setup_bash, per_topic_timeout, +): + """Liveliness for heavy topics (e.g. PointCloud2): ``echo --once`` per probe in parallel. + + ``ros2 topic hz`` often never reports a rate on large point clouds (decode backlog). + + Parameters + ---------- + probes : list[tuple[str, str, int]] + ``(container_name, topic, ros_domain_id)`` — use the **robot container** that + hosts that domain's graph (replica ``n`` for ``robot_n``). + setup_bash : str + Workspace ``setup.bash`` path inside the container. + per_topic_timeout : int + Wall seconds per ``timeout … ros2 topic echo --once``. + + Returns + ------- + dict[str, float | None] + ``{topic: 1.0}`` if a message arrived, else ``{topic: None}`` (metrics use 1.0 + as a nonzero "alive" placeholder, not a measured Hz). + """ + rates = {} + + def _one(container, topic, domain_id): + cmd = f"timeout {per_topic_timeout} ros2 topic echo --once {topic}" + return topic, ros2_exec( + container, + cmd, + domain_id=domain_id, + setup_bash=setup_bash, + timeout=per_topic_timeout + 15, + ) + + with ThreadPoolExecutor(max_workers=max(1, len(probes))) as pool: + futures = { + pool.submit(_one, container, topic, domain_id): topic + for container, topic, domain_id in probes + } + for fut in as_completed(futures): + topic = futures[fut] + try: + _, result = fut.result() + except Exception as e: + logger.warning("echo-once probe failed for %s: %s", topic, e) + rates[topic] = None + continue + rates[topic] = 1.0 if _echo_once_received_message(result) else None + for _, topic, _ in probes: + rates.setdefault(topic, None) + return rates diff --git a/tests/integration/README.md b/tests/integration/README.md new file mode 100644 index 000000000..05da4c689 --- /dev/null +++ b/tests/integration/README.md @@ -0,0 +1,36 @@ +# Integration tests (`tests/integration/`) + +Cross-component tests (`@pytest.mark.integration`) that wire a few **real** components +together — the robot autonomy container plus a host-side component — **without** a +simulator or GPU. They sit between the hermetic unit tests and the full sim-based system +tests: heavier than a unit test (they need the `robot-desktop` image + a running +container), lighter than a system test (no sim license, no GPU). + +## The `robot_autonomy_stack` fixture + +Defined in [`../conftest.py`](../conftest.py). Module-scoped. It: + +- reuses an already-running `robot-desktop` container when one is present (fast local + iteration — left running afterward), otherwise runs `airstack up robot-desktop` with + `AUTOLAUNCH=true NUM_ROBOTS=1 COMPOSE_PROFILES=desktop` and tears it down after the + module (same behavior as the `build_packages` fixture); +- **skips** cleanly when the `robot-desktop` image isn't built locally. + +It yields `{"container": , "brought_up": bool}`. + +## Collection order + +Integration runs after `build_docker` / `build_packages` (it needs the image + a colcon +build) and before the sim tiers — see `_MODULE_ORDER` in `tests/harness/collection.py`. + +## Running + +```bash +airstack test -m integration -v +``` + +## Adding an integration test + +Create `tests/integration//test_*.py`, request the `robot_autonomy_stack` fixture, +and mark the module `pytestmark = pytest.mark.integration`. Keep it sim-free and +GPU-free — anything needing a simulator belongs in `tests/system/`. diff --git a/tests/pytest.ini b/tests/pytest.ini index 03fee8c32..dc8c939de 100644 --- a/tests/pytest.ini +++ b/tests/pytest.ini @@ -3,12 +3,13 @@ markers = unit: Fast hermetic tests (no Docker stack; numpy / pure Python) build_docker: Docker image build tests build_packages: Colcon workspace build tests + integration: Cross-component integration tests (robot container + a host-side component; no sim/GPU) liveliness: Container and process health (Docker, tmux, sentinel ROS 2 nodes) sensors: Sim and robot sensor topic rates, LiDAR validation, sim RTF takeoff_hover_land: End-to-end takeoff / hover / land action tests autonomy: Fixed-pattern trajectory path-tracker benchmark (test_fixed_trajectory.py) testpaths = . -addopts = -v --durations=0 +addopts = -v --durations=0 --import-mode=importlib cache_dir = /tmp/.pytest_cache log_cli = true log_cli_level = INFO diff --git a/tests/robot/README.md b/tests/robot/README.md index a4409c4b1..3961d90cc 100644 --- a/tests/robot/README.md +++ b/tests/robot/README.md @@ -1,37 +1,17 @@ -# Robot-side unit test proxies +# Robot-side unit tests -Layout mirrors [`robot/ros_ws/src/`](../../robot/ros_ws/src/) autonomy layers: - -| Directory | Maps to ROS workspace | -|-----------|----------------------| -| `behavior/` | `robot/ros_ws/src/behavior/` | -| `global/` | `robot/ros_ws/src/global/` | -| `interface/` | `robot/ros_ws/src/interface/` | -| `local/` | `robot/ros_ws/src/local/` | -| `perception/` | `robot/ros_ws/src/perception/` | -| `sensors/` | `robot/ros_ws/src/sensors/` | - -## Design: co-location + proxy - -**Test source** lives co-located with each ROS 2 package (the standard colcon -convention): +Unit-test **source is co-located** with each ROS 2 package (the standard colcon +convention) and is collected by `pytest tests/`: ``` robot/ros_ws/src///test/test_.py ← source of truth ``` -**This directory** contains thin proxy files that load the real test module via -`importlib` and re-export its `test_*` functions, making them discoverable by -`pytest tests/` and `airstack test -m unit` without any changes to the CI -workflow. Each proxy is ~15 lines. - -``` -tests/robot///test_.py ← proxy (re-exports above) -``` - -Both `airstack test -m unit` (pytest path via proxy) and -`colcon test --packages-select ` (direct path to source) run the same -test functions from the same file. +[`../colcon_unit_test_packages.yaml`](../colcon_unit_test_packages.yaml) lists which +packages have unit tests; `tests/conftest.py` resolves each to its `test/` dir and +collects the non-linter `test_*.py` files under `--import-mode=importlib`, tagging each +`@pytest.mark.unit`. Both `airstack test -m unit` and `colcon test --packages-select ` +run the same source. -All test functions must carry `@pytest.mark.unit`. For adding new tests see the -`add-unit-tests` agent skill. +To add a package's unit tests, list it under `robot.packages` in the YAML — see the +`add-unit-tests` agent skill. The per-layer subdirectories here hold only documentation. diff --git a/tests/robot/perception/natnet_ros2/test_natnet_ros2.py b/tests/robot/perception/natnet_ros2/test_natnet_ros2.py deleted file mode 100644 index fc9a78bde..000000000 --- a/tests/robot/perception/natnet_ros2/test_natnet_ros2.py +++ /dev/null @@ -1,32 +0,0 @@ -# Copyright (c) 2024 Carnegie Mellon University -# MIT License - see LICENSE in the repository root for full text. -"""Proxy: re-exposes natnet_ros2 unit tests from the package source tree. - -Unit test logic lives co-located with its package (ROS 2 / colcon convention): - robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py - -This file makes those tests discoverable by ``pytest tests/`` (CI) and -``airstack test -m unit`` without any changes to the CI workflow. -Run ``colcon test --packages-select natnet_ros2`` to also execute the C++ -gtests and ament linters. -""" - -import importlib.util -import sys -from pathlib import Path - -_repo_root = Path(__file__).resolve().parents[4] -_pkg_test = _repo_root / "robot/ros_ws/src/perception/natnet_ros2/test" -_real_file = _pkg_test / "test_natnet_ros2.py" - -# Load the real module under a unique name to avoid the circular-import that -# would occur if we used `from test_natnet_ros2 import *` (this file has the -# same name and pytest adds its directory to sys.path at collection time). -_spec = importlib.util.spec_from_file_location("_natnet_ros2_unit_tests", _real_file) -_real = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(_real) - -# Re-export every test_* symbol so pytest collects them from this proxy. -for _name in dir(_real): - if _name.startswith("test_"): - globals()[_name] = getattr(_real, _name) diff --git a/tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py b/tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py deleted file mode 100644 index e7babf9d4..000000000 --- a/tests/robot/sensors/lidar_point_cloud_filter/test_validation_core.py +++ /dev/null @@ -1,38 +0,0 @@ -# Copyright (c) 2024 Carnegie Mellon University -# MIT License - see LICENSE in the repository root for full text. -"""Proxy: re-exposes validation_core unit tests from the package source tree. - -Unit test logic lives co-located with its package (ROS 2 / colcon convention): - robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/test_validation_core.py - -This file makes those tests discoverable by ``pytest tests/`` (CI) and -``airstack test -m unit`` without any changes to the CI workflow. -Run ``colcon test --packages-select lidar_point_cloud_filter`` to also execute -the ament linters. -""" - -import importlib.util -import sys -from pathlib import Path - -_repo_root = Path(__file__).resolve().parents[4] -_pkg_test = _repo_root / "robot/ros_ws/src/sensors/lidar_point_cloud_filter/test" -_pkg_root = _pkg_test.parent # adds lidar_point_cloud_filter/ package to sys.path -_real_file = _pkg_test / "test_validation_core.py" - -# Make the package module importable so the real test can do -# `from lidar_point_cloud_filter.validation_core import ...` -if str(_pkg_root) not in sys.path: - sys.path.insert(0, str(_pkg_root)) - -# Load the real module under a unique name to avoid the circular-import that -# would occur if we used `from test_validation_core import *` (this file has -# the same name and pytest adds its directory to sys.path at collection time). -_spec = importlib.util.spec_from_file_location("_lidar_validation_unit_tests", _real_file) -_real = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(_real) - -# Re-export every test_* symbol so pytest collects them from this proxy. -for _name in dir(_real): - if _name.startswith("test_"): - globals()[_name] = getattr(_real, _name) From 55d9b887d983c4837e350b1102543897f307c4cf Mon Sep 17 00:00:00 2001 From: Andrew Jong Date: Tue, 4 Aug 2026 14:40:20 -0700 Subject: [PATCH 10/21] Add waypoint_flight system test judged by a standalone track checker (#378) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add waypoint_flight system test judged by standalone track checker New end-to-end acceptance test for planner integration/swaps: takeoff -> ordered waypoint route -> land, per (sim, num_robots, iter). - tests/system/test_waypoint_flight.py (mark: waypoint_flight): after takeoff, sends the route to the local planner's NavigateTask action as a nav_msgs/Path and captures odometry throughout; reuses the flight-cycle workers from test_fixed_trajectory.py (chain guard, takeoff/land, odom CSV capture). - tests/waypoint_checker.py: standalone stdlib-only judge — the odometry track must pass within --waypoint-tolerance of every waypoint IN ORDER, each within --waypoint-timeout of the previous arrival. Success is defined purely on the odometry track (not the action result), so swapping the global or local planner leaves the judgment unchanged; the checker also runs outside the harness on any ros2 `topic echo --csv` odometry dump. - Waypoints are relative to the robot pose at dispatch (x forward along heading, z up), so routes are spawn/sim agnostic. Default: 10 m square at takeoff altitude. - New pytest options: --waypoints, --waypoint-tolerance, --waypoint-timeout; mark registered in pytest.ini; docs in tests/README.md and AGENTS.md; VERSION 0.19.0-alpha.9 + CHANGELOG. Metrics recorded per robot: waypoint_success, waypoints_reached, navigate_action_success, route_time_sim_s, worst_closest_approach_m. Co-Authored-By: Claude Fable 5 * Calibrate waypoint_flight to validated stock behavior in Isaac Sim Validated end-to-end against Isaac Sim + the stock stack (4/4 phases pass in 3m20s; corners cut 3.75/5.13 m, final goal error 0.63 m). Fixes found by flying: - Path header frame: an empty frame_id crashed droan_gl (uncaught tf2::InvalidArgumentException in its plan TF transform); the goal now carries the frame from the odometry snapshot (fallback "map"). - Dense plan dispatch: sparse poses get corner-skipped by the local planner's distance-walking look-ahead; the route is now interpolated at 1 m from the current pose (mirrors real global-planner output). - Route/tolerance semantics: the stack's contract is "reach the goal precisely, follow the corridor loosely" (droan_gl cost = deviation - path_distance cuts corners ~4-7 m). Split tolerances: intermediate corridor 15 m, final goal 2.5 m (new --goal-tolerance; NavigateTask's 1.5 m + tracking lag). Default route is now an open 30 m square — NavigateTask succeeds on distance to the FINAL pose, so closed loops succeed instantly without flying (documented). - Settle capture: the action succeeds on the tracking point, which leads the drone by up to the look-ahead distance (~10 m); capture now continues until the drone is stationary (max 30 s) so the goal approach is recorded. New metric: final_goal_error_m. - waypoint_checker: closest_approach now reports the true minimum over the remaining track instead of the tolerance-boundary crossing (arrival stays first-crossing, ordering semantics unchanged). Co-Authored-By: Claude Fable 5 * Raise default waypoint route +10m to clear scene clutter Validated on both sim backends with the identical default config (open 30 m square climbing to ~20 m AGL): - Isaac Sim: corners 5.67/5.72 m, final goal 0.28 m, 4/4 phases - ms-airsim (Blocks): corners 5.93/5.67 m, final goal 0.89 m, 4/4 At the old takeoff-altitude route the drone collided with a Blocks obstacle (disparity was streaming, so DROAN had perception — the corner-cut diagonals leave the forward stereo's coverage). This test judges route-following, not obstacle avoidance, so the default route flies above the clutter; documented in the option help and README. Co-Authored-By: Claude Fable 5 * Add waypoint_flight screenshots from validation runs Captured mid-route during the validated flights: Isaac Sim viewport with the drone on the square route, ms-airsim Blocks with the drone clearing the obstacle field (collision count 0), and the Foxglove GCS dashboard showing the planned path, expanded obstacle voxels, robot task panel, and live stereo feed. Embedded in the waypoint section of tests/README.md. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- .env | 2 +- AGENTS.md | 1 + CHANGELOG.md | 1 + tests/README.md | 121 +++++- tests/assets/waypoint_flight_foxglove.png | Bin 0 -> 242959 bytes tests/assets/waypoint_flight_isaac.jpg | Bin 0 -> 321155 bytes .../waypoint_flight_msairsim_blocks.jpg | Bin 0 -> 53896 bytes tests/conftest.py | 27 ++ tests/pytest.ini | 1 + tests/system/test_waypoint_flight.py | 372 ++++++++++++++++++ tests/waypoint_checker.py | 164 ++++++++ 11 files changed, 687 insertions(+), 2 deletions(-) create mode 100644 tests/assets/waypoint_flight_foxglove.png create mode 100644 tests/assets/waypoint_flight_isaac.jpg create mode 100644 tests/assets/waypoint_flight_msairsim_blocks.jpg create mode 100644 tests/system/test_waypoint_flight.py create mode 100644 tests/waypoint_checker.py diff --git a/.env b/.env index 00cd4c639..c9527e1f8 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.8" +VERSION="0.19.0-alpha.9" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/AGENTS.md b/AGENTS.md index 884fc1e0a..1579006ec 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -222,6 +222,7 @@ Pytest-based system tests live under [`tests/system/`](tests/system/). They brin | [`tests/system/test_sensors.py`](tests/system/test_sensors.py) | `sensors` | Topic Hz (Isaac: batched sim + robot ``ros2 topic hz``; filtered LiDAR ``echo-once`` + validation script), RTF, sensor stability time-series | Docker, GPU, sim license | | [`tests/system/test_takeoff_hover_land.py`](tests/system/test_takeoff_hover_land.py) | `takeoff_hover_land` | 4-phase flight chain (PX4 ready → takeoff → hover → land) per (sim, num_robots, iter, velocity) | Docker, GPU, sim license | | [`tests/system/test_fixed_trajectory.py`](tests/system/test_fixed_trajectory.py) | `autonomy` | 4-phase flight chain (PX4 ready → takeoff → execute Circle/Figure8/Racetrack/Line trajectory → land) per (sim, num_robots, iter, trajectory_type); records cross-track error and path RMSE | Docker, GPU, sim license | +| [`tests/system/test_waypoint_flight.py`](tests/system/test_waypoint_flight.py) | `waypoint_flight` | 4-phase flight chain (PX4 ready → takeoff → NavigateTask waypoint route → land) per (sim, num_robots, iter); pass/fail judged on the odometry track by the standalone [`tests/waypoint_checker.py`](tests/waypoint_checker.py) (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`) | Docker, GPU, sim license | The pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures live in [`tests/conftest.py`](tests/conftest.py); the shared helpers are split by concern into the [`tests/harness/`](tests/harness/) package (`session`, `discovery`, `commands`, `containers`, `metrics` (with `MetricsRecorder`), `sim`, `collection`) and re-exported through `conftest`, so `from conftest import ` still resolves. Each run produces a timestamped directory under `tests/results//` with `summary.txt`, `results.xml`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). diff --git a/CHANGELOG.md b/CHANGELOG.md index a5ea32f38..43940f0b7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,6 +14,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `ros-${ROS_DISTRO}-mavros-extras` in the robot image (provides the vision_pose plugin used for external-pose deployments) - `overrides/l4t-px4-realrobot.env` — site-agnostic deployment override for a single real PX4 robot on a Jetson (aarch64/l4t) - `integration` test tier (`tests/integration/`, `integration` mark) with a shared `robot_autonomy_stack` fixture (robot container, no sim/GPU) +- `waypoint_flight` system test (`tests/system/test_waypoint_flight.py`): takeoff → ordered waypoint route via `NavigateTask` (dispatched as a dense plan) → land, judged on the odometry track by the standalone stdlib-only `tests/waypoint_checker.py` (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`); validated end-to-end in Isaac Sim; serves as the standard acceptance check after integrating or swapping a planner module ### Changed diff --git a/tests/README.md b/tests/README.md index 93a07dda8..6172d1e98 100644 --- a/tests/README.md +++ b/tests/README.md @@ -24,6 +24,7 @@ Pytest hooks and the shared fixtures live in `tests/conftest.py`; reusable helpe | [`system/test_sensors.py`](system/test_sensors.py) | `sensors` | After liveliness in collection order: sim + robot stereo/depth Hz (**Isaac:** batched ``ros2 topic hz`` to avoid bridge overload; **ms-airsim:** single batch), filtered LiDAR via ``echo --once`` + cloud sanity (isaacsim), sim RTF, ``test_sensor_streams_stable`` | Docker daemon, GPU, sim license | | [`system/test_takeoff_hover_land.py`](system/test_takeoff_hover_land.py) | `takeoff_hover_land` | End-to-end flight: PX4 readiness gate, takeoff to 10 m, hover stability, land — one chain per (sim, num_robots, iteration, velocity) | Docker daemon, GPU, sim license | | [`system/test_fixed_trajectory.py`](system/test_fixed_trajectory.py) | `autonomy` | Fixed-pattern trajectory evaluation: takeoff, execute a trajectory (Circle, Figure8, Racetrack, Line), record path deviation metrics, land — one chain per (sim, num_robots, iteration, trajectory_type) | Docker daemon, GPU, sim license | +| [`system/test_waypoint_flight.py`](system/test_waypoint_flight.py) | `waypoint_flight` | Ordered-waypoint navigation: takeoff, send a waypoint route to `NavigateTask`, judge the odometry track with the standalone [`waypoint_checker.py`](waypoint_checker.py), land — one chain per (sim, num_robots, iteration) | Docker daemon, GPU, sim license | ### Unit tests (co-located) @@ -55,7 +56,7 @@ container or brings one up automatically (like `build_packages`), then tears it Collection order runs integration after `build_packages` and before the sim tiers. Marks can be combined with pytest logic: -`-m unit`, `-m "build_docker or build_packages"`, `-m integration`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, `-m autonomy`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). +`-m unit`, `-m "build_docker or build_packages"`, `-m integration`, `-m liveliness`, `-m sensors`, `-m takeoff_hover_land`, `-m autonomy`, `-m waypoint_flight`, or e.g. `-m "liveliness or sensors"` (see **Bring-up scope** below). ### Bring-up scope (`airstack_env`) @@ -373,6 +374,124 @@ airstack test -m autonomy \ --- +## Waypoint Flight Tests (`system/test_waypoint_flight.py`) + +`TestWaypointFlight` runs a **4-phase flight chain** per `(sim, num_robots, +iteration)`: after takeoff it sends an ordered waypoint route to the local +planner's `NavigateTask` action (`/robot_N/tasks/navigate`) as a **dense** +`nav_msgs/Path` (interpolated at 1 m from the current pose through the +waypoints, mirroring real global-planner output), captures odometry +throughout, then lands. + +| Isaac Sim | ms-airsim (Blocks) | +| --------- | ------------------ | +| ![Isaac Sim waypoint flight](assets/waypoint_flight_isaac.jpg) | ![ms-airsim Blocks waypoint flight](assets/waypoint_flight_msairsim_blocks.jpg) | + +![Foxglove during a waypoint flight](assets/waypoint_flight_foxglove.png) +*Foxglove (GCS dashboard) during the route: planned path and expanded +obstacle voxels in the 3D panel, Robot Tasks panel, live stereo feed.* + +Pass/fail is judged by the standalone +[`waypoint_checker.py`](waypoint_checker.py): the odometry track must pass +within `--waypoint-tolerance` of **every waypoint in order**, each within +`--waypoint-timeout` seconds (odometry clock) of the previous arrival, and +additionally end within `--goal-tolerance` of the final waypoint. The +criterion is defined purely on the odometry track — not the action result — +so swapping the global or local planner leaves the judgment unchanged. This +makes the test the standard acceptance check after integrating or swapping a +planner module. + +Waypoints are specified **relative to the robot pose at dispatch** (x forward +along the initial heading, z up from dispatch altitude), so routes are +spawn-point and simulator agnostic. The default route is an open 30 m square +flown 10 m above takeoff altitude (~20 m AGL) so it clears scene clutter in +both default scenes (Isaac open plane, AirSim Blocks) — this test judges +route-following, not obstacle avoidance. + +**Tolerance calibration** (validated against stock Isaac Sim flight): the +stack's navigation contract is *reach the goal precisely, follow the route +corridor loosely*. Stock `droan_gl` scores candidate trajectories with +`cost = deviation - path_distance`, which cuts corners (~4–7 m observed), so +the intermediate tolerance is loose (15 m) while the final goal is tight +(2.5 m = NavigateTask's 1.5 m goal tolerance + tracking lag). `NavigateTask` +succeeds on the **tracking point**, which leads the drone by up to the +look-ahead distance, so the test keeps capturing after the action returns +until the drone is stationary (max 30 s). Two route-design rules follow: +routes must **end away from the start** (the action succeeds instantly on a +closed loop), and legs should be **≥ 2× the intermediate tolerance** so the +corridor check can discriminate route-following from goal-beelining. + +### Phase order + +| Phase | Test | What happens | +| ----- | ---- | ------------ | +| 1 | `test_px4_ready` | Waits for MAVROS connected + odometry publishing; per env | +| 2 | `test_takeoff` | Takeoff to 10 m at 1 m/s; asserts altitude within 10 % | +| 3 | `test_waypoint_route` | Sends `NavigateTask`; captures odom; asserts checker verdict | +| 4 | `test_landing` | Sends `LandTask`; asserts final altitude < 0.5 m | + +A `test_waypoint_route` failure does **not** poison the chain — `test_landing` +always runs so the drone returns to the ground. + +### Recorded metrics + +| Metric key | Unit | Description | +| ---------- | ---- | ----------- | +| `ready_duration_sys_s` | s | Wall-clock time from test start until PX4 ready | +| `waypoint_success` | — | 1.0 if the checker passed the whole route | +| `waypoints_reached` | — | Waypoints reached in order (`higher_is_better`) | +| `navigate_action_success` | — | 1.0 if the action returned `success: true` | +| `route_time_sim_s` | s | Odometry-clock time over the captured route | +| `worst_closest_approach_m` | m | Largest closest-approach distance over all waypoints | +| `final_goal_error_m` | m | Closest approach to the final waypoint (asserted ≤ `--goal-tolerance`) | + +### The standalone checker + +[`waypoint_checker.py`](waypoint_checker.py) is stdlib-only and judges any +odometry CSV against a route, independent of the AirStack harness — useful +for judging waypoint flight on other ROS 2 systems or in agent-evaluation +settings: + +```bash +ros2 topic echo --csv /robot_1/interface/mavros/local_position/odom > odom.csv +python3 tests/waypoint_checker.py --odom-csv odom.csv \ + --waypoints "10,0,10; 10,10,10; 0,10,10" --tolerance 1.5 --budget 120 +``` + +It prints a JSON verdict (per-waypoint reached/closest-approach/elapsed) and +exits 0 on pass, 1 on fail. Note the CLI takes waypoints in the **odometry +frame** (the pytest wrapper does the relative-to-world transform). + +### Running waypoint flight tests + +```bash +# Default 10 m square route; ms-airsim; 1 robot +airstack test -m waypoint_flight \ + --sim msairsim \ + --num-robots 1 \ + --stress-iterations 1 \ + -v + +# Custom route with an altitude change, Isaac Sim +airstack test -m waypoint_flight \ + --sim isaacsim \ + --num-robots 1 \ + --waypoints "30,0,0; 30,30,5; 0,30,5" \ + --goal-tolerance 2.0 \ + -v +``` + +### CLI option reference (waypoint-specific) + +| Option | Default | Description | +| ------ | ------- | ----------- | +| `--waypoints` | `30,0,10; 30,30,10; 0,30,10` | Ordered route `x,y,z; ...` relative to dispatch pose; must end away from start | +| `--waypoint-tolerance` | `15` | Pass distance (m) to each intermediate waypoint (corridor check) | +| `--goal-tolerance` | `2.5` | Pass distance (m) to the final waypoint | +| `--waypoint-timeout` | `120` | Per-waypoint time budget (s, odometry clock) | + +--- + ## Metrics Reporting (`parse_metrics.py`) [`parse_metrics.py`](parse_metrics.py) reads `results.xml` and `metrics.json` from a run directory and produces a markdown report. It has two modes: diff --git a/tests/assets/waypoint_flight_foxglove.png b/tests/assets/waypoint_flight_foxglove.png new file mode 100644 index 0000000000000000000000000000000000000000..a3812dcb5a57e0a7b2d2a856f33104fa013a049c GIT binary patch literal 242959 zcmZ5{1ymbdyY(O~QYgU+6nAKg6^9~4iv)_h6?cc?Qd~>W;!bhb;4a18o!}IA`19WH z{q9}spOrOhW|B-!&OEmFvrp((IY~_Px99)>U`l-vQv`sQ!T^92fQkse;=*Ea2_OJ~ zl-OtG?+b^^&fiJ-m$IXwoXA3Nh}jv+?wnPyci_y%IRz;_4bTlG-6mfx1;hDWxbQ!%js< zPydP%6tl2ZJj^$QkB=``FtqJpKT-%rr3?Uxz;_R$22;rYIsE5*OIiZq!#~&m=S`4< zcn#O?LSLuG9O?5A&5XI(dH3X{?Ik`WAUvwYRX5`Y6mZh z@3Z@uB)|}+sHEmvQu2vSuw&cBJ*Sy0dXX5PmQDEwf&igi)6y@u#`Tc#Y$=<>SDy)s z!bDi9i+onohdU|;hTn6!J1d4ipwT(HsoXVSEP zwcw`=_Y#JSQ5C&~7|*ZUeSzc_?fbM*QwMw)kgJ!ZK=+HN+}+*X$9K?SzoradxBmo6OfSM4C-DG;M6?3Jstpqi^K(?9jP7%~e4)A32*kyl}RZIPO z5ust#2jU?6Ue~m_5sLTOWzw&2K0|yR^)WcWUvQ*90>Wb~pwlL;_~GK|yLSU7l3@HQ zOzaMh&!`M62|-cpyPIO`z*(DI!gl!0+kdU>rw?~5btz}4_du}gw7fp$b7C`PG`6n_F2~ zd44c3FtE3`*XtJTRwJRo#li6ff=575aL_dU0HHBYR#j4pO-=Re)!iQ$9;U!UWI|-d z`i-4sZt@Ks*^}xp*60Nd8bE=E2V|Oevi=Fc8&`joD#J{$NrKoKYZ6LhZTdw4Z<6EL zD}IhpbfhQVvH57W4CKpzOyI9GWtHjgC>&&GZwV^UjY@T-Ly)%r!R-Ldu4%|7UIS#`G(jYbUNZd1J553ZC>uxMsUKLjM=D=aL=>a@(tYX}+6 zr$Fyw#n}8Ld4&cTg^5CFjQ-$~LS0T1YgJTqZKrCow>k=yDU)8>yTW+hQ z<(MWG=`#y|My56MTPiXsaasP%Nw zGyyg1lGl^-@D+VWBzv)CKw|`@?T*QU?z}Bxeoos3pdyZ1G~{& zb5F|O5y8x>2tfh0BPoD^hoCM(jZra9Z_b3D5U-@WLLZ^ri>*#UTT<#;iS+i)2VJVSPJn&6`_bFL;^;@2=F7g zTM666x*p;Hvz(-~8VNK2a{#_XR}R0bPn{W9R<%6EwwY@%sx?`ZeE9aGKUb(9P;xS?h! zYQmgi|2n4%SxeQtReXFr)b%8t$G)@9dJ)M_*w{F)$U?Q>@L;AC3FY-y73ZM@rX0!W zC^4#nt8aMX@0Q!VECcQPj=1~v{jqhco$Ocs7BXg=4AphvIOtK};NuSSVvr{8vq}p- ze~TIYsf}Ky1GV1@uyo%p*k9mPQUnG5q@jBshJ7|~%*)N<(=~`6YLk~MNLXg}r}JH0 zTv}h$RzwQbC;ONg(3f#8(>UynEF_fTM;tlR0%GvRs?`;>WRzL*4Aj*YV8swssr^Dj z6#%y&N2`N}Sw^XepOm6pR&y05!^!Ji!Q^~Szu{Ii^;?0Bjm=c*j`uO?L+p=|3M{?x z=cg3?)2zC&*?@z?0rN8g^1oq*tj77vVY-FbY_Ov-+ctu$+Ak5GVV|u+%&gTD@}fN1 z#ilqO_xCXP(?-1RueiDEWs_J1-@TJgpv_yR2R`UqY(qD-KLw;RvOd+o@Q%nw=X-F_p1+PtpH%8*auyR&VIz7 zsT2=olY-C$0TmD7SX^Z0=qkq#X2&0;AW1Y@9M8{jNqI5)zY#;WxJ@&x| zrj}-F9h?~t7mc&Hj_Ao(%un;|&!~>VbROB+Ei1npZfKBh7G#WxYB(GD3C9Y-u zwH=@3jvpvWii(4SgHa^BR8opkQs!=Mx0QyH&N{`##fz1OrXnIDIy#~hARZ;Ox5bLz z3w|biBW#u_r*vKQxNe{nz)!J!s3AUSJx^dhZE))NhMnlTNb0)e21^r#(w1p77BQL) z;-vE%Ab%DSqwc&=EepECB1(>rd24Da+Z*&ouVUvXxO7M+0vyjew?FAbYJ`&6?HrTF z!`J)EAX!cIPXOD@+?=|aS|6cie8r(lO4qs_&mjSJOVtLT8Z|wZB^nEXsFrGOU*Gpd z5IJ7$;-_*U$)K*02(yMZa(_^BwL+ImE|X0%h>kL7yQdVYNoic#;BR}=7+7H-ZgdKB z-!tghJDanb=LjKqArzb-E<+=i!3R6Ku=7G3DF_xQZx%msRyo;_b$ay~pu739+BX?GOjG9v-V zhQ}!6TT!Vi0s`)PsoYdjNadvSo4X0y{3*-aV6x`FIovbHpY|hUppUI64mndSJEd<~ zT^uwca&C78mt$81-lGF9CoIPThivuDtLsjHFLsFsth7Gd@<`EpGYU@`7GAT5YVqTE9wRpRRvJ1jGf{)Ets*To*YVSLJt_4q%W?k(z zEBhlXgHDzzip)0Bu9x_*ZAiYuklJiQgiRDztX*LUz)kq<2R+yO3v8fbEml5g>9i}j z{OA?nHN(t~ffR6TSOz8Y_SDbMIWIUac50;Bs5I=!`udYC)Ny5OtipCxPzoX&3)L?B z^9Q3z1Lr>m`xO3YS^Rw@0BzIoTXy0Hjf9w(@lvfUgCqz|VAp!BJ#;mlPG9p4#KSq?h2O<~tEEvhsAv!30y%e6oyp zF1CxLVMA-c#O#WonO2dLsSf6K0*8Y?9NKR3yDfO6a}Hc&>Ce${0SfUpQpwMuYuiix8W7cv$*Va^dr|>D=+c_xrYRbf`)We&#ySl>J+y$jf`S`7txilpHa7lwLEyU$oP%noafA|KirsT~t>$}~F$%M_ zwtmt6l_A%BArGa32L>n(uSYtn9g9q@`W*Feo3ecFzRmC+)Birm?RjwpdAt^Su!|O( zYl0Cb`Q?QtDA)c5DqOEel1iuu5O=|nMTi3KCT1}ej6V^YpO(t6>LgZq&hN%LdVId$ z;sU3z)rW`QF)o{!lQP`irvpVlER^>4OC@caul+3V=^wU7pIH$IljsP-s7U#p=28}2 zcc!vzAGd!AfK85~+(z3@x91LQ2q)h^$+z8!|9I)~rsc7ts0fA3yMaP=wP90TV%B@! ze84LhLnhoY;c&Idz3VlQ#Cjgpgh~E*UR$5>bn5s>HzMnb0Lw2sKy9@#;0q|e&AGv1jVv`F;%W$ za?g%0ZAP}{b@1e#(_Go%FKx8{AZ$Iikl*HSVX18SMVhRHKWW~Ciiu=>!UVQ*cxoj?3?yhF%09npq&zKqr~alWlgBzb?H9ytRQErAtbF1?CQ` zmdhAqGv5GEM^s_S9e9l2hw5TkEPBKFq_kIpdhvK(v+sc^p>U7T4LT^ zJWqAFZi;zwXlUs8__&|2@MMi0PwLN~AaZh_tn*o|&2vD%D+trs*%_YouiRVStCd-q zoAWB&x(pFbw*2&Kn^v~k2_UkY^ip86XFy&$%S;lI!5~7?32P^2dVHEr+CGmQT~RqG z%H37p%T<*S)8RH*aavBDchtLs5r=^p8Tz4DO#^jV z>%TW9UHt9dV7SNQQTm$! zo^o;1H&+Y4eNz0x=aw70Dhw}0^5gV!InX#GYzgqBM#}NQb9IRw@>(iM7F@K%*|0Du{^{;bNk-b65!s;rWpGAE!D2AjP z9Qp=-tY`QbR;j!z&bv~#XMLI{e^xA=yPFgeOnXcufcDUPKT5Va_vT6RPYDSA~NL=_x0|Rv=qeu*r3(`0$NQI@ zngnS+#4r!&nxNL!x@?verTUjC$juimdd{qHP-HG2uc0H=POEZs;(A5b=&Q!I)!xmi z4kz?~W2YTJIA-YC!7I8NERp9v(LGT}GCIGKCPHQi@RbQoPUVn2%Qb0k7)kpu1fg+0 zSy?W6rbbrhYz0q#8cK~Wuq79pl zC~I+ntdi^NWeGmd%ZC~eGACM&W%?0w+nn7$^78Vgu@kbeh0S7TS5X-k6-`f1i%Uv+ zw5+_s^htjTlY9xxx4d+NwVjQdGI{l_8Wk_!o!*~FF)mjvt?haZJGS+!H&|_-_?-DH z(=T5~YV~PmD3TR@uwKV)+5y_11nzOFpwhrg6e$Rp$%qvSMg)X=7_+edqTr=0A^;RX zJVUv|_b$^m@6+mz&9VXcFK2wkm_T>kTWuF2j1i(;MB^8c!eWk1sWWBnpQ3u3?&7dx zpZ`AN5R|Lmr;vN*JYVX081)JdvkE)epg-m%t|VXVmYckmLIMeSUK|^HNPzd^5rK^6 zbDQTQ-6#>QAp{v^RkD>(d+nt)f6+;Grg z&=q9A*_+0Qb#PHhgS@6K)<vp!FLx8lJ|h=B@($mmyw zHqo3;y1=m62rp8N ziN|r&_3}+=`n7@~>(tjOK~8A!7C9-s2X zrctS0iX&@V6d_YBjIL3c?0&?chDRW3>YSF5h#bTG^6&#Dhmq>4Dv1a}0~Kr22#uV{ zml%X&a15`sX+tU(>lcWD?7Pg&^rXC6vmj(1q^M$Jqy9R7PuYqA`3p|KkGgr=zb%W1 zUA>DgRue9^kZ&7@5T`pDZZV8tYC9D=dv&%CQjj1FqNRlecSggPqs(k<9m#wbAIr%7 zkiH-VW0H?2C9R8l*j!^%OJMBjAp?87zfK;W0`Xr7ZjrQJc4JX0->=`3=V7L6S39~& zC$69a*NNnX>~v0}FMwcfK^At3uTOWbZI3CdEq~Kker?I!u0)^2)AVt--CpWFua^@G z+BZNP+FVU-v^$Q7Gkk8fR=o#p3I^BQ1-}{Cz5ddu!q+xjraFws}eY+ziBHK!p`G_kL(YQArcGvc_At^K_yO+?{ zHp?W3(BFD~7vpo^6)oh{Smgn3@abixM2>B(8dvYFu$&d?ol~ovAox$wx8+miSZb~b zj~RnQL$R^3@R@D?ehN{@kvLw4>diVsm&slSYiVgoM!v26))QvOQ&ME1tUM*P84GI1 zD}nv2e9`hUm@2tz+hrbqne``|=;TIq(=tT$H-niSs8fF=32k-vP!R(htBdt7kUccvQxV+u|@+&3LIdc{n-*N zTSgq5w!5eqbwh4hcARlMar;JN5D24v&b;^Wx!~pFcTQT(G_89MVieKmZ88081cde?8#k=bH%JXszPAh3S=tGL>yx)2OM*DZB4N)Rm_CHmwRZ1Tq&5uU@VoIGQp$H$CMWcB;D{a?U4 z+TO6*+3Ao$gtqg|46plfeqyda#YN{YCfx6}v}C)P-oN2iOYH9MNl@X`b3D56IB6@V zu5x4)=%_{kF?(4jRt1|DDLY^s+yM?M4n=;2Y$>QIZ_u_X#ZByj}F zc8zPT@q^1?kn;QoDBqIg7$@| zS0}f3uB6Na01e*jN1F~*qV_Od4Ye!~$bH!yKZ^d)djB(rM|5tJGpg-}ZG^#z0s z6RDc#XX)iNty*S8f{~Q_OL4F;C(HdC;hzQb&6}2w_a@61o1C>M*7WLa(uTGVS6c4j z%w|dBROz?Y$GgjO_ZCxhB+BfyCA0G6iNPT=!kC zJIo%de`Vi)x(|}W&}SRt{4%t`Nc1rGdFRUCTv!q(XqmF*wEpWUHrJ6!FijSS0$Kf* zBmx1JIhSc^JIV2T_MV(?`}uZRun@5Y|9TH^$Y4em1Z2*U3^Km&kn8wh2~N{BIGSBM z1;y8Pn{#6V5n1>t>p5fRSN__y+p zpOLRDx`-g*?L0YI9vf>XB<8YWpdu4+=fDYV&@Ov|GgBw4C#ggARNZ4;;iQDPU$sOQ&A))TeA!Mb#2M=Fo=EOw~Q81d-+VnIgiqeV5kEw%AzllAJheN)BR{U0!{Q*6W@%`ITm3egNS^F9l9-cI!^8 zP_*Y$nW9Q>CnZOT5(@yb=909|(=TCy21@C>3Ohgu1$qdDbY_*qbajsOXw$s&uPFYF z4QIA{9_W>(u=C1gZXnL4s&|!2i z33%5A?3{KO3LZLH)^=&v=L()g1F*sG`-c(;*>{mM>fHDV>`&Txc~bS0zv@Lksd6}3 zk^h)ieJv%dr9&x&BS%1!Z&y#yNg1Z0y~k{c?RqA(mL1-J`>!cHtFEGy`D!wVpr&M( zsJHF4xUMQZGBFI8P3Ifnh=#(+G$qtB2Ciw}<`vOw68(TjUE3PQhRO9*4i_;C2Z$VP zs66Zf;R6670X|py(fHw4@O3qQ9_bkC;P5jcYd%u|4&FWB6#TEc;DMf(#6=1+lGD|T z00QwB^;srMj2E-Vj9$F5q{%GvGf|=BXBQLB-NpYuIYNMl0i{@ zA+{l7yI8G;4-XGdy-_1Bm^80nEF>)>gY(KbpoqY1^A|JTxot1s9r5dV9Z0FVRi|EJAR0K)&H+W6-)+T#qq z|2qrXG?d7&rMr0!umwou3C!FO0tI!tzoMEja`|`gA|W=v!>o{m0R&+IV+RC2#7r0h z5j3~Bm^#-{msr##R3YQl{8A&<5h9{`>bZ>V|7t|Ge4?gtPnM|Ph){f^SbN?ie{OI- z$4()&ZFyYKvTa;;Pt%MTliWG;25p^dggA~68b%-pTU#$5>)a0wy)TeUNrFI}KJ(KA z06eWXQqNXC*dwiicg+8IMt|NXnOY$)r@K>Wuc?EF;9ezNp-2CVBr*6L(iKE~u6laB zm~30B=QNr4O@P#KCnR`oSU==^RLd?q8Ar1xG@jRzeAY5T#M~e<C&tb%kha^aeYlv`Yc!*5MAa2+M z+VFnHeXyvwSg+Lz?4^VB(dA(mBn;P(ybcy1L`U)kE4;+oTP?61b@Y5W`peJws2KFE zFG}G4ij3iX@!@J){j!a`oLv6_#Il+(q@T9{JMe$z9d@tW_aZ%MdWARDWrkk7 zJMr1v*iZZ6YEAk9_tgW-!>_Fe;*++g;135)?)St008(Ck{fz1!lK|oFuRP5ju-3j- zZ=Z7AWnKaTzE30XX0FbJR+pOHp!f+JOHMo_NGO;5c|^znnV=H~E$wY(`7(>tp~&I@ zUrqA@u3PVI6SvO?w-mb!qN*md)DOxEe?tT+ z(b&qOD3S{;lhNb~Nab%+)e|!d4~lH~(q0#T-uo2^H}l3pC~lh1?svSS?=?J^8d{}k>;KUwcAhb`Z_)z$VAR@;aB@Qvl{BgH*6#cOgc|P~6_O*_R=sGHZs!vuf znXro%3KC&%li&Gs`Tf)Nifc-|4(CCaJUllhCXhirPvsX{H+$wqgsk;0m(0wpBBR2U z_^PZ+h=pN)zuvzmt>@*@Stz{>D=Mn`7E74UMUrEgO!GcsUCOn0(_FG|)ny{B_k@;{K_xPzbcP+mt#U;gICgsu`_m@05UxhAauu8$?)2e&K zxU(VL>aJIOJf5FgMqxTEt$!EpoKrk5_r0?UT~);{=f)vv{MWs`y*0Kv^0LwmCr^v= z_X+vEJ9BZa)?CD$c%Kj#=;Svy7uDoG_@SKCJT)4PhZ)PL zMH2BpeDEW?O-=d0Qm$R2Kd*DW1c`}x2@EBcuGC+H2%1etv(0DaR;oQRRtb1iwUc9G ziO9)a@4o&5`ToAQ(0!Pc86G*b^*(6*j)Dl!n=Aas>9b3mrM7)qbsz4m<_*pZZC4F! zpM{;J&tZt_r&sZ|Oz&<-u*jkmMsC*s3E%zGnVUpwH(l+a@(=W%GV5svQDEM6`g4C4 z4F&2T%PF%ejE3j$XGqbPT{geiB$s%VNJ`$_o?Zg_1yKkve>TUbdvXGBOOHm2fl8(4 zEM}LS``O+Dvdu%FmyWH;1rR7gP#+pvLiTCM)(zs?D2i{BQ}J zNUL-S%cVZ+)Win#K3$7n7fw@F&*xrTln5e01hpQ# zP>40SJ;B2JmYpn3B9<7D9Y0wi`zDvK>x*ZqU8mM@XZ#JcF?OH;>_8Y(qe7|6j-8mx z6Wa97k-Wfplv)Kv}e6&6)7XTM_jBH%o>jK)R{ zd|DQ`%O&N34U_WGm!dNNd{Ok;r{=I7Zc&*|3&SIp)gR-+uaBHf-_W_?GxLbL44${2 z`f$(2`J8q14IeYFx*G*~lATge%8^q*<1&$9sgB2yXGwHI9zXjzz#0kgM)NS3iJ+xX zn6;a?Ua_*I(kb73zsY{kqPC->jHV{K?-++tLJ|*K@*7lS z9@x5Fw@Ednh^m~4{u7SQ?C3|c(+CIB@yxH#3-=zneRC!ytK2jY%P#*(6de38R4mnGH2` z(0j3nL+jH*pq*x5_N&?~j@0sp%XoAf|+wkTA=ev#E%igU~*^ zlue|qO1)mOG%+_e_jX`-NFVTJxcFkOQ9DeWn1E+p`tH!StjIbNEEjst=gCQ%;nS+0 z&6F?yIZOvJ_6JQz*zGH}mQn6}EasJ}hn&$jZ@gf~aS!QOKBdG!_RPgbIC)-vH&Xi! z?JLW?h=NvbL2bvicnwRBi~evm|Jvb)s6O%&ocBacQIMtad3cU?(6HfEJxZHVU0q^% zgQ!oF{Soc@h<~odBMY!vH{ zmgF7B=QrG`T`|*$40xe$Xx+lk#@S0ds~U&m%=n%j{BrYel(i$bDm`5<4Sg@g-~EX& zREA#~g&z?C#H4sy{QQj(X?XyIkReo(976cIrl|AHrnAe+`AkAWhzVjg*;S&-4D=h0 zB5rn#-B;YIK4TQ1Scq)f-oG#*L3N-k=^{=Y7^5=KVXdEJZXiMpB6Tqy-MK7|3dwC4 zK&wp0_2qn&d@JeyhAQ@+H)~r0k4X z(Z%GfaDZjq4%u&ZhJ)VNhzI|7DFfD9MOBw%41KehuQWL5cPSQcaoDQ`xDFD+nVdNz zz2OAzWTF1tYL(MUV(1V3hc(3Vp*I%RMP=7L1^GP<*7V|h-y7WK&%jdijl(LGj}JvZ zsQsO3Q)!8gHj;uEr7*BZ@Y-=ifG-hjlUs=&D_6{2)gg1g>V_H{;x-{P`3ElnX&ghG zgCz*5@eGk6q&w5AwX=G*f$!S0{K=i|zw*)Mv!qWpaO z!L)0q>glK3$UIl?#~7z_W%rWql%!)}fc>imdHT7epT!#=|en;6-mDDI%*_;!qD^c+{0N4!=uocNnUe zxcK7(OrHFco&In#xPHZfA4X@01E6+bYYdDypLjM2zUQtk%(Cx}(0aLDRj>EFRYp(# zPM6JfWhOZ{Aj#4uwbJ#rDlNW9>4w8~2FW#oiLn&pF1h}MsnJBNqp!S8qsSeyMD>Ao z#c`A)anIv<`dTyIjB%-R6P>)-`h%#GTZ`Y|Ba2Uk!yr?A-*=L<;v=|vm*=o!zstwx zHn#il@T-jIc+$bk-ul+Nz4X3}=E=Z~q3cycaomC{m$tLf>+K}5mORRLK5J={!N*#S zX0|+gWyC?hN>yyz>bLIrZGM-OU2_wAyjF`MQVclSllL{dD|)`2kW?_&Nm@`}4vQAN z^;K>z{%38`vbeYC=Ig1N{vs7V8No1J(6<(x(l{hOOgwk_25m#G@}#H)zDiMi{uBN z-)7IoEFbAIe13fF4(R?qatp8FxNR4>yn3gYZ+X)e@J3k!;@OR2f*@YTt??voR?EiNd^CbW#oGnzf}HWwwGbI~^@4TsK}>xed9CW>9&-rD?Cwzn&3CRQUY zuJWuRfSHihjQCF8V)GXDpixn<0HzM%=mG9;`M#2h&NDX^z4G0d{A2aEV ztTY>(WIiA)jbVSr3|4Bo^10Z-&gArN-fq7^NhkTaZC+Gv3NgH?Xx+Bho~rcf2*F5m z==xK_k(=<@7kaWzY3F;>q0S>^>U3XEZ`Tn>u4A+Cxl8? zG%B?1JSLcZa$~Tr{7!#52yL~AcR}6l_s0qDtjn*FcVjy9i*K53hlu{RM|OR-d>YDF zWhN3B%lcyCiv&;+Chv!Zlk!#k9zfd+dV@BSrPTQvz3lDT4bd#DckQ0#!TR3sZoM+h zJo%^+F)Ab|pKs%%4b;(OcqIP$=AnJm=PK#<{Pogeioa)4iUG_+m8ay2#D3Yi3p#bE z11(m4+TlK_K=3@O1wMCeDdtucBGzXgVE*4Jw2>!N!9tfo=o1kBmb7% zr_rt}r#|{Pxt7xHqMMmk@X+1a0%hKJRT`o0Q|s#fnd zjSvH)dl)2`bL_pWpdZon?D_ly>>D$I{G!#>>r-yTvyd6BbG`^QdSH}YAfbj2O zXTcSg_e8*;v~*7FM`Te(_{4ZOc4)3!``9Jfk6=XM z+*+I21(hNHPg9y*c}8aC)0XHV@=`A;ksWDz-8!Fyy}}m^?feV~A^e;y16~a^=5w5r zSSbRwf$rt=3u!%lgIc(PoPG9&jJFd z9Ki2rUwS}0}En+~(=GePShNjQ;d>MU0YDuy{EBN@-M^>RpclXFWW6kSyenaN( zXmk8_f!M zWJaEyW+h-V%m?4!G10OOjixxMKD8A zTw!598WEk+NMnLy_-)D@Tl?QX(MhP8w#(iWl(%P%Mfa745=628{f zH8TH-w?@N2u)MOlnME1uU2-3djb6pGL{aiBVJK>=%+^KntvY)P#iz#M z8wmVd%KK{0q*Ga&JL>DLQF*i7Sqo2Ac)mZgmY&vrta_gbx(=;l3F*B8D0l!4wYa2_ zhByW^;kdYI;~iso$ihwK&(8ia9Qhg0NYFa4R56K`55bz6HTSK`=1`8Gtwo;wlcCmw zd))Lq<6%o;i5par?hk-s94C`?#p2U6)rDry)sK*v-qda$TmqKW<~39Hb0(DJmiBEUwfnb{JJpMJ4#W&0pDB3`~C zVAJZ{EA#BR8K1bC&U?OKBmP1)Y`%@rz4qJf0>RT=`O%^auZ~==U02g^dCOgy4dDz% zkJ>uQS3%gMfrQE3nX%C$>JhZ@H+p6kHulqe;8^zFK7EZny9_q{9TTzzPIl{yZ1XuTb(XU4g3`m&901U8;;3SNkp?fg zHrPDEO?eW`i5#gp6edmU{R(18d-q5WFd)>ML`h^aWU%G>2hZ3HH$_nv<= zEa0;DEI-qP5R{Rscz>HH!%ZhoI8l3!o!p5bzL5WIQh6F9kR){8<+*uwB_oC5g>;cu zi2kmomwPG0Ok!@lsVqWwW36%_x4gH9>t^i4By>7pSZV5vEkS*HrF6+C|h&86>?LS31=DH9KT@xtvzpiYz1 z!0}A6>1w*$XFUq2WZP*D3e2d0uDb{S@<1rpRxnmO)>QUnmGcGgTb2>dBRgy;K3>bH z!xf#fXwsE5wDpiMxl!53$g?e%6O&m_u($Kr9NHs#Y=CcBOOs5@8$Gdz6s}qsY2|OK zHZ>)jH4;r&B%d1eX6KTA-{f|0#oheg-G#I5b7tHWGGFAR!lFZ^zA*Mjl$I1lkL>b! zEwB=aTga6SW6W^gqTgc?FS8H;ubOs}u{jy31$s9lWMrkou1%be(%%@K-%d~YlTFfL zIjUow{TcTWwk@F2kUBu5c*oe{;Vtc0&C;T^YL5Go74+wLZ?%nKYKp-vwk(Z}ma|um za|w-d%VE{7o%N5kmg{R$uCaGnKSKCjkKro?r~YF9fL`xvPPMQKP#=wD<-nBy$An*l zP=OMeS!Sv^lCk+U1!{NKp|C6WSS1K9u3?Vh=ES5@_L56}L(^71r?RSKJFoqIE?zm< zp=ZaK4w+hgE{B1Ix~`~5kc-;q&+cbNdj)2Br_tla$Z1YRr~QiqdbcA*?Qe<>&9i%v zdn=smU-cS|IyS&sE+nD z29{qs%fnLnbKo*sI#soqMuX?r$?&Z3h#FI7YkY z9nMz?HcKs*C;6Bu(Nh6X*HL*H87b+I7*O^m?gX4g%+7`qY~Ln(%?sQ`k}dF43cll` zqZ1II!`LXzxrHC$z9Z+oz@jAIIQg0UA0@ zJi@Mhv{$>6I$Iy%+a#P*j@z+VwRwn z2Qp%ZC`z<@i%ka0&-k=o3F%bbFKB!;5{;dmj}nF#t+`SzVd#^SN-R!ExwQ{Lms(3?@P(Vpm|Zm=6F#U?DT|73&M@)!$rBC}B7sUv zpN3QFz^NQF=*Sr^j^Ce7E4GJJ%XMoF_G?0GYn`Q}LVpBbaDyF3j~_j*4$I4mi1@vy z+WowqyQ){lX!!qWpz_{Vi!AEV5C6S73O@I^K3k}0Jv58lKUA?384nwAW|LDwxE6Wp zce5X*hiSMxcE@Yk^3~>}qL1cu!jK8P zee|ymS?FX5m*Gv_q4;>myORrc&!PHN(uj>5<9ztSV=#6~9E5O_Z3Dux9L@wpS?sdK^uxTsl241u^0M1=9&YQ&N#^G@cr)x3W z)!X}X?D+-g*&_i#l;^H{egtPdx1VL+Fri*Yx8L2LwcnrgBOzTf|0%9Z%DUbtpzbwU zd)?qxY?kxzn8|wmvq_fq`t5p50y&vUzwU;6>}i{kh_4)!#954m0a$#)Cu!Vn4nVTz z!-q*Aqi-7~ilg2JB@I;*nwdi^9!b{%h&0{oL$KKRdhEhk~5b#y{q!?!5-MHRa1<9?_<)0ylQTI5dknbn@k7^@Yji`h3#lkSuy{i;Kq2A`Gp5*5aRJ?=D)}lOH!qbn^wMV z$(S|OC0nO(gOzP@?GY;c>Jz`aiy8BZh zF6Nw^F719d<^Rdwe~${$Y`3wnSZUtt7&nBWmYH<%Ihy^>y(ZxE6g**uD83njW`!*Z z8?@n}&-h2uOlZioUiU*y zL4K3#{Uv8~^oW{!nej8ZZ~|3WHz`s`BP22xg>o}{4+mHgn{aN(iN*%zK&;3eI=TgzdB#T?B)o$qIqFTx*dvH+nwVYtB+^<|V zLhrkUQ-R9?TNd)-GOtSB`J|cTf9r;TFO>W}|A1+PX@HfDf)rA|dnu3oKLtp@6TF%6 zsp#3yXnVaK9qwh8ats$!ReMFnj+!zeW^% zPN=`uStEI%X-9n--s>G<>}{Ye@mL?<8WB5m=6%*{;~otBv7>bgh4gW-q=EaBP*0Tg z=ay}suk(I|h8QAzPR4qflD1*)9x;e4ma1PW*;@|+6$rW+E}WPq-mw4Ox#n4Efgn-F zl|{g5-`|V1L&0_qxABle$*x=?>q~|0EVSpM-Q^pInSZ+xAnKwncQ$dlMj%!(dcms(}HxeW=|lz zOmWieK_-QDj)(Zm((I*gA6?q@V+-Lkc_W=)m`XxjG$r#iqvmynDjZnbux#3+OoLMQg1VACIL+2qlMj?tzuq%>5~$$Tb+M z`1ZOEC)pvnCpe$SE2)3gv#ba25I`G~{Ic-+&IWQ2PzFPNLc@ZOAjTuL0!hGI`?YTA z`r8o5Z6=oaYS+bs#98I`6W)+!L2I!m)dqG6*NN0KPEf;DVN$Tbea1@fCYL?&GAGe} zRAzO33g~;#G%`qugCG>ynxjD(+oT%JW@g;PT_o!6&@>pYCsE|)kov&WP9jRVO;)bc z*`UqOLgynIk{k~qDr&v01#wgG;gY7&R9QFF6K>xOp<=>7)?LMXDQf_yO!~V5s@OOO zE6@CA$Y!uh=5DcJT^DI9V4Yk#vY!;rOy9u*VS3K&&p%Coy6V&(B`{QY!ZcUSHyKP& zb}i&xAATS!&o;JmR9!05OE^#t`nh$r&OzMu66n&$N2-c5{F%feC)J^ z`6}j$WFm~rJ`xT=& zBEwKwMs{3-d2~4GA}t+o^cZ<*`3X+?n1hT`U<>hY0ph@*9GU|u$iD(D5|&ul6!MXR z4mTLoEwH7ZL+kmsxK9Tj6Gk6wiiQ_PZ+K;h4XRe#`fl8Fzx#!p)!!d}ttt7{G7bC- z@}=X)#Zz=xNRXID7>!U(s6PyAa2%3hD^qhn89WGF1ewgH2GJu}EBYW=`h&rcNMUJ& z2oYLn)V8f6RS_IfLXLY8ybgJDG~qll4lES3JcMv%4^eI?kQU7NepomZP@xR@Br=md zF<198T1lh}9E1ZQ;LSe${8o?EYk#vute~n}N6U?2j>9N#+$peL^>)w6|C>9mA8kR=? z1(_B$$}S#XhtdC0N(TvWvX_d`mW-rANFfe1H0Plfi=-&DT++L^@t2mii;{ukJV&5t zj=GzurCjIFWD5i5BaXb({b8ob?)5iow;esE@!r_-GZyz3#%9^T=0%0{-&a+ubE2d`4|9eiYYZ0-S6CI>7-UzGQI5C_tc<6CMA z8;ukDAN57?dy2>-17{!kiCQ^&FuS4{Fxp76l9QYfou+h&pr(L^)>yp$+VRz44KvB!~qp--+e;_V2{+ z_&|;w6o<6doZ5a*f(B|rbcIWUW~~rrd@fyJkP5Ed^6M2q#Bl&(J)+FtR3GR$(aH{3 z+OCj}zP|O`>)@Al=!38|W~H>l<*js#34&xmAIJmGkF9SG=I+nGO)iGtT{vI#X;0pG z;ww>w?uq%3lq|<$_-E<{6LbW*K=2S?wt7b~k5duynIj$7JD>#SVNr5e)f?oNRS=r6j`x?f zUjfH&`g3}nOT+Z^ij-@ZPZ+qXJDjuH(jhdLGNnq)kh5Xf9#S|9F)l?5!r78C8A0vY z`oAL^z}$z=hW&`2`GB=F4{h7XG)T>_@`ceqjZmWvk7Rs?P7nyLFIx$}$ac92&dbR`pf}*pmKM8_^EP?X~)&w+Ax+S%rgy4!=mn1SLz?Kgg zVlY_n{fs0HO+}D)birMnO$CC^_Cuc#fd~uX6;s2bhRIlAu5%~EDhx3Uk-^na6-0xA`}}cZYoVbU z31CGDgU6xz&~f!pDUs|Mc1D6l-b=hX^XMR1p&M(ZHJBxYTj6>g&5wStm+HQL6O!B2 zbuwQPq$`-IN(E{#&}(+zC|b(AcUL#eGPLun+P7T3- z|B(sm?l)LbmXpQ1Rlr^KtSpZ?DGn{vXfUiXFw|%u@qq^kkChooqa`M8Mg}RxF`Q;G z9lFd5Q^LMcrefrJN zVG`Ev+qe1(<@f6Klqpn|MuWei^7|gq&xLKt9in@~F32?n5&dhfe;hK)F9C)G>+mG>7I`@10Gyy}Q@e?!k{t;zsMb5{IwDpf|jp>$(s?UNbqvS_p zbz`2lM&E^FEax-G|->MNrNnHlrJwK&HW8NZ!6Ux4QeWNw%C{ zD7s(k;ZEs3-Qy~s1ufBBeE!bF6q@X3F35$_f@ICmOD1HECgWe|>~7tVQVs8SiRu(i zA~YZM`q1Vf>lUsMXs6#@Z*3CKbFmQXu1x%SucN+v2jRBTCwjdGSQjMA?$+rtw*o1G zdcHvGG@+N1y@|vO5ew9nsuY34UXKo1LWrOy(#%8D5a965OK3gCU?;s<;I69cNp_r9rVbbz31{B&FX^eU;D!Wv0-%GyO{5s59*PX`qTjjI{Y{jjP%^ zE)n$>71L>MYrtXiKHb9@MB0#44Ug;C9SsxGRndZZqn7i#qaupCeM|yJ+Z#gK_SMf0 z;$1Y3_MOc(ZfaB9|{jeaZ2)C?tI7B5)C>gl(<&hQ&5V|~`CyrG#4w)d$YHE~^ zR75@L2A98f*~ZHEy9r$H2coCPt;P{}Hw6qBqw_190qlo=+{4j}ZJjlwESsLUWn0~m z**Q3ZFBpC{Y%k}iyN($)8RQcG_*Ak+)W^=l*g;6^*S)`OVN`>J@rhkE+R2WzH}WxB zINkNCQyid_x$dqWKYEkBbYqCI}jk8A*WT)Sc$JD_nv{PXTPk%gUI)a<+zeNg$$^=nLvO+2%m^>pDSO8evM@+40=n0$b=FK&oXAj z{gPN(9A@DYoDU@iKD0mkG~o;>cuCN`)uwABB1;FgKchSgKAjF*>of3_T8h(T6J`1& zjNP-n?k}~_B9f+cCHq8h3wEDAArQS!x~=Ty)-&09n_e!|A@q0LJ_eCi<29Pwu{C=yf;aMK)Qg zHC#Y156qA5ES}#X5yFn44Zqs&Hm)W~#wr(@CoB-sVSjP22l<({KZKXk_aDJWF)i6| zU@W)cyjI{BC~oumkNq|p8#TRv+dPG$#E1ITqgudFQTjzyE^%4Ih2wL@pTMVrOz-s_ zvK0=ShwEE`>t+PsbE~V4Clb!f^Bg!IFIO8|TjoP<=kxOk%fSa~ZOaUWxg1rp6~|#F zH=mQYtu)ISL}MIiIsSgKd$sZ&l8_16vfb~F#;m*o45p{*S~6&5b9ZNI>_ z+=P*(AIe_7VL|hD{z zncX-h;VEiYvS$4m=57o{#F2r=`gFnwd>S{I<&*$aYG_{Ci;!hF#?&eDKKM(bV1BtPkv2CbzADU;8ihE+DgpXM*169yj~^*24Pnr?9#@5mzWWQnklz!iHKG`pU)yauJ{)xd1P*+BNTw&K^TnoWL%^t`e zSagzKLU@l!)nKU$gJatZ?r|RfGCR|xCDb*t3m9~ z9MQoG#tH__q^+QfVA?m}9xGI^v+~d)am;otIug&RbNmQDX|9mKyg=KZlWJK|G$&+S zh>nCwf$HU4zY#7(%W_ZkdDRkhf-hO@@D)IaM^rJ;i=-_NE(W{nR9rv=6R8D|;Kj!A zp?3Y@$>vMnV0K^+L}6*qv8E$C3mun*bfL^EA;@7ODz6`HXU6TJ4EE~j zTDutfaQ7TmGK7wakW7?7%=&1*q^DXGhlQRWL4&~D!3qWW43|sXK_&eqF#DPoJ}wKz z3(Z&@{%u(4PmM%)Jv?a?qn7o`0EmHOcr46Jtk+TJP9PHk>$0jnRBKaLcelF=@@J~Z zd@(smdwPTF07el>6dr-|smDf@(1bS{7_e~UB|*OxnjC{6xieFzP-o#0Yal#pja4dhv8^{$$@k$ zx_&e?f(!+FJ`pekq(Fdc)0b|C`}IK#vzz6%^N*l`Q}@{Eo&v##SOZOvK!nl~xRj5o zw{vYxY>Z^{%ItS2-%~FUBP9s7p*1eMgI}s+DFfBA~0B)e8(3caqPS6k2aBmjk z5_lRh+jq1mO?zNGXao>c%7~y1@t4FB)s4f0dW1#S zp{Zu|z95S1+ZRGf-C#6}csenMzAcW#;1QZ6Hwm7R+1sezh6K7&>0oRT+thTwGRYm3#9_40W1Kz$F9^~l2g?l|y ztF)$QKG3Yof!0N`N0Wmm=w!{QBJhs3A1j~#vNzv<UUggAfLI=RH0mch@UV1A_Bwu-CrkHr7`2=(Ah1@8YNoP^e&ag>U8V4A71) zf}N!?$5!dx@l#+agr+T7>%zdW*|K=jD(mJjw3r%S`Lw>`6Vg!i2z&l+v|Rl=G}PH5 zM1siR@gf_A_Vlg0l-H6UTrtkP0iI5=Z=Ja5&_4+VW;*|$2j}bJFTFH>!pKLFUs$)P zB5YIbv!S6njsokGs`OX7g_itLOASEpnye6Nfpahw)_GI+Y9fgu@unY1 z#!{Aj16qLVZtku1_G_xX;K+Bx&2!wF3%z|dXsUx?j|E64O;9(y0zVJVx}p1x*v(=0 zW;J7c7EHCE2?8u*tJrT3S z_~T*%*Kryx2$!F|z1|yhfnYT*BJ7|CSNap$C{1$+07(8fh3Vv#^M9aoBe;$ z=fL-*ukCU_hY)~}tn569gkgR8!=9hH^ycxJccQ)%&F?LBm|8W99=j^Ru=>4Sb!R%` zgN4V7c%!WZuhEY8>i*_cm&xmfo$G<_>Pr6-7aoqe!3l5I8KXU=2k)#W*WBHYf-;EH z>q4qN(oyKi>3vg%Kgbibdls^E8iu#_*%j}%b{7ryc}!@l6F;R@a}x*2C34R#>87l! zp(0BQH(4i*HIg);z!inpiO7o#sp|>l>iw4!+dx``!uh&AGznPMyBjR;?8vE?Ks0VJ zlJVcs_tpqZfl70Wb>^Ggen1gh&)w8<*~n-cI)F(f1@2(D=ld zTb)5llb|YC3Y?EB20Glz`cVvMAQfTmY^zM^`mIr<@LOTc_djS5IsI#)fF=mx>e;Vp zRM;fLX(I9DKSb5|PzEr+%{2Ukr6Ss00#d~wEKQJq%XsDgpuAUCpY+Ea(}Zs#MCc99ELsZ7@y z^gto|nF>BjS$s#I{H~;fP9Fv(kt+ply#N?w9X1Fn!HKYjcp8hKNa%QQ&)bf1?+vRi zmoTW$aTpDt1$f?z+$ywm)%{F{2a~RxyoN^NjF=lu@X~E@K9V{qY%f|G+Px;_T-1^* z1r3rVQAfF$Yws8{367Lx#qiukEC@)C3EU2*SP@6*0!9s9=rpYKZAZWd zRtzO@Iy-&4m2%Bosr39u?KZbps2VR~BxKlb37CHLv$b0Hy+a%#H(niPm5Sph%F@c3 zVm3%}M(TFxo8FyJ|1ZnXnl|)h*3J^WyQWm6{NA)|zaV1QCaS197jh|FZ$@mG_0Kgm z^JoI%Q>L7zD_(FOx#q$oDsrld>*(mp=f-{C$DFJ}lJcf^izg2yL!mL8VK6sy+=}Nt z!7G)}Z!{K=nAMlAr!WJ4bKoa36PfsN(~h z0!{;AL$s-EbbQBS3%oD6ibf49pX>AQbMO7z$miZ{jGNNWa#_pK`K8C+inS?IKe z`+>~MlBV^u?>UcxP+T1JUrCMui3EI(yLANat3)1mFSCK%Oz zQCL*T1zc>u0(%igbRRg9vL-)JdmNBF{&w-uo88|4_EX;bEZG|B9tV)rBgGLstZE#U zJwJ#)R70!T5Qc!_N8R&r9f{v3K8y0j(L5x2gaTAKENnO`SXG!2*m@Yg5hQQ`D;W3- z8$Lr!967UBt<=BX`L#8C_iOtc*9dWM9uB0hwI z?ii3I6=TX~#o#grW$~#!p9?37T;&qZdj8rTJ8Z*yWmgGIXlG7r6e;Xx73er?u5?PP zcqLlmN?sil9z5%-zL@}Fpuhu^a|D^~JONFC3=F;oq-PTv)YsPP_0P;-sH=@Gl@ma9 zN^7Ts{N(|X%;_{u0N5T>pI1t(As!Nr7KyrCS68bUWUpJd#efG*$;V^u4+<}^(D)uC zX2CBti)6!PztqXPcr&zk5(i%eP0H_Y{)#$Nd10ck^1jbCkopg`~%-j#KpdS;DJqmvb8jg0*5zK1#AJyrjffo#{rHj^vdA+Pb0CD8rDrqSst z#p_sye97>VNrzT|D^&NwK`DPQ4A|(OU+NqL z@&8M&hhlq(`lv|(D0pf75OH(A@$Xi_I-Ne;jZ#??(`Z237AiC)^jFxuPugMs?xxIK z4!WeI_-812u#QA9s3*TKG_)@%HcK%k*hB&&@PGF2_%kv%BqJ$J990C|xX45Dg$Vyd z=6PnAxjD%3zok6eL)XJ$oT*NS4H%+jl%iljML>5BHSxO8okC`s{@*X|*!H~`xuiQV zS~1Ut_M0`%P=H_41byM`L3* zKAXMQ04{s()w^Uj#ZkZJjH{`-1jyb1Ew77e^4uUl&0=VC=HB&Cqi~DBiN|O@&zPxD z7V>58oZsRM5(NZ8%f8><$#+0v_H@Sy3YUHx@}JXi{N|bnncx^vqTRmsq?S1(?%2%G zb_gAPuXB`DT3U)XK_@t)39UgZlbo0p@|P-j_77 zeFiJfyWLH*!K6a+A(+PG02mKn%K{Na**@lr%hT6%_;-!gV`-dW&-Ei2^jVzf`8JZQ zs!KH%>19V~VPbPG@>7q0PEz9_W}Lw;(j-DDG|5zTHb*f)I;Y{{4S!m)O5>jZ;Vyw{aPd!0kQ7E`HtX4s_fSyK zJu+eT6JGAe8WuM4E9(@`r9bNId0I@aVgRzn^z>(|W)f{>Rn;UD2Es)TF;OtYALYoB zh@PJQ@%n7#eo`Djcn7bOc=&(6eR^W?l%E0!%$W%p^VC9kAh{_Hly2L4WB_hfy?||{ zuDDuxGX^rYKFfsBD}T$tF9!BkA%Z~8zwAByAfv=kRaTh7-t*(>k4Yltv8SRRVz)QY z51VqBED~5*u;f25Ya-|8E-E{^YSZ9wp6^JU{yZ9j(^!qm?|yy`)EkG!;wNNfdBrd# z3?pHnlkutP=`B>N*Z=q-4k|1zE}|`mhK*fPo#)g~ve@SQ&vdZOFptwT-jKkVt2E~# z-g3L@1wc6GO0|J=5J(=T9=`N&HSiJ4S~?NTn(!(>4MGG#0@CnaZ=RZ-aYi_QjyMGExp;1wkr<`d9!|mO#y8PPiD^~&JLVY|$bgIUHwZ^fx z>F|(GRRwNOP5S4X=bSecZ)H$GAE!P#T~+0%v?=x*(QvbH99IE%;Q0}~sJPDcI+T}e z>?Hh6+DYC9O1E$*NJ)LE!uG`UmqV%i*+Q$`SSHM@P${kPQvO6Ff6W;$9*GwiMDi*S z3&I6;#EbO{rm~keG)#Ls%#}T7`PF6TwhS$n8K#h+az5s<#1J!|&c|K> z^#i&xi{bg6c>w~?KQsKxT(c<19>84S>A7!wcUx6f<;09=dA?aZP6<`oZR0{SGLYhnB0GlB%jmplxnAp$DIc-*!8kqUuLjQ6@toliNO^Ntj#Xne1nq~PC z;Ndy#k5A~eHoe(Q8sP`bADCF(=JI0qbcqL=S&X8r?S{^LKYsQubxO z_Df1hefI;I+>W*eg`pJwDlIQAQIM6@Y&JDEHoi)bV2C%gk$+wu!&`M+x|WJfvRLlE?k^0{tNEBXKctkX>YmNI| zcqbq!CKa2g!*1g3D`G1nbAPCMB;UP1Rj=Ou(%P0$FRr{D+Lv!n>o{yPcedtN*i`H~ z(X}&WCRg;Ubgj`iHvb?9{`P39e$Rt+G`_nxLI!cGFBaAxXx{}>c$c=pAtychykso@ zM#SnrUadWwube!xOVLtQZFRlwv&vV{s4@@fJ{afp8LEkh!e##FEET`AIshxaPNU<> zM0YP~xmQ;Y+r>pCAg8z+@EnGTxm9nsf`bAN)2XDQGR#yzI^TY#M2k<)M0YmxtFSW1 z0k|s@?4I%c6Ln=NBg76OiUO7Scbz* zf0p0Ypj2V+zxI`WDhQV{DKW9In9p*WRx^orFHbW1;kbJ2gRJb;)s^BBI;T7!kFjds zjVdD}Gy5U1(?C+PPYnG%EJ#^d85x&R347-_Jtih5^PR~DJw4J4sir|qcuFMJYca2S zGG1qc&f-#nkd+x>_C7qp+r9RiE3TJ|GqPZLpOfMpHi@ma$8ncx{rmpTwAs0r^RJ~Z z+k2UhB*dEM{oHMd-$}*hdhhG-z`&}-DNl+f#e)Y-cyx5k=^j@uToL`he-Ty`iC|_GfKdSx#l;!SKDTWHc!tKoyI6#Fd<^R-#+&@Vg-%pkd$K!>BQy z3XEy0sK^4nFlD-rfE>v14xi_b`5jkOB0=-jWr?QzHSeoG4-cWL470qtF@1gcJw3q@ zU;C_=D}eorBmwGQR+fiqWg7e@i@~Jtn|v?s1C?k$D(giINXC-Z87&YNbhBCCWdpY$ zq&vPhO8v8m$ujGemaQX${6WSD45+vm7-*c_^))p$NlA$?!k(t|l&AA;kLv;5+`n5! zw=RA^8Y$P?!m+VUv_9U*efVHrXm>odOy@3efOG9c*D^=x8h%K;VWJlk=6cn+Xbd5c2&&fvakCSLmUDIPMl zX46fR3^nymf6TXmgyqRezR&wW$M63}-r6-T5CRc`m!4a)yQ!!fn zho@>5fcVOGkmdle>1+$_0I`zNvLYfj$8g*IqdA+y)m8xOPRLQH@0{oc90URF&}}5~ zBF7$olpo#dZlE(-UY_OkY9s_XX=ld-Pz~aB+VyD7B(iI|InOcHu0Q2K$t$ZP1yN$u z*41S?>1lnWV6)gR%t6Luw6U~qvhCw6Ep2;#L_1lnwF5etxR-gDn3-Kq_mE2s*!Nl* z8UV5`2X{TNRH72_br=FLg~l?sEK=97R=f60xrE){nf#yUJ7m80F={mc6LZs>Z(nru zW^r<8hkz$9tUDfV<0*~8>gzNcojKX-W+4C=1<{waOTZL!SI_1W3+o1K$7ma!D)#ALs;OJSpD zOGtVwhg04885U*LxbnEUrjJsAZ&uhz#`Y7rsBguikla{9OJ}xO@fPUvfP#7a{p@WB z$m{8NFZ%~P6Qg;Fi0H}4G_IIBHD=)Ks3Iad=r-ekI{X8CIOLC33i_+4e|`K2FzAWw zU$oV+clP5qlXjErY2Udl0(c~Vx(bhWNQARfT2`Fu@I0L|zwi6(b+TN^!vB_?S+LBm zT@MtOmZqbpH%Ntu!N5>VMPswlqB6x%-+yv-pP^uOORBrIwUuj}i`?wG5eTfEAx?t8 zQU!1mXSq3U&epV*M+-7zI5Ee=qhceXVpm!mw8E{g1a$5 z_VXtIA5>6KGU3PFU4w4(4Im-O5J`+EWm#+mJZI_gsj~(8<*LJTltxBIPi?IT_{Ir9 zvHA8z6Ny6D@D2xx%3L(CD?%D>RJ1*1)Yej^xrqcN+8U?+RRG!e@E|GvB`T_RAu%Pz zJgl2p=A)=l29*pjlP}euypy-BU-B086BC;vaT^6RB;EJpo1qMjlD-3rmZl~F&@5Y9 zd!276DdF|EuTxjI*=lcfd6`jn?&$E4h%lW)m4%s^fsXk~(gQkx+_;2Q=q)?@dlr_0 z+8Ln#2wNg1BHFc4S1c55xxrA@TXY#%4mJen4K>^4Wo147Ogw)0Fbq&Z&u_HJMjL!B zSs}+lrRq95)?=9xK!=%b)Asxc*U^JAHXMeHtI7I$8=#w|`i&D8BdJ(iF9BN_qauqzp#4Xopt`US z0s@|l50S$_?8r@uX;}B!POE7DIlV9F;zdP)=ACJNL*jw<*_ByVz=41I; zxhbFM3mZfLvMB(0$Lr>iyQ08r*YV@^_RkoZ-*ygB89|=EFP%z}M_!9sX=+*;pNx@h zJp(>?_S2kJd5^dhB8A*nDG8~2Zzu|Js$5%*c3$krKgXKROC;P#6svx2TzgLfx59JR zdaaSRttX)fq)iNfn>EGpnGsVt1fo!oS5-_1qGzzmU=HgxyuE*lj+W22;Nq&)=sp5< zY+cV=5A_Th_bKAPw^v%*->048aM4v)RRxz>rWIQbC{6v@PW+K>mpW0a{o-}ztD`;# z+S67p(jR;tpCqfnIE@%!S1-#`vLS1fcVsmR_J8bnESx z$e5cpj1tQcxftL>Os~&Ly?w!DftJ0{h~AHqM2}XNGSOFXE>)D{`zQ6 zg&PI1P*to|Rck>NHb-TVm-Oi?rug}_LN*c?BKD)4s@01h=A0>lhj$XFxV(F^|1LU2 zT(rE@GTbOrtRawm3oSbi)G3Ze2qibmQ60Z2kY#PZ?+c5U-p)8+pS5eh^X{B2WzFp?&Bl)zK`SMGE^VG}B za;VqfyRvIf1@!f&CZzoV)DbP40J+JzTJ25%E&xE%-dIw#XUpy(o{iS(VqutQa>?7O zcC(>$TXq8G2eWQVRa0?YS=~!tdf60*u95Y96=yW6{KUj+#&QYZc2IT)2DSiw?SQ=m zsC{o5*4t;|d>6XNU)jt*YBFWdXRSqxr}562}T`4)$dM?_>? zzr?9J$6>Rc?OQGQqfcdZ6yo-nHDl;anTVwAbYUJ(fdl~=}1{9t-%j4;w5WHK@ z)6A-^sj(~`#dn%N0eH#j+$Z*p#GvdWNd<+QMO7l8;WKlN+U&7T_m{ezxus>qrRV6L zqg2mse$r*tkk?r=Cmk;$2pAUi)(e7wuhlMezVk$_lCKc17@p4VbvkYJHvb1rxo&l5 zu}3-W&^}u?(1+|}u$Hsf^sr>tKEWYB8lpP8GUUOw6qWBb{Vg{54X~EBdg3spv`?Qn z!gc2ab_ppY5Wb~_Q%;}wRCa-=sr~HC3n@?nS?Ci~6ss*c2rAC}fj%VLj#Sp0ig#97D zbW{*gL|8d_Xjm{^u(t&wB6Zw2Km68e^EM5)W);&y)x*rn_tZ5qC%&v58X}lF_f;0X zUFW9sr6vCFu0o!>slLzCG7w{HAXK`rumC6qA!NQ3UfG6qv)Qi;f3R?My>Xe-t+$#h z1NzTqZi{PbobNApq`b|$S>aI$6B809(+#%AU1(h!$_&*VP20wP_QEf=8TRhJH#M>F zqp;9Quu@l%wS9dF4WG7IOHI&Scd{sLt}9!tc;D`P?%z@)ymb0rx5ZBcjW?}zc5yK& zF3teIiBETVqJo{8lb4gzWUef~e(i2O*Dee19KrFb>bpWg`>>r;W?~}k&*e6sFlJQ>VKb8I0v~SZGp| z@nHYurDd&g7X7)}(NSj5`8_7j-cmicxg5iOG6Kl!wvLA80pHlB)JA470yI858=saY z2VQ9NLD*}_CRd-T(X%yFPu52|kd-@sg!!VJJUp{Hn?uypzl9dob&bl8<%0;!@*sny zmK;MfHMORxgXJ0v+1#-`XFk-h4M$#W?WLuq#cHRjt{i*E4VwUa`RNRkB&ye zt7>+E>LZHQ=JSKRfYvt%=IhqoL5rEBZ9*FzNxjz_`%hM2NrK^$UfHQpisQcCI%#Yyo0aK%fDTVFr=oqm*e> zQoen=7lbYdFoWy6H^8va-C^kMeUz7b&20mcz86fq6AC)FYc1XHpWoO% zwGj=JJG)LJHiw}vr>xo?8rsr1oGu`qSCQfy^w!kXg@|rsWRz&E=(FM{i1HDKPVMxN zg#X8bI`pE9mnMgh2UAf~%g_N29U_HZId@%xK`bbWEuV%+D@lT`^`veI6B83y%N{0W zS&A@deWlvIIkxf$u4Cs&G1>I-~FqmSd zTgpzRC{gqSNVG7G3(>xYP0c!Ls~H$_up*9~_^4 zvL&rw-|QyQzrUl5^}4%2 zf87ybA=~X@vo^GoM zn4Jx!6C`{oos1Q%rNB+zQGUmX{W>#4v}Ug zu3;$6-`RyXazIeYG4?NqwugxESl?mBSN~G{NW(5Lcq7NigRhiIh_SQmp0jI^8|DA` zk=|$79{ug~Ou1f)`?%aJJ_4`z-yKfWNJl8k>JDb}?KJq<+qxTm9xiY919G|LWGhN{39i5!6G$lvkZ<%hu z(D-Zr%s4WU&tb$-w@LJ&RZKYsE41t*?a!A)86x0u>kQg}ULH$pl8*>J6({re9HN__ z-0<4O1kUf|FuG7}1>Qn#=ATySH{~bXCjR?G#+btFdIS59>$!L;8ADg|D>lkr1XR%f zZ`C&SJ+4qycr@JLI@&wGt9o#C0UbJO==0aAojm2G|MQP-_nhWC8e+;YTnJ{1uOa1> z!8ucc+@yJb=L)_{(WVj26$F2M?ElqNbDV<;)msQmKj|st$;0-vZ!Q_Piq++~|JjZ# z4$AS}n0#$FdgSYG(FHHA=HE!jzm3GK6fmToFiJ&CI6-7 z@2AaW(ipfjiVg~OKn47^y!XyH)Jri&Pf)!=@|bk7+hZ|Cvc!QTd48ycy9+lL@ZRdDxvfFHxX}N;+ue z?rku}gyTJdcCUSo@d5*xBw3naluRa#9N=coG>uTjP*bYUuiOCOZ_`)zv?ByiXqdh54uaNnqKO>yUn5UOw)sw4TrVGmq;_g z<5%lIwoaNtytckRIU-Xd_iLUJ9%9F%n)vJ``0oyFi2eCRs$4U|^x@EjiiNQ&N7oi& z&zBMt`pQLf(@A($8qw|qNPfKk{LY20oBNixZ`YsPJPvb;2!46{`QDV=s;!Y6y$aq> z-y`$~jMCr2EFqwSj1)YN%w z+G)Bg-wlrU1_9v~GHD7i$EKbn;Mha+Q(+tZ1ETgb(1pCDq$DSY_n_f#xsJr#UkAJ^ z21cWBg29%DD?i3AgpW|UhX3j5{8&4#^WtV;( zsdTkBHJvG~IkXxrkvRBSpOuwGCul}VL*qSreIhZkq_nhVXDZmr$|^11DK=L8yHiDe z5*ga_7xo3JcW9{0qCbP)x~y*5N#a+_MFhX&LSLxFktViiy-VVFDwyw`fJxI_Pm;Te zZ~D#4z`Lzftlmt+KAZV#&sjF85@v2WkR;wXphT-R_3f*xqLHH|s$5&=?HwM5+JE-* z^Lv!@DK$TTt$0u${FZg+>UGA=Y=EeZ3m@fHsbfCD2Gz`NIQh6ClhB#(-?_SlNWv0aW!feM!mI3*oOrg%3iTr-z+YF#V|VoWT0Yp-u9@={Bz1O7NPPJU4ovgDH$g&vhnh z(7?N@eFnBWG^gb<0UhoxENJ=Eb1vTMohjflSBCOUU^p3d(WA>=>m9;m^|h{&$j!+v zaOXQ48tz2B52Y98Wn)Xp$+12<*i}hWVDl4+?~II$92gk*db$j@C;K~>sW%VaDKSa! z;QY(28i=a}I)2llXGyNAI)ssUA0KWfxJ(hFWo%-bWazTZ71h*x)^8Bo8v2-Osf#eY z6LVbfK3Y#`=c_r$saR2V#tO+FRzSQ28@{r#a>&*%4Cq&cEi|K!;bI^^K7<)sw(-`{ z)05{=D|8o5z$D;rptaiFcNwsKAb+h_LG8!6Yi^3u#11hKFYyCs4+n2 zIzLp1K&LA+YHp&Yrg>U)CQ`+^a;+&ck}{?}Wf9tp)Wt9*R%aw9C#R>U7Z#pHs0KJ5 z?an4VZsV|c)(bPC6&;n3FtLv_ZoNGNKKg*xgY?GxcBe1?8Ckc+imeKsTAE|i5Ui5~ zpr3E8%hMARZ%N$_WU@bAONuGlvGYgm?(Wj)+ScKnz7h8Nwo?7Mo`#l zB>8!v*TFPX^Wb$>R#p_+y57L|ubAy|=L93#kbLME`*D<^If~{CnjLv{^ix@x2oKLPv=Qmr95gtMU}R(j zyPuwsfsv8Pv)YGX#78c^|nZbZx`_kSM9-_5ba^>v)BLr zmM@Y8qiMn^_^vDm~4D?zxuH)R)0jJkkMhpB)D zsk{|SS8@j{;D|hZ`gCt^4<=s=TXbRpCv7G9Qc)Q_Kz?n0^F3{5W_GkUSo6~J7sHP& z*jF&h5CIN09OL2e9EHW=W_#=XSM`$-RgV}#|M~+H8?UZQA;IpOzFA&o1VlG4kN+t7 zp59mJ@e4QeVV}x<9K>v#KnYkpnD*@x!xp8XPsfC6TTbgLYKTZ$P)d(*=OVh|;z zo_x=9l+@Hsgw5h1e`k-KTdfHnaWi4{zV{yX1mHh0)^3a0`TYx_B2S$@s&ss{@dLTf zPyH$eocq_$?EMJxUjzvpeC02OzIJA66PWm0zRIs#YJR+yko@_I=&wJjV9QyhUGPuq@!`br=IVsR z$m`c(&!0bsKH-~3Usgo$HC@P`p~N_c*2(|=KsB-I#Z9o>H{t#bz41%N&Tj2V>Fxq9 z_g_l|v$`Sg_!A$43=a#-d(icch{?ltZ(|hk+TfFJ#u$yJiR!nnBGiD^SjI(yyV;|7Nx+-x*i33R@Q4jaHuo^w=QcwgZ6xl z$sHLTttc<2HF<6H_u1V7Dm@g#r_~TP!pp^bb+{bZQcC@h_7nL}BaESqY^*mD8!^hhU#G4?w$I+(l&7^cnQrca{`N^#6jNW<5(178BJDPot3ea$ZyJ z8eE@?_kJZs$`D81<@)5H7kWBxi%k_D;cocfDk2m0mDh>-O)=?__pp*`O2k*)6)_yQ zFg-?^1eFHO70QFa_kIGFejY&g7}* z%CYC=P#6wk)SRggn;&=!0Y(mvj^H%NW6R6S-G+I%Za&_hA}i^8Xkd|)qhCU|PeOR> zzx^nd$WT;-nbA_lY#y!4U5uQ1O>jh<5|oo8H51|KfuTqM*4`%vyM4HJO*Q59iu$$x9(@OdG;s%1rr z=v4#-or!_RPl!~oMaFvh9|rMc!laRTms3*BZ^HF3u{Th<>cCbPS*RQLh^T9WCk6ZXlHe!ZfQlQG%3+QH&A6U26Du=IKd|F zcI8B>lt}xWMKKm|(QJUz34G#^)KU zX>fYkiG=XI)5k%uH1LN&JTf2@qE7C}ZWee7LU!v?cW}-V73W}EkrDcG=A08Sz5`bM zV0T^mB?BF-Z+TS4K}LSQ61E5e3Oq>#h0b^4fGW6euRyKqbn{qn&Mr#7{2s>#e}v5u zn@YwYk_WLF#FbzqQ&LRF%IsoOd9;gy?BBh6chhd_PCY;FlUnu!4~WV{9Ok-Vwy%!g zAKV0AFlSk!G}4m|JZ=xC{6YN-q15L|&TA<*=rE8#xJIN(rjZhH_rIO@U7 zZ}+Ci@wCI1d&_V1bVO_Iz7Y(lxVRX>5ANjD)D$=z@DP7=A_$IP1cUwkR^#Qx+1VY- zPJ_R{_%!njfn;A^UWO|h3MK=B{1#;lPV>dPA(uHh3#{C~R=Uc|%fmoH2lTD>2{7>R zH{k3zebR+kd2QE2WDWJ7MPe7BGPJWE-Llh}3$-`WDoLH+9}r-DA6v#WDWIANHas@9 z@seXa@D*aWM%!v4Sao1np!SM|UAgt#H7O<_Y4Yu}=j<+=A zfofKfspil{7y>g`EN&j2cv1UQt5Fz(!o7Qy-=3aAvb4wWIg7<-s%Q5KorEBnnVAs9 z3bOpOGWN-rkW37Y1(biac=U*@UU4`4Tp?g;tr5Tl-aFCpJbJ{1_>v$YAuPQ8^b{|E z=oLPL7L{v#q72rM+EFR+fYB2Lk`q3hgGoT)i2E`dnbyUXVz`nCm8( zg&Gr*zsi%}Oa^do`mV2(loW5wV!ws~v`K|#+Tayt1Uzhum z5dpf2s$Xqfa1@u9mX_w`e(oQFVuQu6Z(!ikrAseq1wN#wn?r=1YJT^^8H_nR79@aj za=}KlfT3>fdO&P)OF=>5GUdWxuEDKax6IAcyJi>X=7RdYl`w&1^wF^GVV^^gv^-jR z`kc^ZVPPn(n_5*R&OmP8`6?m;lc41MpQcXxE(;QOA2viK*yARb>rgW z;}Z~2a!eEJyCdBMdn2=cl2-IFYl_26Ta;W3Z#y6;%9I-J9cK++_ zfN2#ef0a5tKq8B2brq_59K@HZ`Qg&$mt9x3q0hern?DUG2!75{gBl|;2<)qiy7|e0soobX-N}f?(SaE z_F7~%ZXH5@&|X>ebG8@U+?z{^zwVszfj{b>bn>};X;!~1?c$8=Y%9Rlz>q<3`{Kn5 zU!v2zJUmu5HX1mJ+%^R48bK8 zm-^Cpg+?*$C=hI~1x)s~S8Lq2t=kKtxwS6%4S|_yi{c`yAA)~{o#dd`EcU;EAs%T= z*LW*!-}8ylHa&l|e%%?aJcN((IV3*VbSAGI*xOmr1Po#hb8zqE)EwfZfh!9OVf(y6d|Pt?B|3&99^`^>dx8BrOF;o~0(^m9Y8}ARa@`!bYdB0XnCC{c z(WavDo#`rRoo@uwQd2Koym;~Y_0iSsw6wI$ls!0&r1e1rkIQq|FJ0<4eV};D?o)SD7$Xau&qN1XLxSoqk?nUc~ z+=UW-qHc^$m5&qYnY6tJ-wAI8CdW}M9+bydw*vOY$*F{-=dWJ~BYJk0oe=j)woWNI+`=#i zfSi4A35U%9>NJ2eqO<-x+{k!jawMAJMr{N_8JDe=O%WGga$FI;(SAE5%_8@MNyMDE zPlGW4m@cnfE%IGmEzAhb%38gl7y4H8Gwc>Hq}%M#C|1Kou5+8BYS07 zryhS&O^L`^glJ|HMyApstGcOW@sFdkTEiWr5>7NQTi28^G3}f1FiI*aH8nLjVNIH> z6iRE+rdoipZFO5vB&3=H^Inp($oVMPiqwSrn!AcTKPjQ_iKA;Bo7%Sec`^?>|2+HIlsjYE_$&Pzj++xu|&KyY>vnV;gTVw~}WF;0~K-?;y>stkm?@fhGzvl`a>{9z5UmY56{Qv{O?FfBkfahF9D{ zk0tcCVXOj2ik~teKpVyS0;| zHRGKw=SSs^!alM-ymXxYt81qd&kK2(xRGZ5AZ{vGO#kr@C-Y){F#;ur=x1J#QXBl} z?t0^0id+m>W%Vd)Aji|~R2$!%lP3Fs&K#mP#``~K&yc)&B`!>o*mBNiUn3|QNh-^- zO~zh<{<*LA6hZnkJmK^OUc&t!L@N@;+XRRq=Tpl=*AyEe&V+=xHSikwOI2ItJU%;i zKO(QJn_T@mjb^2w6Sme3HJh!fTurn-TVs{Kc-nivULq{^mCITqjkA-}Lix($PsTd# zuU|CGk^HwE*|TmJ*IE#__soC)a^vwNX;0UwvxO~LcU_D~Tr0J&@%l@=WK;fpOEAD=L@uD6k^n z3<5==nR_R6W1#>`6h-KU+SLPod>u5*J1i7fIZGIMeg_{Qt#Jc7ji5v?z7bEs0XD z{7HoU{eun{>i;id6W{&c2cOIsyU_n7GP{Y%-j*du!g015En`B7=>uBPsP-2h`FOe( zv3SR5)JcS)>CmU!t^ruc7s}jMPl`{&8z#P@C)8vo;a~%d$5!K{q1&IoZ>;}}|KA^2 z9MJ!NZ(~5f;|{Y}N@34D|-Z&xab(a+ZFG6pL4gX`gWryDERZubp)EXx`Uq-W3H>k6@Fz;7!Q+<%5VQt!ZTWe_}}Y2 z6$_z3A55P19x5`&N;7#n%*H#P3jo%cK&$Er&VZ7Nl9H>GxwZ9HfoZ0b@YMw8Bf0=1 zm#VL;9@`)Kiu%MW-X?hNO*kL?hzSwBW-}HW|MsfKohVGM<>-J=_BLOh5iPK-`h{ll z$47^=T?rGty;tm}AZIKV%oap_P7V&-5S4+Afe0bh+&BQ>uzHBh%wwVD8(JQ#nh@sa z6N!3uhxE3_Gc{%usJ6jnn;mPqFZE~HtDW*C&=X$!%fZwBfjhftwqkZdKIVlY} z>fF~?A62KC&&;qrNU+!DGdRzYVjc^wh=rY<9mHXQOkN@o+Oph%czyUm0^_qgPD?{; zbwo_yknVWg>=do$g^8Cw0dc$z7Y$CNDj?yZBmuW`uDsf`of=^nqNW;YGphbT3s#br z->~+3tBTkDDAD!n@|$bHe|Gj>`Ns_mX7^RXL2^;i#*ep9^n2Vnw`__8E3{g^m;-|x zoOjr|QV)l;Imn*Zk6#{JV$}}~kG29f-_n_+#ffnU^f(VY^Erqxq?*r{5A@F*fAgh_ zF|cbTlQ^C%FQ$*-(IZCWxAFy|k)RubXak}qV8ZmOJzTZ4-hiL1HEBDvy1l(P$R{pd zm6M~;7`;;ta3Cw%$6mm0%J1OdKt(~caoOqq+iTe$Kh8lsN6&94se)YrEM*HiE>9RJ zDLr8d_;XId_nYqr%J~k5m8E6e4cA}W)!X9j1v${02w)zFY74eI^7w1TeiU+`In?M9 z%O#O^VvLc>xr6nf6}ZRjC&L0>CFl$pNV?(+f7awyF_IFN+Dzaes$8ESSyIix7mDK2 z80hb3yL723M?+-xd}wn4w}%u8?YXl)P_P6(H4Ywzo}M1!0_RohE2I~}b(WB*0naHp znN?goUVs{jFD5nuIl0C>`3lByOe+$k^S>9P`x(zQ`=3Aa+j z4>5X9TK4wq@GKow)dt$&s?(rew6X#iMbpj)PJkck!8=zo6%`YGeJ|q=xgS4@K$6tF zLrPlOgMa?H#LBAcI->^iHG>*2J>Vet-}e@qDG{o4JvDBZ* z;})5-kEEDduh?;o9HJ!sZB0F;0&9D;o=EKVgaUSO9$fEII6|dc!^j#p=vUOk`?Irde z{?K}m5E~vo@OF*PM9}QZWe8w-cxK|7)#Ftsj6*sAI)I}KPE*-z0RB}2YTe0!{( zZ?f&v$Bc8Nh-lNdsl)sI_g=o;8M^HBMxfgb{PPOU@A> zHCZSkh8Tg>rN(o()FNY0C;F;K0}670Xq&DM>L0bTC)kXYnGWUEL6`L88;=R1-Kh4r zS0y0wgX9hf$AEAIkX>&O+|{)K9UB7QTTMd%`6MltTTQ#^fWhEKA!RsBM7<7by7drT zDc^Ok+@;(9)yqXJDH-&T6Mx)PUis;$JZ-R<+IT{9?~lh_=M5=YQSyQl-UdQf&(tHr z5D2u75m5HQ9dGL<5gfO0QMe@?nS*y!c!DxVKL}~>>8RdOR`#(+P|z5iLB*MwKEkFx zNyRC>g)K5qL*>#~zAYPP?V&lok!*TPBrBqL=-ZCM?cayjU?WHiG*e7&bBQN>6ELI` z5Zi$a;4z#BJVc6Wzy2YJU7j<3-M$uBm;Qhq*hqzq7OoLJ#Mn#gU*K$gGN8iXaIkFd2UWo(3Tk0{?frwJ6HUj{b&LMdCU?X<0}+G zk%X7#lt2FLuN{xe7&@|Kis|cC~qo6%E_epfO z<2MH6nHMimq)#Jz#$NQEx4B8-e#zRITj7HBX?_wvNfGXgXB=@X>5)Eu^#|wKLwd(< zP`&6G)4gWKjZ_%3_8j48nDzbskNS}JguNO;W1Mv1C(Wq8?!h>n|DFaWy#U#M1*F-K z$z^2Z9d$Q7Zwl@o0F;JJWAv^)=!|1J-f2GBd8y_ZTbLA$JZ~E|$8C@r1Oxn-r-`077_V=(U6vz zttY95*Yp9+!!P=UceC{5JctJlc0ZI8>hdDKwZ}YrNsh}b8l$JTf3uYXr>&OXM<5~$ zk)GZ+dr?-l_VrM{FDrZD6TD~b^81^3fxa%{taZ{6uWn%_NunN}vB7;vieP4lPE>SQ zV8UUh6cgJ?)y{W4RW{x*gUeysuHWjvv=HI$6g=+!{j1sVSHd%|Y`$7{yvx4EaWI!u zOWV=)AS$Z%lh+TeqW6;Y5;v>2ql-k5IFq(ZG!%-uJK(c{5oW|mQN?H}DXqax2*Lfy z#73J6G^x7vcX$76ID4Prwf-g{p{79)N!pY_+HJi#yY?hE=wLj1;~Lt7I-mz6XdKZ5 zqM{>#Ocufux`HNmdp;_0EH9_#Y;!*vUo}zjjLul#Y?8#K6|HlW=A49gxCXFFPW#s-zD{>{w^(D^ocYu z=@lJEBmW<=l##O`OqrU7X2NCqMd`THC@3u8?!`e^{P-r2rs0D#AiF_cdm_IKJPN~S zsh#}RB_4C|{S_W4Fec~Iil5+}6%*r@AKl(HrwuTT?YY8*hAn`a+C=yiS)$X2^6S6m z;W%`-9g@)z2L}*VyItkK1Z@Sekv&f@<5~-^v4!;s#fINfR(|M2-rAKkEGl&UZE^~Q z4Q@vZBIXpS??by_1STv_nn@3dU=|f+_FGyK`-L49mf^c}lo`@$8+K1PF*1sBb^oQQ zuhnh-mPe0F<${MkVfCxO939x_m%bV7^RXJu)WAdZI7YmdQC2Qmn|?N!nczyz!~M^U zY0p(M|HwqtM{2$N<3G;Rf฼v( zUOkL;ZDu6)C89Z+lfKU6QY*IE7uiW3jGGoYq${*C6t1Vsa4M?domtSGFLZ*;6!~90 zH3DuIe0pOuj0-tn5%vgXaw5A=GHzn>3aN+t*h&v=W%T}bnwAzTS8!@_SVO}hNs?b= zoUc{puJe^^>;loh8jed|Qh9sQ;S;^`wNQJ1q!KEHEE^wrEnrc|Xe*vO&U$@B$|`$E zpAXNg+U{a*wtuHWzNdR~2{q!v{hP9Tn2twz0q6aeS9KovHcbql=K7#>{sH=rzq6ZqSE2n_}R-`^V`Qc z6WS__a6wRHNlL$%wdnr$~aDZ;$27`O1cskd^S@Jvq$yV9cSmRFlhvea1+~;1X2p zdSHhAjQ(KGZ=t&JM(W;hKii?6M=M3XeT=yC%OU-)N5a~Ujssa*2uW=3P`A-Y&>10# zQ2DrqU%zB5Fb)ZR=$>moE#z-!effQt@1+`t5VHVocp|-;v-?3K0~rS=aEGTKikU53 z*?EqU#Z&UrqPKkp?+wTPPI24jvUAez73+;Lm(Ju}vli0PNtxjDXO1QG3;X_0M406I zx|S9l4~G#|R8jJ4m$MI@wyJwMd6P|XW(x|r%6W~r<3;Hny>z=HH<_ocaF{WWovZ{{r&gYJBaDVMOFMS{zvD097V36R=_trI_epIJ`RK(mk zA0^z%D=}J_IfDhwoVbnK571;Bk!VI<9s9_>m&X?T$;f9^J~-IinwN*J_HIZwAsMUd zu8^px7`p;bZJI@-aVUkHM|JYNm%(-@rt6UE!nV7ss%F+G?4xlV<<0*)dao8> zH?@u1b2IUA{Ln4bG4pu1!t$uh?jGeCo(|xiN4q?l0iv!ObA6#C*IY@sI0)tO5a!y3qlQ{#uYSLNw^AKq5r4{7gn^vsM24}zF+jqI zh*O}_ZtdjFe+2uJ+@KN4mN-8TaC2WX z)~{;Puhi@t91W&{U($$&w{W;P`z|RMM@5a4t+Dd(XfHMSlM*Ifwa(W#=~D*>*thNk za_N4CX`3UxrN^!xTCN}@q**vS)E+Y|7xRpf?D4ksHwFHF1O)A?rF<{u{&}Z z6prW@1r7H7bX;80D$Ld@e1eL7+up`Cl$QvvmsJ@;cA4M>4%`#`@_Eaiz9#Sf$lIrW zOFicBlgyA9VuiG}?1rz@`-NRQdz)Khq8{zkN*YArNd8o_qVO;GfRKub)Fe-{8%}wd z!6Nw#pBbIwJzb2=R5*+*DhR?SyBo^LZuz9OWsOn}A*CxCm5ROPJJik6q^2S3t=MP% z{p!`NsYZ`}PW=jj5yV%18O-x*7L;Y=LrMg+2M8Dbiqa|7PVl@`VnbA%!I<)Gt^3P8vG=OXn_a+ux@ESRhh$;^qgS~ec! zU0ikSc)@T>j&KfKj#I&!u38%e%RPvjH?RB&2s{~o%d4u%Gg1S> z9$GgNOcb@g+kLY+83?0H9LYb68++H|=3$XaasKqs$DXXpxE%_F9Qp1G95>J#3q!F!D;%s#Jzs}R&0H;4@*rrTEAx5%s02lUH=_;k#2 zHF@cs->iBg5aBfk{*;t?M~B7F&bnXop;5kj0cybe{9WH_;@=45M*kE;GJYt#woS2- z@8NLTKw82GmwHQ;ds2FIYLD09&|J^n_tv5;-2^cLgwH9uQ`u9T+2pW} z-N*NGtIgJ;w9t19FbIB9tM8Ys`YxlCi%NdQS}3WQbiC!W5dJ0qHmh6=NT3zMpEH(2 z{8k6}>dN;D?PV-T5I*>HBsjDC1Cb4Fzft(an?%3wCj;W*WM-2LzvnhX4?E(_y*Co8 z9;aBE<8(ezHl}42jv-@AP4J4n=}N-GiHpm8%V{^MZh!?|%^#6pi|L8D^^`-NG*5wf zjV}e&yBg{wx$d@^u41&H+Ds-KH16K;qr+p-9O|<8g(BX7WJ4E3wCuk-8VcWK}c$sZ0z46VcrP7hq@?fX`Xp%M>+2O6j!08 zj+0goKbL^gTYmkw{O1w>=v18ebCp54=_NFL)D=$Jr>JY65`@ntq`i5=DyTv)7_Fv$ za#-USV&5bZM&J5-%P(AfDP9|4k-p?fl;ENWxj7GOCeAYBqtKQ5%*;+pL6twBUVRi@ z^$+o#3-8U}FA523&LMx{qnZW#gH6@OX8oi6a@G5AB&m0 zelqEngi9lAsN6qS-vka-sr6)kzj1v)OYBvIWJH9f5exklReHeyvh!aE$u8VvLLpdE z%%7UyozSNJdvp8yp-$#)HclDRE_?HdG+&<-AQTg!(wi4sbCY~3?yT@VJ#C4KOX{}g zhOmtB1^OF9I|HQ~!F-9LqLPw#x#vT>vQZ=6u|^(vE*LFW0(zmAvZ=>(NT`Xm3x;w^ z%3}tTZ+W8;)QvCEJnA6+hb<)?K3p{xQyqE zOuTj^ncqLpOO7nhfig0h^3iQ2ekjEIk}F$0#pB9tJM8bPRm~Zyiy-w&`Pd1!Q_X=* zXY>N!0d{XmaDTnaRD@I+7qL8H@K&t9>^p^3FRL+aZ^IbLrhpCg3b=^MZfOWolFjON4D!? z7w%V~Tz{BX8{Ou1?!I)dyGvX{V*7^rkU0)hX~PIP$!gR zME$QvXTe2PxG!kPZ+9act)0hGi?0Kz|OD+1aMocC2#<0lzVyUe`N3SP!aak(YCF zlEUKXvXC><Z=(l^-SFJ*` zL!cLSTJL-PT5D!Xu&KSy4^R5mG)RHv3e-3b(ekX;-?e0rI~L zqMJ8@5A<#82p|TWD0f_h1RHV$(3pV|2a{4XdVau;usHO7V-R}bB3_Wq^Xs$3#jYFm z!j=dQSk-W)_BTz~8WGI( zXNLPAU@x5QDxK|`r5{))SYLLve`XYx%xd*r20RFfWg%_NmbUgjcePmRp=+A|?vu@T z&tHp)i2+b%Zaxpf@J)bQPm+%zZUqSGNm7De+OqdIX+KnJYioRbe5m-=>#kXo5YW55 zpch{GBG?s@pxPqpTrqx<24LW^)(C70q<68ivIG+BjZ&L7$|iK}_7l79(;r5vpZMfaLJKMMsWj zLaUdUt9r1e$a>*63j)z3<5cpsCc&$7_zH^B(Q)gue`RWuewk$7;9DjS#Lrth_wRQ< z#Z^~LmXB-U}AU<4O`(AGZ#v6`(l+(HQ(@V`Bq! ze7}`kb{fZ))Ri6(P1WOYxarkU-fFckN`RuO7pg|WQ zIsV&e%Ucdp43f+4XSJh4?jR7fCNndZhHQwI-##~7NPN02nU8k%9JjKxAS9MVc66A+ zckI|&jftqMcVY;AI!{cgB2t8?_}Yhzu0QbhekVO^%Es2H&GA|&gpLLVy1E|A#e-mU z($8MHr+D^=+;PqcIPlw>P{h^%vbGWxANG8Z;;dRw8cvDG0ww@Z>V_3G$L9Va$mzhg z?jNjooqQue<>OEvF9gcySdc=pQMsWOHA}YqT2Z@C3hA0EX zD7isazF1k(as52zgBS^&)tdVb@gfXBjO6S+nLgI*;^f+v(tuNSR&ngQ;0WS8LMWd6 zTtfi4Ku)Fdp6>^zi(q_c_~7>z_LL36KDdVfQS8gBtF2| zsx@1koeEbq@{5brkZV8$?UcaLf=V{PN4*0v*W>Rk=T9*Mu`b49OCA|LL$yyvTQFk9FbRxA35Ei?c1<&zp_$W0J&*>Af1LD$LL67j^5U1 z0%Po@M_)biO$)(53`jHiGr32mreVSR4Zmm(zJM!9M{D96x}N%E=06wE9KE!^sCWd2 z+Hqt|j5DbA8Vp=Xr(a!6l_DZ*5}93}?c(F$0Gp!Js0Dd5FfdT5#2L{d_(AFl7n1yI z{=}pt6VNDuh!nJ&4MArh!v=m2TWD@+Nf$KYe#B?v6lOtg3Vd!tT--_4VudbmoJ0^x ztePpwFAQ|JFJF>Jb)IF8W}IIr(s#cxTOuLKz?*(<xFH8WNNfW zQ=7;B_S25hC!xI)DZS8ZvU7Jlsvnw0n?E&u68XnV`*=D|C*3$ zTG>`S%Ym9wlwMPzTa~U2GZCq>f>%N^bM1}}W?6gK4{=bqnK_CH2lzM~w~%TYG5`SM zBj7U4*10q8`*Wj>Xa$ zkB$OeW%@Tbo$!h`^Qm<+dZIX~RLJt)yKg0xYb`yERiIrY3pD}|F#d8MlPra&2R5~3 z;@JA;b{1tsnQm{%=6a|7;gQh*p8?_Rg2Gc8w08g(f3t9OJ&_dugLvWP6g==#KY0zj zxxx_6!{f30SYO4eE1?UPge!VfF8YqXTR7~^r??P*b8!vc7xn+~Bcd;IPH_!lTkGF~ z2=!FLG*&MuQBdz+ zaVp#yRk>k(8<}|ad8FZJ!y-Y`g<}tB$)IPvst*$vA750^yV-qv?>k3tiK#UEU5&Z7 zY)3(Cw2=sA{D19&pzgU3+i3Nv*qZTqeBI4J?@sS474qVUrd^64c3DXG;mlf!qbNQud zc=Da$-gA5VD&cAGD6Tt&6$uTGWF$DhTYTp}^sVj+<;zaGJIF>xt>yj1pgaFu##hEQ zfv8O4_y#zPwFO{@JjS)%*C?vps0^xNZN?tvYMpi6qKk-34rP*-Yu`}0pCl!fl|_nt zrfz10=W-Gi#cKxMU@hr06z-3Sm*L=PCn59&eZfi6vplvE91U>cSBX~7i+cS$EcjFH z00)7fktCMI6wP*$9sUpo^ik+)ln8_5oBb~DYhFL`KG2IiKqY=<7+&6|6OgAPmxi=( z?ka;PUR%S=$saSPi;MFKe+~pPwVu97DzS#&(P}g^Q)J+6^^5hIZFe={Pjs7%E}l!& zgFmqAA<<0>wOeQ;Py;$3HHA{*NR~v!7XFYV3)+;=oGgCY&bG%_xc>k#=~4qSPc;HM zy2r5>z1=B9($dpWB<8d47R$Hw)zvTIvCyqe$KCRKx%JxG;5C(SN7(n-MJfLt!k_Nj zzoNL7)iRf~5`vdUm%Aill5x&BX{GfkCmOv6fa`2_{D5;4_m}|-N|w&Pk@qSxP-?qOj#Mzx6mPd4j&0gXrM=S zs-U3o;H+H;*T*?=lr}op*lZO%ZqkwZrQSb%yqU|)w&uc2ApqKt+4veGDgT1@(+FBU z7KgFWoh=@bvfL}dy$2rZs%5!22LWUrGbHYHxJnw$!q#~)JTEGLk@Prf>wdws7ZIO@ z_oTlOWM=lHiJ96g{k9Zo;mJwfNMUG1^~dT_87Z|1!LaM6*RG_3P?XBhv9#!E*(veR z00h#9)sF%#^kTkz#a0CBSEZVV2w=>)t1ck$k-y*h1tnLdb{?}!I}gO?5w9;-h?$$e zSwGr;vECJv3h}00Co|fh&gdHpeeLY*d|JY#ieXQcnwNl+6>hSTk=o1 zzAe)gx~@Yn8bQX;a7p1GDXAiZCt~Nvgga35wI^xQb8@17Ea13bUD-WA5|X`>)uHi^jR0t z@KTPJNa?$ieY^U>95|7vX!B|I?p~Y|R9e*E#W9RIyR40y`{NxTxVzUxEuTdjE44W= zB3WWn?K<6)H3FaKZ>edV!9`@)Riv7i=Lj~*gw|9Uf$jsf*EGp+nN9X4#3bJ~<~{0g zBi?|!^U$mM(Vj!|Oia6?rjyeo2QQlav(jX>d4O^oZw=3_6-UkU7u1i^q&OD!pN4J7l>O$A_EHt)$lR)~Fp!TsYJ6Yyj)|61 zRx#CiYd37#>AqrmxZnCx!H^Wo<)?5bU+trnj<$NyqY9&kCe0Sp&7wI~cb#@!m+$K9 zS@CH%Xk=3t({~IS2pf3Ei^K~FhrXaqOz@=Up?gn8hrV(CJ$_k`x>>p}`k+qwWEz#? zbQ;uILLBWMAUMA$Pb^vaQ3YqH3TEM5I=uou$TKOa*T}?ahjM8<_J*6f9~g6~`N+O* zoB~Ex_8x1rl*i5OY45(8ZE@paxV>jdFE~&H<$jscCB*G{%-L2nd9mosMFtH69_`WZ zFA0HrsCFx^dKf-b(CD$OwqC3ui^g4?NO(b)Np`;Bl04*e30)L-qs`zfOb&S&5~X#b z1rhVpbW7pPcFUsuC{~%7qgEx7Gpu3Of3Ye;IpeGt`A1T zgbcAVMNBaCEX>G2NIE&KEm%~0Rl(uOO`oK{HqzgJ1`+Ws$=`n@XmaFrKF~kc9`diT z`DnLwm(&$(F)^H&_=ghtv~u$F-{bIFhDgzKdQTLxen?7ba26?7P4?IH_{rfK`Z!*Q zHynvJJ#*m8j&2S$kA>78_$V}m=Z8cVsJA*~KN}H3I=_8Z-u`%(uSrdUIb>4V z7rDupA9}shl%F>HORtdCXBVr_JDPYd)rL5^IVChasL@iXfvlGh`!yKlO=LPrd;#v!$7dW|;Ve8tga>E~uFXk?89vtLk(#yDLx z{s|HAv8#YPB8=HT=wQ(x@%oCNZBTM@%a#*34tl30KDS{^`B`~R!FJqx|5eDx=F#lc z{z|j#Di;BRQYg}B!KX{>dL$bAsvhvFmhcnOCM$%D7I6*_5ypJ`FKyg29jy5VqP+5) zPvgxGR)X6buV1e~vV;f(y){A7LJqDX6DvMaHiBaJ#KBp~|LWqKoZ4#p+Uy^ZwENCS zJM2(gY?vU@SeyVofh4--lSUb*U^T@G$YJw+FYeKiDY4q$-uC}-^_EdpZeQ55%SLB&89M?vM~RErQbBNVjx{ zG)PEGNJ^(tlF!=b|Ge*q=M2yJaK@0a+4s8FnrmLy^_#-sIY8p$zgcmV{Yx2}$)W%H z^zWGKFV{+bt7uSine?*;x5s$%!Y{7hm?ruS@!rN;Ptvci(2_d?H4obgV`4E#qdLSEl^0^{|~LrAk$6-_r8F<+-MQo^H#9 zuI+iyuTU5*;P?!K%yW!R*#qUhI3KP;Th@YmU zTzPHn0J;8hn!P$d1c-6JhYx-a12N5agw$teVxVSh{M7CBi%z*u6LO!;q(iGFG^QcS zU|@=v^Kf%Iqe8R?uAU?)xOsNO%G+%%y~&rFUfX>kuxS8bwr;}fsg0$4xYuUz={Nlp z`aP~I8BFwyqmRr%x&pXyKTQm8F$HfszhTZdCB_ST3d1CSpTq~`$DaD|1>8uvOdk%n zdlVN(9!2FOcAq-MID?1Ji$0OC-y)HOn}Bq?@TIeQ8OtOu=o8}B#KxUXs!ytm<3c^7 zhPf4^nF3=*)xni|K(nn%;*9EC4xb%He`uM20<<07-xB=qoNAF;qqzx?hBf40CYvK( z?DvYkp}j)l-vfh|+_lya&A#)4g(J|n0twLW85=CVY*^rdxNTl##-MKE&nt?2iCgTfUa$Mx?9W3&L|JWlhnlh?G@XZ<2u`epEa^e^;RgCTbcQaiguTn z_NPp##+?jmCSL#O=t;mmGhnwZzktNd0^w0}`x9ZI3p`J~rVf)oe|iC5G_VH&qwWFd z4gqM&0zmV+7YVa52e{&=ITB}3G9q1R!1FV2-e!%ah&wtf*|^Y|fhOSH_7y8IhIaq? z6AReEJ!=UljBgJ+grBk*Rh#`SU>9$^^7$H@KWp^fc4W7X$jHqB;MB_q_d4-&A3Hlm zV+gCX&kcKqe#|xCnEm*el@cI>K>N&uD>*v)_;1$zb3Q(;ggNWfCl5lVER~tHei#ap zZ>F~_9ri{L0Z8W7#zwFd4FqI);2?b{dOAJ;sPa*iBg6pOvy}O4(@ofZ;j;{Av#3ty z#RFUcNQB1QF95F{v6g6sH{sdcEA~aMF)8D|t#za=z2(b>6#GX}7{n2+$5NLSEi&>L zib}B*6n4iZv)vE(UKI?O=w99Gz5@&P$%!e*Sx=$@s$Fb!c9!=?zg=eVd57!y+dazL zyc60?kc`_GGX-W-|B*njtf+)Uo5J+mPZY!GxZuzP*Hh%qklHdUly{wezKBy@(f@OO zrQUQ`6vM#bQVT#ex&4qOLW1($K#~e1!i4vW*0LGl?+@Y&^6SbQ8;{o7G=xD`V2pWqAP4NEw&uy2c#uW+*!>{j`X6B9B zESH3~GR4d`qF*qnFz;+{=f>yd=L4fXzd(ce+qd4&9cW~@h$r0Jik&^x9O?_c9=PKc zTQg%}86_C;@f~8x)81ZhQ60psrSCN#o_VLGrDdqSm6w+XQ!#*7_Yq!L7HFIjfsQj+ zON_yGAY$Ft)|Q+5ZPiIP@wby82oB%+`OC_NRP2F#A^UOqBz^LE$J(0KVWhUF5RsR#3QKsW(yDpu_>1vNDeCMM9E z<48iJ+(0nY!oor=|ok;W4 zO54Tj`)evib&Ni9x-|fhF=$v;i;c}y#$8jVmD-MoI7-O}H4ds$*Y61i)hPzmU@`|- z3#peXA1F6$XXYY^{_fiN{N#pA{{5q4RVI5DsHIt@7zZ*DwMft5I3BBRD;C*i{%3>!K<}sw<|NY3?kr5kvvq^TuZrEH> zRRtEqQ(3K6WeDCIk%)$zlftuQZeAe3cbIuHSQdr0dvU@T36h=4h9A9&M2 zA`+xHD!zUDc7A>iL?J*#JB*R11j`vL2SDzTS5yqQP6h4Dmeo))29@Mv(EacPy>j5{ zN31!ZqyPf>{hy?cjm=Gj4rFa@4btFgWE|P+A2sI3$3dh7U`9sV`BZrr8T)tK{2qxk zI^%^(wo>lZq;whI7})(%;jS@P<{=aQe%7+QGg&Z*xR#i@0pw!r=Z5q5twrd^ySr1G zw~;)jJfjeeta3z=(VLOf-r35nX}%Jq=ZLN0e6R8odCn&i#Gw|19uthJ)ynadLHCK% z=l|>qIdT16*i}>Hd|D$lCS8F**)m20!_Pnz#?XCQE46^a`#`Gc{_@9BH3)n(jP3JC z4HV3mgNVvpv-|bF9(ME1-P+nWFfgPbkM3bbiJje|;j5bV0`FV2W#NUimD^eD`=;|S zophJoywqKHRaGAs*P8p=N*#aqVRbNMsl&yBZ_D>|dlia>X>ey-_5PV&{&(0*Pak4y z%g@VycsE@xK0G{z8}{uH!)E}UzX7-fcAte%Jc`phAD&Usqew?%?=GLC_s^|f^n&%} z34VZDVfTSY+wFN6Vog0__>{t9>&HaM!NIKg=+jQ+A>a^7Owgj*#`XWg#7G1}tieKIO z-H+!;BUr!r1qJN@DvXfGK=O&#d>|UZQexGqpbJ*X5^Do(a6mTzdK_XQAGQ7+MToUv z&(;S$a6qC<$;vVY9~FeYfffm{adnsUqkpB0j|=rDYd1GH!MHC|0FMAa&7j@)U%HU1 z0S6dq^LzDZ&q;7CQI{wm7#Loy66$o^4bTms&Bp(+nY%lF1( z+VzRyS{i9RBQZDk!9K(DKs4ZEPT_2~dVdqc=UHLrm?Yv6JEggHMi7-oj1E!)xKc7Y zp?xS=pt#!Z5m9|4h1u1;`k}yja8Pl4i4PDm7S50SE?Ojm(7{Ir!ion-_k&Vr9B2`NkpoCj~;f|)}TymHL1B#HCQ&qb=o_?#OK0{y&R z6?_IjX)sd{$_5Shym%sGmtU0*&9>vwLw`OZkVBRo_~=eBghF1h+zD(OBYRG}`a)7@<6Hq5+6>{KP7ki%SheO;1lR`V<$6 zjeQgXQ*F5jei{QxmC0(hw7*8$5k>WTh)r!7VYXA!P3)Cb)i%3jJ01<2xE&&NChHarK!X{Iwm)cfATw^8H)@Qo)a$mj*|!+}PGn4%Du;dbHAlL3wHt`C zR?b)lIxtb%kI5!-5(@wG>8NR5=aVN|EP+i5d~|SVgVh04Q(#H3Vfa*Fo&NXg${QjE zMerd@i{T zpMUh>;q9SU85|9hM-S`k>zqeP-T;F9+E^GCP5gvz7TdxmWTJQOI@bQ~4()AJ7T0dC zf19OV{ScbamJQDo8Z#a6vU@LaY}JY1v{%@=yPtttXIu8KwU(bdKq!J>I;d-0!6!$& zvLNIqNX`MiU*K18=_7f2vnkN=v*xWjAo>BqSvCXN%M)c_6nKl%zz_gwq?mW2cD zr9fXXfEN5J6Rt5t9Krs|D%BF;k-#Pg(0$$@HDkZrev3H2{AT#`HAO3RvCD&lQ41&k zeWAK#5)$$P5Sm^4WA$FW_mr z6v<+(pVdLE1mU{)%35T_Q_RGe8cMJA7k+tNf7!WMZ1(-PyOsHGw!Gm!*Jhy8zvI}( z^yR&UvQDOdmtMELCZf_ixm?DMcmP6ITbmaB4@+6Zz~GV*1fePN2bj2lyC(o@+cE<^ zQI{z$2oy8B^jINMES=mV3i3X0SEmBg)A#*v|B3^E+;Adbk`5_@g2ek@q?@5QP=)(G zQ(Rx|Rf11D2~OXRctBy8tS-9)voNW|Zoc+YUIWBqRziY1c!Jnrqf{nu(DQZban%@|iD$l#NekS^&d$Vcm|HnAT3a8A-~Vjf z=O%dufnMZQAY8kSJlw_a0DI!U$hIW+j{6;b?+P089c z$785?K0Cnf4`FJYYc%??4f49cV-#u#S)z{O6UDn;;}5shk@w|4*9Dk1jlmt~^RQl8 ze0535E@1EgUh^Q}>uT{6h&aE>7iTfu-rfdRfIi?Sz;4qw&OA-=mCP8LnLVOI{qxuL z&D;7k0ee+t^*Xkey8QC8)}H!<)YP+t1dC8pz|C#;8v9jU9s4cU4>2?l%!33sc1cMG zP*Je$5|NpAU(Kg4+GTO(IP1LX;k~}#gy0g?jc$Vvwy!t4dhGs?B{FO*37^Kbu#_g z8!*X97Cc2TBaOElK$O5+Egg!pv%kMsZJ}&oL7%TVSqpBuv*0vIO!RL11=bHP89%tH zfi*upGqc+aoL;)d##A`y!^6WuTVUn<0=04{82o}u2CmG(Fi_I^bu3r28HB*zlZXQ< zT>^dGIn{hXP>*#R+1BD`yBnFzA8P%9`r#iue6DHZ`R5G8Bi`VW!X9m%NxyT+8# zrJa*^ysEDbzJdcD0b^UJW~*D&rz3cVBQ2`E@60}|QG+&ZAnT6gU;ktFF38sfLpMFB z0KZ}QYI}X$umhM&>DX00JSHM>W>Fq)CmG7Y2P7b%z)8Jd@1}qx-jnmx8z=*I-iNU;*lP^^RtoDLZeqM(DeAyPOJLGJ48U0AUM ztutsb;~US)LTR$p!Q%pZZu`eb+WYb4O-(=7Pa2mF5zu8|ottqtbQhomNp)~Ni%(Bq zoSRFIiA4hgfbQh!Wtd0_wAb07T;{BG^H(EjU4ZkB6neqK`o{RY(Z{Z@n3QFPaj&z` ze`USzk7lqM6QeDho*q-u7pwXUkK6M5n73J=5{w^wq0q5{JbOfGcKat^9MV_=t@)gG zZ#U3rWI`*91G&UJaq8W@gM8(Ke}8+f&OF@f_O~IhY^}b?+kWXE2R88JXSLbzf_%BK z@GQ%B6WW%8x61jNZuCfQa7*gKxlvIG-VX&z*LQK*i4-11~z??Qe z7!}z8P8W^Zi8i53jg^OF0zZW!X>!v)D0rF@p$NptAS=*95`oN;`yFk&^zk+cO1>h% znJPft>GI^sc@!t#_K!)Wa^^x8l!gTg@z=f*pw|t~2|7k9y3So&{|0d`=;J~4T!Y!{ zDW<{NmwZYv`)ok+OA(4ZgERBo$&lyyvYN}vU*V}uB#9`Y_hHRGQ8{?NS7BZT!Q(`| zLfgQGT&8Xre0T{u88jp$pm)5v)pB=?(f`no`YEC;D07v;JkvT6U6H0KE+E+zW@mTT z)E4w}O27%}?fc~0jqWpQ2Ra`yj9`FFBcrp#%fMwxc>%tM?A*f{13NQpiHRS@;$0N`|&C2W+kl^&9kk+Bc6 zFfl>UB(H$OA~3*u7vXGiIRW4|W(Jp8h>rb1m&j4%^LuhGISq}CUu|Gd6Dm}H+uQr& zv(B6e9w62VqkcAmAxqKxT&l~^(gIa&zo8innTfs~6-4!*8Per+Dq z=m3dY*Tl&|U^aUiiUP+8eFrj!)hV2MHkGdjf!X}|g&iuqajCXQflNq)OyD!_d_0E=abP&+ z4q5gJbc5#a93cY8U# z1g{iOF91`+cP*STFCI8jH-S~vd+wBl-)hYH1R*)MJ7^$p^{gWyugX(lFL^O4p2oxz z{+ut!axeJXeqC46g|H4JgNx3K8%gI=8zP^NqI`(**h=+V5*w^nMmY%w!oMG=(T*LGM=b%TV+ zfsi0QuGAy6A?k1_-(;H`f4LB#hr-<7>TDc0#ApDN*D)wM0a0*ZY%GdzO6>JJ4sa_s zW(Dbx*zI-OS#-yy-b~Rn^E~Aas$e(ZtNy%K$rx$63i8bqcgJ2GLuDphuTx~1+1WsJ zn3%|rE{NvGRF6)@N<@+_Hqi&wG@!(d5Z$Y`8l|Qcs-}EH$DLwSp##Pl0`qO<6w3jm z=hVy5^M!%*_zFmh(1e5rT$7d$Y;kxHV+J)r%GkW#NzWF+hc}wDZ`al2=+OXtAL0jtmj8U zbxy>GF9nDxS!13(^QD(L3O5@bAlgFyANTA3SC2JEScFW|(sZAk{L8Zs$-2?Myt_81 zlkyFBIWn@pc{h7R1(J!00ojohr(e5n z0WXf1yZgq@J<2rH?n2GVIXU_~0`zi+l-ACW12S<}v@+f4*5DG%9s$U{Om+U4n+!|k z1# z5_i)hYC7w%V9jwlD`X}Vh?+3nyyNZZ;)CoNz<`2d23zdr*1{)?<%qCp2PM0ReVd5}^A3Gfn(?hD6xeX5)rxR@j8tt^C z#?|NB;$<&CM=h>a9ZL*VGZ>-Fb^_{#zQuWCg{W+=Lkow!ZS?E!&{iH4B-4dmWGk1 ziP6!z%!i7pJMm7y*A;nbj}A$|&20b}P`3{kWIOScJuzFcdN`?*=_U+f1+jGOTWG0O z`G#`j`r)ABRjXdE2!`cazvdzaHgn*z#{S$V1d4Sqk|;aG5LMYv4TiWgb_(?y28^Eo z_ZN#DT`-s#2CleVh_Och53kP8{l~MdPU4fsf1#hc(j02vjxmIXgO*_T>g7Mq^p2~? z&VqT%%ftVe5j!T6JKc977;>pRH+@?ypvM!fvU5Ka7(^TH6> z`QpOrp$GQ+f=+@~8nnQe$4!ntN)JcbwwU<$q)G;2gwULVc%emb>dBgC2Ga?zqra0m*7QHjUPdTThRotY$nxAFm;xQ-Y~PTYsAj)C>iP z-;^)~Xf)o35wKRN5rU?#P}eK%;LlVG^*%qI9<$~N8_CkHGG^f{duBz-DA<(PIm|Gq zyDZ(q`5n?vG0dsqAz;?a>GaOal*f3((Wnp?`FwU4$jL$jCJOt%Mv^$U_GW%!ZvxhV z&pF#(jXD-IGRF8*$b3SE-!1u>uOKTEbvr7uiDcfFh&*{aM*1f?ff5+u{AP@h0l`+l zPQw1EyTaIB`^Q%?zn(`&YAbqSMj0YWnAaNMU^@r65RAX#>z_4qGR$HXg+$$_Xa7gJ#tIi?AmTuK%zhuh3~A=a?*HKc2*mtBsOgd1LYtW=Jgn+;oQOqj)6#zBQ7_VO~D4}c6dtuCS!`VCm5 z+-@Y~fU9`B$amS8P!r|S2@?*{(EZ=o;++fD0;z~gsz(xMI^VmqRMUtpi;AN25j_(W zLA}*D&WFjKL(J3CaurmXtLh31aSm4GQ0HUxr?kWhlErlL882dPIrseuoK#j5o;Qoc zqVO|Q>)H8erg=!gk=7x_T4tO>H3jodBsj9O%n|DKy=3M~F1k7H=6CBvAJ%}H_};yP zMX?MSjEItwnB*PxG5|81TiM6Qro}kQuWB$!-dxwY=%7tDP6E9CFfA%f$`{M`V1C8E zVIgZ26o5Uy10@FjMHmY@wq)cL$Rv^WSc3^8Zpc(QC{We5X-%#Dj76QI-kX+G-|(~I z@T`ZNd~6Ea)fiLSQE44cmlIwVd)g|=;(N7vSx)f8_*zwq-()PTo9Qm2hh?Yb^eCPi zjQCg;Fe`qmdUmXs;17)hW%6S5>q9kD7A{3RaA^_q%lwJbAX&4pR2|}oq+nDGtWW%6 z1)U=1p%7`J7#}m=c1#WKYZhv3en=pBEunGpE%k68amv4I>fqb|eb|qIcLbu&R$%F+(XT*l4 zB$_CjDMlG%D%q>*KQo(~_|eUDYyBtp6SLV^I$ZIb8<2*d^2Bo{3UvaW0t26YHZ~eIZ?8x>qJL zq^P{#fgNnShj1?dZ7HIo)HeQBJ#xboJ-h|nbXx!+6JovNpI2-y^xlv7@-gu ztvr>1(K4Yyv|W2d)Tn$4aTV{k{GV#XF}UYMcSX&aZt9Gd3W|rqm1e32gd=K0Ag}N{w6DoH9$9E9zk1Z}M1IoYdr(1HL%YjUga;GUnc{GZL^4@4i2n4)}65P<|!#=;HJF=;zM> z(b~VKrmU=`jP>UI-vK(YAAgO z?gnu3k?%vqf>kO?#SV{$iQcpbp4;@cN2<4LQY`(jGz}y9n?1&R2YO|*66}0d`m2oh zPulOrNFKzTy8T>K86H~nT(*qH`)QB#?cs+2ut^r7Ix#X{5$|;TK$q0G z@ctz<_V?G$?Q+Yu2o&A&#rgZqzrydAPTPNa<19x6kAgf=;mf~ENVxYWc}6qe&+qU_ z%heUwKr4}O%Ydh5*zPN{OFYVyx@zu4(#qA2_TAg%-cwrc{yxTJYy|#Cw+`+rFH4}~ zXl)cJQ>Wepy_b}@<)??_`_!W4bTC{M##-fUWF{ya9u|ZK1qMH@5|JVuY_|umpIhL^ ztZgDHLy7~!ZmmNFE`Qm@$v`8Ch#dO1dOV+sD>XzTjZPZ+W2&})l`Az(zq$EF zODmzd!gbVs0UeBs06fUsI|;8|_{K(_y?#UC+dQu}O6K1H^9FE<1{{nu=3xAwKEGp{ z$54;o?H{0Nfe{)Di$GSpIvZQA&aVKJj0|mxSWDHKn)tp4cr^R8}^ym4edtTQC+#B7!N3Kxz*O34y9W9t&{B0md|v zly8VD5~zn^-CwOI876rd)zSxu_YQBOUKoQcA+PFQj42pb_@?xQ=$#!|x>~O7s<;@X z*S_I)%KnvA8$$X=Zw0c$|x4^j~T3BuqE)G~4{0u;#;nEDKL z1hfHGCJnDC=wRDfvQ2anksnkX9!q3D{V^N6sn6#NAMGL`u-Kb1yx&;=+2Zj@_qqSK zZ)ni?s)r*d+iM}0saWS%{F?E6zX=bR2BL#y+;*!v@zGf%8|o5r;UT^DO!HFg6g%Y(wDHJ|?<9wn)b)#!T$U%n<5zOm z8nEJc6eG|`k&qx{Hss*IAb1YT_}WD+btes+^)X^mY!8>_2{rkcc(rDhR{r}X>UrP( zrOg$jtAQ;H*z{Wb?-#^1UcI6%XtLWW$+|gBRTodGb6DyN2pZbkCEXm;zlk8L)H&#c zR{m`nRjaV_vHxC=5^Fg-H}(~C&0JW&JG|hSH<~>@T{dpqJ2WnqFJ5l%uHN6>bb`zrla9i;@=~ z`4y3yUTT?Lbt>AGVy53}t*)L}Kk*&WE8m^X++IJ!?b`FR8p}}j4F`pzy*{9 z3JHNa5wq(Qu%^{CkXN1#B16p7d=OD5hvKK8Y0}=$Yp5vEvD|C3-@j6Ygl>=TBIR@? z(pEQpc!t-^?yl^8BSk<d-GA@>S%y7mkO6k4lSRgUX<}GjGY0>j=F0Mqm%H2 zNJF2hCSL;TfJ0R{2UakIkp#6N+iWv2?>gDX>h;c=0nHCN-H1xP#Xy zKx*FM650Y7d-1bc=Acp*{qy>gG>a{B!;@QOFmLI8)6;!n7sO?RmqfP~CnqOgJ;irD z+pPotb4<{69%%EJP^q?-j&`EgqgI=~wfN#kPdIeF2OBuIXF2^3=06EF{S)Tt^+j*h8?QqKKbKsXp%!5ElN44|pR-;HS*Y%>k=rFf2x4 z;~O?77{Oue>ixeq`}g}h0pI&uKsJg^YgqpC>Mvz z9bt_)5b_74s(E2mg$EGj#4g?4U72u(!)A6|rdJY+d`E6DR}JPjDdXBqkL_HJXt zyqimK3z7QL*Ixi=5^G{~XuH{Afj_?%DEovDtWy)gq|3RvqCMgE-j`GofS(K|p&TKm z3Ie|e7$*wmfFULUgVDNZ$N`o1HB;3%?WkWH=r8}0eb+Ij50xY!Ft;r|$vA5CyN8W@ zk2ds!I{({X9(?6jqva@jd61DxhXpCp{!r6BU!5;F@^4j=IO2|@#nL84U0lJEauvp$ z=V3&ihBA?%#=t-&#k^YH@@E3Bd+-RlPFL}!UlcIewArm$0UUkd4ilfHqTat@6}m1% zWn&wUqC)wai|tGdUv{WkXi;VPsGUWBwJ#>E>7qa#S&I#%KJ9eS!#MDJys-a%jK00s z=jMIDE~-r#7Hb(6Asi`~ZgHh)9p{ZI6g$|Sd1xb`A>dX~JLV;F!^(Lu(Wyf1qocbe zClRL+2a&gymhn{UiBr(3tqCX$DBsMtmw(uE#3qGedI*Atw}(8Dp_`hHOOwmHwc=5w|U9QDow2qH#RBVfP7-0+Z z+xu2yYbzjYFZF#kvBi3TaYQ>c(abi9j_Z77a`sPD=`QhNSEblaESg=s$TAFca_B61v= z#|E>N814BTEP_q5+d6=sur;{K>w>+M6GkI|I=g|uPyY&R1 zqCO$I7~>o;tSb>Fcor8M_veqx-M=Uua=b5XhX#!{*)W!WV}PPH(2=_q7xx2DhN=8d9S4~e zMMynvS5lA8+V^Tx$%w@ZV6?SU<3UJ;4?Yx?S7qb zBwz#qv&Dg*Pb^4-6fX=+hVr|b%mNplN_%fR&wkdgt~!@p7_|2{jEV}>Vp(?9KRDZa zfC;HAkNCficiac)7=DvAo(c1Q0)!bP36vBi@Xa9)P@7elxZMw6tuZVf?CJjcGATrZ zkHqYkNl(szrW$>G|Muwn`bNpn>3!=KN1dlmrDC(lwHkpGaj4e_Ik*NxstI=aw+WD7y9ejG6WQn z!+F1Xy*(u3jQZL9Z{zyV@5uP@e)nPf8Rx7)+eS8;zsSi(zuNQlPeFJT3N~MPgV})V zzvs>Lw!ha!a%eryjw0Jf4|X%mIPPb82EnL#!7G7fjF?WaLl|peun`@6MRA+b4c&{& zAo*oprt{s7e2TG8o`@Z{N`+y^_OO*@;*34-;f!E`>y&Xd7Yk#lif4#a&Rj>%k81dA z#rSd-={8j~+tZi^%J5lBABiK(UlSvgu zQbkcDSopwU;p+N|Bgy#gd}wpCKt_6zcSM&hRSEZQ#YepT)R^ONn|Sy;{n^{=bBa%& z!e21OE8%)q%uc1Fc77jDsziNb(60-9`h*LrEg%idxVQOSe||?x86NE8bM0p6Xr%z@ z)_5DTBxF8wRiYy(ri@<~Oi?M2ax~inJXrx- z&ith|=-_T@M)eJiv`c4Td|Ja1bupJL0-06~6O(9i{3Q>2sWvCK!~@t0 zk`?GWL9C`qmx_O7r0>`iJ_veeIg8y)ey^mZGql1VGJGul)sT%{xGL~H5;gJXu;&mi ze(2;rg}cHXe`xcW78BveTg_&I%kvPIteeCH6@I!n+%C>qnW;J5x!k?^^rjl<(%6|| zVdVE83W_%(!hKgFmo5Zw@QscKdgXg)R~X#-3*7EROA~1p=|J-f`mCSPBxGtVM=(9g z@(Su6UpP8Au!xFy4fU9W$-BjT00Y;y1@U-1-QB%1x0)T~<-;7mrE*&gmFd;kEqIX2 zPuBX19W@^USQFdu(AM8w`iePme^>m2jNg1IH1vFQ8}*0#M3q&J?J2o0Z5%f37b0J& zhZ~%+A7lq_pRk)=PpFsH5C+Qu9@5NcEF?4!nvlZy)oWb;FmG?^RXx zl0DSCG8j2X=tRNfQA4?tO~L{hD6+oNg@JUmv||tb%rFN};^`y^%F~{o9H4Mqt@m@L zHoHv!h`K)u4m%RMNhxeUqe>Eezj$qA%*B})qk&_sOMg$Sl_YIelv^d7+n(D^@Qr#g z<@!QoXW6|G)w&bO3x6niASM1;bQ{yv5ycuV89oco*T*U*1gR+a$j=mJ1~*AbMR8DQ z{39ZKt2$r&t0Dvo&fg;ICigM#6a2?B+9v{6QTgQFzE>oSJI$N@71wf+6Pb5%_W`in zy8M5E*!R{$@!>@eidYoYx@t5p=oFYAxAqI?7`6-i5KSb*r_0l8{JgAKvq;C9;by&nrg=Dd-6=FodYmYAtOK%lzmn0TZ#fUo`QBE$K6kWZ`C@R6k=!lGulQW!>OfSaN`=|^iMPBOMxjH)A`G1s>a z3uAhUdcO5gD=F9QwJ~HKxamkk3?G(w>8Qz_h*%ho`*Fq%Hh%AM9S8%)T4GsCYNd|@ z?&t0uU%6Hn3+YW#YQ5h-7N1$nvhr;Ad^l-YbziZkmddNBv?KUgqmuBWXlT4iK_7aP zz5aPthw1(5Lqfb)t~?i~+x-+&z`al>gyF={9~7k2rU726_k>7!$yYSByKu&-0n-vZ z5qW2kp(zdq2K8YNF<7Xi6$YaqZ)TJMyXzibPjoo#qpr`ERz|(1As$M&E?(D?b*0}D zwT>fVqJ=R(Y?!k>)w5FG9J35isMj?kDhla&p}O`$fMxjc-g7yVz99$WFz)sT4jhtY z&yX~vlE&X-B*IY*knvlen_&>E`&O=s;)RjK|27ZgZ^C@}Q=*i@#E&k~@|9;M|K1`b zA)TD9D7af!{(_0{n&kRkpyI*U6y;A>LMfR>@qB4rj40{MOkk)hGpn z4J$!LCywd)P|1T*9aX)OUnL=TU#bfl2~X92Pf$LaLrEEsi6Mw4H9+2qK1Rml!Sz6K zf20T@Pe@{pN6e@CQ#NcS5Mo;NHgC}u5~tH`lhQIYjMn+(x(2SYu>Q9!1orQdX9mzR z-Wy)MoJf%Ry4C)ZFo|@rk@@2GNDrvaeEb@3P#l*X(p{7;^_M3e3V)a+;bPrU%6Y+b zE{g-$sC3lu-)!6x#K*{2nj?bw)%jO8gA&p%54(TB4%`bZ$x1Lp^OY@eiyh8mjIRwusfNtEEMLJv|<+EcG}+*;3OQ zV-iJ`Bv^c-(t&`?z46)Zqm4|HU*v@&)m22$suXL+2q*uPiLDc#;1g1F+?v^OYP)%d zAC?e7eV1Z|3ePJh#A_Q>;`I`&OPA*lLnTj=Z@0v^I&STc4jPKT1YKV-#e6VuRRw1q zy&)%^5%aVz;+K=eoI^6B#yL+Yt7Cc>r!>6&vU6KgNn-)|OvLHM(d8@sHZc$WiRPdX%SRDws{z$L85~LbY&oRt{G5@{V6oS162$(b?} zl*S|XX^G74J5i#^D#6`Ww6=5$FXOuf75rNj_0z%>dllFK!Zjx(#%PDa*>F?fC28b% z_UA>$FFKKrDi=azv24q}C~%>wK*%(>&RA)y+yazQwnN;xVS<$e8Xi~VYK`K zbt=zrggi=ay?W z2+RrhFSN~;p_F4?wI7HrpU=BEPBj5|8sp~!Pw6z|(V*TJPmAK(LQ1smixTTt=7u$# z&FWs3nz8wrQftY2Ck*NvX!GlC6Ffw#*CAiy@+;H(tAGb7u2|2EUP#4!Lk^bW4zZ^G z==V}r9fp^?|8_h`G}+$lc5u^Ov#MZ_jg%r;v-_=f@i+VrtF3SRI9b4*Y(pD9-}Xw? z;U(#RZ|}nL(G+1MK``v(-QwTpe2*XWQeLC!5LS7dhlH+rPt8#a*pC|DQUIYmUpF!p zY+LH$Abfq{pH1Z~_;y+&2&|K@W8D(O?}B=@U(iM`(nah()(MxXV^`2B!7a(1p{HRx z%X|e~qTC9(&QKu!=+jim=V+6mDKTw|!G@BkFXN*jr*h;rQlwqJz3rivfRE#>lq}I~ zA~VZIO0OQKfB7CKTi07z-T%BcP3^LTw;xVn!ghkk(00?xDO7l7Yfml1#gy}Da)xw! z>zld4LvK=c>4u?b-n?hIe#F{2B4lUF%7Uoi9`^R-;V*u%!4PSs&i~Gri|dTkz#l;X ziS;&2>f~YHS@s^`r=OmSF)eEknrt4|)zM^tiK%Wo`7x}Oc?z2BeE%&PjP~a0C-J7d z1UKSjc$IH8twarC){TDO7h@`2Krr3@a}UTQaB2mp*cn6o8?u-$&7ySxS3*}UwdxW%Zl=nOT{@ORaWycXJTI$I_XQc(bH&WYuI;cJ^GiRzyQaH zB$n=%&Cef&(neS?5~aRkJX7T%$9vci7C%L>)qeQ>w{h+Lk>y9Z;`|WL{qfE3;lB6R zec_BxB6XCS|NCkvgz_BAf-x;$=SWc(PkglDI44@vm0o?IITs*-GlI3z#eh{LJqNt9CQqd*cV>Q9U6 z9Rx;GpUOwzok4{w1?tqXD3j44541u)TC8gSeo;>75zk4XOY>2yzB|Ezjf0aVuTv}` z|AXXCd6AS?7Z&r7b^AP8Ae(R=W-Rr`FrBY4*I{kL32w%#5&^ zrhx2^l!%A5EREP}*X}1BdCWuKU7w%syiFbmKUw!0wzTgG9MIAsDa;)DNnRNLzik%d z1C7ctO6mf(%{P;$m01j@#CbYjV#}Hd*&E*%jo|xDp?YxGrmSKS2aT>)@=-RD{DXyH zRCyEFJ^EosJ)U8coEdsIoZwMTbAsvqx5Q@VDu+pE+F-59Y)2Z^&LF{qr<3XM)kNG7 zL%_$xS`&5OXAv=LPXci=>vtuO86H<8gk5fh%Mn^4k}xS)XDvUCuyKX7cULP;nBnQJIbq^{(7|yM2V!DyZ_xaFgkXS z!c2TNZpS}}($F&Sd#+^^;+GV6IHl;mYU?!qgj)150i$&Eb4kr*BkAjUA5(R80b1K{ zPF22^FPDog^49*`YkySINYoxEuXwcaI<#!4-<^BMc1S_9eZ$NtuW1dBG<%vuHoZ$;f5F)a-^^xIPtzp$Z)0?2LQ>9o zadUq&68g?W;(G?^?btp4uM4AkJ4%J({QqvV_Y%m@R+wkn(|NkZzgVHCMp!-3CQN0! zf*Yt?Nx>2fl&8^x_#RW0ypq~}TlA{@Pj~G0mjH6=pJC>|ocN-H3r+Xt0&ySwruR7` zC@#5{>pqA5_HoHTS02XH$jE5Yesw4JC|Pj=iS5P5)xa{1{&z)vRkX7#(PDi#0pD1H z4LGE<44yC?J64Oo-l2@o3t;Q{xoVtlZj>geg45K_QjtS#1wG;o!0gs~hB(f@j(yGY zRcGyy`{Q?ANN9=vA|{$XAKcc=EG@NJiCG9+&;vxrZccl{kNi*u3#8T(K+O&osJ|pK z@(pQ31Fua6rasD}&JP;=djWklCAdPfM zx0JMWH~epWe(zfE2S31q<#65SKIdF#pS^bnlpky4)uH$&t1>3ajVNJMRqgolI}#l; zdr>gwYQ=%}w49dBN040}g>Sg!H`oU#_+EgQ^<+3CH$>8+?6UNwCAb3;#%coS6u~Rh zyjT2jbW*_b=F<6V=oWz)tPjOgAE(`VuEUQZUU5l5P50l$Z^cyPrd(?JeeJ$-gOS?b zCRTu1)cjcomHN*~BcNbwA1+31F#F+0mbd6D0@|l9ohkH9O{TN=Z){RDBDJ4SYh~>t z$sSZyiV3^bvtlQ*pe?`i@E}#OYKiblzXmBBE!2^Ynplc5Lp@zm^G-c9{PU~araLa^ znEuvGSjkFwCvoBf%J%pq+7g*wVD*BTc0gf%9K3xJlClNealF|*68hB>a+Ud-?K7V{HDzLMk1@HK0@^~i$ z3xcH<193)(tP>ng8U(K4*DQidX+Xf^WW)1Stz;h?o=D!kHGkR(@R6pc1hdGi$x%x< znJFsc%UM=+1fN>{aP>T%`#W<_Avj7)vqGkM6fIDcpP%2{yjWKWD7FsOk@t{-wn0&E)vaVmPI(fF6zRxcqz=G{VP zpW>ng&S*h6-;K@nwHr|x_Zb^Ma157;g_q;RmD?MYe#N0KN5>dywIqlyv?V8&F|6hd z`t&OefznL%Z)RtN)~B;iDW;Wtky4GXpJKMZVWW8Bl1?QE&w1KKT~=Zyy=*l#hK5s= z7={sSaBwALLgj+Mf*@8g-h#~p9YTCsFD7ZaM8mxDrW}={6Xh zc6&&A;men9hf%7*&Bu3rxAJf^`6A#?WM6JVj>iP^(S@LL#hKr?^FTdy>(e5R6(r4C zip}zrr`5QCMzfj9U1Ta3mECB+*hNLfMF{H8g2i#shhEb|pP?+|lL=^~Vc#bu$ zI+x!|Ad(I>c)Ib!PUn1(Y&ry%e>W`sc)a8;l>KaJ8}}|^OvO-h9x{=o@KuVlw*&oB z_i3nG0&}iP)j#9$7UKZ1WM3_L*a~9JYGFV+#y1!WFWr6zym9%&)BD(10O!g3C-qs^ z)48Y-9Wc<8NLI3{o7ea8c?Ifoe&aCrbGWtDFJ}*KjQe%wND=bnf!P8`C#C3pQ}O#8 z#-ekKE>QgFws?O0)5YAGAgF4ucrraD#d6l|?iOibK@Z+(3A zN0h?^1#aj&ZLlCK>41{BAq`chv}5;oEjo1|RZM+M#gbw{SDqq@w$CBZ>rK@ARX`{Lwe^i#?zslIeicE{kw~_ai=|jJTsUoL2i%F1J^~Qq3~!D}%N6GezxML$P_1U5Tkz#W$HJ z{_b8V^J({UZH_C?PqWJG_0SQhGtF#jRoDz`rQ=Y!Qns{LYN=LuMYd59Ay_^!MXN1N z`zpA_2ew?)OO%AKa{7lHnS+;u7eqA&qyfT$_pZmTl;C6RAR^-vT{n0tinIx>%`{J)T z=ls}M{D8UOXp)*zj-Xdp80NFo3m_;n`>YoO5Uqjl01F^D{nEk{i;ERy~;;C;hbGMYA+7{`BGBTealU-?xTQ~IYPtFGnF zEk_^a&0vM2-9o*^;cwj{WzEsE-dKA_YlV%|8xFHYf!>&LSD}Q6&9j-m(|a#o>*1h< zE-Z)vHVpdTY!6aK7KvE-*a1RzSN<+nvD~ixQp@Yy2Eew0a@Ota#+M z9ZUOm&NeD_-Mk$gUM-6};0^pedBT}|*6o2tI@Gkin~gB*jpK5k@MUeLL(_mW`657N z&~tsf*_LO1D8id!VFq$sf!_29LXAX+5W!Twxvi6-xW3ieTUam;3yWA*xdPuNWoh1+ zIK5Yepz=#5Jk8H(h2F~8G!Cqa8P)HMQ7o;$PZFd+Ffrg7i&&jwCfbd|?|pCAqL2{= z_6VEQdawwob&FQr3AoltB0w>#f~$dcPnX{;RkWQcnU3>VK*FWioM|DljciXvcPG^h zrxH1^5?U%k8~h*an0R&Q=OxM*OU9zK9Y*{U!A2`kQuYq}%*}J+3*0<)!*KaAVx=)j zMjYnb_PKoj%Coea@yzGA5dG|rI_Jr<52di{c#sz|Vx_vM9(I0+0WRcAl*-*T zUyp5Q>g<~Uw6m7D#>+K23;gta??X!R8Nyor=afIw5kBZv;=*6*8`MX*5Pt2#%h#%&Uze zd@WBgl}~@vmn~zwq2;uM`4gJmgylHcnj5v@5q;x7?HTKWcSXOZ*Nc^FpdjAilz*nL z;(LW`kyrlNJ(<31R_UU=$nevFI4BS9p<{ewscQ2M%DW2pMt0q2BI3o+anMJ)nPET1 zDPqZioUJ;kF+0vioi76pCcltK;K$X)M9zHvR@j)x;dkKhn>1&gf9vdlSBPgQ%Ivv( zipYLZ7>)i~2t>2e9pM`i$EKaJi#`Y$??JVz0VL^fdAwMGyV&a zvwJJyJK0(1$zU76Th5AWzSeB9FYf4i-mN{I$LDrEPUu0IO6Wz${ydk%-Ilxl?Caf8z~|}`<=G1a zthD*5kfcjNSS}0EL z&9Id}AdFjS(9D~z@T`LYxEBH1w)gmL5``7u(TdeB2BU~|a&UC>3=ehV& z0lJt^DcaS(&(AU$g&{(|1A#(*InD$F1m?r5Liz#+i4S3b$&MiZ!17T;_^~<-QB`Hs z`P5>lIAUGyt-IZ6@i;g|9t4KNflB>>`ED`laaZgc%tJuY%fae9W_}gOq4irV!wpWz)T>!*MIFD6p&1n0m`i-8m@t^dT2i!#f2K zCy91sBT$Oyxlz@j@T8(vJNnj%2F0Iue5!9M;2cI>es85|-%aH6kzU5D9zp&+!ZBeA zs`XcVWW08yJIW*3Z<+}XvPRzp-0dZm1$K&K8L->mZuqjcn-A0lW2d*fk9ae0*6*qV z_T!5yIfr`(?>vw1)459E`nOL3tCFq2GwcYdLZBEacCRNfC^PZQO-?pA9E61;Q>Beq zu6Vz{`5B6gMs4W7`;COFn1`Ez*I^FuIzrEnzgfnQm8JUw0-UqXf1XtO~L1L8(lVBdcAzW7h9e1X(?P5SPsyxHhk!(jnheS+sO~7bneEz z@m+RXdfs0GY+=T&U4g#0wY3%f1#7+iJHK@8{VVpbHW{>6hV*!-E3kRM`I? zzu);wXYz-=Cs*H1#(b5KqCvZ7jNI=50Y2i>?C3m%%rm=(DE=4m6w{)#Pyn|e`on9P z9-v_s;Tx0$(`B07Z&IeuH#Q2#FA)9 zSLyNBDK9v-u7&X}aihqbPtt(5gB0=Z`?>)CLE+b;lpV}7%IFV=MzmC;osf<|8(OvK z&hi!rctmuCV7qR9X8I+p#p0XyypCej$oue!`dHW?Ykn=WsQ44EGTZlda59qG~=17)Y@@v%(hIu2D3&7N6Z zfEw8SM#hN)$ds5E*w&yN9AB3ZM8Ry#@Y5U7BewauNKE4M8I%L2H~U12Iy@DNwkvA0 z6JJV|;Jqy(^(jQfo5o<7Vm)D<*eM-ZsUakQvc?q zvsDS)Ig`H=T7G<|iU>q7@PRSjQ_`bZuH7tk`YL-HUb}IdzXwtbJY^tai#H9W=!Y?p z&YNMuy}tiu=i`xV#s}E+A;bIHi<)+Dt0fX}#}(-5Bp7%a3vhXA3!*l<(3N@I`!I4r z$qNU<655w;uzS<4RpCyqSTJ=6plPQ6%W*}7hYK4E>sAiPeS}@wOqXXyhH@1^te{Dv z;t!9}okkB)tn)-LA%tWxvi+UE9f~yO-qPk~h$a#Tj6EzfQW1G_+p=%U`==NE&%lD!$#FthG* ztNew`hTveV+q=M}m6#oNWjH-C6^8Z};lU(cbZ%eqq5AxY2OfrF2RAAuLCcc!G8Z%~sFwex)&&{~;2qtuGb(^I2{#G2jtxkRbuXlFVfn?7PG zu#Wz)f?^S36vWo|2c9H`3ySW@5S%Ruh3T!odye)OO&Re2eOLeo4Fij<#%yF$TncvU zx46B$eZ*hG(1(G!XLXzXj4hN8BYvqYxIeUCo@55`?69iIR%V8nV>y$dwvOenj5?H< zT9B8@WiV@3hBe?)A8UlBqw5$$6xT4Z#XyiK?}jxt;C;j8Bf!BOG!y#G0#fe?}E zysgh#6_qik@w9r$-JoN{u(pN7SY>tiO@D-oyA!@95-Y_u+uq@`8<(Witf_++hT3Z` zT&C5AmuzH}I#8z)!~7*Gmgyx!^@?&&b_MF?eq0z`eS%QRNW>4fIXX*taueCjr=uUyXHN=m$TKLC=|)`E%Yro7-VDu zHHCy>>w7AIq=h0?4KIJy7sVA5xy2$dlf>`gKVsn>dvoCaC9(~6U$;I@rkPJ*7^qn zLNYA%Oj!^PeO$+I5Jc2FB?;pXv{CoMJF&A5V^mzLb^7_dHS)ogn8Xy9xn6j z1x9aMl{!^z-X}Hd0*Ic*9!Y8G#hLQoiq~KnwpO=ytO+=wVCS@Ep?H@?4OyNF%%fVC z;w6<>IrS+7_!`1^1%|xw!|HgxMB#l~ZTakkN4!`tCJPLMk%S-fDg$ky1>EIUs{L^D zuVfO+b&x80m<`euJO>GI)nuWJhdee>c(a#(`z<#P$Bj|YKvImz++i$g{`Q9_8`fM( z4Y)KnV)fGOfFcw7S}m2`4Y_53kl$U=%LJa7&UjvF5&Be=vDlXdcfl?+q8CfukG-`p zAQtk;9w}!bQtj*BW2Igf@yn>+*_oOhAeE_o;^s0#)lZx{--T(9x^<;!1sun8HpQ@! z8fF&|H+A2|ia%)H{R;o5wtU zaUIa^@p}gQ>U`K!H?E!Ad3Vz(rCo7i01c7O1 z!atMcCdc|vg(Foqk+$H}fPx^O?YE6*Z(A*kWQs*eRIp??FhRW5pc*KN6qY$$tAdc~ zS-eWpxvx#IPdW?a51Rz>{wwS~&B8fOF=+s=Mz?)Fbh1Pg!3 zHI;=c8kQV;Ex4jZgWynrEx|AlB%cHA>Ngmbujo>65R~KcughebkOh`Jc+=+DF($dE zHKGd^Ett=4S~8ITrR+;q|NiOI{CheE0M!>LI0>OdO&Y@4fJ?{{wtqJ@p4Xfl^XHv* z=gjz2A9dF5SIEvF)@NJY_$;K0Y<)hhcxw1p$aI zUvYW$H-B=y5vS{iSB^M)`&=zA%5FPZZDeWZyd-+!+=@*Z50EWc_?VU@uTt98%&}@M zme;5UZJk`_7&a9LQ5zXtNXg;JB@Bp((f-WZ{k3TPUk4hDS0hC$hQfndxaXc85s{4g zZy+1h`VmPduR$k`URD~?V39d(65b2o6d*fd`+;AztxfhF!)6EOzmIkKNV!?(>I-xw zzQ)5Fv%_31y*X(_ZQ=v0$djw(e&P}q;o!&b&*q5+Y1NLEUsV+|&FovwOdoD3$DmM( zB5AdXm-UL0+qs)tL=yLkT^th7p_Yp$PXCjl*>%df(Vw4YdYq=>A zj}bTTAl1-_q}#5j!o^v1WdQqKYybz2jRv z)!o31((wG49)%?9QY^Zs8UZvUet-wS|~mMr13A zJT9u;b}T}m;+QS&mV#e&lCt>Jqj2Ot?sdBC?_tEER4N39C{8ekcr7(UcX|?A50RE| z%}^=pus=&>6ur52YgAX2h|lCeNLVwXp_@PTN}l)9L*oVxiC-y}n=33O!Fqgj|zm&qb(9 zN=(}x{j;%VTiB=3?P>nBe}C0je@= zNH|YY0y=F51zwGu{89pJuhHAlvc+mJVVAM%%%fzg~~?#d#V3w-BDI?K(H)$POA$5Y!kBd(4IN7Y;_DCwz7%*zKqs$U=n? ztuKuUAA`E%MA|&(>oyaduKgG(03}aqdVIoUT4r68Q+t{s$C#5!ueY1J9$#2pKsB82FPpCvy=_PhZuA7I))2?#B0x%pi0p{IX=~i;gLZi>R_J)2J z)4*6w`kC1Z*e55A`fhPCW;q3koc*O}d{LcRwRhwD2$(9b)Z3_QYtI7u!ZsPj+g|hn zpX;dp?FA9|2etP6gg1rlJyojuE(AVZecB+aSViOZ7&>z-3==hl=h(;58iw_LlfXp6 zC7+9QOF7)yIA_>^v!8V+^btj1no2uqzT!UMHurn;66_(XxJb|Zg;K!YBn`uJna8jE zwC+@?!JVuvju_z@$q8Xj;p#O7ODhz2YgUFGvX$|^?c$3gDs>8ruFIt28^3)ss*h5I&m>T6{+&ETbRs4T^J*hD>X5yr z_WMn?TtLXTx3?FNXI=4JkD+(wB|zJ4q6B2BmZp7fP%gLopys}pYSo1P61W?r!g;y+QRF1MQ(jC+VmQ96KOp5 z4l1uI~!8!n;-9MS>{Y9pcvw$dy(Z;W>ZT z=qep-bJ!?(L&U5LKGZ_8KfX8k_n@o4zM;@*3pts|bfog%phl+Rq)HRDqvdcwvymJ%d{p465)iVj*01$moN?%4 z2Xt}Wj#{zpb=o6kvYh^K0P@#uM>*jJql+~r>MW$Xp8fYFxxT;Z4glVFaJIp^!^5Nj zW=!OC#mw8-4~Je>?(VURZGL?ZSC2&v)sDky-@ghUKLYtiB$Dr$ZtBpqTWrnk4Y)ZY z@SP2FiCmN^XS*-hwBJrwhv19~o(+FwYik4)2k7Nf0OP{^A3t<8D-I;n-jZX9+-xKP zKw$Iz=}^+Z1}bCL@v{*Dpi>}m4v23vBn<#)ahC5Xz*=XmXOTBxey*(h57^aW1C(C? zeQE?&Y4DkM6CR_7kNsc2&_yCb`8MJ1uZ>pg#R0u)RBBO-M01&b&&tBt zRf<@;Ft+Fu*85No^i_EN)td~`gYm{AgKq$6EnH<|&bVY_dxwI2>Tvtxt5kWmKAv7k z&6LpXtKApk{iOZjJ$aptg!rg$<1eqW-#vet=(JX&F1J6jeM09QM%C?yy^NFrTd=oZ zGcpo^6J*TQ!7!dK|5mZAo20;7W!QQ+t-Z{5*+^?OoOM|uvW&W#(@dq;?yxT6cmCVF zZtZgM=+!s(?u2k&ZaO8;6%I_ylOp+4$GI9=xue~ZoJ@YV30X#}r@nU%aXkgw*#R@a zq8tJZf>n<9V^Z#IKl13?^Ta`BduwRc8D-o~$eB>qfy(N%VwPCaS!7bOQphaISAjW>Zw`-L{{;`MB_gEDK zz`$Fl7O#Nknap9f;QwL$HR!PE%zyNZ6&Re3j3_QP9LBzu+bsg- z+F$tv1ZJxY?rP)wSnwi!0n4`b)gW|IQc~gLKx|-?I>Po>!?C^9X8@2L$R0>yWK0@( ze0&7l<(busf!}6G9Rf1*JHS&pSHR=&S*Lr%GMMn&znlF2Yhw1mk-TpT3>_F-f4l=F z)PhSoIaVz{yx04h0%wX6=}JDSz53Vj54uQCi9@%@1J+Ud3u&RyV55%VhVfcTGqv8| zhsE7f5xu0`QeXnAgo+3dk|O*Y5UDXJf9YrE(MnN_0uFIeu7>9n?aTu9|_c8y*amRU~TCQCOh&)A%L7BORXryT=C_(?XG-r0U!s%|+Vov!Sn$1Pe>ucXu4a z=0#Sb<@e09@!{X8>%c}ua?DU%;jhgzkZua$M@hJu5Qd^unQlP(ewXj^+(w z!+58sN9ljPHj?><%B7#WLhB2j7+Obj1Dr~>YWs48H>UtYiSK^K z5FUTi7hF`%8~L-3=c!*U6>~W)Xh4f~_KOTO>=R|KOavJot66|B!C}Dd&JSgg3w?&9 z)(!L*w_gA{9N!_Zo8vpkeB?&n_uby|mwDae)v+A|IxNEFhrY1mNUq4$W-4G?qQGS? z2-FX@`*UfaFDiPI$|A6x#l`Em+~hXI3o15g0ZRXetzV?XGCbULO}!q#rqG|qKscJg z&C}R*oCJ$`7#+>5+lL?IAkp*GaG>v^9Tj{KE1+Ipm~pwYzanmO9(I;)JGp%)B;H~wpnVx)%ftWn?7 z3u5z8J(GGB=1sgwq3q=P0jQ6+lp!P_SLBS4wGJnUAV3~QjzJg`;>LoQ=2Igs1#->t z3FF(w?iDA;Kxv&(tU=N+Eip3l9_e-J_@LE5od|tfZn5Z^LS+p#TCtzG zVbyi16gm>FCsxAFvlLe%=c|P|fbaODE|;YOBD-1dR9J8KaQdXLg*gE2ja!4EER+J~ z_~Ma2Ph;R{@lRhHKv zMZ?O~uR$KG9Jjyr4a`&r~}epHGEm$2dAdfvE8`x^C&zSO7Tk|*LBzR$G%5+>p7V-bN;I@F{J zrYiL#2gNO6HIkWq`6VP@f>E~DY>Z)uX{&#YDTqYzy!AT4%n($yGs0S;bojTBAP_hh zg_M@fNrKT=)Y&*{Cz7zVRR$QL|{UA_U}UXz=h5(lovP>7joS9$6k#%2jt9=6(JhZ z$wy96%d(vHaql0w69)W=c#!Q#b;!0 zSo=n(gLy0BG3K_zNoRAShSVSJ^7I0Ww<7pA%(!2TT+os&&7n3BgulpM9U%4Cu~E7} zVFt1qI8+@-vksdL@?V^;E=FJH2Lktn!fJ#Ce1W*U>tqS;BWh-hK$E^vz8OFDrzI3Xr?wnU1BfZfi6mkw$7K() zGiAQAglzEv)AuGRNR(Ob7nfv0sl#FRcO!y%5W#(ZRjt*0*YR7Ft&b~F%Cl?K09h!{$O-l8;D+y-&HhanIvrjj` z&j=yULPSl3A#4P4_ah;tPrwj2MBnDc4Zuz0-RZphNu#0MRbX=0{@w`tFO>`}Uy4)> z*MT7FaH{abCW~2!y#!I-gZPZU1~43zRJ?415o&PXi+zp6Z*Dst&1Llk;pYDi)V!5C z5Wy)pEtHM;iN{e?_!5gb04jPCiXuVdX;3!=oX*3axS6^xaY+^CuVU=X&~_K!LEXVh84k9G{BpCv z5`~eg1vJj|nX3*#M#cLF8$uQY2Q=totf5%0-<_~uOjvBdxObX!PDnp8i5@Pey&61h zw4;zn!OJPh3@hDv0|)Dqv8Abe?-hm#G6%IMd80+ZV?vV~amf)(SJ}s)?463n`0SBG z48dqo+PN4aY?2|XF*T!IajavD28J0G&ZiV<-%(6+d&7?BFoEp#so_W62+ zKaMvv+gMZbxw86vRDEfDus0LuQD#_Hv?4|3PyPA-&6$6eus^R=sS&|DA<_kMd;Zvm zc7JoV#K8{Uj5_=)PE^I0bh{0tJ+m*%xY0M+Fw&6@-x5d%pO78n4RZrs=Y7TKk( zx54MKbg7YtQkEWabZ8ne9ko`Yh$@CPHOd{B_P->8YM|I@9UMX4r%G9+8oJkHi}&xL z&b?yCL0Yz~MdT*v*r-#u7+Jw_To7|ec8jZIcUcpYOxr4G#(dW+=F#zu8v^(Ft-w*} zeSlH$G7_mBls71lK_+n7QYNs?@|^c)_OIYCSr`3?+<}PMW(o!R5a04s)~eDdnLLu; zk}jwpbxw2zbkh1Dv#2CU$bv?lPcE{@ne=)DSxXK3RCkE1fb=J_)q zWp(%_yUz|@&>*?yed6;(&>(r@ZaLI)46^-Rb9Xh;-1x31ID6(bk&g@vy_6p1x%W5% z;4sywbv|1>DQr1~e{pN-*a-twBVw|XK7bf#lwH<0bcaMp+m2BrCO`kUUk-0oe=N2t zH12S%F%jfM-DyLS*nEj2&aVGT$Rf%49-*edjvtKV1?rrH3nG)MS{d@KXxJrl_JJY0 zxxm>*&9U_SQkVzJK(G`*?FdA93P1F~NT}vMw=QN24{bIuXy+5XdvT&lzJ9`@SaDa@ z34;FnK6-_u*uj&p`ebAyvM`{sAo>lU6V_7%b;%SXk6OR)&kUpIKp_R+_dM2Nyy#}C zwA8V#vcnjb_~}MVit;&d4^=0FL2wj5qsm51M~T=e3J)XIr%k15s1b^v2ErZ-#L)pc zl+kvXS7FoYVdf@%D>?M_(pKUQ!V_cVuVH~`il_XmE+Sa;#xX>IkT?o+Q3mj6;~s)0 z-DuPhe*medR(YMVvD#R4b0YYBZdi>Elcrn9zszm1B5C1;gPghIuW*}Q7xHtoLqWnA ztj}5~7FTCWq3G(t)LbY4PJ;h|YC z2ouuN5~>Z zFRdop4q@=@{+y6Bu+?J6@G_J|aQAQHK53RewjF*j=Fa<|iu`ueA52L2-K5`p78y5+ z#7oO$QN(@?lC6hY+D;^HRPb^@k^Yp0eKS+#updMq?byqc?Euq>BCL8HG%@f9(HB)S z-8^9;U{FdEo6CLh8Zay~2m>c(po}|h+6^mA;786(O2btt=gz~Rf(>gbZgbD4r$3qsrpdAa+np z9)h}=2^=d)o?PoO45yK51COTXpwmT!#xojel*a_aY>APYGn4oA3P0c27^j=%yVUii z0x#ZvBytG$X)2^1il86SILOYKM@~IjAGGT*Bi<7TLt8M|M3KHt(0?k+SRhhq(2sl1 zbM5fCKH)$ir1Dn$EAU+RHJ|ZW<1IVh3aw^!rb|e%4ba!~VE>C{GlYZHT{q_ zy!{TWpoAf8$`<ktE7+Mf=KYrFs75;) zX2*aclj=52+}#+Tr{P#N9+roIqR{7J5Ik_N*y|vj{d>1*66E@{cm^O)PbbnUq1lxD zra#D)CT_gShFY=lf{;7K&~?B2WT1gRMlol0Q;iq#lnfQ1Z=+oV?=v=|8*RM0+#;EA zxHt#+?|?Nc^)yCr`AlZ{dPP_n{swL(wa(do7rEwx8q-s)_S=7?#_H$2grL2q*mb6^ z2(enm_Om&6CIURM(z53}tslY5lvO{C#_Go65^AczEm&62Js3teC#Sqf8M!_7!oon! z#^(YVDq9#|5I<`;`9LdV&OFOo5i0LJ$1jG(-pN}}kv#8kC9*7X(CGZ#Rd(!H+$j+0 z_>0~$Qk??mJ6eWANTuFFs>p`aF?nL9XyQOzUtJVWBrC0CL~kL(TdXkkQJsF7`ss^P zJXXT1bH#UtH=cpiH-kq94DpEF<>>StX1g6Gn)&(emog6!cgaPuA9T_enWi}TC4vag zy9@+R#>xMh^XJBhxYZhOo>=(0CF{Wg}nE3GNqsXl})yP9O!@L zq=zZ26aGA!Z%4PENW%O0R&awR!V(UXZgq|q=}Yh%I)hqtIh5IkB@{7U#~-o++?~e3 zW$B{P_TZJ+9vlpLy)0uqV3##Ybf?QSbuqiUh9-h)hZ2qvP86hrZW(|J;>&a5P;XNr zCgRYq5*o~|iLn9UKq(;|OyJ?T9dE*Y4SmMGa^Im}J$VT}Z7;L*1?z0P*Nk5E@FKGJ z_k$8(0so!8%>Q@#2gkMvkwIp#@HbdNYk?@;3KCijgcNy^t*Y7xCrafv?r3ns?5=)Q z9MX5T!!{635_=Y-aU3KL;&_Z1^SWy!)3aL~#385b&A>MovzrDYo%PU)0r`V17gvqL5UcKEy7*jMk1x+0*|n z^!T@cpDm&M%od7uOfRS@_OE{q^;76rp#2$=oG62HVEJBcFu_XfC;{tzM)6*gg>LgW zDHlLpo^>lq4!9Cfur!zr#fe?l5FtYgCGRV1Sg3!htTs8gw794qI%a&9P5{dK(NfhR z#129X!FET6Xiw|w2R)! zK%}!wvTYXqg2y%MHuFBGUOX-@gdt%BA;EzJpzu2>oY_P(`5Aib)v_3t>%-Hk@SJdU#komyE2cW=QLg< z5afr%Tx2o7=EPLGUPC3R!b+&AlOm@VRvnkTQS$f*_sTJjWKpXY8$Jx85vd}2VVhQX z9|LeVZ(4ZV?R;!j3<&c>aDsz_f$>l#hgn$l(DiGLXx3{po9JOX;@&92S9*GSfMJ}_ z=ovO64Rg^`o~z7pUw+&Lf9n^o2ppPEau#3b0W(h-=2yjj54RVuU%xK4N%>rFybgGN z@BRHtL(|26{}6njoy)=J6woqm&7C>~hmvEh-KTklRD3^|!B!JR6{Nu?~yYUjk|?9BDf$V{GInR*q&Ym6`w5hre8^{<4V88wPw_ne*EcADkILH zv^H%m&^TaG++Gs2`HI<3As9k|15;uz8pgKdBVB0~S-nX*mbPgbB%tPF+VHmN!SR6G zfgNPVNyQsWYAVm7|15Xro1PbT(Je0));uaTsT(W*?g=YgPJ9l6tGPFA#&=Y`0`BLA zSMBYCmLEi8I^L*#;towr#`!HKdEdv@e(vq)C;|gi&2}H}D5W}+Bs&Y`XjDTPBhrmL z4=q+qG{#+eRNEBJCO%d#`CN&E1gpO%atf*q>abh_KE#5Tb7l!LV5lG^;J@-LtFxj+ zhj)@IHtt9m*C!wRN$v5npn*Od6|v02tT9!vlC7s3rKw zV(%vo z93viTpYt;)e0b=fI2loDs-IDe$yMbYrMWw2&HdK{z@Eay@+Z&%JW!%=JTbZ0Cc}id zs3OmQXuop>VY<}jSrY1j92KV~F&+Vo0turV_Cc^(iT&!>huGem`iR~q^nRlYr!MTe z13pLVnh{ZFYK+O#+`?J^1AefOc|uGy_ScO7s0|4m>#8wR{72{kziMsM2!eh`#f37_ zle6&@d_N3SkCzeXo`)NOU;+Y0Gcb+g^XA7BdHuJ+7aFty4FURrlm|a!DQlcPK5{<; zP$!4;spL92jCZ%UuUgov_8fHD%W7-$e-sU@{P9}8y~qUEeIp~GuK^;pU+Y7*0dTO^ z#t^sJ>?-@m7Xg@IwN{P_PQDh+0VAuI;03roZk!1Lg&Gj?`& z$rLb^(&l(v%wk36KE2YW1ctR@D6?)xo@=4Me*FSMqv%{R31EJSPu2sBA+MYgn}rOX z_qVsVmLnM!8O4t6Q4%!32zcl4@X}rdkV-V&nl%dW1H7hznLo&G(SNrn_0P%mRGE5F zG6Dhuz+C}gda($=rCLv`NS>b@t6KMKaY>2QOr>tE4olKNV^yY72{JM=w(#{T;O@PB zc4N&AFqAVoJ39f@W=t$B2E0ggww4;e6ui1xd9of*&Xi{ckVy+&EhYlJHs2e-n;So} z8=xZ#j((J&0dkNJpdAjd!`a>U=X`~!&_ZwT?u>j6z5)J?&g;GN-$#b4+iuIN_P8vE z0oFg@k__0}1_%Il!CPBf&t%lv+VKGTmg_$sV`5?e`mGx|<0rWr{6AC{i|#5+mGKoJ zIN$B>w4)-_%7B2BPh|xJZtVay&{x$hi3$wB|GU~$WkL+B2?5Ls;B+S3gpvkuR9RCO z0U%_5{wpB5El)drl;II)lR>}q{hX}BIV%9_Rr#E|E=l2!BT}tJjqu@qpwy{?j>Q*4 zS=6h;9Ewm9$Q968&?-op_$$X`+@hOU^P`poJ)PO69Au7;D7yzhfeo;_Xu4G*#sM~j zWGFFvxO$olktFMi;TAmD97fPVAFk>=47M)1E1D2gxoOe#th3$vrVhO)WhfMC{LnhT zvb)yIsNwuWO#4;6Jf4~Y#F8-ZdAuywu`sZWB|3YNSSK#A*x znB)38mO`lg{$`{p-)drB{Kad&gYzU{4Pac`(7W*4m!f8$zHQ-v&UcP&*#qW8B4>Z1 zNhX0OfAxPvon>5>%hSi{ZYh;6X#oN0Mv#yO=|)1jK~hR2rMtVkySp1n>F$PS@AH50 z_|X?~kaJ!4?#|54eCIdUOALY+3vvQa)AnAjw-vocqBJQjrhh9< zle1X6>Ny?$VidqFpvl;5q|lMT>pBWk@Ukv1^+#zz!9?v5D)VDrg<&5QZ(9#5T2@_M z9oUf9hYL$})@}QRMSljetsU+=30lidCo+pl8wbbgz`OSqOEiEm@F==voKDuWgGi$j3F^X?&uXP8{i1od0yGX&1%>-~3K|}oA+tpiy54vEu za)$hFepcB2@v#nQHd9g4b_C)Fe2=GHfYE8W+fI7>_U-$&$Mcuxlb7fFxwhv^(Cc}- z_7m_BURyD0Vf@$g;a-oMP_fL~jd~q0bUCF56Q-9}bGmKdm1yKro56_#CBB#Js4s5d z_<%mRMKIU7>}$owBlz-h{buWVOqpMozV;lLBGPu-^S%}QKNDmCAeYJw2DXDAP5N@s z_FS~%3enA2TwJWRUKMaYUVi^$9Tm)oKG6l;{tfm!t>E>6KKBEpN|ZBYwi>?ub}C+d zsv7u%)t9GLQqcbm%Ky<{@}Rp7D&AVv>w~^G00jLl*YEl_5XCj}o21LOzjSOsqA`eF6uIhdg|sBg3MfFAweucsHb0q8Q1HIDh= zDrU6gBq7l!$><0cHe)^zp7Cw7Z$IP3N%X%$eFro;tWm?W$gdoD73_%*dW)v}UC}cf z%7yx3m~T{J$%Vh@Wd0oS8(k6Hy`Y7aXB8cU-SD+CH*J_($D;f~gou_7P4vzRCUcF( zfY=m)VvR-5`z;b{In;&_63cHl)ec>8<3fx%Es=`M+tO^;8G?91$2khdsc`er=P&RD zDW5fB#xSE6b)>r}eBl)T69q#7X`_hAIpvp*ryIe^g_*Noa-ODt8$$(~j^`v=cFNlX z?u{zx?(U=oYV^Bu`<}0>cU&RL0gpkVNq<7;9#0o{Qa$c2KYVcm&gqNbX7<@Do;XHz z8jv~lSo4t(yso&wi2raAwsg|8lNt!&bP^n{PrESr0fFT8;^pOeKJX243cZR`U1#Zr zVwv04I>W20*;g-@1WD|6eZgMatC?GAej2+Wa+??Fvqk?~fBPW~xQM&w8_&nqSK%`p z#6rw*UYC*Y1YZ2*K%etoDeuW@z3oN4tzq`^FYlgVFa_Z0dMVcoj91hGg~iRj@R!B8 z)>@B;YhaXVaee_j>;pJcV1c!HQd+_BncBO2CC6twNSF6x4VLt4jd_f&UX4Ld@J;g$ z|8Vk`Ji+I)a8l4onReU#E>|wibCi4M$L8Br$XS!)JL{GPa{<*DTcsctNuH7i>2r4Kwd5Y_;`W{U@4$C?XmGHZTH}yijm<>=n9D4fUBbo zH2}~|*rrS(`(EV`vpew8rESmmn4p8$Qz7r5ag6^)i#hIsqZmkF4;Zzcg<3zZAqtvW zSX5bpS5l_u@Opk6^EkE#hwA1-!Z7&AJRk-LJf;Mx$K_sLnt=QDckMaAH;e%XPeX+9 zjV%FUaV#w@SMNrhkHJR+*0KL4r9k)N$#&>_zjDTd+$5_(ld1rHsD6{GJ`(rC?9Wlr zZvXX8oFGDB!OY_0EL+?M2gJQ{AZTlWdT|!O!OuqMFx84{xM5^DI~+hiZG=)3%-SH`<iEB^m+U|bF)D1^)yAHyc93K)voS`(SVeOHmiJ*)J3x`WT`33(N zFCYVM2z%Rwq^7EW;t$6t*4ETGIE4IdtkIlv>|+0DxeR`>8xK;AT?p7*oB1!Lq~PJ) z!^}?RNax8UbPfXq!0p6+HEC!oF)0a??^I~c5i}XMm`?l$fGmT3X4_}W+_;;$3S23s z;NvOS++IK&;1c}0xfx65{tz|5&cRVrT`fu17xH>Du=rin=RRARDJ|fCz##f#RGHEP z^F$}mue^P6Cq}{QxIYb8(l2QO{AptkFtd%02fbj-4=Bbe(?gn?SHM`H0%h~V`3iJ& z^wzO{mJoND4TV{w5&rXZM992BF_Po%-NoG%B#2XKb6Nt}0K zAI2^)_)gn_CI|=>-v(4z{pIPi#o)gj2Ph!$S7T{{ZFO}wp&wW3k7}2o>Pru}!i?2Z zcZ{plaijWY%XGbF)xYIT?F9q`6e#6Ej7X#;Rf5cYUf8EB&~Qx*fAbOtdA<%sUXOt&tq_Nb!69B9LTmxa`YbWzqkR>n> z6oWx5fzaK^fExHIfcPBjG>oXvpXG3t5!M zj-Mq9`-}3@e;4A$h&zg9?Ekna>3_7zUl0qklr3yzu4Sj zBkHRhuCKq@H<+?h@Hbff{iIv4P?7a6;k%wmY-K|ZUCf(dWBbHhCRf9~LA)ZC zN?PYWTh$T`$9B-WtspAe1!e=`{}QunSRRaFK<_d&GgGg#O40pv1EvIh2X_Oo{lIPI z3B+TbOO|yY$^jw?mf+d|%v9+C3;-xxQ(xL0{{r9~7%X=XeW>ZUZW+gEZ@&fCi=}$o zEZy`va4p&1HW4(D1X3qXYzc?Q$FYi>?Cd~j<@4N?4cDRILEI**y^_)txRW$CHDyig zf_4JHN_n-&qtRXD1toN?pMua1Ad}gOJnmo;j-@*Qb7W-!S^Tu{eU=~uL@%*2G)e{2 z;NPVJsu?1c12zvNth*AUP)KAJ#2r5QL=pKFyjmK28&HdwomHjF@!D5}PF&UOVqtE% zoCIO(KHzI#wm-Jr&6Ih@B(&X)SVM#bauz?o7;ic+d9AtXFyd8M0j9bUO^;a4RcwO;KHYSMqP~<~U5*IdUl2@!tyj zUUHSt75US6uWxr^^J?QnmQhzIeT9)=u|-0#-gPtlE2V$>UF1((SmCXgQ}s(eZymej zUhVw`zf0|4sFrTin@>wX9wI)2^+b(Os-++lSIOxAbF}25M7ta?htO0!Oxi^Vt z*cS>vKD*)uV5)~5ucr{K%r{BYt%)AcdW>w^aSZ-Ot&L!{6Bi2k4UWCC*F*0`*g z;}{&th%m&nB6XiPUYMVU-+@@`-_pyEvD{G2*ygiY1qmiLS*59(E$g!8&1?6$(gl{> z7DpiO18$_B=sD(rLARK#ererqOsLA`zd0OAYrQq3?R7!_@U@G@&CQK1VHn&Z!R7`N z*B;Vd?$Ur4=76FzW{%?pA^LI4%?j!BX<%b1z%0)}_}lv8vfk-`=F(hMbpbpy&|Rub zdkwt%k9C@)5w}#t_R3-znyG~av!&}U-~kS+#;jIb|ElSbxb0?utkU(8bzN;OiNm;X zVlH*m=e98b{{gamKSoW58-(_d=8QOPrv%F-F!3V$95@YdkAm>r;GwRD6Fz`z(JYwH z(r85!rEdZh{LRGx*T|N-Q=-#y4`R!kt!8C$|083ZTKk=dv_8xiOZyx{lMJRazaEy~#xY^lhT)lpg&UbxF^d z1fR19I#Ihm=O}_>_)Bc#ZYK>i@=U$8ZeD-iP!G}ukS9EIc?8#;2GE1>+m(=?8;9b8ZRg-ku=nSgo zd6%+}Ac{^3SF5BFw9lKBpITBnxXz)4X?`6pF@2H}O%;kLPM)nzi5Mj56(fmdBI);C} zBNcq`eywN+sKwG!j4AdvZ*YLY_-_8?-inyK`w-RyMHtNB0C&ytnVAYhiLuU~n0&{# zH&8BnJ#*Wn>~%#&Sp_+JhnT4#D@hR<&u464Z-3r?XYZixOmtIRwJK=27(JOTB}w;l zXy!BB=2`uF`WA4rl*^j}3+DIgOHhvih+rP~XNsB{8@Is*v|ecj0SK^@kn~YThCaC9 z12wwclM_L}v;zG28tju}kcPCju(;qWD=I3gs!~E?1z97oq=~r;;1WAA!3vO8;0%BV zXVj?PZ8`C<{`PGLIQMX_jUe!KF$&uWae@-vw&z5PQrAPUIl=x9ByponkVyp|9w4xw z>tY9tMF%6OzAu3Tcs^fY_~ehB_a$X$GPt3fZw-M@>@oiq zB)|Zp4XAtBDR2u1G=BUVYO!DG0Z7Q-{b>EyY`9%pTMIrM@R9zn?}7^FmkXbOey1P! z;%GeUCL}D}-{;v3`2ZOQ>G|Xa2-t^}8!K=@F(iG6Cne1li{-y9YInw9yFhAD%aHG1 zWr`b8k5wt3_Q}5dF5P|`&jJan4Y{ffoGOm!5tt*l$U@djlM=|B?M#;2lt2UYFwJjDA6J|>Z+)+d>zjS7}t z>o&Vf^!95a9q+lO=9&pUPmF=oisxX#$x^5iS6I}5k<`-Dc_V0XlB9z~`*^|gLnX-c+7f~C2PxAKh%6yJQrHqiATItOL4hquOONx5 zD{Ql|RQLXyx+)|72ed9npr`D<6nS*5S{~{C8Z#Ho>=hxuJKBJMt%u1az%SQf-Xs za1nW~`5-=@gNyiWpWyWlC#@gdv#0f=z9bdSGxhG#5iud*ID2RU^1u37y}^bnFP_%( zD0$kAx6F__^SyW2zs?nUkqH=-$=lwbY^Wcu|l&UY>e5^tA#Q*u0e z*@}{?1&U*8PiS2%vQu(FWxGb@Py0+cq|$M%?5BF)=xOpC_9q1>z1CIOH^Qe$BVUTs zu3CZB1Ev`gXMl9vpEg`~?Ef`G2{7vQba+iQ+lwL{*%d~=*YmgyuRmd@LR>?cWq(6- z7%o`;ucjz0L_P7HlcM;pndx!QH%}i;0pC~>BxG%a;WrO+1WI#GU2F>4f&JaN!bKk~ zCq1oXTk40taAzOg@j}Dg3-O~N?~9Vh(7_L~x(TCwN8l&BVn@j&g~uhuovF=4QKQlR zWpJ0F@T`o!5I7X7sjL}X_|5X|Cu_Gk5v9DPQ@cLXK(ydB{uW) zNQc@B@rets(u0}0PUPKkGq)zgXY7;$l(^E~(gnz&X-P6A7_?+j{bdcNRN=$@C0 z%uk@m{lp6>su(v@aXAfjvR<6M=BrpM?59PR{I?)gyQe2Kt03Clgd>Qw>vx!4$V^ao z`7b~(HjB7jemA5J@`a6^(_N(S*p0Y))9+iv- zjv1=OlsMTfWF#LI?24*a<2E1C5Ybr@&75O6AX%7Z)M@8K{?7w>Re5iUP3cp<#2uPY zat~SImfiVm%~0r#Jflqo9W(-mD;zjYa`-PF2!`IISnMA`#AHx__}$ncUDNn^jBQ>6{izLG9oKG0fH#nc5G-(x5bgx zA9h~>5*R)g){pEoo!vn&CN<_F{jkT!yQFBQLT@g~EELF~Iep;jp_9qZN4oX(KA2*S zvrK=45`zzq0fJ4K$nTNWClCpYVBeG{?=iklHktgPU0|ZB%+z5mFGaPnFhAF-*CSz&LyUp$sIdM5TqckUPJLWqM|<>?h%t_J7!WA-yj4 zh0sSRJuiUnh5gwrjby^AZ{9ud7Yzxsf!5B}@B8f+I&vQea|3Hd7rjCax$?q&S$o#_ z_F3EU77rK2&?i<^QlC4GSgM|5$!vZDCJDzmKIXW1fxCzXs2g%YxR{V@$%M)>nV zcSr`%9r-_l09kt4PoyyY*v_vy2?yVn1v3^GnBUEKMa}8W^15u2iB*yXkeOzgkN?UK!(zNE z?V`dS{R^J9B4q$H^fB~np&+uSoHITrVuA^*U~M~}4g;?rJ%yKYuzY$ULWDA8C4?N; z_GWy;i*Or+T*&i4(@dU|kGXL;pGsuLPKFglQNglGWkdmfa@IAzBa6-D_wTr-NA@kj zS=9MVo+XPG91n6iEW36LEFbj(NE68!=KqHMK){7yg-_(Hn!%gd&FWeBEI--XOEUCT zet+$@L1(|(XckOwL#_}uGKjViv$tD5w6cSPQ+mommGs#bVE-41DS87i)QRK|z1cEq zq6y)&ZubK`|*#CaorH>~t2UrmSlY{r6^ zXNh(e*nfn$ASBoi!KUjiju6jdR+frd&<1P;_aC@|t1szyej^rZgK zEPx2_1bg6tD8NjyNidw99L-bJg&>gr@|S)zu0cIdrn>hym-i?-R>%)RExb@=x1jV| z5wcz{&)57+{ecrpA+7k;R+gU^Z4Nw|Z-u|$J*Nvr68R8~vG$sAw5X9pK)ob=6ByJ$ zB!>=zlGO_xhYN#Nuz7VWj*oyR1p{<&(f$42tq#Buz-NpJ215IJ)0C4Tv0x)5Bf5(F z4?*V$qt$z>6@AmnIJZgouFUB!E?!va8Sy(oT^XMbui9XyKRa`fkYQS}ps>u{)!Cv= zBfz4FO9#=^pmnC4F96-)XBd?k-NS!VMyLTM8mwmmh-XAVJbT={UT{kRTR(#^=hi?5 ziRHCI03{0&JH;AC;a6@f+BnLf^*yLHGkCmD(N;BuI8D;bg@3j;2;-aIOvU|#S{J@p zdqeT6NQUUW^6<={oY;ixN+!8go$RUFTYD`)B1tAy~D41#?mb-{u)j?H>62jub{zAo!wAEMi zy7o4b?WoE~v9)UWn6v895%i^+?Q;QZDmm*q@*L-zMY^Ir#6fG`8VT>Pn!GVaeYEg4 zN+DYun66O#>}U|W|8JyIY9~v`SH|4!|A=S(NB>nmAzxjbLzP4ab4Aq>)oH1qB8^hl zMj=!x{e*Inwf3q+0sy)m9H=C}B?}n&LKA0+_c>fhteBh#bq}4>-XB3VNOREpJROc4 z!RE6K{k||1ADzFke`l@{mrwxE2F@C%w&!0D7~EmQXid2)`)toR50%)#Qyxr*25^?_ zIGlP@$^1+WPTJBpZUuESI{AO#IH~k!)QSg3h4HF1=>%skv~5$0l#VuwuD>C-!_-1; zdNT@~IVpY(-pF4nYIvrL4}B^NTv%l+jUq|Rck|*3k#yKILIKudeoegVz?_r#O!HvT4r2OX!~=Naj8XU!*Afd99oizxVwYdg#>kgBN6eI% zlN}W0g#XhNL-|kE2(l;TLkvKl6ZGjPg+@o>UOyFbUmGSwd2O{XD%$H2LZUUzyj!(t zgD%XWhr(*SeN&~CH6hY($U`Hjf(Z|m_idhOl|ouGW4l%U#zOhzCf$o|uNHfKYMWbDX z_0Nvg{p%qvGeg8-m;F+a(m3C*NI_Q8Hu5#f zigw5fE%TK0;rb|i={#YndIz{)#loRG`V~K=eS#B-&w$o%hnZ7M`tf&|a#yH2zf$FK z3JSIuw1985e>qF0!78oL(T7GB$v~;{Mwj%3Yxd^b$%6hFlI8E*Vv^^L8-@5C(W3{| z5)loJP6_Wo(5~U(VAMB8QF3l0x0_tD zVY3!3LfXCOm;PTFk~R1&C%vYdPM&hj=YY$C9yY8HF0fMp+7>Q<3n8=6SNP4j$xJzt zN^?smB$Ths z+E4UX(I`@&lC)2JWV&n>tmPF|GXKD3wi|aYzzq5*%5aMW)1eyq;6}mk!W`w2Cp-m< zKr=>dloeA$p^B<&=jfY<&en92!Aw%f8}G~hr|V6&zJow zL*wJWb!Ie$CklbWXkrU3erEgL7&lG*_CnXI2Aw=}0Rdot;2UM14`Av%rIVKyzx5f&xK}^M8$;5Qc%pbZo+D>RlRB|J36Wjc|jeG z)}>5u>*6;-8<)xV2T*)XSeUE-A*eQnAh>)A>Veq;Scno zyi9nne%iqwcwIMX=W;D?TUq{ROoF98cR}3;siFAETXq)S8KTqWNDUV{H~Qi$4P{VD zX7PkZTb5x-8D)fpu~&W{s?y$A4@jcFPY^q6dy`HO&}oR|q_B$(1>5qBPO^3;-|Y4| zmi{OgTEkYr$RkkWGi)U^s@^yT^YSpa5B{)FqWh_qa)wHb#c%(<8k}Y6Ue?mq0OeWy zIMRM)!@r&Sg!?3x;lwiee`59UkA?m67wA9$sVY(E;Nx!&IiuDUsYe!}*(YS_9~)A9Vv;y{7Z z`M!!$x9QA__7_JElTTi-y*6KEX1T3jS0or&Me<8*#q;3^lA62u>%3}N_ty!u{F_uQ zfE??4&00nXn@Zj`R`GRMP=Sv`jwT8)%Wm`IHHyCmc?eqynVbY9$s0ct3-R-ri5-VpKSvX`-I3hQBA8uqDV);dlmu ztQh;q1>Y5?(qX~Jnqm>z-{80$zU;Q+>%GFG=(eC`?gBHbYW?Z6BNT<)WG4!%TjGhv z*l|&gNk<*Sa?US&g-DXi4Zln_D6gp<*bm1QT?!q?!11-i6OpNvuGOat&M@9K= zZm)U0*13z#$bjw^#e_>|bsz%@mhU4TEZ2VsbEV-QOZfAK&X*9-ZOKVDXms~*XH>pM zClZcPBv3)?cGghXx^A=}P;J91jP)=od=SapU=U-uPnbS}pp7AaczvG|9BAx;2vv&1 z`B@HAfEA-=;|5qb{-=NxLqyZ4;fp+!`mktKfiUrE4}hH zaZD3&G!4M8&R5dz>~A>G3}*^e09!P+We4EQ`(x*}q!AO3z-d72b1FSyW^J z@}s2(rC%!yz&ij))@9?@J5U1N0pchikH50A0yqVL2Z7N!M9#~$ph5-FLjkI4D~pST zW{ZHAfG|I?v9X{WmmCBTI+D;&pdJt#GK}YH z8=A;yZ$!3sHR%gH&cw&Y9w9HO0J=h)c;5w;4&uVT9*#g6R2}^^MGbEVLI; zwtg3W;Wug$#f(yZ{hy(SD{lJ;p^b_A&iLs3SY*2@OKwtJhKtG=6L>7o!59T8XEctg zipTf8fDv#Yavu1T12i@?TilLXZus~d7qfw$*?mPDe+v-Y2O2A0w;gRMV3zf8Joi!K zON;GBH*=rkN!v>#oe~Y8QkJZnb|$?VTqd)T+ZxVpYBy`E+4-ApdjS3^$+o~Has@2Q z>wq&*=v)P&Ql$saeO`}&Se>@X4m^M7=D;)!zzD)(@{#56ff^2k?py)VCgFn5w@PyS zS702~X~0{nc~B<=ze^jo6W@awi&MF>5S21V?KA<8dr?e&hhf&fX!>Wct-demw75BX zdA)$V-#vsbEAPVKJ#B{tkmq(BkXYeAaUBcyyv!P6ZjwPd12y|AP}BklUF$(<6F%ct z78Vvj4m}+~0I@=qqEf&vfFFMeNCq7MK~u+RRqA&I2tKZ3LW*DjFai3&9}+@v-rzPN zey~V{SRVi?-g~F&KqC@i{qBjzgXnf%+v6Rmceu(IOXcLBB`6rGbg+Gt7x#Hf zz%X4{)eWm+r~D_UzIWdd&(U9tLF2a&GfS^&Ut4=HHU&bUuqDoFW%U4@#YcD?gi5V` zD@=L`W$B|01A5gxJru%0b1BL02tO<=as)}-Y0g+^@^A~eA0-3t1>xmMNNb}Q)K!?i z%EfMY?%ikXdcxul{`ny^|Y1w}$3oiGzhQ9WZI8Rso*5%t-{_yS#(aWsn5HC-k z$NHoktf_^yw#%tcMg*9Iegp8^ozWy&KOkyiw@ka-b zk;JRG=b;{;YDMsI`LcU+EcwdkCeENR;AOHERJH*AR^8O(4hroO#E#KmjY%c+L zN=QVM3<`c=fk-d{r}iAcUg8GZXoLgQN8J~U5^C@mIN-hE*D@MbwVTf43hl??IH2QDsV4L*-zekr2@E1p3OsyH5vdEfa>1iUu+ z2!;)=DO8=$szHX53X#g)dQ7Lfkbx+fHEK;*p?tKjXAuf=h*T#M3WS?W#Iu?}n6r$44DwDBzV1t<2 z+D2F$yng{9XI-7MsvX_?c~I8&Ts`&Wt)S;4s5b-H^~Ce0>4(6H>k3dN ztpzi+fbQLVEmKZ1033kdlA823Ro%#!|C6BGn$GPuz zv2YUp!Hf|;nt?E08z(-}a|Zu(aL{8&0LhQY^2J{4y*)fD2N@ikFueRS@^98& zF7GSKKZ>!@xOwm)NK}4``+o{{p10&Mt(R1DnzTHW}PF z9K#a9T6n>Mnqog1+MCnyI2hbhk_dbAd>Qq^hSoK@{znl-*vZ|D+|TzAF>V?HSfd=)4& z)mg8$f`NN0E$%xz|EW3gyIn$DReAsj z8Hme*6sgf)v;Av(d1wPL23BV!qq{zM=9KCt)0o)U>!YO`z(gQ}z6q}v4_8-LGAu|n z86x@$U{C?jGa!%_o~8CZ9GN_G!qnFGEadeE2s{B~|Kw8nxCzmml6(6J4ZpvoGd?`nDjGb_s^g?LTYN|%o}^#?0utud$bZ1 zn8ut;a!Z(7!}kgUoa*>J*i%pD;Ztr%LG;~G6-m|YvY(pri?J{ZE(!c=;U|x?XFq08 zE$|gNR!komtpWd26w+!qilt%xWXUSrhrTmgkiyP0c>A$ME@9BkdTJ#qL@3){wD9jG zKWq>Kl0cww2rGu(Q~t8m%`&C{=$o=+f5SjeUw_?b<`nWP=!d{+PF&iH;LC}R7+;3j zqCY{((_)b|lXi1vK|@Y{`Du0(4Z{q+A=}7o>`uY2F|9_6%MQvN&(ncHywzgux-$?f zAn2*bYLoALk0im6aJxQaHRvvXcsZB+Y-T%CyKgr2e$R+JCFL!L`E>q7mW0s=t6XH8V7=ayoUddA7Ytb;2 zCL-vtOD=+~kTN5+jvQa!=;CZxDrlTowe73%80GmVdn#VoB4i_Pz5#bdf1hAG(*D}l z1Rr?(xlt~WxUKsCsp8d)j3ZX(cyQY%CTElqMHYvUsnJ`~x5&D0Yi3ts*xkR43+Bw# zF$v>FFe7L~r9Uu5B7;Gpe(*YLU<6MX{S6M$NiH)tH&#emh`s<0|(9#`y*Qpdo;bcL)_*dH_J{>zE&m_y-GBd4SlBiNV9e^K5$oWGhhV zeK=k3a=$(_%aaGUk>Sm=%cG@_k&@9dF*$67V9}mDy@Z8@L5m{j$nXRsDFK=?<=6rY zMq}yD*4EK-LzP50Nv!eA?5xwltn5^EVq)UP#)hx2FMxFclt-OKHJyV zj&9M^>WF7dldZfzM4ig%!Q~%xQ8d@o`5W)pkP>X+{OXk8Jj86cbZ*6bWz`&Vawg@d z;8kwa%e6!97?gW1oZ`xhrDTOXBE-rRxO|K-ho%EtzY*o$=pg!{SIvf@+^fzMe%=cj zk1lMmWU0zYti7Iz<1y`CBN3POR)v;*<(1aoUT!xnKj!>|QavyWj&>SzB=^fTE3o6s zQ7+_eOi30OBME;ODfu-vCN2(iUHsh`)Dd|9UXITA#Z*mT@xU2ORq%~z;pEj14m#XI z?<%hv16>(US`VN0f90ArYB4u8IZ@#Z%zt}st{{DQHBEw{$J#p&eF+a$@6E&Z&K;X< zx5QPrWI^$$JD%-WeqrHDSUsL2)+--Mwm&WV7PtW$wqWrv=mKK zXDr~6(X}I~4Z%`$(!Df(y7<-TD_whJDbuy2ameyV z-`l?Jwc+7AeZ+E%Q=HC_Q!VpC;fMXX=Ly%PXMv%xhHQL$0e-`$(V8u-m4;`27p4Mp zI^G7l8XR5><5kI+Juq*DcN#DB(^1N2CeYWJJSJPR8up2M&g8_Nh0uYd!iCSOP4OY% zsasjCScq(GS*7&UKyYxahg_kT@FBqy8VWhR38znzTS*jKl4@wFxL8P96^rU-?6tpV ze{vvmUx=rW2?4c{Y)0U)Ifro2r;SogS&wF2t2kV8b|r;hYWC{$S%UADMmp~QBpA9&?mb5B%U^ zC9!iDYW}gU!XDSy#4mqR)0s|6@Spn8K(b2I=r5$ImHcX~q;-*DR5~aTta^5%P-+IW zjSCI%_-4dw(8Y(Se?#1p1)U=3qd#m+b%b-rt^O{MOj)&ZVkvD&S9C99HN5+~h;JAI zHpNOdR9xn7Sq=I%-(Xl<5tf!JX9;){ZV__JI{REG?nd9d5yALYyl>wbO1LH}yv0q{ z#d(O$k*BE8$?9>*Wd^%RMnsXIaK@nt$kO4+kl?6^BAaY>>7pD%cA_UHK5uwFmbaH}D%(4R}aldI`Y` z)q0J4V@<$GKd=SUV!cxlpWPy1$I;@pOSkK;{8N%hYtPBsR~1!3ZzrHzlV%sgU_ha) zVG_fJ{XWP{8x;@NK?s{s2wDfqJLda&&4V%Mh|Y9+)^bpe9zhHmQ?HK7+U8$y z&F5Cy%<9jCAFjU`kG+jI$CMkW>w2A|7)DiA{aCx~%(E(+S?&C;M6Q`KaFiiRPAvjX zy86@2Vj*^wQib9H0hVp`rCJ9s6Qx*?$V(`>LnfTpDWTzEULQ2$|JFs2tR0TRn_%3V z&l{H>Syl*H&zNQ}dwXaI?KFRuJ+tW(f^Eu>=q3_x+Ca)r= z`Lj`=1(nImKOPb9 zzhgwWwDY+gvqt8tntQ$&9eQ-$xVV0kcV5ZuXhMI~!ZAJnAMpib!G=mZHY3ha#V{4_ z*-t16VvkaoL|s+qlJ(DMnC9V13R4TAM3b3|Tx8m)FCXEE6tqlP{qmM~ z`{~|<`HT>uOa&biV!qe^y)_lP-+q)FX$MtKfC0}wYLt($kB2THHfHhTzy0QuVj;20 z`41oZCAn!{U%Nm5R zUMp=t&9;g_#37>pV8R=^TY679?p&q=YjJIef+hbJu9Y%IV$k;w8L|u1`)IKyO8y+@ za=ToqjFMg^O392$yC2@w+*irk!yg_PkR@MD<~6_D-?cHd&efeczgw8lpGzUAPTyQI zM#**?ffOR9`H4i5nU1BHWgX4?n2Ux2qQ(&SbUib!{SRo1-a(kIOz~bg0(|gBJgJ$B zzUn0mYG$(Ihbz|#=x;jyjS+)GrZngL1L+!DBmJG zJ#`|;@&f$=hMBWe6gxk>xM3)xlZncitfCFCd3I_SaUzv9oUg0lD-TfgPoQ2SLpG5^iO z9V;J}jzgq9?=RLS8^P1*+5H!ZK;q?`Trh$*%D}H%&sd($i5+G$OwM1yp!EqZwT2| z?Vm{r{v*uHJA5;b=1?YVkBnnwii5fA6E`ZMk1E5Y_EJ1eqImr9i!Gde%c|qV)HPP- zwfj2TWg1+a&m_X(Ap&)kr<`33bFKf5Hy76CI0*5N+_}6#v*GMCD-1;~AAr*qYOy2j_h$;lsb9zb zRlig&?&o&J)6ANe#!R9+S;@9{C2F|F+sO+VAL--?P0{fh&>~=D>SZJcmtXIk4e^hW zZ3Us^I@OdUC1ibUPH`8%PC*HlUq&M*hZl;-WV+4&GlXS4%4$Xf_^lLxQxuqbHdv1Y&cwf(}^ z-97rp`br4bA%-Q4h5?$LE*P3u_`PI{PGstHC7Vu>h)IpMySkO4(8meoV1~3$uKYg~ zZbf%h19Jk~34Ts5ZohKilo|_?F&=h|cgw59^fI|KVI!NlrX=H#EIBg$Ml+zGVHNk3 zG?S(bN};r9({zo(gj@2^5xu1R;Em#TRa8`d;DY62;f0l^qxM_-4*cDcE;89CgF^X3RZXifUuY8V}ij`(@L?c79H5ai7Oj6~jtp3Nl3o zcL!1@#+L805>%@4puO#cU5!MjkygHVMlCexdZZ&n#w?!D?&1|lSHpX-X%yCtOzj}i zFZhth2xz8N^^Q-L8}TugaGVTK-5T`&7Up?}<)I?kUj~Df<99PNqa%d}cLo_cafsL^&MzA@5T9JD zIc2DkcfWsSc9Y~shz0GSsEcHu7|1seERs#oS>wq-H5ZH4_3w}%)k$stnaEWNN{?RJ z9#wc+^&M^0$lL^H26=u8hmn|WC1)X2hTa?e7!jd(DJOvfF6eG3_f>5n{{G!`bpEYl zzOCek_j47{xaSllWuwn(DE&szB-FZ|qwn=%=-Y?SzHU2xxDPibstVy5KSa_t*~^1L zg2{(ncm+Ez%n-#}|DktY;Pcqf&}LRUA@5qxrY(jxu!_&2P^d`RaulX7-83 z)BY+**Zi%1k-*7@r7pCT;Pn`;{#V)vezec!rD(jlUK%SC*;eB9_+50hfkb2zE^E!G zQ>NEIT)|}Lw|_1a+$&8XKjjwuGov?CoH-xO-FX$&P5ew%#lpcO3Iu&4HwqzMJ49m) z(*b|?A;Sy*{B{=_RX*akxWy8k#F38Xj?In~n^HJYkw~fMw#}uP)PY_OyKrbmZszYF z)Ob1;dD&Rd@ z*Oe8OnEI7I@cl;N*CC#A^d2f<#OcXiHjxoJL)z<#IPUmnpRe6B zaTIZa=!0Q#SfED~I^wcI-BbyO8jLhOmE6LwYkU5@g6COM$Pv+|FY<6VLC=!1KKo+! zu>Bn&O+(-5kdfnpjJ%3GvE@rayVtfsNQDO}V)=i*WZrW$EIiquV5g=QmnqILxc#3& zw2sB>y_Up9Zb+@V<(=wYT@(ZPkoQc_d5oK}2sz^#!W+Lrf^1&r5~jhaW1=AXs^N0W zxb*b?X&E`4d+B*%m>}pb$<{n@<*UksT7%Y!F5QXQb{;;6#}=U&93C4CGTk~%pvKjL zgQB?0&(PxuhKs4wqWC|)-U2GBK71D)U}&TyrMnxXVL+smlrBLUB@_v1kQONk zMH+?_5TrvwLErA0wRN>mi2@4NlJ?|<*P=d6P+m#mrDv-dCF-y6^SJfSQaYvYM* z_n3xTKiMeAoY=3um^dv1NhI9k<_JC*-B7Ck_qOv(zPpj0xECgyB-yLioc3Tz%QBVM zba$7N$2QdGo|?CQ#&)IhJxcWD$#v`9cEg-p(~&fLEI;G1A)`b3uWQ0`ywVqsO_*vnHRrWc zZQHfIB&5Ynaz)8hD^UCK4yzkSSS`YozlYcv%Je51R!-&4^TL;Dqc<@-J;IO>Uu*OSF-~Ra8y3BLY7)lqHKOc|3UQ)y%C6 zX)@53XK3uqkDE_T$)Gu5Xvo_q?BEaXBjQaxL9Tsx6QM=p|x zR)|I4O3PMaUs?5Opb_DEn2>8!vr$Pb)rh^FcC#ojI6;m+mYqiErqO4DLlWDF5d7?7 z$(Pim&q}{dW?&Tx^d?Zo5m~w2DMYaz^P!m%KZ=+9t*~}M<5_uR=DF95oN*QSzs8PR zQ8ZB}w{R-oyr1ZhHYi8n-g2p?AytCZp*CiA=WSXst=`k~XU1@C*dKjJl_rbCJ2X=28Cp=YhlIp|xSp8x5H*KH7vA~dh#_Np86n^kz{LTwwnNV%bN6bPm zb8Gbl)M)OWx9yT?`P6VFQMk~eD2*TY&l65(%g|}xvMbIz&u-AsDtOGX7oL}HA|i=E z3u|Xy)54x3ChFmrYHTx_3T;r(!ad@MQ%A+0zEWrr?0T%d-u)q{yhL-Ze6*A{9kcKw78fPWUg&dIoR6s5EP;It zH;0#uSf$z?j6vRqADIrpX#xkw(0vL8z;>vBJ&U98KBCJQPYxmG*x;MlEp2El&8U=V zY$0<@^6Oz0Eq?PAm|<+Ktb%&+>8H|eg3%^J5s61|o-fz=eL|dpqAvVg8eMhdTKS8~ zvi>wD_r-HtJU8oH+Q_ED2j+DI$pulbf2sHU@@P%^YU<+nsClb(Bj|;hG2Y%iiqdR0 zY82iewsh*Qq}-|as4$TvrWh(SCsY>cUDL&&PnCjkFCt1`hZ2l`anjho-msNMaw~ez z_(0qJ%tyRAj?a2(hPU(GHcig2IzJQ)4eQw%MU59HCi>T#3@}|ijbUcE@bii!cT?H$ zbeiH*at9cJ+z%g~f7<`O4o!@}>?Axi*eaJ{Ou}IY1a1I+@V4#n_jGf+FYP`KG*ZEO zZzVYm{WAvJW;Q5IcO(yM#RvQ=D=XarWLb>WP;lGq7g*>5|2b%t+)E|vud(N{KhB8h2Gr*N#bOxYi_P)kC04g`8 z=H|OVBGl0E6EyVrMMRW8MW412rKbu7!YqWNPSWWmuF2bcGyAZNn zVN?PlN{65&sIoif3nkM%gscS`n_Cvbf&pwPJAd;n@ROb3-|mHL214650@MO%(D(P& z-patU1|t^&9o+gVd!GTWlZ7u+DAhJYp>H&Tu1+F1ST*pr?Wy-3(VH3`cYccX<5*4_ zeNL?TjUg*9-U)Y9+nk18p&MNHvu%WNG-kipCE@HvzjD$UX$QaQNUsA&SisrN&JJ*j2w4@N2Fy)N zrmpjF8tb;re*?hCSA@t3h}E2PzChDwxk14(&`{lFmj4Mg#@d>iP+-)8?Fv1nK2UXp z%HGkCSa;{i@qt*!MgvrY@|A;+KT(7So*eB0iaZKf99D^a7wk03?7_4JE51pe_9!sHEg3KEN=f@k)ny9wXu)q(=B^V~e(j%H&3w zmUAOt%k4|0L@u?K_-S^jUn{P2HO(I6D}Gs?bM>%JqWKuxe6Ho&xCU!cbulNts!v6! zXjPM6DBUTdsAG0pYno*$BA@3Gl9YvvHB7_t(oHedPeR15^37U3;p$2V8=1`b zuqa529h`cQTTMkRAr|}S8CzJKAxvw9~cHX>|%@)4{`k zYOwW3@Qw5^05vi)GD=0sH<{`!F1~FfWD60eBnsJIEl1j{$$|IMkY5_Ar>OaMzk_A5 zaNxoL$Rx}Lz&GmLXMlZ%7b~sz1Zr=(0^JK+{vDvW#+jyVYn#{g8qh^v3JW!{Ii{>k|`+YBCpamF*Ga zXx+G-_L^&mXcR|2>?SL=(W!buIr*mGK^2$j6@fQdS6mEVmndqb6?WLXaWhBH@Tg>DWs%Jl@ZnbZ z=*m!FBaneW8~ZIFEbFw}ion6!$bCe`r5W4s-a1WW{+h#wi>e;!Zr4A(LOUmL@FfT?ua zH)bY^GWp3-D(g0SUr?gsmsm}-_|e%0yKBtxIK(|@B1#2m`nd=~pVLYa=Diob4xPG>yI`zHVMQt>x;p8ci^KOD_zA8D&L=m`!?EI)ic{`|1*B@QzV6*`G z)tN?juwoq<9M0r)VI*Omnd6KYE5Ex@)EbNEb&Gw;Hn%`p2}~<<6SIADHz3ridv0zP~)+( zP9(-}{Y1#0_WpUbh~jIi>%LeQ4(|KZS)`?f4 z%HrEIfY&yLbp)2u%q*+D)CSO;YHGcOYM>`Du=E4C&-^!MnxHC<0@2bruK*~YE3tY* zxlwZ{Le&4WYl|gbz)*nsv#<49A75x_WF=)Qnza~ zi5fnUgneGBuZ-J+qUv1y0C%Q<2?{}1wOrpweLj4h$W9{BoP=BVjAn@DqZM%sn9qxR zk29jwa6ldNlX#dM^SE3OH)Sxth5Ov_7KSS0;l}i?3FBir>FHncqGXb;Mz1%NGOrq| zujI*luVZfaJ+oT$aL~YHsZZSQmtkExEi#8Eg8$gY?MBI}l0f;H$j9-NIrHLY6VIz# zFVJUvkmDtFLD!|7zN#cEi7)1|i5sm*9lRDuN)zOriio$tgW=TkN>XRYxx9`&q zM!WMdymFljJaC$%=itZ0^LD|O#Dd&aB9SCvXV0@7gg9|=P^ z3`K&X*+JYw=GnVTi;3@V3a!Ril%BYE?Hs5Ui<4aNzDk|)i(A;ZRbA=`!$oO0(RuOV zzL*8{bJA$vUQARQU{EyNC{73_oQ(GGAG8~&T>z)hU)Ovea=C}$CwNcl9GMjdx4_mDZYe`rm$`cSD<+2ve zBU`)EE^)d3w+u#RnZa?{j7%aA1uJb)PQ~F2g!jxLtv0LrHG~K< zDC_t{yI!z9n`bYxV$e4qmvUkEG3?OKM)JMXV;qIi(a>lN@p%y4C`6Wqs!C{lgfa`7 zwyo`WE7y@9AgsvjvM{D6omD)oFX^$;#v*NIo@_11#gO56{(;czKw5OaEkgri+@szR z$$q8yFB}`^{LC3%*%xpJMh0Z=v`CXS?|W6TBGW9 z2Wy6*{2~L+T#6ym%eaQAxQ36>rCgeCy%w{%3^VC;FHo2`$4zB$wXZyq;7)L39SI{o z`9T<+XuoPmSM{Qj)J~J=vzcd8w2Hc21DPS-MLc!h7NS3q&$(GIS(&I!If&uQ_GZ7P z@@37Mcsjk!MBqSw8y9uu)v^uLoAL9bQY8k+X_$MUL<&!#%7^C0bkPvxXcNdj+Ldu` zGQ*M<2lg4B&uXc*evpnCi}X-Om11iIT+?aN)fi_?4XPpdT(6ojh~vaiA5x`~Mut!p z3ZzES5k6-z;v1YZ)GQQi8qHI7smjYd%})?ooOEJ5K=xJQhIh%KBhT_xBAntH#?Bl5 zZ`8l6zl&sH$H}O2;!i9zep8*R)t}sC6m-MXL*l`KAvnOZ;s1dNj$S)m)SbHMA=HbH zMPa!5m#+UZ`xcVjfWmJrtRfc4xc+TUZBh_rnCE#l({b12TZ~TcxY1nv<%PJ|8ycZ* z0#TKY!#U%5FxksFi`WksD#E?G6~ zi<#bE)jn=;y&&ypaYnmWx0(J)Lr#;cxmYs`k6e_^^a~ipsmPfB*22mQd4W&*?7BO@ zqDT_a=N%=?**=N-^F$F$Q#}k)r{Q-KvrAW`t)9PIJ?DIz5fa|y-^yYaUFLpM--cMq4iYVlslw-A0+Lw!Nv z8JDw-ZwL5G7U)ntY_W>b>QUHU+Bg?^x|fXgx&^2=$;z6vIeGI#5GJy;-o(=Ut%4+J z$-|tpyou0^!uiazed{JePTJ-46!BhM>4-uPod@h`LJ^tyO&WOzGwye`8rf=bt=Rlx zU5Qe0hp{!I_i*#&I8`Xe@74Izj_c6wV$bHSUug8qs~l8WDe=F(vs*MATUOK*n)xD% z;42zCW83C+W!2%UWsDmuNz~Ua+DKft1`#^!_~p34{21|RnxyV80FZK3Ut#>p=ty`r zIAf_EYQ7e=V#MqOXSm!`jfIGBQ9|ZQ>v}+l73)LxjIt7KqcnA84z|aV<{z)EWl)8! zT@<9dA>%blWWLP(=zWYH3wt9?Q}diIxe@iA4H-&y9DMYk z{a0}k4*X~|ceKawI47~itCwllY|G~db+P+8*hKV`C$%PKUXxxSgs#or-lvKMBG}Kb z3vuG7B(VGBx!CZ&wmPptB;-Ysp-L1fWvRxR8g59>ZhO12)HZu$D^o0KEa?Y&y<4Tl zMAy-fVkQq%BIR)0K)JfwUq@<+7FrwMbj!3VyDeuB z^z+`5k;lWC(YB=2_~7erNLxS4^W)%0Z#%S71#nr!-=|1Z>hE$BzRfdclbGimzyHQ0 z>OKXPYG0VfHsy%cM4a|hTZvWew&IN=w_p+Zou|GDR)Ujv@a=I(;_+MtBKg?KWpu0J z$fyXS3SE9mr?!-r{>Y-`TKtP4CiD3hd{xhV%8h`m2M=XfgC9^+X)(gO$>>>SGP|^S zR@s2Z$r-<@;hEZlYU9lSfAjdgnX{ktdroT=Y3!a!je5+U?)%&5Mi;lks&*h{z%w7N zZ!tWSg8ilj_M4P}j)o(tW8dN`a>bHRneCrC$&D!ebxNZ2$B4PDW}Tb|N+1)+heCkm>Y(S8L}WMny-7MHUfPj_IGxocB^qZSHy>0QixqQM~`Tnbg<6 zR&lQzsE1ZnGNcjHM!50P#*k3*5b;`_NMU$0(vpf=1H2U(Wyiww&A#}Wc92d%)zqMQ zEu#2+)w%QR>|b=6%LbgotNMl+_S32+3FI>?I=-(duc;1pt{tG3e z)J%b4D(Y^y7FfHLJO3UTNj%f{sPo<5Sh`&G$W)8RGh07Ji|&giqGZ=7Qg>QNty`PA z=+oj?HG{{TEMr{(9u4b+_6%bp2TM9C`7PI^dbR}R zdN2*$_TT}wB>d7`9Jy~YOA|D)UxFg5u+q)5RGc+g7G55D!1wNh^5(G1eY!3WRn4q7 z-a^5zCJZE(INs_1sq~#>sQh9-n3n4t;A@U{jwBEYjb6oV8hCUi8ok>HPR;u_*+cm@ zjMPR0=lsWX-~LJAg?hcD=*tv=tAt@G!q**|C#>rF-Gm?SCR;^rI+C|$ zAJm_uw^F5P67hJYr^>SCxZwQ$Y<}<*c1K+-xu6tZhG^GYN zw$WP%TbhwmhV}X$+`}*{vcpevSF<2&e|3C(9I_G-TGWVpHa|&Ss3t>At>J}{!^OC> zyz$NRF86YZDUg&+&m7O}vJe!h?lPs4tfr#H5xm9LRMT8@*0_n+bNNGX=2LZ#P!Zlw zfenK?O%XY?_&CJUk<^Jay(aOf^QGi(gr9WD6EZ^5LO3XxK5l)oE=7YAlI-=uL$|KG zsajpWtO^am;m6wr5L@AW2S(lFmGI*g$dVr%?Jiu+IqSb+;mT3;WGXd&qY3LJJ}|}V z=HX%0*#KQhn!Bd&FWS(jLjZ zM3Z4~hz*}a;YVW>)y|AD36AKPBNI_Gk1}&~Tml6FC?Ob-F-rD-^LAGCa5Xx`rV=t% zs0OL-dOQoriG6@h5hOG8^A%sf>NtLc0|T^?UJngD0$TWmC$6KQ6oe}WiEg34KhVMY z6*{9q1wb!X?#&h)S2s+G3G*E(qg5RDVd;|Dh%8O?&5EEmt+k|Go(DQg1RLH2u z#9N{8(L^HAXp;#0qRs9{t}WRYmTk&@ANyC~`hGJRVaqJ3GkZF?Ej)O1!=GGED zwz4fFEj}O;KGx2Zey>@Fn!Dd{VRoDl8F~O3Gw;Hh8TeU-;Kmg9dQR;~IP$ zN@JtwUaOMGswORJ(hz*QQ+$=hL+Im^kDUDY=h=y|GlFQ^re?<$M{B{!=9&F|MJp(x z)=rz5>Yzqtkq)WdMr%J&?CYVSt8Q)xfhQ2zCcOOS9^=%#b5CEP5{PeMxdLftwlQny zuQ$+Ok~lax*z7gG4e^9KA^54%+H@{ zd}E*1@zM51z1zXgW;d|zo`B4Z7;8%THz(ww&!D3tP#hR-6hR^0c?^Aw9mzRvmTN`Y z0#8m~_O-Nq;y=C6)zoGhOo3g%G&l#FvYG7kynkwOo#<(9^rUD{`gQ(9^&p2mj%|Ay zcDp1Cfj;N>3=T8g5~4whKNwaw(XR?Oap*r1%tg?DM|ojNyU{leUFpp{n7hEbH6Q#+ zuA|m_n=9$8tD}=3g*Qa*^84F-FyxG&u_+8_*kNT~(A7mvy{Tw#SFoL287(N#=GIJR zmJj^?5gK@*6QdhsX}nqvplSEMl&ij;-Y_sj|7o}<(d_zjdz+Agbq;%hKG>Kj`L-08 z%mn8{TMJytTeG;*ku|u*GK$2j0!AajGgt3_sNZqnFp-z)MDe8h^a_1S=*^7ar5b$k zIpv4y3HWHz**)W9CQ$>D8rx$wQEsg2kX$LMqH}0^e-D=N&AA?JYzEqT9 zh~s=C963FLO|_m-o|Tnd2wb`gYfrJE@1j5LCq~!YA1}F5__5Fxx_}jBKlSYEY>QX~ zlnlb*$=O1_t3l>50=nrCC6P%R-(jE(4~9Yw9uOCErM_N)tcrd^TuyOu@nxV`4UtBQ zXVsU@S6%0LDN!2MCU+D<2y)+ruZq-5hP{3&)nQ<^lY_&cz^z8CeG1JW8d$>6Nq${c zniLfsZ5V;8!+$E4v@E%mu#BLxcY;b#vtQk<>fO()QgoVzh71#hV@y@K8NGiYMC61kMakI0Le8NN9gLGbM0@9XyZ>eoROn8C16Ei_ z=&G03Q+NmzC`KGZ-!rp}hY6JQU@?cNTn1$eIqxW#u>U0keccDf??4pa4MRgjwTvT* zY5pCx_VKcA!SB)OaOD?sF`F6EH&aL7*^u05ZpED=SY(YpdUbFwO+0vC{Xlg%%B$I{ zo*VTB71}v2^F-uUQX$3xNqI2L&fY*j-nh_?o^Sr%JwNWnLc^kzKHS36rs`W&*CZ7DpiHUqiH(u&UH>11ZEp{=D{hKipY6Z41B-ad@9!4(j9$0t# zDt7WV4&hyDzqy*;-nRjrgV2I&Z2mB(47SB=3LQQDc9Cn+=BF4GRRmZLXi9!sQ4wE) zn7W<4I?*2PA53_z_lZ&)A~ee(MDWpf?Hb3TjsYvyBdd>J62@a+GgRFJ)>z?RJ zv>J+-9ha7;KY|FpnP~WN*a}_dXQS`u)i5m{_v5tX&v2rmi6kn-)uM!Zq8m`e3=#9i zLnA9UaOe#4=Nyi$T%dKpaq-`N2uD?cK|gCkDtM-00$} zCgn~xlR|b>5hlVA^;Ioz^O-|sqLz{gZ77~_`mY`y$iM9v?xp&O)QOhU|I{Ps-WjnV)C3|BOf z8fRMP3*6O0YJAQ=5h2d1*~~a%a!bzXR+H{p?>YPb8XeWM|3cNEv*SkOm56FTT(yST zdOY^2N>is@=j>O}!nitx+D5Eams(P7RQ%52oUvDXZY)Kr@I}ZZ;@x|3m0PM!NB3uN z7Mqc!e_Zn)aD7Te6Lploe;L&7s5ty5qi;x>3ro{EDu7C!ss8nq!U`a&Viv-WMxBmN zBf*tWN5@m;L=aUHsFPB1)KK>6jWk=;keV3bk>J-^8W9wiBGWl(qHR+~@ULJ)6-_M1H7s?+hebF??P z1 zK0fS^pD;@AZCFyaAQsIY?ei2J+#)i>Sv+wLLblt+gU1XCFHUhpAEhnaPAO6*Ir^Rk zUgzIyPpRnN^l_e2(~l0#TWvJ2<&e=kNGmqi6$n<9vWx1P%_)PFNj)1AmHTQmIhzrNFJuP*))O z@z$z+UZrx2EFuDhg7)y%qhI#otx&a$#3Rhg$O!rMMhl{|GiR?^?5sN{9r;vVUjn)v zeXQPtGD|bFv(Q-%z>R6?VQ`v|&i~YNeb7&L ze(BK{u3`{lT7)8VF0h-GxoecJw{IJ*s7$q7+WlAkdxj5V@aKGW#Z@+^^<(87{#pNJEV^t1f27RoUm>5vH0QMk|naYv7z6e-|sNRpj z1q5yq{G9=3&)m!m+Oa)>zXfFL1Vuzb0h1I<#&`orIdCefs`f$RMTvD3oTpsyQOdpC zbIw8wczSv|+?~E|<@!EItaQW80(|0r8L>UI@FQS~D16dni1$pY=pCPeFNW z)~mx4XtMmN3bu{(-geq~+u5ZTp_>C%xLAkMT0eRqWxCQ&6ubi3n+S&j#E~V?$B(crm|9p&0lNl> z^Knhn(dTf39*-hoQhRnTiXfByX5-X5$? zEkGrJ63GEP;_m8OR~Q|L;$RPHIyyQoJPkzzcK~y+1fsUYD1TqOkJyair0zeb7@bTj@yF!5KfDQ|4)NNoYrH7RhC?Sv- zX0;t?tbW4k4uWq)pq?;L8djS?0Y#ro1n+yu!fr!%p z_GSZv4G76}%lQJqo35^IgBW?cfjRBdi}7YPZ8 z@DOYipjc2^&bS1!jgg3a3jZ&U6kd5u^$n8JBOe3WiC5wCJ z)zSf2H(i0D2B815v&(nSff;~3s#|$@cxW#^)g68UWK5sI#1o@&`SNa#!wMixd2w$| z(|cF!Xy>5Cx+I=U90PknCE`!iSTIOLb^ho-qR#7?f)R8>K>1bFd+U}t!D{uqGuKQmH9 zsm=^cA24MWFhh{X16xo6;<;azml*+I6&rfj!!j%H5Tv2qb9NLqlYfG4GC~jo8x*<5 zlM#ih5Rv+y)zZETFcWE#E^52UhX@dY>H=GWkg}6O&&=VE;57ZcKIwgYxQCdXlV9bk(*>8_N`qmRx9^G2FT2D;odAR~dq>BJdPlZ|*uQRq zG$v7uoV6KK_MmtCKHc<3}tUlMR2

hk8HsA8e_2JRcY-3P6vjTldas?s`u(-Zhf_=3^2gG1(L}HZJm3dMuU+3ZIOB7_xzGDymeLal# zrV1L8(P+Ztf6Zr-AN7hDR7U>YPxYZ^{^#MqyiTdy{jW@eT;WZ+<1k!HP!X+{fO0S>e)z`Xvs$75~Cf-L`iO#mAY z+-&DLslZkHpI864iYRh#fDrqqk?#K%4^VJ@4WWc$fwitW5jFEGK-eG`tq-Lo=10m+ z^#wfG!gZRI>XWu1&=Dw2xWUfHt z9=&K3b0klsQFHQL8=`RJx{CH~O8tz;Me9VDRpxp6k8RHJLkrH|Bnk4Zcn^$&t<<3$ zqCL;a$vLxpT4ab?{icf!1S-oICkhbGy~!yk`fYLhmW}A-aWLh@O9M;Sdv6Q=dqWY; zaX63kW2+n7a|Q_WB$~|+zZi&;|>d!vAIdrxo^ddW@#S@pi zwdp|P$fcSs{Ys~-BT5_2B@(*j=(k@l**x)};ejpG2y~^tf%8U4!8y8;u?|wa4NCbXb zsX-7daZojWJZZNAX#2OY*TCYuWW+AE(mGxXkdcw80g|!K&O^ZYewxt#XZ{%!A1~jJ z8Ebg7QLUyp@Y!ZSJPA$+BvJrH%VF^HMGy($dj=}*|6gx7kyeO+g7pvz6MkDCLIGor z0ufF#BO@fSg<<~=?py!T4*<7jXJ_Lf#~HjI@sB5rw0W0=JPQ7dqv1ptt9>G{mV z7@*!wOw6~d1MhUBn~)<-oH_x$^4|)DU|`1?LCu5bNPSmRnKi{0kAVm++WIj{=N>P% z&jL9Sv$C@Zd%(9&uSskCV5=|>;RZhY(EN2{Xy-Na3_PtTyaRvcv zEr8HFmb0wE+_r&tGPM3XKm2$PSSEjhGq_V1ezZP*K}_Ut5$K_6_%RglplBea_ZA7Z zFzg7UpTIKYJA*(~aC(|D#CDKj=v8tlVAqeJCj}Y13yUg&7XPd(N5Uz5t2GZETDi4Ds01|JyPF^tY)_Sz@e>$P5lY>#qxH+O($WxT zbt6K*4F#)nCL& z&i0?)5^H*%#1!1K>CY6~+b#01d11q&YMb}sh<0EAuBLe}y|)+N0&ZW*fL{GG_)41P z`^E9OqbBA&)YTZy0Ec#gv58k|26AMfB>hpKVJ0P8+z zWFuBtgEP%u+->D9C>+9em|4Doj~hAhoj=-87mA5fzExIPzoGR-`CtQ7%cp|fPJ*GQ z1-7j>5L?2}G*hNAjzzoE^5x@0Lto!Jlo}d8EDlG*_Yb1#vI_Ge%t!KCcDJF!AX$Uc z2mR*XF}qf0BQZ@?;aU*-<*mWWyUEE8n|)}2L-ewtJlSyxM}R}UXe^FxWWM$K?V;tF z<)1->LTiQViJ`7KIWZx#>6O6)vg;ZLZ5CNXGt z;s+A~?PNPwD1?KR%iRBxBpVA-5*2P-8SQBwpz#UzHuC?yNpBJzbFqpEA17sarna-BToq&ylW`hZT0y|^f zyuk7L-+qcG`{i+4tp{{9+p`ey#dOrsnc3=^jXIFz9aHl%-8!N`uYRVfD`Cp3We4vQW6q`w%OXN%^3tZ z`Eq^w5POPnNp~NwcV`%bL1#k(a0n+3ft2?M7^YDl!E7FcL%Ege4P-VTiFc{R_#vXg zyal)eNMN~81XND`tfRN9sj1|m=HYp6KGKTblzyhmeIw%zWH@pxgh6;)6}q}0BAMCr z4_5jBXK#GC=fA5jn3Re!_6^Y116S)odu%+g)5|f7iivH>&6Q>V>tI{k9;kXMY&^cs zrInoYVu^*1=Wt9e%{OkNT15aGnrUieB;otzVh2m`8t3q|zEG1m^Ib=Uf-5}?2lA)Y zlbB?^UZ!$Pm8|ah%Qrv>1Lt*HVWIu=L|UY;41gez?m7$0_W2!pA#8iVRv0MIe+DvA zXa|E31|S%_fl827j9l0%fMao{>1qrmGU?LUL(4Y4Rik3TPEqsSt_nLYv!J1MBkrxQ2!tHz(RR3p%OnvC`y3uB%9K53;so-Rp zwXn7IrX<+4RwbGezyYt)d1hP8rz;6q#Oh9Z=7(DgZw%xEyC>YTB%QBl0aeA?=gx#A z377T5km_Iz5mm~IO>HyaOAK`d)deU-*RDf!OGyMdF9^y6u_UCVo83}>j-avEN^wDu zH3e7{o$Kf#%G@_Mn_2o(o-c+UFLWE+54=CSe7t>fe6(|U!NS;h?viB|QhJv|!soZ;XH5l2{OMS!pG6BhP|P(BBs2okn}ODJa7`3Iaa zVuIVyNr?~^Bl#$Vef+*-^6K=H703c0AP*#s1g)rYu)1(=g1-fHvv=r3bem`{p85p= zoP2p^Q#Ozh@4@qQ6}#?UD9aD{fbUAisPOgVc;8J=k40hrjxq=-JWB%&5t5j{YcD~Q z>yY%>KdW>0&IWZKB9IE3sJe3clYfU<&3@2G?rw{B-3Mps!I2L$ZN4k2CwnOXPYF`W zx7AE-L`bx?LM0B9B?oq5B4y6ph29(!hh}F>|cwAUk_658z7KNK`!gNe>J|CgMmQKX#CP?Q38>*>*A7sZN)3}K!w?npjRB=lQ z1zEQzik^2FBsepboKM!b_n;}obJXn)rq6bDNX&m5LE0QRX{#zm8)!4z{{U32r07 z^X|8fTFyp32~Y!;2P6`soKLC)0}EkVd`B^WU9WY#>+k9+Z`pNp&2Q)s9%{HlnK$!r zp40X8E)Arew7COw|BZu=v_bimk|u?Z?AiCAuJ&5S*azC&3fnduaQykS+2pmkyxFsm z8n8G!)UZMQe1hgQD*k0HVPiGvrrKGVXZC9&Nx=M~ zeDc6ZUQA)fK1<}F?l^CTlI`k0+Z1_mXKQjVg8u&+j}H95^&pld9T-ei39rQwh6Bce zc!B42fDE8$;1g^TMtH!kUB+mryH(zLfRIu4;yenC&qy~gWt@G8ms=Y!M`bwq`zSPD zg(DiyTI(ltu#Bia`Do}?7Zye!A6{tHH|n(?Te=?HWrywIV}-cy%YHINugZ1}`?(^@ z6TRE@UGhDv?O;yqaq1>Z*7w7X&&b$L{^@erKwYx!Ql`5D0IQh4_hFyoqBoojDH%xG z{CH!ZbmTOtg0PomIqL`{oaFXS*&SI3U=?20FD@!VVJ|^@V{5Mn0!pa;>|hg(6YDG? z#wAvf{ikyQkbYFhPuSq0{$sKDJA(g*;@H*QDc={ml^Z80?aGai88)%%_aEFmj&v)# zp4Ln#$jjb`=~$n86#+~c$4_QzecyVhD(FpDcMO)h9;{-1Y|Z5=^B>I(#1(9mq@78c z81=JQG)f>WE^uQ9hPiJX9o1Fr2dXIb)cNP)?}mJ~o;n!oro6ahW3#cnw;>KB#NZfi zd7XbN-7d_4JKKj)i$ z>Y#koQFJ+BC4?KB8hi`WT}szZ3YRk5gbu#6F^0eOJ9Jo>57;C;em=vR5%y7YH|Dx~ zi%OB|v{s3qxN)KKnJk5)FW=@);mO1iqpb1OX5pc*#4`6tOkqO< zbq^G(F24QDKzH!q^81;ky6xLz4>;bnkvZp$@dx8%E)B|GRX%xoiE$vTIhV~?k^SAP zEfwqp_=m2;IZco-pFdLiW;rF~$&6E6WH0C8L{8r(N~pkUH`BFdEa-@SQb0qFE+H0L z$uzG$1;FnQ)8(bgyu(>vBG4$yCtdA4G%c6oeeqekcalF$zrNn_q^_vMu^Porb&Bv= zrLAd%BpFxJ#gT>Fq)_*c1zj{Q$56w?w}Wo-@Hk3MyAZ2JYXQgKPmr z_fKvt4?=3|gP)b3pWn_hW&nmY&RhU|@XK9*u!AVALq>G-{nyFfCf5tTJ*f}aGUmEh zSdOEb6@@v>OSK;i535ygmyEH7STxX^kwj44MXMV{ytZidu-5A{W(gOUnq2twaJKjR zyH78l45eA(<3$Gk3^gXvwbHCL;#rx^tRG6y=BF8vI1}KN`s#<>w;z>T-|~x3E<)fB z&gvtJ`eJb0x|~7h{ZkTOqb%j-d&>I2AB(`k?+v*M3jI`tHd=CfzaaZ8>YlZC3#$9y zFBOFDFY^_=F4%oVU+;G9Me@wRK-oZS86R7?+FK1<+av2I3kG@lWmi%?v-_hIZv7O* z0tW$75<;+5kQPw}y=*u|yGK`omov@HYHchnke~ZGL+`hCp-4_gM<n4$a8;xqg>@G>O;;^n4VtVcEV2gj^xhI$SiwL&o18w&KJy7 z4#)06?DhH_h01Y?R>7?wF_Ty7Ee3M#tw%cRptiaW9GsWWqWpU$nNOQ<^JUbt^+^vy zjsz*4rD>xv&d$qFe1f{C|6ad^u%-qy^Y1|+6jI+@oDIU&+u_;;&|?fGhuuG1!Zp2@8+L zNNB1q&Y#7jfUHL~O%&`@95dhR$}d-4lPt_IabKWc2;2J+cjrCdBsrVYplZbRj8zR~@3ga&6l)N;Y~oh>I}d3X?@kCwubWY-YoWHH(Sfx^ei^Zd#2&b?-tcdhBXBZ8*02l3R%Z* zUS7w|x=zD?)!`uNTJPL0U1;~|`hgd0L5E!i*_vdTNexY>UQNek5CNjH-u{xQpQHcB z0SToUS!ciH=86Pzt&v9W$Y%PlbA$&n137jDduEQzs8VB%3cDwt_a3n6VswO#mS21b zJsLjsTjBuKn4Ib-<99c0BBh|~V5H&h@0~hXD$2@mzb_qkB~YH{Yha-Fz?aZdjZXy#Ex1EhM)IJ50fEY+y1JHa_Z#FI z(*YnS4>=*vsmHw|DV9)-hk&E+(|1u>kCgKW>=}aBY+=g=Ydj-ss{yGxxV8I8e-?Z(2v928vd&#Ek(w|y%n+!C$M5q%&kEf%=zKGd0IK>=YfLdr` z8(T<&qm~F&jQXcjOlNDXI~;nWc=E{3feQ*O1N=NUg=>UIXDg>;o~xz ze*|canajkiDXxG?J~>=bg$Ds|5=>OR$2-B6Sr+oR?mvs9ihe#n7N19AdTNz#w%LGZ zF4pgPuBqx)7^oG53Tf79S2ukFTSze z2Up%RoI2B3Pfw(h;^j`&83AWGnzQMvi96lHdLqWUzz%z3Ini4s=fGVlUz?VSjOFat zL(&mu`5X7$b5?Q#_KEpe)_(^f7Z|RTK}7ye5uD56MVXjy5#gUhMHdDtIf#voQ-74! z83_BBU<(*>089J;z7@zf31A_`xDAgR2K)^#buIi=))D;t|3lMRKvlJWUHgD2APowN zbV!3rcT0(g(%mU3oq`A`rHCLPEgc7t?hud?De3O+u5WSw?{@~nJH{0coV|aw*P8R0 z(8}9|dZsbdUE!C9Qi9w0PTRHY(+l>E4dLDsr_N##c*=mxl#QJNEdTpiHWU$Wmx25i2z4xD~z$bkx#Z_n{b0gWG&2NB2kRWzPRzYF7u2~js%5x*kj+ggabd)7| z2BVJt$*J}p;md!6|3*m!(4?%&Z|2qL=?Y2Ri3qrn^38bv3tWjqTRILB)ns}DdSqTO zBPqz2K9po~m20$z-1b~cnYXso)<9{Bl+vg+<%jMY++WM#c$$18kUfZ$%B~-DIiECo>vv(}t5XLS+Q(a3pUZb~7uB121`aVT_7@15Cjs`;b@kysGFgqkTsUHT ze@0Kmx}FfSHD=AFNy3#R{0Ww-9EY_dWZj|9E7nA8q7f8rX({U%ugmt?1K9vqp7*aF zM=3sF!Rqi_rJ^CgM%$RO5)8F&p;hdr=975P7d&=Dnz@PG0`neWc&1eR7h01wS)+74 zg|Q|&4CXRZna;P3cLnS8@9!SPi5v_V;UPX1qxu!fhqHaISRfNKf>MfAB>ty=i(f;` zYtKwsZN<#Olh&M$Z@Cph`Uw1v+qar^(ZfX62INqwQW~G3MRE`q=9Jq-#4!_M|57xK_@p43 z!0Fl;S7FFZbI&g)P@`UC;)Cp`LBe2!)md5GH_W$a)WdzVUMEs_LIP+XzrsMFxBiFp z*yxUsC<=HPXe`N(l>C5^5-lc?aaovx(D-1%`1*>bJvxIpcY_1hlD?unjqXOb-+=2K zf|jO6jix6}ejD7~=&ND_P&k~OHA1|*g;Ts|DZr0^O`U?O)&Rt7VK+l01rVsb7h}!q zz?~3|o_ZZCqU3wp4{eja@$vPeGxdBe5UDE(Ll~M%*bsYL^b5v)5RQyBPq35xJLxoY z^`)NOaihX7;dq6Gg=JQMa{1QJ8)CiC8ylB*TWLlNgpT|)z@KHU) z(<7F?dyZaR94My`{L5GUXQnJR@xmT@4SR&4@)GW-ev@Df`hC2IxK)7Nx8V zs+(EI1xvy6zO$pd`_N!+y`9Btt)lZaOd`CWqOLb@p1>Cz_6!Zh%WZX0@?bwY%sUEE z_!t$nl;~>~!=A3+rC;H3pap7!qg98+kxZ+&%^eTcPLJIWzU0pP0Z=im+Zi28vF9TV zQ{+FXfbP!N4T39(S+B-!zAmA{4c#Cr23fTPO2yJGfr%Huce<#w%0z*?X2q>EVv)iBH%_8OKLhs!`Lj1o;M(rWbdx4sjG5SNsoOOt z9+sW;?JqriKgQ2(UnyFg_1&%Qz!l9pzlhPFs^DZND)O6qnB+3Bot{)vzCC6|Z{K$k zbL_(Iz1OTtQO>?wttWKQMZ+8IwLck?n=2f*>$Kv!bCY4w5@U0ic-OF+O=8F@m9w7+TG1{&i`P>5ezmYGZ8zYpUzzUl>oG$T zPtjGesWqt;spzTGNEA`TcJ}XoqrV?B?B`!@=Dr`i#8SF{IxeaYuq^{)8}afwQ@z~c z;?Z5_Se}a}39cTm#RYPQ7#YnA+(f6*$FxR@gB+LEE-J^>QK#{wshPSy4odk8>Au}o zJW@0iqPrz;+HlyE5EJEi}s-+gjEM$TR=e;Qh`yiUt~-}7pa zmxq_mgr?7W|MRQXXr!LLs_`x2qcxX-ly{$>rJsy|G6s%Y0GK+4b!mg>Za-HL&Vzq_ zeY1|0pb>I)-=&ObyKZ3t>Pwah)~HOy_(RavmBlm}AoHP%0)~(BJ+W&)8KCz-`jn?r z`Q)=~7c_j~yEYBd4mG$y>jR?E0gAfA>Ch&NatIeXK55N#SER6SqkI;b@6XbZ7h0uIAk1&G+3Z?26fOOuPDXE9V!Q zAC?=M!-WsB-l=&%S;{RfZ8<5hQFWeE5^>!PPK7cJGMjsmO^0*g+v8Mcm6uWjg*Fp; zu8RymK59%Vu#1XrU+@kJ#@C&@>5cGb-l4bBi{o$myRU(osqZxNX>+tIQ{M)ojj7)2 zbRpAA;4ExVkmgRrf#Cjc&Tw)z+HPcz*H4k#oTP%wSwu?rL@^o`!@f?O;`Zh>yosnK zj8bz*Qf|ZL_*P$hIE$qO&pi8IT}e(mJ1*F1Lv7V{eO}iobo(CVFEH1n^;S%GWDCEj z_h=3$W-Cf{9SP{Y&@NMdxIg>RsMfTtl|(=BT}@=4*IoLFbS64~ySj5T=Y}VUA7xa)b@i%rQams&55+6M~M$* zX@=hY6ka_K8|{$6H0_wy(^1-^AFTme~;T+>jxpXR!hKm6~f(vg^M zX?;({jjjERva&Le)j^LsxNR;j<9^^{5Gg|YT~iG@)h?BR(0YW#OX&R#;R-%W5IBI| z<+>ZyZ8TKqqhbX$4-O7OcJA_I?HP!hAs^2+ZV{BHZty6u^sE+#&9&^$2qPEdAq`VgRjt%v$2BPO7782$`6F~Nt`8vBCyxi9n}o+S>$t_Ev!5a5 zN8hM!2c=DUm)q_wcJOcc_T_8y6LcpYT<)}ydmoglV?Pi%o+w{JwXn5iVrH+}FUA54 zPr30hasdNt~icNRnN*cNO~W>O}*yhg5ZTlJNyCrp%> z%`_<%tqke9Pn5=#7)4*^ZA0y*__DH0d^2n7VwHCTmht#Wg{g#&e6n+KYvTU{6J z_K^A#V=wP*{mm)7#w4_IbSd3D~M3G1w%B);!^PH=aEimh0&K^vGW3 zKFk;^Xb(~Gtu@Rneq_J?Lx0?3kefv5(xq{UAiM4xr_8?4eCq>tQ4f_KoCoDqwWDyR zx>Ji=xstKM`A693(sPT4LFGwr{=t#cQg~d`Wg4>DmU-iNagV!Oeai7bvHZExYwc1e z$-VGFg}Q>)uS3tP%|=HS|JW2~oy{2E!ligKtQ>?J{qnLaP5osD)Oy^65-qrH8~h3; z0D1WfKm-DUMo(`N#AoZPbB7|whlQLFs0ho#m~UM1&q}C={F-SFS|W>CFNLtoM3+&^ zXw$Fpcb!kGtlytlHhv3>lBtkg#lWcRmX_=lv6Xk^Ax+@EAo-T{Z6)` zNb7?7-8h!#SC=6XJUlbQ#%)K-BTFUy5gkM6R{0|~swyv^z36pYdms8xHgI*7hAU6s zu-mospGvN^-FJun!6g?C#}?8u?;895_u#LY>mG6mvpaJdQ7bQS8F(KqxOOdgt^l%J zclW=4ND>=Pd1U=gPA-RV`d`J~1ns3R{Wq;J$$gtkY#rNV^z+|Tw;aYr6LD0{*DwX; z>+WlQ9b5=}0Lyamt3BtN24jo)b&&-(FPms_DK#G(Y{MyfG2LzOFtCW>JNHKFT;1h* zvyIvbH?NNS;6i5p1ydRIsEylSWQ{u0^9RYJLFwEd9! z*0SsHyme+mrT0mUwe~p(Zswxn*++|y_qr8#;B@js4_UDp$YQSJ(bScXv$h=bQ2NC0)lw0<}U*^EWE(zt&wBDP1~F)Rjc*UF*hcYW2h-0*r-&!fVf0 zSH1+^sX1$$t+BmV%Ir)W@!mNLN79p` zt24X0e($cwLlbTbu`)OwSNV8s^fZ3a8cnF5f}f)LVI*6X4dL?bFI(GIRr$O4R654@ z7CUdIvc>2K&du&A9e$h1qIN(Gim{~gydF+ot4a{!cAWnWlYxYon1`DiluKX;dBHf) z$S^^{4tupH1gH{juLG|{G{XpbP`TMuu<237(Si~CM zV~?@ko<;%9M$Zzcf)bv^;Y)x)AtuHZbeI9nGjM3XR$qX&ZSTYjd^XXh|H&!eH736V z?N=YmFq?bHzVQ#eq3eF+c|xj{^j^#Ox}i;HU?bt_)qoBH2cUEKM0Vp9y-Rz#_Ki2eu>(B;&m9sC3z|E(IkYPB>IxdU!UVVKN)64^ zhUV@+cKr6z>ioFKTFsXuy^G_w=jNR9LMCqKkPa`)qqE00>w2-`0VN~p;zqnt|h zT<6KuZO;?w(#f8fOy_)^x|}OA55%T$>zbE?cK+)KccDb6r}NKsLd_OlVhA4%-L zGH|rfDIG0R;_rOKeK2Cz9B32t^|b1cT*y)EaTKYj;ktE_%~%QbO96Yg`X|Vj-yZq= zidW!8ZRUQ-zjUtrQUb#>#}q+_Mi^F#LN=ax2UUjh@%e;#lGeZ6quw^k+|J*JaT%?H z3T6e`R?_kB#DytB7Gk3DR;S(tpW-b4Os%_Lm-6MQ%h^zK#L@bg@9u{-nqCc}o_GOU z#wb#!-zg^{dK349k3$rKa95Dl5^JMN!Jj9t4z}ln&v5t-E96KEO^ws5OFWATS?+(S zoZjDMK;&x)@1WyWb-`i*ZJZ#XHcsH>v; zcW>1TC@`;M|I%`Sg+Y^Zb{o^3)Wo6XghUPTh(6ey=mtU~8ki8Ez@zybtos-p#J37e zp!>U6VnGg#?Z~ftP@2DVhXc4^-8Mwkrlvxx65J_ge}vl zMxg=V?x6&|_RLTAkQ8-rjBvAQ)BR4)&ICwaGT)2%c+CuQmvE%gnY#~XRgG6<{fZ|^)ghHbxP zP3rXz5V|ca@S?Dgg>m09ZhB?Ed|V{(dD3P&Sg$TjRdLRX-L$^fWToz0{DwB~{>Cn1 z6g395p*0@|ol>L)=RNxcE_tt#Rv^7{Z>C)Y4JA#fVDsi11~RntTcN@w+qvp%sD?>B zl0SY#N%T9#;A&j<*}gkR+DD_VoK47wyR z(hZvfHyo*EvGXRvt*zpsleRcmS%3E&nd8jj`_C3t;HQ?946hx0_i2?=*MD;YIR4RU zS0zOY&<|w`5Y-l6JtVvR{&G7=^z~R72nv*d)wFgn>e~QBt4kLy>@?7m|1aP@E6ZZ- zUp`2fAzy=B)aM*}a=B3gK#C~?y*#K)9Oha_@~yl@rl8voZrrPe8+g~^`)xNN7wpcF zrcXuQj{;$CdWlSxSoWVDZTbGwG*|rgh#DVbeF7ugQcJh9v*5c=dw02BzGZ4U{q%u5 z?dRaAkHJPo6{cD)%3`d<<-Fu!Ej4-(mRh|Q)=ELLT&s?i&-$IJh52*4HweTq!21|m zm670bDi;(cI4oi)ym_2vTe%TOEr|}k&b?k)6|LR8pGv)4HjU~E0r(&;KMl<$(Q#8# zw3)!_sc)jxXgyr_Boousov=pMnz_C&?Ac>xVSzHfP07@(IMa0l0{W8xpYKT&t}ZKc z0Y4tJa9TUHLnhG$g5*U^PMWNZ6HK_+Juxkev+&I>;dnO?US~oeB00NB0bwh;KZN#ZLE~&xG=!!T}6MF zwI9_$OTHiDV}_2~O8$^OS4o%Kc@fFG=poY^fHJs0d2H2t|^8Rjm%3Sg)jxh)Y)46-1F3 zaw=Y3y+MbTXD|8cg@wQDFiT9dsq1fl)6biS7^njoY|gc3M@6(B@_kYLxD;>7D#&~4 zNi;`Yp4a4eF4Z2axEeE%JGZ;PlCPDpY_g$u;kor5Kls)IEr}|HA+A+Hul*ecGUtJL z<4mRcAL^gYKn*%yjfY`1VHi_zqTumNWak?^Y0ai~atNIM^`Xmu!DzH)_(t+w3L;%c zkK>Bz=G)E%Fpgha_Tk}gVt*LKy-#!JYAWAjTMTRO79 zK-2OZEJaJBK+4F61SE1`$d7(WALGo(L>2nu-oaxx`ma`B6w#sXwNk3yQ!&q_)IXZ3 zcdD_saJCx~F0lEny^Y*qQJN&dr%RSLk&5vZN=GFAk7xPGXWZqJTnFPsaYO8GGYO_A z^y#0)6xL)3F(fLS6~*KwnZ(=`JoKD9!(R0Wr<|PK?5V3m1grDX-`d|W^JV^0b%oQVppDyD4l8MPQ#)sDKWU$*@F?M^S&8{b| zo+552in#~*`LEm?ya(V`Ckl`A>ieLs?^4y1|3hLeQA0l_OT45;7;HDnDG05 z6-jpttV0;lP`L)B-;NQ0gH?WnjM5JH5jm2nTrFpqy!DYKl8t~U+pVFi0LmQcQ*JpLb`U>Az z4&sQLlKJRA`0wp`=tf)Kp4cEZiC798DWj$zoAh8xSCY}yr42xIjRqn<--3K8d6X(` z%_93}gan}$e6}9CRri%XEfPq`b{3ik(lzMSo66C2&mH`z!;$n$Mje@G%P-?-yl|uH zlr+ITz55zp7WvTNE{pMNRJg! zcvkq)rbA3{^CN)0-JDLT7gg5Laq6&S5IX$bA0cu+(8*CtphUNRf-ZQ%W~WaF6;=SQ zon6bry+Ie&%{>H!t~zj1x$s-B4h?X1=N431f$RgEI6Xc#_7Z%8J-^iEm zB|l3wZJ*QxQMmlx>#2>5Sz4Q@9_r+%EAu!wLjL(Kyg02ab?1ZBr1#l7yE?tSSuCWw ziW3LpZ%Xdd5|CAnVSr|dd0>CXV)h)sm>t`pdPCdi|~W4Zxx z^w&fKXgB9oR>lU`fh!#}fK!|9wb?OW zFOQk`Bp`aI6+YQ!h6+fujk_+(E*h089bu6e;%VrEq5X>{cdUBNox#iyUKw1`AN2%E z>*HmYVZ0*RMXy_Ox7hAKF*8AmH)ExZ>0KH0=G^U^jIFml|5b$WqVN(p`jV05v{Lmq zAOlm;x|)xRXh*iBbV+(4Iwvv_mYGl|o)v!7t(E>715Ij;`XVj9s& z&UCx|J_&#EtT9}hixArLR!!sRNvp2)@7g%%MY+L@c&kE3R>UD|^f_3Vz$@vq@p%fH zolnL5+FgaI3!~=a4{0h~cRu{u;8PMJ{Xmax;jUF<6i%@z2HuNY zIjSm9Rc!z4QMa?Y_sA@7vhXn9y;Ge{d%tnkMn;L0XtYdk(Rm|L2{GH7Whjq$@r3kt zOB9dcY-8tQ4gc1j^j6*BWO(w*zxH5hV+SGx(|ueSoEwv=vkhf9d+IjDUG|Dpz4iKd zhzx=VDWbPf#(znWoH`TC^tJZuH$%+68h3ZK68FZ;J>CtA7wktrnq|Cr&Psb0IpPjl z(G``ks?G{Ajd@u0byUZ6ey&uz@2RhPteq8DHl<^sO4AZ}`l5#l{a{qCwqMAI7Te3D z^eM}45prrW)F$@cZudu$`0P)9;plq&HnOjYTY;n?H?K2sbHh9f%rHO%`0MNq4?%Gm znABitij0XFg3iiht*7h5*Av6TeBhY7%c6z|mZHHTW<3T^$VS|+?~)&1=71O#n*A?3 zz(NNTdcybWf<-N_Fq7$8{9~Sa0-G8%BLBkvcxa+SgJPhdYCMJFjbQA=mYJhMb!Q;s zng7D8r+sh`cckZ$(1Cu)jQ_#yJpN`$Xek?>8D{JH44-50n$~Uhq9d!%t25bbK{Zy>-m^@epPD=YdB%edYcn z3yQ>TRRMoS)Fvf-PxqU5s;Po8h}S>fE$mmpXQHWTsUKR=cvqpEv#K=`d-$P)#sC4PEby)9#Zeabe~WXt=I7@#zZiUp}rtbI_ zOIqyJ*mQA*cTs#?&guMIPwGB@tf~wync%B_`^EXuuxkF;zmF`A0!uw{$tFv@b~M|0 z>7OZOJLBAKDAj8`hg_@3^?77@Sd4T}JKb4V@ z5!^Q<(oRKX;47CD*q-=|=rK^+9H87VSz`I6+pPMTaJZYSuLcW~HRe{Ol27VPvfEDp@;1{04x{^wY!JwDK4!ZxPn=qyV|*VkX?tHpo$!ufjm5ulW|wx%FKs#Gf> zHul4ttLY=IzXBYRa*>Q5t9A>J$5zr9(C3V3OT$hI^&KwKV3q$%jGk6Xs;J?HA;Bm_ z6S`;b=l5@)hsm6oIf9ZyQ8sjAl#PViwKlmGz^V85an;FXgmn3-X>2KW#j21doQ>HC;~_wVT+J_MY}PpJMvtqDA~V0-WQ{o7)! zEMGpos_P(Yo)4*8trqzMo)a!tTE>QlmxU*pZJ(AQr{-gZMVS9Gb_ScNP#dOr+-+Iy zDl!xaI;6~dJ@1J_hQEMZaPK80FddYaiAa$Dz3Qk?EgNeRNz|l7OdPE~P`cGyEonpr zS5GLbYBrS4-m`3pT*NVT{&%k=x`M@W?F!SX6!J6Kc?t7gO(%U5AsbWHk(3Ur~nt2MrS0Q^aHl&Pk#-BSN`;f zzMP{w#a&;rQ$cTQB~o0%C1>Icq@R3D6%=_hP8mJu{qo?QT;4Ug9>4QhE5TLXtonjV zCv8LBhyoi+I@-t|s&Rk+8gNU#;IhHNEm~HkOOyMECG2zj;$);1SNOCfY=?;M2Y2i{ z$vip7?hcULNfv?^m84^K3-YoU3{$s1(k;@G-oRS#@w&=i~8@fe~$FEskDMB}=sK`DM%) zF+DP+vT|fJmc@nENe%Q~oVb3R}92yn|!HZDWM?paW zSa0u&Q;gMPi@5Or_|aCdLiqRqUHO%Z%OT`oe)@j>@;Y(36_W$RX^7>+#KdfDY>e5e zq?LPVW0S9zdr^xABrz&F&xO_+T5?aPK29|8NYZ!wti$p&BciNNIe71q39!dY+`?PU z&cOKavW5PaJH*cA?Ce*1i+zjGvcbjp`RHsfB+mVyq4CDKGgk_aW|&h;u3LBEU!^p?Kj_gF@X%{BBg!`w23+*fZjgl{)@B&*+V6q!|yKqmz?}JIC=5sf;4@)mG$~tkrC#==g)!} zSp;;{l!wZ+k+_(C{*0E`B8JpzgwzVjgDwR%SwB=3D7`)F^O|hpZBDm&92$#RztE*~ zYD=JA;M3d-V`pX6kZ$*wnl>PcP0}A=wX~xYYLHQ&jy%M|-Y;J!5HHk{eDUH_Z%>YN zLC(X1&HdgQUGxYFK~1;q?QBx@=g+kkb|Q@#o}<&3ZcXrikYZ%zN>0ri=U{#EnE$iG z5b@WjcG9aq38Skjblq9aO_C+7*-tc5{jp_khvcs5+~?PvQLAm_;CjKaA!`3i?w%dx zlZM|FL-`Y98E?!*Z@j~Z99))Y_Sd*gK$p#{?f>rFsPBnHsAGcS>o265ddf*7)SA-$ zjVUP}+|rRh8ryj3eh-lnb_^zc+4m`Bz`FUyNxnoia{g}xU#qHTUd_WN{P{NWc}Yq2 zEu?o$G9d74pgOqgIX{Y-qMYFeeJP=ZoLtAE@+tRRX-Upo`ubsBIem%vB-uvara$4= z#?fH(8SL2PFlyGjZ|%&bO9x>^e9uwr=vp5aDO%J?w{!Z~s@aVuC8H6>oW{g8jMvUc z$CFr`AgH2~gB4E6l}z-0KC9xM`e3#ddM|aRw~28<#C zBwA`ZeZL-)J94Hq`D+xowAej}AQN`~Q87QAVoa)#+>(5nJ7KA9E;&CpXUr47!zf8| zF<76S-EP6o!7Qgsz22<%C{?VHcUh^_b`-cL7M<430I&kUZnGW zOg+wYb=1(*jBT5fla)QTJC{FvJyFO~C)tLey$Hgby5u_>YgzQSeoVScsdcxm(rn8w zjj#n@mKg0$!gtbD9uaQP7Q0E0?}M?>$x?YdqF^Yn4MQv0wVEjN~7f zcj{JDW_s4bh0cFTqEsn7tWRQLVsy3;EDQNUA4_$$OVZeWPJY&*qbra;MA5BZH`lf7 zaCARbRLEyQC5kwSPo}?wRe`WQ!}Ldd={~*np07`0Bg=VRLEiO?4x`Da2sU$FHvzdw z6+lrE5IA@fwO>;(-z@!^0KTZ(_z?>O2I{5Ux{|t}<9u*Z5+#N1+ELq|xLtB-XW^<+ zS}h$6UzgZUj1MUP> z$v>n1k~%Y-I405B#6q6X?d1~Dih|9EbKa}+*Ofc_HPT~&mrI}de7j4%P;`ImskB4s zT)|Go)=(|AKjNRW#Xyv=NBxWE1nC~yd2bd_Se<;9ang~q(sM>^|0-47wY+iqw~rcp2>>E$GX#bggpm$quu5A|d;B_e;lkNG)t2mi{3X8UHAP9fh!muKY@ zd8d#Na)kZe?e}>)*1xTvz3rU3k-W#h*HL=nsuSUhZJ=9oE`QdSx*}G*&K=seW$n(* zccn`s_heJp^@At$VDPy6mKBA z&PkJGG(D#K#GQ+o3KPx9t1-2?ZoO66S}m0tS0MmpUy7&|sn4yK%51ILjC^AWN>PcS zOWEtRcSnn`Svw75=;0y=6)5+Nn3vV}`Ymtj;oIwm&KGsMueWq`L#RIH%15~UiMVpj zPlvrx#Rb)FI_u@pYMIx;a;?n+p59wJdC&j&5|i+oU%p_gkJgbOZFl^&=i>HQI@Z=6hx3wem=;^7HcADmMkkI?dCq4iaky(=EgB+VHg_78R!0;!)`>LSH*W z%+!|J)IOC&WW!#NePcP|)c8$C)J#NopD%CtSZg3B!;sDJ$ULF5Pa~jHLo_8nqy_}l zSvrK0=xvcq-krlE2x4T4(Q4^#e|=q z#fR%Y`6B5+9W|)s6|ZxAuypX1t{sDSOYfD^d2T#MW`Fx$O}k0eGs$0%x^DLNx6XJ%rc6>5caq7@<#ysU3;A`t4!b{FkSsVJm_4h-y?cAa?y4%90lQ%kbj?9*;xW3s-i%agGF`*=}{O=!J|7p=pbtAXb z;_U|oOLw37SOlg14q z#I%_oLX9O=EjNV_QSli~oC*V#kP-!vzb?LW6#7~mKZOT@$WKvjsxMZEWc^5!NQTI{ ziLjp&M_?hg>pSOWq?G*&su6G1Zf1tl1dQAA4%|XaXV7AxUM(_wOl@oakcxrAifB+o zx&HHNALstK9|}Utuzz3{?r=%u8U3tK7Nq-yq5s53{r|r`yc^E{zbe=NFIn{+djJCQ zx9WE>cC#bOwx~bdOs3TJ-@Ii(h4)2RNSkP8SZd{~iL1gd)v)3u7m1eEe*F!Ib0!P} z6oi;HxZLH_tzN&@DKzjgHKkX*!QAdd9|-ym1l){p>4OpAx9(eR|0jLqh@?3z1Ddx`aY)!Deopz zBwfnfqyOG!VFKR9b^rEa&@IH_NfCJ5McLk}n^&%Ov6=GR zx&XB>E~!&cw^`5*jDL8s&k3Lv7CZ{cfs={J6Y2mA|E#>c0jpAhgiTNl&dkiH*LGX8 z5=vm;2|17_5hBy)hiV@o-umMEQ=y52+zOszN!oTG0i`3!>}$!t_{ne-M8PLUz^MX~ z8W)nXUlkbr_k--2hN#~_i72@>`2}%l`AjqN2TQ#hlIjs%=M%j!IyqpW5q(Ux zm%EIjlZ#=>r<1Grgx`QVy%mUb2-XIPECQjkveD07mo!%a8#4i#L$*1?Ictg4V9q;q zJm5k{QUyZ;ZgFuDx@_RBl8bZ#U~#mP9R_%QvfKxpnC9RnXznZTd3E9M;9x%Fj-U|9`5IdaAJUO%|5DlfxH7UHFYR6pGxlk3E@sx)J{|{Fxr0?8s|U9oE-4oy6_pr1TA-mtTpehsYm)GGI5pt-BCTA^%84x* z2_N;2^b95QSKrwsmh)%9B(nyZz)}XCowt`))Zn&ft-2ZoEeQZ_HpltTUA7pWZFTE8 z$zY@qvrJ!YjaT+ta7EH(M~zpXyOk2(%wPmyg+J9bxj;pQP7dOBj*&4Qb|JeB?$`~^ zerf8?&IcfgAOC57T|s{So$q-honrr_xCwpuGvxtre$}6@vXD6)+TPI4W9od;vqupj zquFP4(fLYZX41inUi&unKJeJS^Evn%-lIPca~(BOEFY^i?Ih0Z|G-ZC^^2)H;4ZFl z!X=ikz*-iScI*GHSNx_{HGVOxwzf97hKIo=-`3Jnx=i)XbN2jrE&_biocFMK;y0ir z`<=_O@7n1HV!YGSQvevdEdNE3vD((&4uQT4G+Sb*f}z6;TqbC%J;wf$m`HK=ZmI9p zB}mr@(0ZXM3xym|?ZaS7PDz1^>>A*L=rLwthQfocK!PneRGGgri;7NJaEXAG5y(R; zPCn2OgNEDV$BzM8%`2@8up`570J~b6pGQMS2ZB8ZwD&+Y41Hw)y<_6y;^N>8t~kM{ z5(W$!aL(tQ1b!A5gHUP7o)1XQyn%*?n-lS|vHTW)sr;$HJ`V6{Ev->F%v)Jo12P-% zcu$@@ftKK(;&p&ISv|UUU86E;x&1NKwsHYz%T-a)(VKv9f-@aIKRf=-R=k{b$DB2~65B3Llx_Aco*Jmuh`Y zs`s7!6|?Ul+_UFTwZA#BlpdpJbnodM>3lT>WjcxArJFIG=`XW=&Ub=C^+d|h8YU_~*&k81yW;pTL5^1E`1J76 z-idQ`VqAS$au+J3lK#xvIqeAj!2o>4eyh^k$Q8#*O#eG*hOMCy;PHHu@O-V-S~HsI zBRC9z(f1Ie1lC^hcemaL1lW_I5m26x6@^E}_x4Zbi?gY4y*S;|jeLTgu4h;Z33vkq+@e6_#Hi_SJJNLp`fsYK_)Z5f0K%N0ZMo20j zlxe!Ux&T+@4QOM?iTFx)jTeph_^}F>2p?cw*VG7ol?MqAtMZ*7?EAa$_M1@Bq=uzZ zTwh;5l5Pm@0sxMI@DG{l*Kq8Mvm<17wq}V1frJIzckp&R;CGw{WGS?Wh6e_?V9-WI zMO9SrW%k43Uk@uJj2Wmj!`s>c#~t#KU;ueQy_;U{8r21U9$?oS7immFm{n+KC_qLt zD=LOpoSGybB8IROU;PJFuY(1?yQ@p5=J@*OTXT_qmA{^caQy)_*VWheGGC2FE>h2K z`{+p2p6aR@PgDm$+S1Ax?N+6yK)atncY7a`Z(42x;45Q}2uvErP zSdcbGo!wRg6~0PEn@0Y4%`*l-b{PGfu!iXSoV=$Bh99;AVR3-yjDmmqla&R-AF9Bt ztk*z-Ybx{&^xZ)#Zs}`I*EJ9fEbHGyF7g4o7?%O9?umhc{e_sH5L3{*ZUhqKUoDW{wBjBijIZ`GwlE( z5c;f2t6eq(9Oq>rM-k=-AY}iZ{9!%53c@A7jf?vb!wBvSEHAE5lRMjA`4S)B`REy_ zRIa6v@Oj|0u+pFTll1{UKKKdzfX4#n1ERXRx=1=z6_p<-N^p@`7?AAWuHH%5b>&L zG+<<7E7f3?48#VjQ#XLy>K*2u{?{higc)LqLV;^96r$yGHK9i(5e(Z$vF&fQGm?= zK^`zWvE7TXHx{#c0yhJO(EYGh!L1rjUS9R|mnUauTa&d9%!@2bwfMfGRD_#o_N_U? zvTXKKGk5UYvxoj-K(4!{4#01C3cJO)=T;U710G>3M#8zT1+GgSOfvx6I>P9I#r|a1 z^tziLh_mKkC zd;BGh_TqE!uDY%cv{!Rd2_z{)w#g+dOhfx#n_vR{K@c`5f%NYRv7hsuY zU#Qi}|2MZY9a|$r5h0)6RBMMX0ed4TbtN%+xx!>pXWS`7;fFsj)!J~ZOoVj~YHnaa z`_^qHgo^ven+n5!4n~X=zLcS%p_W!$=fW&sL2+^4+zxneX+=atIvTbyQUK2kKo~>N zWx+5>;IWBoH<1_;@}xx|sOdfuZoaE|JyHx=F>Mr{HM-T;PR4j{YqQs{*C9`Yg_#+M zUDvTG&;<*?7f+V^q6fP+4E#-Rx3=DjGh}eXnQ3Vpzat{)tvCL_{3pg_i3zJhvB_s&LSr1ZKVK$QS*}u(LfjJEqGzFqFGcwFz zg@?rhN<$G55tO1nHEna*d3mf%OqjQCH$`{>6%23?&U|DcBpfjGc6N4zdLX@TbZ+My z@=;P!Qn(Za$#8|*^x-^`@jPL)%tkws4*CoL6ZjSr0|7T>ChdgEcZ`jUAZZjL1Av)6 z+Y&NXY8@0L*=pMA#}m{{-RUSRiy;v_$ESS9!6{DCoI6-wxkm z{^z@2b_EJJoM`0Sgmi!jN#V6ko$$QeSSGRC%(Z&LrJ9z81`e`dQ%O!vK0Os6Lq9q= zz{0`lUnzm3;`7Y3qN3gz`+y*sd)OZO`t?9x&B%B-PBr~6!q7r4@^b6yatmUd>LdO7 zq?I3$hB1GYkNolRBb;$Y4IY6cYUUPtrVBJ~^z}nJ7l@^)A3gdzQ>d=4KKIs5Qj(}M zr_1b(q9Qki#H86z{)9gvf3&i&+3+~O|KB6C)>rH92102E_`FbxPG!`y3WT6Sj*qES z7-VfTDZx9EcM?+-+})bxG~gi2rHFp(mVD$uy{~VnmN4+szJ2?4xG@IH25c=wFg3ub zVxn0Fw-hFqRjCK;wy?E8qf|*jAs#YkAYug$fG2PrV5%tsNE1H*IK<&}*BkU9kbrfI zjNU)b^jt}g`F}ip2{@E(`@SNLh!G)KGfLKyN}{6BNQxvRQbFuzu%)o%}U8?-#HR^tjh^aqU2<6)Yl)+X$e^JKJm)U%?;OtMx$9-{lFfV);x?{f1Dr&X=^z+DtD5FghrhUdV71( zb1mZ80|fW}!3l+7uTOQ7t3Q0WZ@EWAgn{FC?KZaZZG;18VqyrLf}Wn9f7@<`(vxDO zvfc?)Zx~EdgxecSa5P?>RTwAg`1oCv@?ZvH_ML%<4x>24#t%Kl>6&&blWpzVy|nSO zPEJnfpDirD;LLX(>*#e*WEWD#@F26OBns8JpdI5!pI-dl+3E4~%c;89y)y_jPHlR| zny@6()fFKv=HZc6V}%k8nuLQS#1LK|ab;EeFyf5c;eypOPBnzRfBaxQJTQRdy$Z*e z;NVmYGkWvpJ4%qsr=P5H4AQPCosprJSQve^*-3LZ45@mzEw^=TGUI=H#p9FtUm3!+ z65-lCF;{bNZW7-E152GI&ItC3VtNGbZBgEV{V)~#?_lSG4hQReP+U!a9sBA)i9*V|jl>$L<@Lu)^2)Zup4N{dWQZR?R89O?k8lkp=BI>b(R<<;Xx(vgK*5aEt znrx;Kt^HIlix?>kza&v?ig+tI4Q_M&&8JExMtYss>6Wvkol3ylO(QV-h8Z0Jb*BC|Ee(m9!KKVmz*FeKm9)HBi{SIF_K z!k_ik)jQK1rAyLomdI1zy>n-`%Zx)%mVsCdx{{uFh(c2 z;@tK~ZEVc(Jt$F7IslJSHam4nQ#beRV~D#RW@egUR}tc`Lzu9>KADMANJvMQyV6lQ z4f)1p=QHMKXS=dK%OKm8Y?f|N^5=UlY=9c->Nk|povQ0P7u+2Az7zU7_%CRE#mM*{ z`Id!4qGz-~lRuC-xH89i#2CoINb3|WZsWd%C8&yMalGn|-DP*MDPbh>>){OZ+1*}~ zlattaSUeToW*vlug;mL+e|*g!qGefCXJnqUb9mYZGilGb2e<%*YTLJgZ4RTJ`Ub{4 zkYXGaed*)n+Y~B>q=gUst*_sfBEe8v<8h8cQYGs=+GnW{g|cH&;VKHp#f%?=%*v58 z&8xCpEw4j={H_CVIa&JB4baU=vZDZgFJDspOdQ zPv;xs`84BnW?_KPal8&cHmj2Hx6C1@H{%=(2tStgB9{%F{}`A`p`!Rc{a%25OU!WG|^6a*|}N%N#P;b!G9#F>de90ilz-Zix3!xM!=AcJ6L0ivpk5}2sH&Lbvy9Nek;J5iU5;S!^QC{s`qQ)+!|JLHIjRD2x zHzwWl=Z@SK5>R_nW_qQ0UE{@x*_l&Q(IPQSr6unZze{k5-w!O%l#dG^9 zYp+%W|DEx&e%cf15GcI-TSp9{jKZ{lXa-$Pvel>d+J*}|nm?Oijjh(+@ zF1z&coM1A&4rNEjel{37bh*1vFm$~?uC%KB-!dLo8us|f8Uj+eZy6c?p}zhFvY*~n zRyO2zU%vV5()LABbzPbrp#8wmh*w2NquGpnw($^erS#D)2K*s+4U=lV$taq(DV)Ny z*b!TEFN#sm`3w02tX{Rj?7NFg>q*ILkb1=EY|)lrBYxJw1u!vi)7v5>UG8g2?Kt0G zS=B^gt*NOoGc1iBDLxvFkzTG{8`MM@|6?;IiJO2SeeibpfQyg$=o?!J@DNL^xJ3?S z5>LsH!9);CkJpQm)WR*JYh+*XJzc{vez~y2{O2*^-)~7VEi(z%&MIKF00067k|g8# zgTzqJe*iZY9*>(IShn;|96-%&ujDCmNcWa214Ek=@mle)Rv%9>RIcM)zh3R~IS>xk zp@-=Tk=pltRx#wiUVe38F!)(K8J6h~Uq-{@91ORj+5g9Lq;s+)5FXwc9koCR(GL$FQO9b19T;~$f&@<+LB za-XeWH-AI^?U#q>oYA(!9L9#F^ryx9HjsCimQs;Q(t-g~@N%iQ@N}>>ZwNag%e#Ydl6G z{^VVF==#b}>&AS<8V4^{w>w+PJ6*XWT;>DqGQ!60dek((yJI#Sp>$#K=RK#?u%(;V zNT82art>g#{U|`$CE*7zHe?DojLKH;G76{C?dqBek#5M80pMBGkWf}Gps1)_)^LgtrR}^2Qr3Rks%EBL3|2f~f zDqAxxsKnam;LL`Bi7;!xlShrh69)yMZyXTj>72Jo%`wt zAP6-GhP3j!OHUMEEMEc@_!f7EVR(9a`Xvma$JimzO~B$?zkG4R(2n0@ox-aPLRT|- ziU54aysgU~9!mgr{`}way9^zg>)Zd`c%n?;bUC665u8pWZ-HU2E<>{FQd}{j1evknS})!WmPh= z4*=Zaxek$Jz()XWp;PYo_K#0L)t%@4Z#6&=C5OWbx8cRz^yf9AJNXUBnwC~+LE=&eu zBatK|CH?N*!=cjh=2}|g(C?X)>U1?&UNZ|6V<&F5);-j&gKgInaU#BRy98%cE;=@LA3bwPs;Dy@tvCD zILc zTJFMFd%^BqyGB755^Dt@<=$&Q-(^3~{t>3Bg5vuK{b^>VI=A|Q>DX_*#5~<~#m1q> zxz+QG0nPmZDE_0*;;O*Rsn*p5=Q@k65qaN+hAEUPo#d;y_;25eV!U&_L^WM{>D9Il z%hF7BT>vg1p^;@QS2vZ-Yq@i*jEl)Urv>XCdauc$T*bb$y%Ct_iwU*gif_&@{`-sl zKt@tBASmd`MdQCFq4E-`S7(oLtGg)2p=aJED*E>9wk`#Mv4hr*$2%TDzz~19pk4RW zDc^&U0+mN&=WMQA@oTW=X6jzEza~wjI@>gJKf3(M{_5SAJFfiva?!)XBUY!iXE4;n z&eE)7z2~X-X)Ydy9XpfkD$^D`6s1CUoA&O;_Nx?|@#f83gc;A}mh8kQPquH~%q;q} zA;kmz1OWXD7P&CA&Bing<$?h@L#=86eB`PeL+4E8&a3GDkZodQbkDyuon8lj@&8NLwU(v0d51M;flZ~C>+J`T+0g=&?Eh=;591*Pt&l9iyI*% zcvMtWgoTD`Kia3FsF)uge-1Odu!wg>UxF)BDK^ObVd{xwOdJ7Bm!FXV>|+rKaYkk) zaPfhGUWX@L2(z9nFX z(8`gX;IA74H86A&DPov*w|UDJ!V3rzX}r4}VB9e(bq`h%sGFK?p8_`Q7*F~0$bBwV z!VkRd0&YjW6DM$}I_egsaV|H^2s~k8Vj@T`f^TMLkK6CDhz*Ap_XyLt(Y?;hcwo3R zg$jVK6^kg4Yqw(1)XWTE%6|lHIXq0zSTH1R5f_ie#5)ZBQ6*u9*#N#5U;v#`gzQi# zi18o@wrj$zLbp6_V}C53`x%NCc@yn3cE9gvjsJd>3IVgJz#rIPt*;kj9VIC%YmA}d z5H_moq9nIg=o<7fNGq@6g{Z^jT;)X(X0Hxwh2z}fe(@R!i=zc&h}aes5cr&ax*2bI z0<0H1iS*&)M?L#ZOL|pX{fLP-tgIYTRKslPVJD?RGcVQPhmU= z*YDg(7%xvfF!Cv3R}3P?%xpdyWu4Pv7=FpdW)8*bFXz2fIzjhu!&y^g8hj+Bc^C@L zCu@Anosn8lyzTg4d|g7r92QK<$C#}wSq*R z>$T2F<8;{GT>xIZwZG29WH>X@I`ek*SZ_Z4iqmX4Xeh07ocojnn((LDPn8!$mWf$8 z{j?^n?m!L3qdT-bLv8^wdMVJl)~ci%_}^c^h+>#?icFK4dJQp5 zP73h^h2`CBYu3na-HL+;qX*+9QuQSZ#jMXSt3+hSl-PP?{o>!A#oRLxc3>Ix-H%r+ zol4aQ3PV;K1TqB>c2=_&M52#5l#B=!;t>#d-Z)gqXowNcxFZ#gf|mb2ou*b5+(Yn_;fKmn0@d%xuzr4xS z**N)Wqpt_>S~2x&U$`Y=^Z<>nTel8VrGR8QrYZ(uuX9D}GVE6V{jt|OQZqC6gDb6# z-HXAIZ*SeY1w8H6EsWZ0Y-|JpRROVnN{U3F#uPLRpz+pq)=CDaGJAF7CF&eG0-5XK zbLV=1{($Ylu7E8<8EkfPqY-pI_4QrO1xDe5+s>9^wjx=Lt(@RQJw*zELSeGW+)y&; zNdJ5Hfb2(NmfQJYm89lju+8-7=wauA5$6I56+x=`*jT!enV<$y!9I0_sR_D`a7gsRiAiwuNU|*rhi|xwnna@{o%uRrN#1$1AFBtRO!<%pU3xTV-v@FhnXqo>VS@s z26C72%CEmRvB|?Su|W+COEI8A;IBI)1AqYHM)ZJJ-@@W)>&R9K#j|1xvv-~bajxY=1ou#AAB2~-07b{=+SrP5)##%-aIuw70LHlSzN;Ygwa3dC5nvhwm+ zodTWf)3+Jp20}tkn8*&`K$r!5E$02nYR9Nvkz=oqr{_v49n9vy4`*l-hCYkq^5hsZ zk~|f08#N?|qegjQ{0|#Nrx+{m;+OjRhq19upkiNMx)D;94}DX7{3h?a5R$VC%11|E z0x+)V*wcqeD_q#pus1`@vWnF`JuU4rUN-VI9j=e1ozFAIuSc=a~0SU-P8VJ$$G5)i=VUr{zy z1{P22NcFjM@i4#pCY(RTS(n!;h?8jUvFh~>0Q{sU-Q`PTaW^ywH4eyt^L*h+1RF5c z`KS{i5MTw3!tvP?IxJY_piRS&>qzrKTq_Wx7y}z_`JamFQ}(A@D_-FqjZ->V?9d%- zV0!>G7=99sUJf#?~f9li>4z^Gx<0;rrD6|kqh|G4U z;&}$sk{2(wfu6gJa=zy%m~vK6Fj8brPEqQ*x+#z{l3_L=i#CT0fv<_(X6^|ZtHlmH z?yg-1WTu3MyJKE!nHj6Iu}&0Kw8%^#T@GGmEB`j~hhPFaL8L;Ig8gj%@89l!%&S&$ zGU=L`1$S;Dsln`m-JeM{QLme`KCD3&FRKBf1@C%3-zZ|Yae4zRLrgHKlD zRM8@nW(Gc6E8r`bDL;X=>f_CXB6v3rqAuVJQGOp+j^#`Zs zEpIcbw21{PXG6aOus2K#237)m#Uex-i0YD%v0(|MX(Y7dJ2pZu)`V}WsC~5m^3tLk zP7xe&xZQH3cU=1Z1#K~Mg@oN_yFpG*(%O{k9)UPkQrf(0*AINIt=X1R<@KmO63<~W z2xvKcFi)vmwzk8#9l^#I&?e4_bBszB&ngCiIXQz92apdsgg~-<jB zTN-zTEeM>&MfbI9rx6+K@H&P@-|{C0X`uRw(E#4PJ1&02J#yEj zXN^NoFWSug-aYfHCFJ45eDt9pn+($ztIH2l=}((%a0}Yzv@oyb?{iQb9^QDEjzjq^ zvI*~7K5e|X+wPkK$lR2a57?H`xLlL2mNlBJ2F7L--d^a^hz^H~n;Xw!cJtatG(*2# z+tJtLVv^Rwht_n8lvLdYHgqFLH|JEG(=DzMr|KL&jEd|LYKew~N}B0$6m4NYkqwkt ztW+`L?81@sAMz~w(|E@V3nOQxSX2TfnC5GID6{@=z%%qn5Z;}Z4o7kpv@w{D{ClRW zW!1mq<8&wyOp9Daz1MMZL1FPQGSUT@2?3(uj;B(o1b_j?PhIy5A(DbJ83RT^)e>1H zognqFAT-EKwH@o!v);T>l9Rg(krZk@F7@rQvKg>uqgF#*g0DbHs&MqGfbS!5C7i)% z!0=1AvvFac!fZ28uh`!ZdXWk4J2)UfT1v{+&Q2k^1|Mc{U?Aq=4!96ZkaFD)Hh?vI zg8c8^JwSqJq8s7Td;lnbkf4?+!0Ci;i46h-oGtnH0nb-s!v;_U#}>(x#K9396jY3> zg2NBmY@FD7TV`F>2RC~az-pnWb{$GSIwe^<5(6`!XG>#y?b%r|wSdU0m*C$)iDFs$ zHZ(LCr~kp=PO`cqey3#En&8@ly7~}VVCpp*wW_HFylKq>*^j;aF5{BclXLf?B`K`yNEL87Yc8P=eOi;~3lW zAsZ8Idx?PSeK%d@3}1BN?V^KPjJ?i2H(t&G3iqG7zq>mIFhPz!GuFBLLTy`L-&2SL zoZ%&9beiX zFUrh8?B|IGeV;>d$t3DD=cIo0=ya325g8bTKoGj1OW!RBH$Ko2l2Gkk^M}No_3>^s zU2Z)+IzSI)+ITD!@UeTQ{P7H*x*gDb>qJgIt1QRO?e+G!p`pI9v69E)N~mCCVkVJ? zv57M5^akloxsSUTsB~OusNjjYvKUfd-VKNW{&XC|K1q>?RO4il=hMdB*@U6885w&i z>~Da+eX>?ijIDk6@EdYAoD2oVo#X4xVn40}3fXQzENCpJC>nxH>p9#U!;<1U0o z`2@zRAJWC9IzJ-}`Q1OLBM=Xut^LcJr5A5#1vHK!Az+aG$&<8v5QMRr;a^Zlv2T!v ze#;s1w)~l!XU-IXy)EiO>4fV^Ci)1_#)7+kp_c+SFp*?qJNyz7VbnM!{F~mIt+-ID zZ&Ct$fc`AQKYP^UbljY>~8z;vsFFRysZ>expMI+3UA=Tk8cQZKw8 z#h6hz7GL+3*8qUX;HQO!A;o(X8!O1kxg83LaLa`FcsAyPnEs8eS(uZPbN%`-aQoQK zx_`;2lF?`5RTlz&!CHc15lA>jQ=AvR`TGH4DwdDi5VD$adbl~CQTSzu!ik6?2zW?O zP{(YEqxE1h{hpX$VPor4thJDDcW%8TS3HY``YfLn=p8VWKoRL_`jSPaW_o%(0|UMn z>Ls!hfno8n`JkceZCi2IbVYrSc}H3m z-6$M={LP_67JM^ajbxZsx&io{-Y{sC^BvFQn7S`|kgSHWzjR7fujBryPmP`;>zFo? z^dSyAeHuEU5ITKR^%Ymwjo=;TeqSuS`MWYJqfj%TSV{3bck;O`=_%;L1@^luZ{OZv zl{29%N6i))&(GszdSK4yDZ;6S^e&tdl@oIiQ$dSkyOohj*GVR-C>kaXp`%G^NU=2f z;j)+*vH6?b%l!O&)Qy}Ztl;tBG ze1?w59X)XOZcq?DmL}_5l4wk1zw=dY^{-#Ppeih$ct33a>NMH=?p+6T6$WWbHWAta z#rmoGM4{{F$IYZmrv&x>a;^lBKnO`HD=Se#lw0L+F`=4ORaKRZh_;MGkAObA?M0)) z&Cxcj)@@NqD0Xh&zHOZTGAj!O#{K)pjl!d&qPF_EV(KM7|Igl@9z-sx2(gRyITxtw z{-!AgS9+D0UX0pzebmIsDVIXkNtRMpM%+joFNON<#N95@l<`9nr#RCOlK##bjyn~7 zyxYqkn3R>pDY{8g^1D^eu+w*393*%cr*9S&HJ|@O{QmXplMAjt>T-G`z+M?H6O9w@ z2>sCc^Ii?sT`s0zCesZfs*mb^0i%e*Iug}NmLZ@^nqph*@kBk`2#_=QS7_4vqN4Z< z+83*~2*au17Y~i}_(hlXqI(G0Mb||ABzSeE?AA+?>&v%s*N%mQ_y5DR+Bd&lwm&dC z07w9;E7XZ(CgAJMII2PAfeamVdh`DMW~1;4xr6Z%K$^R^e-Aq?1~oo=(QwdS=A8-( zZh!kD8k3Ixo2Yz**PnG>GeXYnYsnHrMD?oe%iGAUlI^z=8?48>PVg2DmtFTmp@Z^a z93_~Vni`BEaMCpn^|{0ixcHXeu9QEu(h%SZfT9Ems0e+0d@$6hS2P#PoFz=_4%rpB z651|Rx`nQrXpACfaEjikQy69HqvfO=y{6GJx70NL6A^A0gZm$OH9BBw z=HiT=j2OBILi+m1clAbY_Qn^|d+olB!*bA#ZWS2r<{-J*zkdmITPZfKH{Z|C59=VO z#UiIANO-@Hu<&)b65<9x#5qhQdcPn%=VnLxK3FU0aa-<+CaY-%g`zx_GaP3;dIF0* zdf!bY_PVWs$)9gU*Cb(mcKh^vNu9L)_wmWaSFM8U!@i}S7=V-xninNCz+_H8$W+OBRc>?UH-_+6*V=xfEWU%Fa@&HAfh_~zX&pZf;eax{b0iio1`^q3~qvH6VIF%b+7E*D%0Q2`H9Hv_sS-tgK zm7vkS_v`@=?|R&x%k@J>diph`*v%nXPy4K|U0X|~qbtJ$0{snU+!U&!fKJ@NlFdTH z$Pa+DBhEMA#z3z+J_+~E+K5LYcd?z{r=&L7fT4BRQ&+d`uA#v-1Ti2+apNO(5@EW8 zz}7`s$j!2QXeTEf`YMft_ZtkgSg7>lsozR5Dhm0aQ$LOhR1QtQK?Bncl%l5QW!k97 zvL=RM#dE1^k!@1SkDLx1CY$4i-ahX^V5{}~!QVoyM6}Ggr%>MZ@ zqXPxQIUw&47Qz{*=j5bqWRz;0UR(7SB%AMZvy$$nQ_0fCxeho# z&_n_d;#IG*e+A00`^N|T_v=^F_HQEtr}_G>yylTLq4m6D@hwl)ONDY%OZN-MerYh> zgu$!4`@rEJE{NROAPg5JB9=Wx;NB!E7L1_=;Q};Rv0~cBUljY^$82f8&nh!*JI=jp zrs3g99;oK3K)4^3}_hs7t#}YiV6Uu@A{Iz!XRl zfbysa9Yd}N&;>Mst!-@!#BRKKv#9r`O`8aV8oDcNt=OFZr6#|JPb&={ATaKD34#l_ zPAc*{pPG?@EiCi7ffTssfYFkrMHK#*+VUroAMwN~IxYZG^uJ3A4nB6)=nqidFV+eW zD(AM2v=?-q%V`l{FKD+19$lo701m(gP(Ai(z?PbM#!VRc=cwSOc5x%TahWIOgRZm8xt5?3Ho}+4nh|8tG$3@-=%TP9|bqj?O<}I zt+D?KJWlQ(?my{$ud>H!upXTQyhXqX2xJ>*^;pSLvGkGoKmE{x{{GoiJ@UuGvLC^` zGhQ4W7$i~0q^h6#P4uA8r>LDL!Y$JcusY*@{fYr|*fsDq?XW)T`suP1t%(&lghbwBxR}oFps-Ts^iMM z(UG(~^9?(j?tr9yRjI%bl`PQt; zkf6nm2%$vA#$L)Y4E2zdx=|vy+E#mWBkBsj@&k%%pP(}VF@w^ZLwTIRJ*v!n>sXhL z{{fCTr%&@KVV$4D)Ly1+w`A2auNQaUQlfvnYCe$HorhN@ea6(-?rVV+!|e4{aioaq z{s~%W5(CGU3kv*8JzQe9$zTDhYex7sY}g~>*WiF7izH^%5l)&fMx*ijXKHd!oYosILQraL$Jm~fg1z{ z@+g%GEGX7E{r&0cPoLg-Iha+f_nUXS7vDEQW4b*DH zDE3iuTjgAYT`60dLr_TpSSqY9WF}OiF}`6B9^B@-q%V05l!b9R5TsWk<9T`42;c}P z0RraN3kp_j`-Ns)fqxo}4SGk=R;WKby$-HnIM8{^5e&dL$I>2@b>XB zJ!5!Zgp_AoF{NDTs9fhC|1c^Fa7shL=o={tg(4zSiUcvM98C?446~BOm#xgBoh5bM zFZR#0O#wtwuDhu49jc{RorsW-Gy3|uScQK^b_lx;ih3TPP?=Zz!u$dE`~ zGovaJe#+M~vTw{_^9{nDtC`t6@Z(3Evdb0BaYtnZ_7jg&6aK8KWFd|Sn8}47y*RIF z>gebQH={R;++UnzI(Oi9$!Z;eJDDG%Zy2WzHWKLZ#i**<+Mu8yWHMsjTq9sRC#9QK zR(N3}Us6L_N02Mo2+-quZWW32A}KNPk;)BB+}kWJe)6>)s6OmXj*b(=u&LaOQy9lu zCBKG+1-Wg{GBa`PmS5$UxZ|YwOQ7;~yS;TOq^o@fS_us)O*Q~ZN}L%P?2S)5owq8Q z2>S7O>MFZrdUxVEiBk=VrblkLrs`))wQW}DXUNr={SIN$WgDBa5KkyZar3K^SXe5G zrZU`4u$6aZ9Hf!2!~RXrHwvNBO9k%|3J4Lb$Em`%k2Ogwo`cQ>)2^|nyu{F45u(7I zM^9u2kXAYh1ZzJ1I)s)rqFTl;K6~fbKx@gG%f|tRZ172xpo=(Y?f_c7)lV<&9pqD_QM=mg0E#Gm;om3yU3WS?L!vVK=Q$r$ z<>xBPb@lCx)$y+rE@Zl()O+k}w}a%VwTl81H&ldl$V?_z+%|WGtX*0!+Be|&=5(a#K ziQ+#U4K6V!{mKpVU!P#US2~FdS#i9( zC^c=gpHiJIU(b=D)Xyi#UT__XK8fZwyMqIjePx=rJyv97WSG95-0dj>@*gHG)wY?B z_9Y)Q97ek5D*s9=e&|v8(cC16&)F~DZdXC}iRc;);Kg(}kGR!we^v@^p{k`axyf!- z#ivtQ5?X$_VT?oWx>9WZ0wOn#b{{M+e11`z65Vid@4L3j zn@nnXG`)wlUdK)H}X0m6e{AlQS8sW0C@c4_C$L$)2`+@_A*~jAS#f zlMoX_?qak^%1Y?8FW>8mP+=%7X}$KTbY*%UBgD3zB6s6c8dAb25npH5A1RIa)#ULm zUIL{ZJ#4Dn&#@{t6VInLpE)y?6VKm9|a&Dd=B z)EuwdJH^)=abK4?X?s~BM~7k7S9k8T%dvqqU1Ey`tp7R6?0=lupua5C!FS#|;uUia zFK#uVrk}mg`S`7%IPnCFR`r4W3=UJOylVJ5zC|p3-IC~3CFk8P3v=~3l?^@zd_RP2 z(=|?e(-L9%@a})=^IM4fM6}Ya`oBRm$wNL|#wzF3dxW`4dmc!*D6Z~zqOSDLG+vJ& zmhd~!@iUBP=Ig?A5Y4vyLu56IYQE+6dwJ6A#epJK9$zm-CESy;eU5ab$(zO;M&Y2J zp|f?9Al@6~_8iEV<_3?RMSvgV2@aiD$f`vThlepeP_9hHRUdz}{^u)iirTuiI z7k)IircmFwsqJ%*$$xv@Z?|PK!>y7p+OON=Aj1T)Om#?L&X{mg%MJh>!hUjv>>Ua< zxiK;R5L$rk*=hIeJh?YJY;-%pNlGdP5lt@W<#y%!MRhBox^E@Xv)1pv^h z)Ny(FN-Rj7v1MTOrH!|Z@UE@zT90$Qz@b~M$aFt47|@uEe$P9S*<79dYl4Or59R-9 z%bSI<2pA$vuNN-($KNensYlb)lH}?T6WY{Vz=w7KT4sbC&z{Loh@UACVE2Whd3yR} z$KBfGn_Gt^ww~n`;x}Ep^avgWdZ?OdK4Srq%9K=ow0l#K2R{5a4nxW0DnZLA0(Vnm_Rt8}q^ZtMpJT0e9K4`k(WQS&ZJgtM!#j9Fd4+>sy zd*YoYYhv_wDew1VCI*x0kiI)LI0%*>x1xuQ84aPK1C0z3IkW+y<=bRdswl3>fxZd4 zbJTq3&%wqxd`Zy-NdO1|r8J<&VVEt8ymroxKzKgZoC!n<+FVe`0l~o#8shZE4sDWw z_xXU*0#2MjG?6oM0TmE{umcAe)KYSDznsf5O#eDLdF55>&7WUVK?k(j0l|C;M*>~+;4eQbylMH-Aq;?W`D2N0c1)5>Yjgd<_6bl?^3#knC zH8cePN_SuS4&MR9Zr*Tx*#ov?_Z8c_x3#mgq`0`F+X1mgfUEt?`GPBvV-1-U?!$)>vkbD8yCX{Xyw4RWm(&v{C7bZV2(u7C z0_c%F@b1{k5yuJ^`f*CiVfs_h*HGeN*JDbsUg6L*W5R@L6O?yQ7#EYMUp$4Xk-(`Z zxMFs`t9bl4N5qxXmNO(zhyNAODdI4Xybc2|0!d3+Q<>Z_j)i_gf=cqNI0q{&)!u$| z*)jZ1(7~bLefF;%b;`I4&Yw?cvC|;oo&s+Pr3#(0apT5oN`gVPkEaA@kZS<5A*?PC zaA8lG>+<5*@K_BPdidlXpGdEun%ULXkl1 z*&&q(N+B>-1mf>q&nj045hUU{TI>kQl~WDK^#rI8Ah(bdz}yf-p2H+c2z8x-Ua4zn zfWyNA%C$}sr~d3HnulP*NvR#bL)fM7L0?}Vkwu3#7g+$nKX*mT$jN0E7RFd2mV+iB z-j;271S};Mn-G2ro(VUEoP@+6f=IBu;g0?W%MnDhfY>SnHh7W0L!t?+$noYT(ix(D z(0tn&4U-VJsL!>kWL3XSKxrlX#$`B&O=UbeW9O|qmHuGDHtw+cRC=3nf@C0LfRSW` zgbVlr1UtaUHd8+X?Ef)AlkmHuD1iO4=EH~5$fL7DGmyojQ71itNC*8dtLK46Cp$ao zx+xql)MLkZnN-Q%ZEYsr^$_yz|JxUWhzRMOJEuX1q0+<3g54!vf}f4q$NZ2k+~uD> zeCSd{tO4jK+@erTp&P5p{xs!=9u~JPR;jjj05pdZNl%~t`ku$C`3{=Ou(w_75v*Ec zMcA}pe5p(om@S2b)7a3EPe1?_`62pK2|pfxbvngU#5&5f*v-Yk!Qsjk6ad%;ao^J* z!9S$S#=>H5WWcQsme@^f9)jtcU=e{p;B9!|Gw${!_TeS&Z0X*ycZB9YfA+&)1|IJ$ z;|^84(6vHt$p&e?E<(lq%XJw9)D~xv4Bi1v#xa%YFLyl3%u}Jb!7QSV%E~Pe!BQv| zrMFff`AE60QK7x9?bB3Y_P){TtB?!9^(^8I9tuLGMsVoRRl-z^eNyAlp#{+WP}U_} zBu)X`v3LjI#572N9J9ebBQDKC9-A>J!5gy2xz_{1dGx3=?UbB1hPrabwY^A5xhsOP zX%L49W2p!U6*~>-;d=TJHx0bmR601#%x0-6M%T`$-a|W#!dI!nb5ExPYMjv3%|aLv zD0wmy;j`~~@X{fMRTO+&aor;JN>I9F@?K%>K%$lV_8n z-uH#s6h75GIJK`7v?CIWA0YyX?;Q`2z=3RMOVCUGVsj1dYUygpFh3De;fo5B+c4H6 zNk>z&I$PQ1OA2uiqEp*OG+4#B_YgY<(cnf9%sS~mzvm6ZEQK%>oKm>h=gyzU9R$rl zaBbzo@%Ll-y+JrJ3IXP_`Wh6AVh8-$Fr5UN`eXB zkUdl?c!QaR@(4IXRfc!Yxy;bqhbS zQi?b5Mc30OsLKe)3Mwl8z#piF;8i2iaB53!pubUek=Kz^BR!zJh6f0#+{*L6auA}; z839c?wp6@aVh&DN81z!uu_XL(Wu-cB#KyIbjMhJ46cYZ2AdA4Dpm0hmd>D0g92_rz zKtdDq%peV3utQ{Th*{OMo;-PbvjOs4~E6JJnQC;CVbE_=q zmVdD!0Y&l@0TPbL!cU*H{o=>h(4Jxi0ROFZG=co$?b%y9*59)A-b<3YdJn8W0?2`p zH7cwXm0`#j%d=A8=aDu%K<~pIxN&2&b(D|IYkb1YUiPgoF~#N2J`g!z?<~p4M|k#3 z5(L)4gNzEt2@CSET+1v&_XYg@AYjoTIS;xLYKsC;k^qpwED9hfu+ zS41SB{xC>8N$pZXhLtKAs4tUh3}7-8lemph>!Bco;XBAUz4%ixtg6sv>gd>SXK#L# zBzGp?b7mctjvYEwG4}1*{I@eb_uW^OUIS=;67C-%A;$)WuhT_?i=V$bWcaPKKEc!D z_F1a!i>2PV6|)6fS4ex;LliS281dFayw=w7yHwrbC9(D5RQlV;orNw(+!x)6PRw}6 z^8eo(Cf|MSHh)LB9KuUH9H)|-YznRoJEsQX+HGBH>E!eqR9d0Ol6zW{5_C9$8eW%s zL!D&$dC0NCj$$%0BW1oCCd><_{i5|Z1jmvO6}tNmXb1~G^4}+5`uoo08+ChQ|7+uq zr+LGF%HLVRU=k7if_3#v+XjILlzVce!;Q6{b3cd89;;4jve^?9LZQ0wQt2phhn!5! z-&$Q#Vv6Ssmp=X27kj)NF$^agrLhy3dml=hbe*%iH)Ze`Uk= zTQ|Eds0*-^T~F+&yYf-3pP4r{`1^-!0Un+t@7(&oK;0i*C|Vj)`TTmu>a_Rm&}NxY zy4*tdEA`T}iX_>&_=;U}Ikkye*HX*i%RX=e`8zQ89-J184n&>X)Id zBBTcM1>^Dd6G)F7sELkS+wzzV8SQ8UDlCaqie2P3_NK=%Mu>SNgi(CC3U<+$@ z0mX4_+>SBHH)e#eL8etQW}PB7T88@1Yu7v=GpXu^PI4b|dA+TzS8(v{`K;c*OqqkU z#kj{D-K%u?85YwI>7r^zeH#h6^xMbD7o(Cc!jRQGyp~xdj;0Syx(BRPkZRZl5JIU9 zKHK@%!dP~;Ud$Uwegm)Eq_8-%pGzIXplgUe&b1xZ+l(UgT=eI6eVP1F{c7m){Cgk90(l|6F0UI%TxzkCV0 zI-lePr}>GQjqh2(V9!9ne~?!1J#yq2G}|SR{_*&ad9Ip*=-2J04A^a3r1lD_z)tUg zWeLRvj_wSU@M9e+f=U)y(i;T@;h!@(aRS|OFg90g@`w-xHCN#0x5fh+EU1JD<@Rp$ z{zw@8U3zr!#GYRQdR(y#*7Lc9mM5e)BU6&1YcPrVZu{0C<)4P-rRKLV8x zD3LKSza|{O4uND@OL`%+M=zT;QMVjY)ib_Pl6m#5N=!!Vo($wUY?fwV=}f<{xUV=xBoOM7aU7arjfDA8Z_O!SuT<*6cPz z*NP4HAI>H?vaRywa-JiKsTl?KpghVb3YC~r{W-B3YKb5rnI{r@WLV~>T``tvdWv|Y zJ%1jxy;qD*nRbO^9>ZhCr>D>4*&xhMs5(1dg3#&X{zLVUS+q+=My9>8GTU`pQrYn* zo)E3lYBs*(1_pD`D_>Nw5zj(h8E%PEe+~y5$QM`THm+jj+{I-{>k`5vWlF z0VkZ=da3&N*%>of+p6<0K4v}u-L8|qeyDV{KqW|^7CYpwv6K*1J5V&9Wh`px<{N`l zeX!ghvf-F!V`K9R3c42*w9m2UY12Ai)mBw)*nSQM zix`8n_B`7GoGNHunrx0bID9~z1BD@OStxO8FiB(kNR&oGRn3Uu>C@GY%Bo~v4&F`P zWV)phwC1=lBh6!9LG5_oud+DD2YncV(q3o=$Dtc$D}UC+U-;5Ab2ncA1n*GFZBQ5t zg8QMzLKa1q`@c76R_;@v=>_ui1IQOoj;uV})!}qTS5QATUnC-`Z#JJP2PX9mivYcs znD^-G3tvzIn#k@x#8rB&U?l(SHpRFBD!nZB_XZzM^#{HA)pfbcHx$F->W1T}3A<`~ zCl1pQI9Of}yrUE75{d;UB^6a%7An1O>~h6D3aM&y2uqzH(r}4NK#;8*CD*Rqttizh zgI@2779?DV?e@XhtjhX`CT>Dr6lKd6lvfA^c-4BRHu&M_mDl)5w3-Q=vBKKz{a1Sb z^<22oCVbzVPg(16b^q_Fsd8JWhrG#^judJvh@|JwwZr?thMRn21p`=Qitf9*AAo|V zh#$l|B_2Cb6(V$L%K>jLMiX9|$&rPXn%TcyV_Qec4xZa1;7qt(Wqunk7rhKJ%W){~ zzQPYT_*}HJ0{|t0!pMbjb%%TFB}NQ-#{(@mrAoPui;3E3p{@N1*_dyfhTI10yD_-k z`#5M#t;D&KrTP7!FvzImk|aK&U^9`;Gq!xiO4H1iPSgK|{mR0~^+)6dvKrjPj6o{h zl{dn)AT>asx6oa|y0&L}Sy&Xak)3wCpwx#0Nl)(zG-Py2Z!cHz;vC{+$(xYAhB3ap z!onzSF1Nni0OFh^TF`#`g5>WY*1Pc%q?H{>L>id)7N7DP(z3Ei7nS7Y^~fSo3>kIe z8oHqk))sv4?RHU-R&~Y1qz!lji|1LR&HMN7M^rbCliWio+m~;E$IEFq3o}CqJ#-qp zp9cVLLPpXP^dq|5M&S+C3M7D*!?2tYsxlZ1X%-gxhYoo+4t@G$ z*9q4LG`@Hx+0UM3L%49g%~2DYd~|O`(6>V%T=Lng&i*<8k6|YtT&KDD(-UmBmW|H3 zp(|HDhbtNCUNvZL07>Cvt}M@6&S=deoQyBbmoOlP(aiN?8DI-=|DcXh4PxdKW~HJ9 z;99@FVp{;jm(B+IrV=JM`|Rzonq7DzT;(&&#KABq}I&c z2yR;Y{kEP`(_w7 zU=+TdNtIij01C@gp3vgZ&dv`eYykR*7%S;&$YY@pd#Ig+O57;ih)DZbuP&;f@TCA9 z_?z}ot}Q|HvxaE1mg7m0Sjtvjj|rqsCr_?F+Q{tHFyahV7pBKjpJ!yiPKR79`&SVW z5x`Np>yetytqylD4AYRlxGD>U`B@r;RrW$Al_()EA@R;y;r`PZMUyKiwvRRrtQML~ zY2p{C5F_H}f~G_cy^v-!%+FCI8J%J*C3H}j(fV8~{<#Ytr&&T{ZD4Y(AUqQ63e z+=xM*MjE*1u^tXW^9?C4alBsH!_w2aUn7|)_T8s`$o%~~{|IA(6XPAuph){Z@&}5S z{9}v^xjr|x&?(3>Yq9$?J8E9H$T4`ZKARe zGW3XuVA|+@A9FMfu#0W(sR=^DSj z5>{@WETI*SA{fO6t_%>89(Om|ym2}Y2lG2elLH5CNmqM{3_4*#57P?Bmk4z-;<}-~ z4(c7BN|&xyu7kMg2>rv9at9J$<0T}wZ|A&x70D#Xzq2sn5yg`rW>G_fxnm_&53hwI zIT6`6bCO~FvB8Zi97gK)BYPK1Z`#!Q)n2liRnA|?qC^p4f-+1-!{vVnEHTA`!MJq4 zWKV`Okq;Mq(D>uvkfQGqG82+Mp~%ZGE*6!L_)u9HuS{cwirF)JD@3QDi-#N(9qD%@ z5ahqf+eDT(&g1BY7XA|0Sn&E;3+oHc(wQ@rcRb87x z+h$qKX!`?UNW(;nA|u%e;cxYdJ>QoBrFEQ{pEw4fjS~t_T3JG7Vn?u3-x&0TRReH_q!bWW^ zt&g>}gqHZq)xFu$VpREe7Z&HaCB#fNqKrbCK3e#+@)ZZ085xXk472}##_I1iPLA8Z zZYHyqC=qd7OL55po(``~VAjPXFAO{s@mu+&O5HXUjX2~y`5WbzuWf$eM&8m%p)Iq= z#0(V6t~K3G_fP71^=!dykWf%{nXv`G4v*_d0<&;@W5gyZVh>B@T$X?GRL}B!4jvTh zQK*lt6{1jW0j8|LZ8tYZWTgX7gV9!I*OkEQS5EEQ(|oir$2SBJM`;Nkl`b1`gicBE zt8QaEP^ZdJx<9eP6C7-t69A>jQ9fG3W41Wcb;L+U@s4XpcOe;)#mW1}}q?74#PG%~N{BXDc8vfo%kdzmv{H%l~Y-(mkQH zt!LWOsQAmsdH;+c-eTRYKG#3;nFdABG83L}rmm}WL|9Yp{^hsGa3F29S3bleNbfrP zx_wmLp9eVCnlqbx)XClwehB}0Dq#Z!6Y5)zwQEnN>O=ZVp+dqeAn@y-sV?_cKcswJ zMXe7E_RY6kqmCo80h{i{6D$Odw~ug~?3|bce1G8r!51{_B|Ar6{3WL>Fs2wfqjQ?F zbi=o`uvs_h<32$7z`F?yca6aUUjUxB+qn_OxK@~UZMb%|Tl_c)WM*(SQ8s|r1H251 ztFw`YwLKIX934f50_fiy#1Zv?E(Zk>cQBktI3sxW?Sntv&5GmUj{p%-zs9~bJNxC* z;n=iOapI=AdcZLp6I$_tutcRihH65cF!5YDLmd7^{{HKP0v8S>uHUqeOkP9(GgPZS zNqzNJB26abD#++WINg&oQWo!+f|DaBx!KlXwgpoxK#;h1=3gPp>iPAHNV?RXjT_4FGtR#dBx_nl6aw^GBoWISRlHXb^-y+u%lqGx!iLFp3{|Vala6 zfspA2S|nZ!;}VeJXp->G6J&f?$d%9jKqpFA%enNQjy8w75nd8!p;usHf_4LK4X!Xw zv5cMVWXt!luB%ZBmAE@l_en!&1Xm5P8C6tOVK#MN#5}m_e8!U}RetkmHgJ&Npl3IR zu>l-ZeHKx(_$@kO_U7O)dUf5#>d@V7sF+~9MOaf~Hisip=ofauD}E5c!igC3F_^AJ zJ!rdYj`ox+)2$;*(^?Hc7VinVJDAiyH28Z-M{r4DM+5Z&$g(VZ_Q((&>_$?2V`?(@ zMGQ}Xeh*y+LvJ-GTdopOlVG{3+(*K$gXM;Z*uDMUg(>1H2G0EWifqU!j-Gy)OX8i*XJ+vhE?2 zF!B7IO8j*DFg|=-w|KO9l*qpkA%IuR^{3G+!tgB3`rFWDS=>OhQ7{lu(9~@9Tb^JK ztHJE$KBn+Mc}jc?$uvn|P6DFX%6j-xRqzgC387fTb;3!^#>^}MPb8oZuWNCnc6Dw{ zSAmrtWb-!7kCZqgwqehopJ5k*h8$~a5{e@`w^w0jS1;f_$K4`NtzzM@#4cua8uKDS zHd1^t_w%nM7iz&qbg5ndHG@FYzCNC7+7`~$5@wE6nu{pjIwRUmSjA-?W$WaQ;d zVvY5B7^iAtVh64m1)ITbJ_##i#-NP|=g9h$*<=Om1#1#n(H;qnz`+|E|2+i&j*F=w zk_F_<`>HrRI2WJqv_c*G{OiXTHU-cNOb>i&@#^@CSU4nM55_A&VT68w@UE8uHv5%a) zb}{EJ$6bz1-ZAcpD}eY%&cEc2T;P^A`_Dl>D73{C8caOk^+NAb_cW^~NQ1hMk?UR> zAQA2c`(o43v{S!cxCuA--CJ+A+tsYg3u6-giHE@lGq=&9`1;-fjITk(BJwA0n4)tL z!HNL{1yU8fL{gfV7#R^bb4J=NY1PLO848;Lfo$ChOa!~Q|K5|~=-tOQyGy$U+#wr)u4ogz=)!!_XW!b$v^TMAl1)OK@U2H!5(I2XkXzXja(W?BVkY0-aQ61ond1- zs3v=9+!cY&89X_Ra3Yu_*R5)!w|{@RZnW`0y~x%SFr4}HO>$a6&S%?q(=qn-O5_%< ziBE%5)K@Ps<0=uSLu4B}K^Rk0RErVmHRw1MegVj#j&!LO3#TQ~dWzPkxYtqdhk_90Tjl z=T=r$cG>xW*8oB!-o$?M;(xU;Xc|0!|<$6ERgT)-u(oeT1g{A!?;fu zZ9B`3DrkVR&3c$$RJ5hDlYI{yLGQ+^9lND@`9c@P}z07wVth-{;Rai^oe|r3kfT#Ry&OwXhEA%Yi;}qHS zu~K0Ee1Zl}L>|&?Wpyw`^(cGT^$NW=7VI`w)&<$+v!NIp?tsq4Sl>eck?tAXhPwZ8 z#WB|KHI&TxJrK%w>fcWJoie=lvo z6k+2=FWHj0`FX=ieBH~YhaX^r2r{{q`~|+1 z($Av!>jAQX-GAko1wuH+cLtEQ`jR30Bitgjv|2zSz!jjs3QHLxpC~X8z6L8ed`AlV z_JngF1}5r31J+1?ztu?F2MpJ7FU*xLUgY&fQNzD;i*Gx|!s2~YR=oi5)t`i$GoOm6 zbCf1rMK+TmwA108Mkp6f?bxU&^MW>zi|a+6g%G~*dQFofe6Uo$Tnw-$J~kHWiC>M0 zvM6!bZok=-v>Kg8qOKY1xXrEBB0hhrC9=L32KDDY0Gl68XeO}#3(SZqF{FwcU4M*^ zyy~mwF4IfSSJL-_yufuOpWrYywVLzYD_p4bgW>{TH7YpFF_iMj zI0&S!d)G651eU?f!Ub0b`qu}C-(L{<(a*5wpg)Jw;y-~aLG07u@Mm0lS`{mPca1%B zrx>5USvW_GD(8VW*NN8wTn=Ze6*>B&9|2zV9J=%VI;-yZ{?Ta1M8%Sls?f*>uKfx$ z$M`tv$5D97Q9vbpWIYcPG};QxU_XBTjC-P%-r_p^eLY!TBNAXuV-F}0?S-O(Vq*MK zQZ?OVJDzRC3_Bs6cg?H2Gd-m&x)9lR5F$oJ-*krX=fXaYAoI@fMsTm81JlM*{?GAghu0?7TU!M;$5d6Sjotho@+qnzkxgku!ploWGEh40pJr19J4g=YBKJ;1b))j+N~>H{9HQggR)DJ=Y(zDLE8jj$#$;YA~}{ZpZe` z>6*kt?us#_1ZMvM2kD8{ak?b=;y7*DbIGa0w(T1 z=HONU8WtwEH`9KjDbv=``P|y-A)NsA50EFq2@!L$JOer<&6f0XeM4_~6bYA45F`d4 zR50+MxPZ73D6OCpRwC?naZAe<$LkSJ`14gvb6%!)Ch?d;iNs%zmSuZm&G~$mUo?muZ;_o7qATm2@ZOL>BpZ24%;?!sl`U#zd!oLYy~m~$ge#5KbSr_ zbL}}l5ugmEF(MzHAq@=dcsQE8h3|uTU{fJY)shmoo>Fw!(gaqD82n(`# zQH1xu2q=b+pC2SK;QX$Ih6gP!LgCIXY9U6kLM#iy_t2>uzqmMs=4=C9k=!N~49IY0 z+w54~`{J(=%CMVz0~bd8j%9t~<}sqmWAL+{CwiMFo4#uh2GgH?w+CrOmu!!>Gu*#a z^IZUKcKLA*rLdcWDBFkzjP(IV>7O&Sxb&lu?WUrlV&Y}{4L8#=SC2b5@=@o)+O|v6 zv$NC%ZDu0OPtFYD?qWZ06G3H>NPX_PtahW(7>>Y6%d_mo5f@8fj4iG!72pe zo7(jFtP5=qp_8U2Unr0QVFpA(7}QvaC3C+OneS$r7zaln*S!8064Lq*6anX$l&0za zrg(E5Ou!Auy*1?CjRX-JU)j`j#C!^=5qrY8_P&CjnY4Y!cXa(;!^~*DKV5{Te>{Ql ze0$N9c-~OAnSh|+;0N;a+}sRYNtmV8)HqFADPf*aR<<)?m!SZc@%B@ci@ACy!{tw+ z-W$1R(y)`7@DS@0FH?BP7M60e^Lp{QZXQkn!}pNVt$xAm3ji30uyBS+c^OWA#6vab z9OEmgJQ@CRa7s+=f$9mRa6w5yEV?V7H~KpL2gtc1D}4^Zb?08$VhRb_BNC<^3fi2Z zZ7W5F!u0BvPa*)hJJMb(7rStq$ zF(MRU&N@0O6t6zhtg;=ms_z)Lf!zx$AYD{hQ1OA^AkDxo*;xZ2RanjKTd*V;i}X+C zMzd|co2`*SVGLd+z&9c@K!xbWyHkt0Lf#nhhu{9O$2&y?AS32i<`)n zLoQB2f}Ic(3Kc9Ny)&460;UI+BPS;Z_b3$nuwG=@xsx{-7NDj~H%ytL%{TcAF@@CD z@?a*^Si`by+wqpXCM#7XCDe`pX52Azhav}Z;B>7gXXsfEOY`y?Fjm2`83PIbE5nUl zoUbCtY%<2-2;JCSscs>T7FX z??@Gs|9>q`U?-;Z30wMxYlIr{$SZ4>wt+K_qz)*UU{;7z zsn8IuC~oAl5C=WA*T81p=%^b-^EJIJ>OYqUA`RMHsmN7k1mu$ zX{~)?+HdexK_V|`QzOa4>IYcuUKl`TeZpD$6`h6j4uDfoj_pseMdt1{3R?|I<`)L> zh#+z*Ja>isAv2rDy9k3Ln>vaTMLm7phS*hXlS+40M&PgwiQO@Ak$d5-ed+L?l^_nK z<0OYmw>XqSyFo0Kl--g!!%(sw*`|FOI}~yYzg9o{hrb@p5}0w@sqd!!2EPah)^$^> z0F2;Z_XRdV*v2c0#FkP5Ct0gl8EEl9UUQXv!oMG%`u=@%8v)LGaE=+eEJcmdgrnZc z`MKI0m+Ck`>lL;;PDmD*TOG^pax`tFj9l^<@ynlWxvo-Dyj}wX1PU~ghKBzTc~W=# z<{-dBobT5KGJBXSVpZJ3D=}9G)L|68t>YSZFj(na7mPATsq*$5Y#{a?d}WhXf#(}< z3qM5&;`}CF1n-Z%bU67wjz5r>_V>W=9X-$8@%vDgB zl`j3f$I3+AwbP{BwNF+iqd6p`rLl3jLTL4RuM)#?k`U(u-5q3U`Q69wtY5z#c`h$I z9$KuvfxoykR*&7ua7eRaVq$;+V<0{QPsC#_jPf}GgefSB6EN_cB-@GKmuL6k1o#m$ zt7;WwVq0yu;i8%Wch=34!Q_lTHkd>Ni{ zZ$LkX=@e&hIPIM7y~i*fG{PJ>XZ73uhDiW1kN*)E^4>yGsHi2i_8D!;x78iOFo^(_ z6K-Mh;l(}8I!==GYx7LgHo6kCb zU48x4|1n4!9BtuNPcMRd3RJM`j~@d!r~oDhfZgITJ*y2rfLRbt(w2>HUq*8`zQ0W} zDx>W6{EO&=XnVDZ7W17>*SD|sDUFNhE6kok4d?-9EF$X?^*5UNIUrXeF{*@0xqj~S`T;n)hdJ5+j3wxfyWlvNT}Xt!1D8O_>~m7aK; z&&iL&IgrbP(S%Xn7u}bGe1cpTyjHIU@mab|y5Il@n|kbBG)RE>wk@;^I()$Cgj+ie z7}-{_78vRP^W+Jp7&Gg+O99APK>M(b;?9zS{9@DRqp=+o);wN|;7tQqZK`4@7MPE| zwUc#ZSI(c^HyISRkqPsuU5t#V4C`RWhNGl8`7sEx2jMhDuxC5Wtu&ZMc6+d6pn)Yr zC>F-+M(6#eho#j+lkM32bf(&!8#`t=A6!!3yD5N=Ov7;O^s&2IBzYfk?LFHXbNa6p z$O=xr-om(T{TdBMl7@x{EvezV+p}7`%g<+oEfZarmaRk@V*>(D(QF#^m{@D!Rg~p* z(ZVxJ=Im5PU9MM2a~<=H@8*B+@Uq&dq9fI-B)5la_$!3UCgMjHf9@*Z8}Mg$MKD7_ zj{fssIG*b*oSf_40f1Ls1}{z>!!2ld)$_b4*XnL?B5a^V7pBuDHQagUpi zVfI$>3*Hl$|)%UB#0lP2E){~gT5=UPRx}I7U8Xj2B$`fRMU_{a)10V730~tlWshC zttC7hLxhCw%d2VeYF{TkCe~7 z2PQ2$em87!GF^5z3ro?D8?4D5{!x2xC3DA30AEJ9RHX3qiyR+YAnKU&mY1)9Z7++M zNWaByb`Sv6lp(?JfVVLwtBY>s3-P3V5w+g#3y*h% zg$pjVBrb~y2|3`2HSy?;!q|TWOn=giABe2XDH&G|_<%+Y%z{6Zq|xhw-bs+a1Ikz2 z?HWlUw%7Rhc=naq8ZS@Jvhk>#M=igFXggPyX9$izjE0|&EYw~Y2e;`k?KKi%a!1&A zCClMKy;GZz>p8Zi2PWtjUOo%DpPKLqQ1mclq%U0q?ye-e2>k8(}c3)ZOEQkLWk!HJ5 zc6LqQB1{$?*!pzK=RXy9E)8p`OPKf8MVT@gM85zR7&3wU>})$2y6W*03C6I9MmY!K zk~Xio2(wyC*=&_y{)-nr%M*zv`Io)|%?6Wg8h%6zczfD>iq2fg2H1^coCd6XA8_-JhY%T6^ z{c(L4#_dL~pLCGpiY3=RQ@*kUOc?pBYslaUgDr#3y{5XlsJPg^=rV$t&iDXb|e z>&SVyr_UQCRg}LXbOlb2o5`>@S-_eD;LtGVx7_EqFQ8*-*o`1X;;i@Tlj@97PUTL; zbBYzt)UE&A;wXGu0bv%6*YQC(co>fiZ0q&~V3*wE^(WV@$305|;Ycmcx z_aAvJ>T)ztF(o)wGi%ntZi<*H`S|!8C&huoLJ35xHU=1h37q?y?k3tYj487+kF$mI~Yp6aN{A@Lr?V z=U*>nXkWCp_P|~a8}zT}2N0AKa2igsF!Dpi$ZrEsmVMz%O&OkMwD(YL(b3WpZ!2yB z=qVVtyio{Gm4vNIzz0~)BOB%!63dA=IU?%UA8A`TOdSV)w?wQjzEivFLy1857s;y zvFTvyt<3nXxOsUkbd-t*d>|f>_x8RML0k#5)7Z?g6r+TvKWd#i3ONlCd$_XLxT4yS zzh^g);m>e(+!x?tG%`FKD(SVfv=p3N**ykie`*l2(`$Rhdn2`y$Fxl5>U6=lmz0!* z_Z-v;6e|GG*bGhZO^pZ+OGrpSnc|6OF|t|EckC9B_Evbm;w|)hz)}jI->uzMH&;*k zN5FOx#EurLdPi zuq*nF{Rf=EF?Yk(Ddz@>zT{g1MKC-kgl>=HglM->#@rWeOcuV~KGN~!)2FFkK|kgp z^~3a3U+BpJ<}qMjmcW$^QQ+SY@%s}$yofcQnt74YT|`Yw=Tq$#p2wMB9Ua(oZt}$^ zvG)8YXYH3punp$)c^o}@beECKk8AYHpYsnIwen?2o(rL?c*?J$5n1;mN@N=b$TEln zAMx(4Brn1P#5ERFcdLI|&9Jl%l`;!toha8`HeGTaKX z{Jy(^P}$=DPRh$g8yB%q-Utz1CpWg04jPTj<76TX-C=z-R8dh8k_!n4Ttb|&v5gSt z3j#hQXe)>V4SyL1@`SCT=pQuA46gj3pcb63+Dthq5;G}{1KAd{#;&*rX#hV#86QoT zUE(o#IsyqIA`Uj!48KCIs!-2(TTEcg9`wDHSFV?UhKG4 z)&CJj2LpA`WL4i!v#z}WyALA(-N(RXqS>+ey46i0Bi6hAY{xT*CAr&L!K_`v@uaG% zx`^%nONqzx?7=y2nSl+&H6YFlEZh`flhNZFyP^W$e89G!k)BZ3>x6;?0xKOIeX_h5 z)kVAkyvf+@w@JY@-KZ1r=Mdqx{OhFHG}92b;XgtCa5j~%+Hf_;+LJ_GS}K*2ah=Ar zLEp`72G_!{^@XjJq!<5AzMZ!qJ=JhOTGZ&!Kts;VjIK9V=J~HgIQzpJC9j{>h$J2y zjmDwgW?C@%7VP~N&gT8;`bYmc4LMaaNHf#g0LoeZwMNCO{}oQh^;>r0 z-K9NwGIVsrBP?}{mHkM$M2PY$@jq)4a=Fni8=hLeF1Q2L&4y-KR~smq|1NHrM43^MlMYXY1YO* zQ=f?3Rg;zZR@4)zMMW%R7o6V~BFw}_?SuW|V8F|saFyh*O6*Jhii9|dDE_d9#Sw*C zAw;}fvcsoMTyY=T+7?ijWEU3BWoBhlXpS>jQ5lnsS{W52AjBbR*q7Ogh(EVQ-bv?N zL&3GZBudGMk1p{>D5JP2gTuvsN`dF~f7Qq?2jMg_b$9RCV)D`h6ZnZfK?k-PF*Tml z$T#_B*h$dOm4hI#l-CmNi-LxgnYj?G((7GK!O19C({~CQ_joKHEciYSTV_~l2UQCC zc@$*ZPy#QAyf9OoacwmcwfX$#f!%yu_MqU*#=lMlI0fY1L|x$9cz)!gZbTx5W@#$N z54`kGFoA85o_MF3G5I`kDMS-!{9qCGqb5{{ynMEBzS>eYB4}K~ME;0v?}Hq@k;Cit zcxnmvA~&=-U~Miu}E_C+XkwVH})9ueE|Q{#^yn0W>nkWKIs@$>G^(k)KJfxW-femDmdWCqjWVNGf%4@P7ccZgbdG;9^IHTY;a5S&zEHuwC#dQ? z7Uaycul%%c)^^c#h*u!j`Z4@QoeFhRqN8esAw_6>HkveUU?L#1c)`Y^-htI97ZfV0>NdL;##c5HuW{+brc9t-bU{~-Bb zQ!cDlP{c;uy50O){`ZfjVU3d-o1)^zwr}4)=e-QGTW!VzymQbEAEDi^%(7#5=;gkN za82G|h9xH3oPQi-=M0X)6x@u*M ztfR`e_9!aq7_X)3irP!P@wh{?5?B1?Z`Sxl5a52lC74a)|4@(Ld)WC%RK2;5PV1Ai ziS@m-6?%k98l^&Yb#T?s&rjM=BYbOa9?vf*1z*W-d2xMx^wNIijqKFRJn3;o4nN-L z@h{h#ET4zTI_ib_@!nH8hJ`>kF2)WN4bRS2!afjG^X#&Dz~}>*2xMj|^St06#Pg*A zQw(fVEqeUF#MOIkw*D=GsO%YXC{%+{)QY@GampbU<3CUb?6CbPHiWh*Ve#W0DMc^! z;rb!rh}`jyJ-4H3nyz8|<2@dc7`JH9F_rB$lyypFdd{rZb6Tc#&rR)Y+i_SQz1|}` zyz;ozFF-(mYUPN=iD?wD@u(l-wkVM2yc2{En=3D%0T2j4ZGl+Al?mBXvt@oLjQNed zL1<5GbaipLpZ2>*9UFE|4l6smDDD3F@qr>4aq&P>@R>v)HSUv5vde!g-G~RSa~juc zrc_s4UB4x7U?9_}@MBx?^R%>Y6W(q4$*<2svBfi+EHS3 zv(LR%#KhNxeFNE6UC{ozI=kIOU_Pp?zPO_V4F%*V3>hj!Fm=pw!QND&w6?=IC6vaB#bv{Dr+c4g{HVh zByC$uB`!9z@8s6aDC)nSVsOjzTBBvuD!wbBPfgv`vTdbUGN5lR&RMO6Rxam~3!APo z^Q|h*BLm5<2+by~Rmc?~KN>9ZVq_Ysl{liN)$&pKYEiaLEDJT}4ngv4Xkf`4jvKq^ zIyC|~hN-SpWDZd)eaH5+Zbv3AHs=g$h0vrltimNe#gmft_;oJdiq?77p}#OO@!vq9Dp&}-`#)c=Oz8+#^xv=Pbx!AY8*j}zbkSt4 z>ZHf4K=zjU&!YTccZ+}hnu&uGi_IR-6VMT z+Jv3ToU-3hB~@)Xtv7qU53p}j(8*;qWSb1r%4?n}*GjJxJH$fmW%8}uh*A9hU#Zn3 zUoa{Bq+Y6KjPQmXxQ$lr^l>LS#QTnu{y3{?;%%KvESGsa66u|w#Px_|`DaYSO*$$dc_)fZ;UyyE)(1ZNXf9(b=Q4tr9f=Pw*i zv&~4iLa6fopQeT!My4o#;S-w5szRUleIEiW~d!|428FoblCTy<1bZuNPPD%Mi5{VL#p>|b#;l5)gsX)w|2 zQdQ-G?5z6jjZA#8X-z`s7^>8+X|V(_l7n^}vKU~heMO%f7?ON3Ey?*r;T=ty>cZrU z(Me<;Z&}h>(~w7uMY~yt3|gaF)Mf0>6lzxoU#z0_*I=5B+ytWsGQ)NwE$e$KOYs2y=p`w_DKc;NC8fdfIw21RMSJ--!wP2M(0uH@~nw>wwEkDsTGbPOTw z3*I40v)sAmvx{7-_G^71YxiB8nQ7-#8Fk1>BQ5QM??uEo%)31N`zzVK&U-W}rN=!C z*cw%wVj6fqI)kKIvG<{2D;>Z7NeObWDa&%@?60Y_3{$o|TAvQd>F!Wpmq?Zq#}YgL zI()mCIBj7-&TI0jr`F@VO^?!;JC~;mHf-2{JUeN%mUAM~R9)w$)(b=5Ii*`R-ir!E zB}O$`c!S?}V63BcvH$oBAHT!2hv}#P%m_>71Ci3|pL_|wl|LSKyDYYkCq~!ZQQ;of zi{GM_7rI&YWr_+<{`}S9BBxSaU8$;_x?SDwLT}pc=9BLv&fiEqS-rk`^NeysSc@?6xd8Tlh=_Vese+frYs(`c)5Bwr{fl@!P=Pd#Y-<<5*k%{%57))AXR)>?iV z&^eIV5GN(d+QLz7wsf?u=qAIbDbchSq}uE9^5m;pOadE^OUMchxHQD*M0{MYx%Bf( z=KFR{q!qNsXwOwo+1y()lPk&eY{)xdq(C{T%DAIRE$eUyqyKaRv+Y#Ia(sJpotXB4 zwdo&P&e~aLAN6g@t-bFrJkVa0VJ~jd_FcwS*{Jc`?NiHUg(okB#k5=WZ+4;EDnaZV z$38ONi8hCYuH)cm+g3y8+oZMg5mZ;MDbhGXxb|zJa20w($#|>kvmKggjv&x#-6WV?u${CGm|?!UD=r^^Lufo#QgGjgGt9Ar@=R^pB*|61WE^vPP?}@ zj?BvqO~z#0eQi?ubod=#&J)eLna&A^%XKby+n)KJUGWl-jCWu+eRu8E1oQcQ_XG=C z4d}o7*w_AJA9&sEXz@TZy;eV4zt3BU@EwPZ#2luuWL?o7#O ztI3LO%Uwlvles_kMLQENH=OUGNoOYgZ_{Prndwk6rWM+9ogeYjKP2LO@qOZ+<%(vvQWd&&L8oQz(JN#HX^5%Qu~Bl2sBE0ZMalB!8-`@k z-hv}(qpqC6eZj>=EMDC@_6#>8R{?OE)|H z%C3Ge@tvATPwaFbmR30wVSjWXWcr_uXwUwPbX`>`+f36~<$mdeeCvC2F`*Td5wLRB2owa;eRxo6Hr0-}#pYEklwbWv^ z1)p7s(VB~1cID1}(o$}0IzgjE)_n2A-mmqy($B?yR@vX| zJmYG7=Xci_`o3o1C{65~kJwf4OI>F8v1R^CuEdfL57pnMWKO z!-n4Dn?4zIi74&0b9`KSdCfny8kx$;8wp3DtvUHr)fAeSQfsJ;D_FOF$KAOpI~&nq zI-79vAC~;%0S_~+S@Q(~j_GO8+!b>l4k{mxd-(N{akAcy*YYuk8!ufjz4FoBS?&r%Txiv%$tvPWBXg48oZQSz9dbR~Sy=dP{oxnf{4Uprm{lL#@h&qhg*8x1KqX)YN6kc4>@Nvv+Lx5rSXOC% z&J)ft^PWaRM;PhU)Y|gZLw*Hmu`G4l`?8*HxUH%gr4=UOG1kvMQamA${58E~D1`Z7 z%pK=|J9nBxYc#l-lk<1)|IS|P@xFz{x{It}`?xVQEM(&pZ{I(L(>lA!JeyvtXY>>e zNBDV0ZeZN>QRL9?EqPW=C8zwRH(DAkjInHqQVu`&d;U*PLE>5OSwXkOuFIlYZqtR9 zgCgfbip_5x*fDGCBksOK-tNLgA`E2QuHC zn^DhTmE2rZ>?SOqIgkmNtYOl(w2rptEpN43Ef|7amWp%S8~qk*QZ3^hoBuS-CzUP7 zaK-sn)MOtmGxD1}bxQgDtK#BpN}yHgah{j2JYA#OOU=?P^jJ1~F4WizWC{0$Y^gu2 z6GA#cdFw91e2c4NzG2I9cPw9nM@@!eLySfE=!l=6Z^zH=US|Z@`L}58XE5FO^_9ce zM>iACiW=ESx(&k(tbc|S%|hUj07@)I82tAN(RQy~RtgagxRbf3a?E^zIX-vgDyu2eK#{NB z;!giHoPX8R6b;|y#tm5=COPE>g4M~GvU?H9egUW$*k1%wzlb z&zXjc%HyHCjx3HY$lBYVoSasV^O{RFFzp&HW{){)`OV>!T|-HgyinmnmF|tIlRSok z(#wwu{wZ1NXd8L=W1J=@#!=OhZu$aU!pp&X^d?Tr&5s1EyV7IR5{0}b1pO8x8bW`S z4VP7oh_hoet={I;UDNUWPg&y8@tWQdDI1m))lXA*vk&2zv45U^gx>v|V++gO1gn#| zChb$NQt!nX?bc$x_u-~Y9xbN=$#gD#Qzjptj1U`qvRbNe*x&B>FU1a{9jr(IWK6EceTv zdn{3+7?J@86Ew)<&djc@8lDK}9C=z(-(uhHQ5k!j zOY_gYG+bPI4slLQH_Arj6tHDSyZ0?>Ro3<+9XO9TBm+Z=g$vN zpLmuH8=+j~*W6mIA3kgERj$u|=?Yx(}@RF9cWbM&!CX47#{c=LbzXXQ^Q49>8>qPAaY1Fy`cL1Ws&&dxI_C*&q=j7)Sca>b0FmB+w9>@ z-VJS?=LdfUrSn@qd3OK2dZO^B=1;0VE$=eu*mx#R8J0NM+4SGMe~x4Q=XYP7EjU%5 zB$@rnu$L$+Jeedn(trO!)+QwBaeave-)iXBVM@4BC&y8e)>YF2$Ut(1hVr`JAHXXzmuw@qpxp4HW;trhV>W-1Bt{~G0|P?UQI5t< zTCj2?#?2TU4%JF+rzF?G+R_!uwE^4l!Eb##&RqV|tuwMDm7>|u6EYi>EL0780p;Z6 zN5TOn+t8z$*;4+LP6NZyzy=0e(~ykJB1O7?w>2vFgGxLuQpT(`A!7sISY8*BM2w8C zQTI_Qn}1V>N8Ydyah`O1;qP26p^-Nh&pNU+)QQ#fdTu%1_)_gh&rauDyT*7oM#?@~ zN}aKR3B6e*HxB&I=YeljeEYtB3(bkLJ`I6Q4B6eO7g@it*j&4_fvn@2m3FwAdCS^Z z>x)turPnlAW{X!CLUPa1W_PS;&2DNsISqXf|KF zBN9r=wi&V-?(Ye?u|`2~YW})K)kU-W!UCl1%#3aRqAa{;2Ku*$)`TzkzS>sp|9kAu z_rgP_q~P{-1r*)~X{t64PH@v1$^MfKs|BsMr)yvG>OqQ9eJ%zjQgmB{ApU-ezornVQmcgWZcTFKs>T+-+ zI^4T&6SoSSN(?Z?A%m(_ROo(!%ol`4!9fw=S}tf7058`cclc(Vj``srI0+sD8R;Yn zSG-Ge8#+{^`gDR>659PPPGZ7m#49n)n)QfXoQQcHgg)Rug1Os@Tutx6vz_zb`(*B& zo;5JYhUEbEN-r2lAjNK9>LjwwOMeFQVX!aU;Nj-xhQx&V4>vLD zS59rF_Z__!6tt;AC>z)zU}Iv=j!EKoSh#==ViEfbXtH2Ye= zyJ!HzWOqPpOlBBR4PZbIBTl%)V9I`zG*8UCL0s5GCNSND26?MY9l(~C^gm-1l3g-h zhxarUBHGaZ=MNhzYcZ~%>{!q#48j*1K|9zNK+L%@Adai;ZlWP$T(bxs$)4Mzk^n@{ zK(U6g!%RuvQCu|5Q>Pj+aFc8BK)T0UMVa$V)$n*=UO$_5Dlr?GRXrF&U`*JC*+^_} z{XJre5jW9|IWz$24tyGX%EN~bUk2BWZ~!G}OqfK1_L75nl8Ypr-+u^sS6U@+91!oN zF;nHa?Rg=j*@_f!u1TDipUIiWi^q+5drWkLiomlY_Crxmo@YyYovD67oSvNg4$cWdH9(2zhXIdhX!qi5jjRlC*)j~T;PM)&d?4Th zq+8I-g6{FcsaLsi;;?E)zt{YPhz$O79K3&WiVDk#vzSa z3A~brhlej6Ucltr8m7w-sUqsR1H(FhQW3`9v=y@-(`8&iXz^WW>pVeIin!J56eKAS zRRgx=AK*g0mKuI}O>>?YjFgrx`EL<;+cyErm>PmS)8W#h7Uv9Z-X*w<0)hs`gD#YCpy$`IBkmSP#$Ykp(5Oymy{tz6B4K z0OXLJGg=uR(OY0icP3x}f)OD>DZp+si#h)km^EMFYRz7CUDW!E9<#m%2^TG&1r*Z3%ZNBk!>)%{sg{ko<{f17JL; zmuphR5*sZ=m=!db%y38`aq{SVe+rKA&DSaT4zWWbJ8!*5))F@3C5+F;u=424O0bW) zOWWz6;f%1hwOt|@T%SJeI64KIP-#}y;I8sng{0F=tgKRfan4}e?F+z%-z9*811Zoi zvls4R>}l{pPOOy7?(dV%>;eCX-SwjeC_mseYD@3wi%WEAR>E?^lpcxI&%4EG zl0JX_3?n1ah}z2YP}f8+wfZf!VtpeG>m|qxPKCtND2=T$dl3G2v7~316oT|~Kdlh| z0f{PMP6DdKpZ@-1cQ*WG$y_<2R!N=NHMO*C`tHqB!^?H%BgaJUs=U5-hvMTe*Ly6T z#DJ`;`U@hRAXKK*UWst#P`LK=bB@W%ZF}#E86Tcr;;J+>G+@?s#FQyB(|~^`91fa4 zeq`+WwX zU=%bAE`}gXk9u$^uXl^%GH&f#BxdB6mVLyCTT+rGRgtjqgSJRqM8pZ9lOtnmQ|<;s1D5xsJ02EEs2myqOy@A0&Si^XN+2ITJ zB!vcg?f71p^u}R{fb;S}#pdi=JY0mi2eyZ~H@nJ4JFse>*!%GHV01uyb*V8>UFCIe zFTbACaB?40HPzPCb#%u9O!Yu4Q^-lK`ok&9y8Z#-Q)P8TttG){T0h`43ebPKPTJuQ zQ`&1Aut|f9cZ(#6Xdxn!i&#~lZX)J>OEp1__B;4nGDx zC^Iu$(=)*h7XS**B`(=HC6O4YE#u&NLC|RMuoL7P;E?#?^XI`T;hB;aEZQX=D#!D} zwip^39QJzrzTnQOb4DOJ*^co>8z}cE%z_AK@fw0Gi^m0w0qoq8p%oIcs6iMN^b72a z7_S;YFNrI9-7HjM13B;9>$TNP=9C8&?#mQCbuWO>i3633PEwf8tKMadiJFQnH(T|7$_1Zx-{efQuZJNtWCkR z^}-mVJRbB~JmENvqmJzQyPp$x@|+fZNgz&wET|kbw9ONxK#Eknx~Ikw6H}q;vJox# zeU*)j+!eK1oeo2Hxrt0rWbw~fknXU3;EHp2^Os(m1O6l_YIE?Zx?3l$Hdw^gJum zaUR`64-VV``j|NQx|jt-y1EJM&NK4>Ev0hg*Bx*!kU=h#9e?jAq_ z?-MKgecu+a9@1}BY4!@rF?Ij=k>peuSKoWCM3kz_!3vA#+1E>w@mN_MUfl^y%CqpHv3* zm6d3Md_$Zxwozp8NlyuQj6aWhuB{xnL3nC<8lT^<1cWl$ion<)m@ou!eo#mcyc}AW zXxPk0^Z8Wt{FjlFi4UMA|8u!6n3>Eq|Z zw6q1pC3W*EIuvy-cNFIx71-fx{g?7TnzrV)-zqbl`l+!JznK$lugMKr-c9hcnB|fg z3#wgn?`ZUde8fJJ1vb9iMoYf1l*zLImsYU1N)^7dK^58?{H^#^-PE5Y?a6bAIc{O; zyf;-7?j4yM+mn7v(=Dm}$8+U(HsgiU;tHvGwF*OyAA2@hJ>BqE6Gr^i!WzEhwNzV={x@yl3%%ZT5|u;$s*~J*ev0k$>csm$f81*G*Wir! z^HcUa>$d!_?|bzX#LdoYKu}_<4HELrx5p${l$0V#UlPd$hEmeEjI&+ls|G zF2hc!^Y3`=SgXg>y+j?9oQ}@YxiAh6n&bu#Y<#>?p4YFJzZuH$KJ}lM&M9f!2qB?( z&sMQ3RZPxoTv5tHeh*vFzvApPkL9Fr(^g=MK_Pk1*)G=N@PqggOKo+5dwfyK`LU=fZ}muoL?ut}S)X4M ztt+ld3xdd(6(7#|H*X4d{QGt~+>Nhu3=<3?xEl&rgYnR&sKIE|ijX>_84tk$i;z|j z5C#bPGleT}-_lUtKKnC*uZEr7Ft)~%*_`BUw9;Z5IU|uHd z(cRN9RQ~x_28)}i0OkJHd?-5Q$)6_$hu@s(_CASQ4L)fVd3OBXMS(3+u4qDew80afnw^yP** z)Gs6o*q>ol=do7^?z|O(O*j_@G5B<7Mr`38AlqFJj^u+#%@%wj-GB6ccyx5Exc6gXpN*3*e_WFx`THBm*w4w}tm02adMF)kr}DjZ}vA{Jo4ayzem5n;D-F zBCrL~6yS1;G1kv4LB`)Wp{+ybPsO_`80`AA!EZM;6`LnCRd1LSoo9&7i8!Enz*xSm z`K_Y^6jyJaUfculnK19^p}a`pYeqPPhh5Zq}Sn)s-; z9>1kjXEn8<)fHfgp0c&BZfji>-}ayYQnEHLJci6Mt4ipzg>iaX+6f(<$726pefRwm zAMDV>k;b##N^Kw&(g^w8JD-?u3)c;DK=bcB>sXEE8E#(fZP#ytFOP!9MO(~KF!-** zP5Gv*x@a)QhPEG#YU~S>8WwL6^YZfIVy~CzgsMd(Chq=ynhK$a+Ww+4#1P@iV4a;0 zIam)Tld~{;FfwXKga=y<)b~V?tcdyMKJRnqsER77*m#@b$#+k(p)hL7>(%48h_&$Y zlA>6lFbI-gOTM80SwnI6`hSfwsi^FwH?l--qm%$vvbuQDvoKe8N8E(foyg2gjRI%- zsN*{|#5a*I#9Bo9L-wzDRIe@H)rc#gvfDA>w47W8ENEdEBPVywwk_Ze$mrQd#bLINB1Q>1weAuSQwvDdI6C?t<3OvS2poFP%CN>h* zZNbTq?#c(bo}=CgPH>MsdukqR`ilDSw-9j|e_b^H_u5nCC}Qa}iyaI$9xCsAsaTy= z%M(2#{#2V6I|;V|zMs)H4kE?LbHG}guI&#M$7g4%1~AHWdodVWQ=WY5c7-w$r74iM z`U^ssQX=)5m|cxvm6dXBZ(z73IpAAc8z5WpJ~Rnu(Hd1rBuan9(P*X6*-1RB`Ea%y zp0N0LTcqlu>;9zlvnvi9Z2)NjsR_?O4m^?q{^1)J`O4$ZmJ@V5;;;!VpsC)9U$RsC_) zY07)OWHQuIQ)X!W?{z?tl0_VyrAO-Di3cA0&k=`1PW|}Bzansa=fav4{L9Tf|IG*d zvr&~yJb(XvlH=R{Cx7q@{e21zmH6cU=gI$nzW@Im*d_dLudJsMXInMbUVX2x>ce3v zex-uyzuyFe%3G{ zY)(1qRD&kfiflEGMg`3WRXG@<%}c{&dR`Bwl!eyG-dUSewec0-TbnYefh(v;pI2Z+?_$#Ev+ zP2F)Mb1bS6WN!Bs$q!V~B!A(w!TI_ZT*-u)7}(`ch`t713^;Mn8<7LjtkbNAeY6V= z8~P?-)E8BFld1b?wV0akX0^@9I+xqbZq|zTK`IolKBDoaC;LxF`j@`wqsEtdzrKEb zyhThj=3R#%!Z4x^W)E&A{~Vj$qo5#Y>*B&_W{9kz5#w~?kWSa+nH3duUpF~3-Dxjz zCzP)G%ghpb=mRS1MH3B)rBhGRnQyE)dVuQH>*vr(kt;Eb=q-||wr}oiV%mTG9M9eE zo81cb=aRpRMzXR9&K}=;U>}oBQOD`1`(F*(v>((}tIFvx?OHoNMl~Air=LV{UcX=6 z@c+)B#>fP_!=X?A4!%f1r*Iv)(^Ex~p_p4fe1~_Wp0Yqe9!9;#Ko@~iG z%E2@8FmGC zqgsy-O(hCr=yo)7@wl1(XT8O46WZDpZ;CKK_7`f`b4Y1QS=2p(JA>hko_zK3@D}I7 zaxhL#@M0}UqSSY6=(*w^!}E-6oER7oCmX8g^-#MZd4>=|5y5k%33w-`q(HO)GX;Z# zD`(EKs1^0~JJ1fQQYasYFu)%PwdrK9x<`+ecDmTpV&dZBKp+Orxox>U(@^sDVgN6p z#LJE%SRV?83$=6!ev{+Jk4v4lR+2a;Wr_Lv^(%`B6mmc90$zc12aAx`ljP;BnFlpN zZJx4d8v0ifX=aCRF#`p`#pbx2|v6MUAiXs%iEswM9t|Gjr<^dM<&a?&#@Dryb= zVJ}Jk$}Pa>0siw)Yr^fNjT5(cw&#R%QkS-ZaB-854PxE!?Z=L&IAdo_K> zgil?`iA`3qP$Btgy~z7>L7RpaLML;$8au7J=!m-rStr2(+NYlhZOf-cWp0w+R>jQi zeH%PKgomHAAX=i8eQkJK+WR+<3MjJ(K}}XcP7WfJh1uB`(~+=b^I<&T$&*WPX1X3b23?DtopZCZZ=gw_k_|>Gh&tf| zzz0xF0pkP=m4I3%q`L)iUg;&DSAPPF`HW$UU(b%-vyd=Av{lJw+YGru;~QERU^s!i zL>EBAfGHpk!J~o)@R+-MsLN|;8iHx@gv;yC+Ca*JCaGka88jXZZoIGYIU?1?JCz}P zRimGQwi`Iu%GvV}K7r-R5cj_iu!xXXV+)_hmYk>VPa>AyBhXd|N0?YWw^iL~ z2Ml zcbR)Jq8iE*MZ-!=GixWj0(38c-)(H`;YJeifhr#z*;5-E8w-SGAn3QEqTKWFJNPd` z*_p$h=S5De+ujTqTnPk>ty>|ehsxc)b>z0?;Rj}nkp=WU=*&8nJUb=;a zMaf()oB&WFf)5_Mv&I=NHuBR(LWM)2K@1c^Yn<=F)U~^%84#ydI0&J6+)QZFsbqJ- z;)NU_w7}@xDL`L_g~5cs#)T$Gk>QOoK|1HsRHY{Ex4*}|gJc5w6Fi0nJgdS&IU0?{ z^g?YH6@__Zzgl)*xnUTN3lQhy+kQ7971XajWmtY6hxPVri3c6Se_03}Oo=qH#P|?N zHmSvu^ykR{qs-ihY0Z> zt~Skjp(1-ve|`I$XZF7SY7ZUc0pJ`eQM)Q(UC9nYSvXvTA&<1;KsjC-aQn`kJD{mU zY^KpZ+Hd#x2ZWS(E{hrn$Y{Ex?RP`4?VST}VvRE+>f2cl$ljra={EWfb`1Dp(g&FC zDY=Pw$8gJ^;VpzLt$##j9;ZZ3*LJE(QjsNZa&IyUEJ<33j!X;T?Lm+4Am8Th(*AhX zq*zKd-MhfzrGq_Q*vBD!iL9p%(CMc+fqpJbCC4VP2lmndDd zO~pJu(WK}kZT3r&*BMZm?*et$MnS#g^PH09BXFAy*zN5-3;9sa3LXyX=X$UGaz$#cJ^+0V8CE%s}U)o++P8r4ii7(2l!2EYhfSQ+KvK=B08EG1>@<;Av+)s#Teb{J3?W1 z&(m}LfFAAxJ2yNo)G=@(?xXPcfDZts{-q576~L;2a?o84hqnb2HVc;{6GtD;szC_? z?vb7Ge}aR80NNfJYK{nfTL(lufT1~Hbs#@R8G;?+J{}{^MffmqLR z_fsM}SNx1Uu_l7nGRcnHi5O+SX1jGAfjMm2zR=FnjNMz&Q^a}fKn#s$sS=iC)06jy z*rm}(!ur2$DpAwFSxe;hc+3iTwJk@x57Csr|M|A*{%QR?r$v=Co-}GkwJ*Moo@jr( z$`O+NL%jHi*JnKQ1ftR%J|WQb1;f|d^MeJdarUwYq@U-ncg48V?U-UWkrG#m(*1W@ zh?Uq4LzNxsW;%w33L4`wF)pPp^a=+O=>T-yfMU;c-43kVsF-4^+=_WE44A_k1yT`W zDj2EFq69%bC2I~eAbKWjpGw(Qq&X(WK4cu&+q_XaXRjAe6r+tUvWBBa>KP&JxrhTMvnhrM)fwV|l&D%_%}2rWp7$Ce zI_+X~jZToh)+O6a4C>@4Vr26WRxhr>?Znr->!XujNN+-k(||nycZ@;*1Paczsn6NA zy^Fh8d2r@KEOT!SUD|pDxu@{5j1hpmdVv`nIzFJ#N_rF(DFalR3U-joYfxLkh{;mM z0YovBkSs87c6O$9PUQsOWdx>o7c(K;1Z+;#xm4szyfF!y2qjr&pjauJl_>IvOM}K(gD*^ILuNSv#*Dh>c4+uzb&4CWNQV!CI z$jB}8|3;5~V$wk5flH+qF_8(bZ(O(TZN;kWEHYs{<;Li7E#J$@Alrvvajpw!xXAnqJ& z=ns=w+#$79+1^NTn+~k-TQHQ+p5x+AZ76D4FG|wq@9rP= zeMumg1DEf!S(jkNVN`H>Nrl-Sb-I$X!KVA9&a3%Uy@xgILz4S-k|Hs5GnBUPrtkO1 zZF%Jpg&m<~qd&qgI2lnNoWA(?+l%dM=|_Uh?Bz;(SD*gCw&UOiD-Q&4#3d(bzGe`q zYss{JD%zKw2Ccf9zrXJH0S|Yhr9}0_jhJrxyv^TQ zc8B-+;IHGqZM3l2<$?SrsmR#AbDA)GB<}a})0mjh4UiQ61MR>7*kCiJ+M%^veU&@* zQM@(l@iAQ#njTI7ZrK6X=+jqnlU0PCm?pKQNlX_hmLbcq`hLI^(BC9Y?9WVMQqXvA zMMFV#G+N2e@OUKMES>Gma3B(B13KpRQpUa+yX`db#6qIF{eF>G!vtf*POY`X3lS!% zB_Q*aEi6y+n*8N8R+{fgrZ|wT^4Tm5Q@B?-PxbbMZ~;6OiO7}#I%j{X=AK$hbzc=Yx+NV}$6I{7G+wZQR zxfoH7YSE~m)kH^{It@igc78YOn)X0aZ>q&69M&+sd@R<7%>KO9dQ^LwXM@9VKDvcYVtBrknK50U)OM5&3xl)LE^gD{;De^S__z{};1 zkAg>N)g130Tr^oHdPMg}?d~rj31-8`4@0~9hSpXgUQP76)%fiwr|K|qZCjR6rDPYU2%>)QohuX>34p(3&Z zGMK-h9N~y`*bM;=w*GzmKkp#b2Xfaf@C<%9zjW@e`|;yT`rG3Gy_J#{&>++1tlXe5 zSQ8gm@|i4ERGda_6IJ!g*!}M&$t+1m=~!HPdUm}yUCGl>Bml{ufTS~BlR3dbRmBl) zJsy??FAkb?i9*tumR0@A=4IsXe>cI%l^8jIqKKXiha11w@p9o@(Po89R5Hh`T+Z z%Ak0Gopt(m*3>ySezaN>8jbQ%ctp?$i$wQiIIR99WHc=gu3lNv0pt+ zmHEOm!!^BYcvzS<&{*nfYyaN;H{!>@5iQiNU5!6pFF}t0AQqjlb%MONkSCJ>naq2T z1#ucarGu2{ukn9V^B``4)(nYNqO(WJ97+iT&ep`#g(eVN{9Qb8vgyBHSK|O)#3RF~ z`rVu{QGvcIdMW4AyDH&C@hsJut;aUFzm5o-PO+}J4T!0#nAuzEdii?02HZX5x5JN^-5;ezPyqj)&IEHMVs*2C5zT)t*j49L2nMtuh*rfW&3AN#^ zACX@P-`ySAC)T}dAKlk}+2Q5A9u*u>`4+f&*5lDN<=CwKozq?@#xzOokti$M(VAR) z=z7uHOwQ49lo#5ZkoJIT&DR{bv71*yHr>wa9DnZ0r8XwgQ?H+K#EWSi!<4zHRTmr>JeWzH~N3;hdSePY^$oySTy zv2GrXXGk%AXL4;yv4JDlQ^g8W?jOVgEE@v@nn=;)DyuQ(oXLxxLp^y<(rP9382MF| zs1x?fvy04#eV$XS>RDtDto);IEL!M9E{#aaFuti3To@FP&I5c(v4N`&)O&Su}e+_%vutX~Z`m9Z*;3YM8#;_}_e?7RPy6NjO%{C^(cR%0~3 zJqemNt8mu5iMhc81KlZ#oHv(o&$qpENwMMQ7cB=TCny;SqZK-N^dGIqi$)-F{Mnzv z*?<&j6?PE!p}LC~VWw&u522MaK%*SCCZ*v^nA=4}G8haPvVh_)#43yOwJy}kW5v0a zG#v58or&eS@b_aAdasSMq7048itu3p>3;LM6>pn9{!Hj%*(i1gAgNFjx2q(*!Hj?3 zEJx)alcHRrFI~_!rAg%a%f}9gHeb+2SDPv8g>RluA^6WmSnm_ENTqTMGJj^}X`PGc zxo10Gk)Ls`mAaL%X1EJ!lS(~xDc>)@3486C`_a$%nw>_{dh^|yfEfjV)dJ+$3sek{n<<^=1uaNk69}z*SScoO20u6H( zjB#r-#!o^0)b}rdZvH$6&=mle0FA;M^8{qKOxrsk{)&ffmD=rq{qaAr;LH8J`hP${ z`WT!9<8R+0&;#B{Oe)tIl?=UIo(upKMVG<9pa+Y%qBt>%+kJvJSI}s(CM$1u4ur+Kr53DzEY`5iwFk?<9WaRVjs} z$p+TG&>_LVuH~~Yi0H%%xo-Ed(nq->_D)Gi^L8@N zX?9(!)|?a9EwEwrBPV`ml9_JZ{CTO72OOhCXgw+~9c=uH1Z@6A57G9}Q78>dy1n4z zD$Sf)hM>Sfv{-&7b%)!nBsZFsfxo{*{nSt8ecx4millcsL|1Sy1^E?f`d)zN5wY)+=XPzfc|BBwn&IoC9dOuHI%Dy#f0qGC*Gdv@+um1GFpOs@0%HqEDSzn6FO zeg885+$s&f3kq-Y^1-2@_8@?lX96LJip_ya8j`$8Ju=zO$(<*E&{ zOiIXvUo!C;OOl4(k5oFHtZ4s4lH! zX)9ClBF-c~yjLfuEM&j9UJbMKK&L21M!}~9S%UPHZz`@#W}iW;o%OPmo!Y#Yj>|aa zI;bl`YvZ5kHfa-VNd3>GoZ~N%TaI=S#UcvvDTEgwmI@iwHyKsuYKzv)$un~Ku4 zEbZ>3I-9-Mt3}Al{A8)-5=lD@zMZ$B;YDf6&vXfl0q23!TjsYFlTt(HlZz|(?YotU z2`}{9hYExJ38OEAs@T>p9xoa*5mx^m>{Cw3XrWP-3an4?;eLDO`P2c2$bD(E4Q)@#~`O)kd zW~6yrQ51$hYco0|Af1D$uZTi-st8GBU$A0iCbHPrihS7z(Mpt-#hm5~!qC34#HM=( zxk6q|#+-vIIs@=uR+-H+^}W{#(j|&%!WmO3Qj$sgO%8OT?ns`R7)(NLlIH9V*Th*Q z9LD(Jtsx)D#&?K0cR`s(sP~%LCRQQaIX6(2&0#YmI>q9<5BMSW4)u$YY0~#9o)Ok+ z<##BE30}l!<)h#XU=uSFBVIJhJU$fDe8D=cdQVbabV=p2 zeuo@Yb1n6)ajsE6p!sA!PrXu*&qnr|GZ&4z)wdd*`_2vipRP}x*RV}#xWfM~@r*+g zA75Rqm6?gd|Gf+KfBABm+Hdy-6`31bjNL2g90)rXQn7dy6y;{eu+xhVkk03keAP?b zvA;SOl#drGrIr+g+N55ny|_GB`1R73y-99k!k9e7^N`YFtO-KXz@%bPn;SzeW_XoY zqRH+RR-K)fGp6jj^(|688MrJ7mt>E;D+PrHj%z}GT4f?=^)dp1JUWtJF`hjCZ{6pu z%w&czniBo=a)E_`m%k*wb2TN(ubU+$f$pL^NM~I+21f$io9>-=Ah7o%9O=`oYhay_ z&}|RUciL|HB?T>!b(c*{})fXG)FIiKc{PH*Ep`f*slnrhNapm1pfyoY$_qfn~sEd#sslT=4C7;b>a$Xx^ zyqUJ>u%bWLH{BymkRWPZOu6?tr7`P+4sJlNNUJfSOEeL8I8nYyMZ6t#tX%xT8olCi zf^&Ftrl5r^F;&nCWBu&q+vpswSI`@=t}mXucqX7UyoZ>NR^goE_0&C!M$)(BifdrzGCaZ?e>(>f(bs%22H8K&Y-J_AhS%+KL%1HIXu* zn-y_AqV2#xn;lm+e-OjS zH}Sm>xq{=#9JeoU$JDlanoGWxVhD1Xt7)~yj^XE$R}cQ`>=e$}O=put$X0I+WbQ5s zpo%lKxeb2uOd+ze#t&J6sf#rwQ}%6*Y?%3ttaX@3FK~T1{O&@=mu|}Te*ZauU0O{o3-*cP+E&{FkXve&$&j0#md+%+K&T-0a?47x@{5&y zR49n^fiE5IRzNGuEj4q+rrXvfnV7kdviGc$<+H|qLJ5X$`JqI z;oP-MvTaSWlWOPBLOfSD3X?16L~6!&uPFBkZ#Er85>e>!;ZNmZ@>rQc;Wq(o#cp@m zSmk#5GmX9e^K?al(vqne1=k!~o0BhX0$E@MlmWNFK_Z@rQ5-5~T}8v;@|uK|n1 zoAS{8e~sX&W+$s*;1>Lzu{qZlXl&M5topQ|GpGMsUUn5lmo;} zU619Yk2THg9~JBNctIbg(MVkfMs*%ycUD?;y4_m8n$w;s)W*s*L)#$iX;~50XfyPi z<-`(s%e00`en}!ird{_(IlIg(^4{cDv}gfR0*^+vH?i}Zs5y1Cq|ZH^-P&lROro%r z?2yAlP06dr5{ zj76f9kG3!m?c9dWralYpz(u=LnibdLNCJ7pJ#;A|vaOuM3hB^wAdQ)gX*4>N9S3M0gMQ9T7#L%;hkFHTd?Ae($9x=-0jJxc!!6d zlYGpNQXal;8eS`ByJ*l}jp?g6rv?zlt(W%l-OsC+Wc*)M@Bi-$)#qhfC5Ww0#QDPe z0Wq-@g-_{%hOe>(!iO%ND#IU zp&>lhO^rVnMWL18w|xW3u(Hdrql6W`01Q`;*_M5NBFAkm$c;A)hZhRZ%L#*ah z6qX0^`(@X&79^FmoL1sUjhR^V9NirEC2^m+Daj>zi;Yxw(o`(LH-EA)+VFbDc$4d? zOC>UUFo~Ht3dWxYaX%XToST9cTo2s-S=*=FTkah-9whTB&^EuY1j(poW(lWd(z}u- zV+KwoiaUcYtmG^I4Z{D0c3Mv8y3p&$fCAXA4Mu;^k>{cX_HC<15{V>Z-^N;F*EO!t z^u?M_&hH~2nzSK>&9jRuD8pOP1?k6Vf|aweKTG5zF=Y<80i6+HBF1k*w{l3=$D43` zyJ|Wk;?s$q!*)5Aof=xV z&G)G`d7vW6_Gv_VrjQA?t)SrwBI0-T=QD8UQXC*lx&~Z3Z`s1od6O62HX*(z&X!QVWcxLjf%8JAXFzK~9oQi>thh=M?2xujE|Tghj{ z>hTlw{wMfsvU7cfI2|+fn}t*U{eDx*$#Fpo-Z8oDf6>PrL?42l<^z?y9?)3rd47`% z_z9&=eA`%?lJPh3KHRf2H}0^pJ|U4OaYCLsB-XW>m=sAY{4|#>8TB5EoG=(o5)Q^4l+Ug$OC(gi^7s)r}X2{C5F?(u{7tJvE|7H^#UO?FWjVYrD0%jYuLw{LxFBLERXdkjvU; zF6-Hg2ml{2(l`}e`Pb455-fM#`h_+ps`w`uKEWj)Uc!G@U>%Wenl zUIDEGEujH(4S{ooOzovJ&uf=bqdSGI4(2>chG2mh~B<%>6A~r+?o%y=ZIc9qm!uyzNJ6Z68Ppp~GtQz4bni z%<}y5Q6RRH1b6S?Yc&bFROQw5zP$sZBB! zu+BkZ{9X=9)E8tNvu!$L$Cm8Ev=(Gau?xug{3rny~5C& z=5kurvF#|yETn|qny<}O^W{SrR^}l;59HQO8V6c3 zRd+lJ&dN=UXyunljG9Y&Nm!fHpxMi_<%6@TXoBv_ZkA}?*_`i=06LgGjV8@Y$WEs7 zIkKWuK}V>A*xZ48|9Lsj>7RB*)h#24|Ef-(U4p!f%7fPY2PXF~JiTH_efl8skQk`q zPHVw
#m4Ct;f!Q(fz!Yp-#IwdkSFmUZ-&GIe7l|{1_2ldAcVIp$IZ7)3%o#`Oh z_3Tv$_RF^DGbK_~74eVM0CsxWXQFeHEDA7P#=9KUaELYSdEI2!de@i(LL#nlO~p;l;mqyF(-m(!D30E7L-Y*m9wNBUL?WP2$Aa<^P{=b?d1f82awV8C@KRIeCczo)I?Ryh2NMH1G2_HHu8m9kurai`eVQvmQ|6?J# zxBAQYR5Bf>HIQkFcyLHVuxgN~`^HO*>BZ#N_{WC(#7*D1Vu zj3;4=xEuijs0Jj;r^HhM(M8p|_T{)KmetE@o_dcsK zdx_ZWs@e!b93iXWk42>t3?$pCbFd!7-dXZrcHR>H__O%1rM;B3Q_Ph&$-S()Xbs<@ z$Q{Y|7GqJBP35izT}t=nqwm?RKgs+FG6EmQ`qf|Psd_)(Pp1eP1FLabzK5jId6y*p z==7y+`KogbI;=zHQrdQW>Jl4bdEuuby%fT^ zIpjJr=Ikdg&J_-VhvOYZd zIW~U7sOb+>HcgOV!dTq7{jTAAm_-8W$e5TPhc-9^Q-BHJHvvt-`=byqfN}#q16SH= z^vB(uW22)dP!_t^-^~b*9qfdA@nPSIeVzKMEKrSt%!b+xBD|hj0(tyAI;ZE`*RS9) ze^=TH=AxcjYK6nvRFC`op%0~h_~B2&zq@_4H_m?D_!;q+R@e9Itn&2I6JWw5MjQfCzCluxrV~LEZ;nC5hr@8twpyP9|_PkggxY4+1gaxV@$WS%h8jyvz zwtYJO`h6w!$v`}&$RA)QbI&L(89w#%jE$;i>FN+X>d5Z|iQNT> zxa`e)pQ(-y9(?O|{pM@xegdVuXu*j?E3B zw)_`4PcozIoe1QZ#PAZPJ)+`%hMwa-=|t-=eF8;@SdMkxr{ombnSQezOS9UaozBZy zL}$7^4i_AbxPjy9{m2Lu%8(7RkI#Yf=Cki+nB(hoCNVuu)<;PY4T+;q6PVgx=!PsE ze;glfa>D0C6zznmZrEKSNlxRxR7Im5`bv+~H|M~%^Z9=qy9*gkOrDYY$&)7!PV^-- zvTm2A9jI}B`AD%F4=0VN_G`Qaa!YlS2xd%>&}7()lc1IWe- z3JQ$Qsf;NY(*x?FE!M0-Fb&SYcsZ^!_{d;Z`<`oe8F>}a5gWgL)sEd;+JMbA$I=W0 z^~a9M`!J$z_RWBp7?=&~VjaM=fvJK=jv4j|cZ5erJAk5k2-LeODk{n9&qd_?(h^MOJ_jRifO6sd5*qo?B8CMBXpGW9!@g3ct&I(n z#G1XUYR6iL1>G@{AO82|44_>U5xQX$v--AsuJy;@M!m zNmMeMjVNATj_5rLiup#jFGWXgUF;6O)4lJ=XbdqGBn6>IMh+f2w2P+26MlS!mjUB# zR_-W+l3B_D4oGS@s6Svnl{sbojQez?3rUtWrIFV&ur048V*6}04m&LiU&o)WZ;PeJMN z9gU@xKa*wIJGPlPH}3Re7>#O3^_)K+5!EH!?csbqqe7@xx|<=0$t*z`!*I3nuNGh6v4 zf5^3FQ8~b)1QT(i8|TRrqjVLUmePn`P!7z^Eyo+*>Vw&o(5(9csG5-AVBm`c4}ZOy zcqqq0ekKkM0%)Pa;R8p4Il}J&&m08S-k`$z_>EIT1ONd{PENuC;4Hw;!0Ib6_Xj%* zNWPQ3)_(lJa|h?vCMOkHgzPS}h?Fjw+*khDUA`wzo&;YK;a3;W7F+y(#$o`8IOpo% z@dF<|d;serf7|>>rL;L_$7P7lfM@|0=?{PhDp7^i3yT0I;!vdgK(z!Kd`1n7SX^j2 zq^s+wz5_axJkOF(=k_52I7;Ad2g(twy35bKLF~k$f~+JiKK{u<6JXoGZ2*>7J5@NP zLty^^zX=GH-#)B_v8Kbrige(yx*R`#pFIz=?+9cV%mIj$91^&#fpb_80sVe1>M`da z4hIg1evq-pEI2YE;F-dUlX3+%%Mol2n4f{60u#nQUI#Tii4JH^9+wdwqNU|O!0rg~ zB-_A`+G>JK_5uS1*hrZxKWA=@5?)0V!Ic4N%<}RwuyRnzcwOCkpw_&8?bsOsQzB2G zxc&W^GiL`xUUgk6gKvzEj|Ug}pFe-T*bcZe?oSi`fCF0)4Oa^;5*YBIp~K;TCxJW! zggSSgt`znc;5ikR!%$2J6@0aSLdRW7NPuX0`m`QeDTfpaUNW$$|N8ulmVIOpZ(}|S1XajK4F?*(A=H&p60bCQ%41D8YQ?L4fjTU%R$D{l0QNhN{7bn>4+7vF&A_t35TA`dHHet3owMgx0kcl^V@ zevgGf>2jgtz=b5-x|`l{EX+aKN~5VtX6%OAl@2ZD$ze;3_-h9N$vw(cP#xSH=j4L$9a>M)_gzYTp|wSn6G`Ykhb_U3LX7uQaCf7;z%xjmuvH_oPW(V<-k z)DlIi{5~&H`(RcpBXc=xVYJMftYbBLC&7}ID3}o$WR>Q7*!)GKU%YqfEIrSdgD(2m=afTDs)9qaYe zM4zHML%La9NwYebU5(&+K+h9}cDEO|FxOLO-hkqLd$-EHhKdxMU+7XsC3}zfGSP88 z@j~q{8^asb)g=MYetEgPg(OR~q)7FeBa;QkE#GXNaYI{<)?HZ1nwlYb96q4(2CB~O zwb7!Iz5ziKuMbQ~n(uSkP6|8R=paqK!3Y!*MD0V3m-`na%M1?0HO_8jYLk^;@sh(0 z^fVDGgpZa+yR0W>S?@IdOx5+5z`M)J^_^VWJzf_T-O)Qx6PXD$=yELmF7zS6isj`f51^S1Q;y)dTeoh_06#i;2!scZo0?#X7|I9yJHWGo@hkHy z(><3IL3Dn7{|Q@NkQ00dNDFUx5fJgD#^27w>lSu7Q)DO5bV%g>hrr@V+y7_u2VV&= z1EYFjs8HZ&j365RCJ4Ht-A}_II8b7ano`xFfvX!9GmQU4^7k38+Q1iU4)E#7kxqDA z_tDV6z^<89NKh%#;zTej!OH|uHj)UyJp#*t!koPenrhgWoJuYh*l;5hKKuhPADRW@?7aiFCuCLC>!G2cw{Nj5!~eNC zfU~*yKkHn+n-e`#RkeaW5A(T1I?jOHAHEOE(UIeJ0FS2}KmHUPn?nTQq;|OcWlZ2O zfI=)JAOMl*84|)k4uYv0;>cjxpLwJ_8prA;F6L5hBj_t^qYzjMe?ATh*LkJDw1%aijBbhgl3i63gg(wDR z0X7%dwC_WvzT#`Xx4vZF9e|ARl);Jz)=ZV7`d4uA&3*fKXKPK0LyM1rY}1ONdT+{l=>N5KC08eSMS z2gr?^Vg&fkygMybdb|y#hJpd@au!{pkCqApIOK;qjz2`nI89VuAqox;K#`A{nCTl-QnVq=$%@rjvt0}{FI!8h`Lp*rYij7VEx_N9JbI5)v^NFj{gU|H=iVemJ+?$Zkh3=N78&$cg^=@ zb##>iJwxV8&rKwCgkVmw5T4FDI#hox<5)Q-ccOJUkg0k@o8#F|SC1{Oz&5d;iICe; zwR`DO5Nzb^%sryXQom6bWRFklHyfCCL%NR;a^1+Vi_`~-B2q= zg|+HNi_OX5MXL^YdC{*S&Gk?k4JJs7xv!UR-mg^J`ADh#kYvU})x^Zc@A4ZlW5bDo zqF*1*(W~}pi29f{PHtZDy?AMS$0hKSH;ZqE(F~mtwm|&^F|PK!uWdpuxJZF#@;c_- zYBPihzyK5EN$TIjn6@w&`3TsF0fnZn6c8ghV3R!LkY1J;z`%%@4iV61+69cwM+?h& zNQBclelE|wDHNu&x(dBN#;DM0A3e2QbR$ghzg4ZD#Nx~@-;z~-?E+PaJQ7Qx6DTcq zs`uO#IVFGjV%3JHklx+{V5>GGu_ zAj^?DE}YmJ^A73?H~enxE++jk%iaJz=jOu$Wj)=}FueFNN1lB5bJ2+hV|Ux}-1^X{ zUh)uF^(l7H4~c99`zCWr{nuxQ&G)h^^@Ik^Z{-^8sYU>>1opI&uaK_+5o5;az4qvz zzhfJ#4Luoa&`|ra_|L?jmW}J<*DGE+lrytPSlSrvlql*b{!N%3s;O9H*SL>oc_?F> zkDN$o;i*g8Cc=h9{ab~nHgR=_xSH~LkA-lNV_BqQn4Ig8eGV-f@A?wFzZz#mFI2sB zAkkP)K5cXzP55>$tVqA+U4!`34a1|p4=$+C{w-s>D0EUtw1j*9+@I``%5ML82o`zd zl3YLVMO9dw84-v1px_sG$%Vnx)>jQzQUzPWaHvCktGn6)KE$nZd-K7k=2-OL*yHTI zYBxM8>uU~Pvk}xxHOI6;30Ay6R&HiT+pEG9Jd?^g&?9UisDhHVK{6~T>w`J3rah)6 zslmahqn|eO*)iQnUk#+t`F=mYa02u$g}w8ker6%5fNM$SN6)_@vagR1MNp#`+FBYN zU8^IgOdArb(Sa109>H9_Cv`D+B(AgGlKB~$a`-7KmW;p@G}IV<8~)?}D){x9XWyVdie^rI=w(k;=C`ARPNzzmPChj4TjCFFI}+i6qiA5= zu6r_)DcpP!qu>_zZHM>wPL-uTaKj|wGxkTvpPPT2dm`HrMOfH$V$X>^YUW&PmJ~hJ zr%fYK$JcGxu;-zrV~t!n+6?D6VhekKvK*GHC-;Dp*y&CEIK!xyDv3Fc=z=D`J4oO1;_iHQ_G(| zffmr?{w@88&3w258@JIMs$)h<%hCxK71lGC{&0R-S#R7w?$ z537AIq%jqI7($9tpC-sF~PdXfg5AY6fb> z3r#Icoe^$nu?G(x{1F>(aKlmg{_n$TCtJFF_s|iEihZkNFABQC&-sa0&)jZh%bt|i zXju}2tLF_;TK3{1b|6S!0kQ`yt5&^Igst}Ufsz&Wpmy(B>FXid1x&v{|DyXDE@t%* zV;#tlOfG)i^kGR$ODnhvP}s5GC)-@B94ki>6&r$ahc2~wo_YG}++o!#sz$YMAGhcv zBDZS3N?%!g_;t(Ws^?+x=TK@DIm7v$Jta~;XzBhYV_Ul2JAX1(v24(W?{-q=4hsbO zid|8TQ`irg$1xvjSa#V;TMnDvEkKe2NjMe#SJt5mtqD1GHJ}>gCHTLNAlV$|AAV1sG5$}sPeC>a?>-m&w3StjkwQYy8fQ;TLm8yQD zhJM4d;a*YsA)=*;5xuM#yYVRIYl^E#3E_LuPsgv$?-q_;R#oE;xot5+M!LD;BF@Io z9;}m(t$Y&ek$&%UVB05kt^6|7mQ=14o*X6b(^qW$^>CnF__e$}Y63FE;GK6|Ex)=v ztLcxc%=I;IWBJ&lM*<3S7*-Xsc~!_^x66mq<3q(?mL(!L(IZvL1JOA_DesB?kJbnc!y=`e9G5%aV&pVU!U^O~tL^A;jUY;v^zUC^a2kz11E*#+Pc87b zsh$wZrJ5=W^vj{E-`~>aj>kL|*LPjeSJQMKH#{F&`Q`BuCxf1MzlxK-#C2qoge&UO zeuSO5d_ws_i#AGR>HMb4Q7{mfhl-xs?>=SwaSQRx>K8KK;m7aZzj&xjs5%kZDPCVNL{%r*p-2cTef|hH`mX`W+|BUI(OXSM9x^Op8*A`Pos|Bb4EVAF%p9 zJ$nj=!oH(7hh42Pqpx4{pv|i3SxPQ$Zud4{T)NV0KS$0HDSMazhNxpzJ7$_Ay%tVA zh8*Rz!im7ZiVsW1FrGeR3N|M&4)vu6jjy-FZ}9TtQ;&C=4-}iGG`;%Hkz ztbp@*GNV-N#HZd+jq-<_%~z`K-*-ZPt|aX=id_SfF5veS&}Hm7zzo2z$aK~q!E)W* z{7p9}>fE{OQHKg_cI?=}o?W8h-+o;OVi!zo4<0|2>j$SAxa`M5AW+I~S-KMl*ruzF zCfC?|ycn?XnxM{8V3VP6z}w|DJU#!tceEuONMEbo-lJZzm;!7J6m}}thFt6w4A1Zq zr^*T4>#x4Fw-fy;AJ*5egX%s4HVStOjJvZlZ|jmWAE~)M@&pElgc0Nd3%wcS7Z>g) zjn!;Hwhsi-Trdw)@t$Yu5kM zr>D1d=-wnXi0^`V+EzJ9g4N0Z;j3=*so9?yI<`WZ4qXTMT5G;GEoq}Q?_ZT`u@_s6 zkGf(7ku_26Im(B+m$ButDr?&meOrc)J%evB%Sx0_f=Ubf9Oy`@XdGzPe2lL$tZ{ZXF`L60Uk^ zo)$V}#XYsJf^7Bf(XlJ7=mx3CVAOstqWDMamcu7S_9BXX#7W=ZMjj&>*{(!TpShq} zdzf5VwZ<|gA7bI6HKJbdB4Uy8X@HP~3a&WfD*PcLw4J^7PW=|)y%kUE-ry_6;0?pYA#;zM3@BX7 zwR=8N&g4c+ggx5$lW>1jUS#W!*e7q9y=A~Tulg8%=F>C5s+eg>z5Q&lyhPPDWXV?4 zGgb}v(d|g9hWBkc*inLNJ=)xSKNlWRA7XeYfc0)vddGZSTy$%UWUU18|39LxJdo-C z|GzPstL9!(AvE_9A`}zZ93Pe(Ga`u@B3G_rjmlNAQH)$8$H;vqM+iA`O)XKbN+^ox z`n~%8^Xo4;Hr}t->-l;fj{`FNHfjo@#Ca#DM=!oAVPr>wu|zM>F`u59S)`8u^^wYN zKYxGXqx?H=QC~Xu__H%`?SSTg=FTA|UE8~QdOMXqaAb1(H#@6Q(w^4p!?=E~knzgz zkWnNxxV^Ua*h-c&>CKeaxy_}Bw|=_iVda$Osr8+7O{Yg7ZuvJRc=L{ zwblWi)?iUXLl4UoNhV@T@v1+b%>En!Qoon~#6SIFWGA}cQ|(6Qmn}PP`5#>pxw%%w ze^;;nJYInMllJo%rO@PWZ|~NfKUGinLOwrI?`kREH`D0U7#?)^P^B8aXFMcJlyHlD zJ1wcBUela_fWQsutQc>kmz~3v{HD8DEL#EL(JN0LRq*+X9VW@9Wy&aF-86szfYgcEy%Je*cR%%Rp@#WnOl*TqM-&IHNJ58|iAzQ}t?j^Je>BX7WO%WaBxJCFd%sQOBBi$11J`;W-r6b=7t4}Ysa|p~g z?EC6U$`W_(*Bs4X56P+r;U|dh%0^z)KPxGn43~k+s8M-37)|@9+ey}AHK(vmGIo0MKvy^ za(5m?UO7;E)qMNvz>3k=LqsLjibx}VtM*L2RAuku@OR(>RviQ?Oi`A_QSf&_6lQug ztQ=ZBuS4G)IJ@D$T5;U((ea|$_46$u#B0~CX+4mpD3R*fdy7TUFQT{o*w_AvtgfHr@n)Dq46|MstfRr7Rg5b)^hQ9lwUI*| zwM_C?P{gW3S7dV_ADmE2di&b7-9nt4ad;P~MdT6&I6OQByf$ZD)X^m~TD)+_u4gu5IBb z&wo-%zFa%oTwB~YxjP)ar91Ds8PS~(Y5Jgd`*-A%VPrXL1(ad!s>zOGMu`ZVQGxG* zh5)G^|GlZIF2XzY;pD^bkuyQ`C-d)y?*+3i&AL2q)$Mw#^`@}&^3Tmm!GNo*()Uy^ zXNRieTVd7yF8>Bbz)}N5b)^GF;|}1DTV*5g8THlGZla5QC%00(`nSw`UyNxFZPYrl z_!X!QNLi+sf(4f}l?>OcM8Jkdd*GUC@%C^TEW#Y}kfXpLUdH=6X{3(tNf8{DH?~3L z!dWW~IV3-0oNRqemnE>|QlB;mV0N(BSYaz(1yRzigmG;}r2Mn+ice~c^EzE<3zrN~ zd)LYi55zl+mNmolPeS&RB3uPhgk0y>XL< zB|>_*vbqI;6@Kh#S1Ko=v|P5()$rj$iQVQK^cQV)eF@kRc0s!*zswmsU7N&!nU#qr z4j*LTeVE=R3gjTvBuyzq9k4k@wo?u<9WjUZIO}7AWpp~>9JYZv$u)LV5lU95`Zw2+ zV)|g6JGTgqgUrilayJ}$U{<>P`g=z?cdFMt@@+Zg=IF^a4AEbgv0)+D!2ApU1-s$(F_1UDJ(<=ElUl`mf5`(O#sZtk8pf zrBaeFB>UYp+LZ5xFKLG7%(su-Ubntc zSl`OkJ~w1tm{&_d0Pm|Ha;X=Jw>{5cQq)R+r=PVL={n z!Nwx64BpP0ne4{hv%T^Xc(^sMxh+?o47nRMWO>b})OE9B#x9N8Rf7UZhkiycl1;h7 z2v+d79IhwJDO>gmkzw<73v^4xdh1f^rya{|@KSp+Un9C^?5s4t86801T_nIu>xcKB z+5c(ciOtMD0?C^$mh6ITs!Tx)@VRMe(jPC^Uilkc&Uahq{v~-4Nr;u7x(YG*6%y>) zkJ-G|DPdKFuB@3~&svHfgE&7pO@B4Y>qHIjrWc9n4bDVXt*s`{V{kU_?UnrPgiq+o z>n(5m8fv$`y*xS@PIzZp&= zPp8Mp)E>^wVT_oH=_@E&;Pge+U!Vu)$4(`0HhjyDJ8<u=3n|8MPhMoi z!nd;Odu%K;-=k~Nz;ij>njt3;3)hvV5O%J*$lLLn2oUK<)dh=$# zGB-*-)y$L?aP#dFN-{-+Su_@V>~1E3WOQ42w%T+y)Q>Fm&5xvEJ=V5B;8PTWiE^Md zIP>;ThnmgPU1z6Ts;m9k!yAN^>GtN|!QbN(NJ4T{+&PNM=DlM`&@d8u{Pw(ZPow%~ z<0tRL?IcmH&8Jc6+Q+c~MA?F=6aUW~5=`sFrHaq7D2e+K70o0{iDUVjjn1lnl~RakIcM z+@BOtS^tKlYkHj|YFz-6C2+TFxbj0C@on`6f&^0j!HVI@K1}X%1xil@M>IxUiN^DV z`jfPi$82T{>PDrGn$d$xX$URlrZGO#Ij4(RTBQYU-RqOT6%{EwIIc%J%UYgjO9oZ4 zgfTJh*1e+5$*!6Dnf?joL6#Pg9kNsm?W7>a0)7PN@Z8oqiJ@?cn}~Oc_aqMn5x$i~ zP-($K?)7Yn?d+RFUJ<;AE&suS>0tPL%_$p{N$_f5vj|pkp1!YGQG!#3uxT_>jo@@C z)KfQ`X*`kN_(rgnysxDHYE}keI%5)UgdPpdx{}W^nrgPKUPqbWmhkEayC02jVCo>v zBHRLHV6ez$7ZEIj=**|-WvEWUwTLF-RO%1)G?T`69P}_(AZ-~+(1V2vlvi5Ge3Mfa zsm>ReT!?o*aX@+tn0#mrp=V%*A*j+iJFfMOpb#v`0eE~Zu4J=x&jhc&&X1o04UX67 zZUrRd%KcnxkAx&x^>;?8VBvY&-^$S%pEV3?*MoINwzt0|lnvQoh4NsT1T>fZD)JO)pEiyHFv=zlN`^+$f_b%#r^EfBcT3WrbZ*#mD-T=|UT$CUfr>cg zJ+VrZJd#WNT~G|ikyz2k2c7%lyv5M`ZW&`o|6!$?{ z2!hqwU(v%Prd7SrNv#a)!vuu}(=RrU#R{Vp9k53P6hK!gi4g!w;1TR0#v%&aG=cEI zK8}Zvx@XY}QJa1>m-ByjAP7DnM(Ulk+UBLUvnE8EI`gM*RuwYJ5b2KKpYMo3u zrltdg(y77Cnzs<6IW3LX+ZO>*G9%X?!{AA$f=-ayedL+uw5$skMJD+C1VTFFmu)E> z8Us%K4{V8Umly<}$!Hc@BNku6oi5qZz<`i8<8AIB4pqD4H%{z*U-!nN;6&J6cLYy> z+r|Ndv-0s~cI8W;V@(T-2}WK{JpzeBkP^`Yh9De14b-NJhqC5hiTC;hBacRu>Wt#9 z$UF6)84n2;6(ak_txr!hDw5}DIlRkKoOc!NxZiO~YUKO2vpXJ%o)}5wuliQyC-89J z%uc}XXyg+#$gXBqEU=FE6{6O4Zp1v~ zwZDv4zz`g89g10}jx7Zj`(;?}o)SFffJc!S3fYI`V8)MOI*aVNVfd9u_dS(d9QQ$5 z&}D%mYjmO?RV*Zbl~Yb{`>aFS%+VO!8Ae?oSTIz{Mtw^-|Gh^T_1@_LLk4wk-r)ob z{e;DwJ7tVhO}Y9qu&y(b!j!E1#%WTEcuLI`Z@m;Phd|!A22^QtnA^tY@t;)bL0Q+E zC&c3r0V(ebL1vdYUwKA;}zMJ1X}dNwsT`Cpl2*dIrHX)p}B&V{5Bj!^?+SYV#GNSkp(U{K0t~ zlof-*j_NJ-u%+xj%B|yi-W0?zAS!Udg`RmgJuX2D6LnSOTdkjZ_}?{8c^7AlW22 zD}VWQMuN2J&Q>6#1Ls=qh&Cl^U^Xx~QGzs-mGA8$D%7T9Q#&+PA6RAQ@@dLa;XU1d zUtt|eF`p_-39S6a^r$FmP+XhjI^?6qxT`fdh_evPv%etajaDM(dyg{$p7fcW8fdo1 ze^eQXALC9p#w5)J9JA%~QH=^!_cj~`D0a@AWyamIxq?#|qdvE_Nm3<4^sx#lE5k0$ z6vj)oqaQ21Zwot;od(8QEThDik%t$ooFRMcS%(_s1KjBYwYQd?+{Uvn!%|JvtM1eR zryKN@JTVHw`2MlmaaT|chIrAQq3=1Ctm7w3V%J|O2Q$I-T{yVfa`^~6B@K*=1QNFQ znsSYcNQJf&)2cFZ&l@Q=O_2o19t8BmYvj$V;!zOAvqkKM)&>;l|Nn)nu zp_pG3(y~<11YyFIKJ8?Ic|^;&PQ0Fn6I6)}=KD2tz^WLQo>Y)eB6t!(}aeB&QbY(#m zEQ%*6%T+Bgpm3_J8J4fDexV-$@GXqtBuvz8clU&|KR1!uh05$HtqC#5MXYR3+> zthd^viQj0WqECo}+>11LP|^N+R)Ao!bEgnG)iXlZ3=Zm+l(Vefln$4VQ<~-xodJGZ z{FQJ90xei)n|D~O8-v@dxHPw;8{~(-5)|x>r%Itwj3W5e)Q94JqTc^*Xw@Oh#8{Hc zc!gCOTFI44R(?FXm%-CbT(WJOt`w8!LcDIEiTOz_6MU1YS3YO0|42Qjr%&BPyg6i!Los3}5^W_9o!~!9;z)w_u)!e_76^Ob$b|ES)109qP#ZuEE=ZxW(*Q^2p?x1HM zkS6!3hK`Zft!iY&FrADEr0Hxe1Ut|T2)@!j$A8&$XW|qE2hw_+WoYII-claBVOnK# zgXcJxWe!TU7~^r)O7WnAmm41=P6dXRXHa}ogy<{UX>>2EDXzYH;&mwr0$+X&nW0Z& zak09@6E>OdX`)0oVKUc6VQ=e;>SO*vvnR zd)pa*$>4EJl20CE0!tDJ`?j*rKdS=-{uLz0;QuapklO8`#7cKirV#GvI{WdWK!3?5 z8+ve(Fhe``mN0>8#U;%s+q!?eL7~!r^5kr=x;V`8#`Gy{q{kTLHB)L8L>dAL=7}_AvrN^ z*{dJ0iG|&V(=UNG9Kj|Fhs7zIk;b&~RkIB5zYd&x?a zPJ+XWofUgAFL_O@~WRTBgapV2-TGEBy>k6x5So>Gok)%!j~5L!*iwDg7RK zUDE8q7-XE1Um2Bg7tA?m$>A$%&>{KJ>V6b9?C1k~KGR1~H}$o+7~T?!T;=J}Q@7+V zB$wv|ok+z=$L9&#z4mp^zx-cKjY_k_rvYU)OLuqhN$;PX-gk=ujcfIWgT<#U?tsKP;+a$3QSRIREDYs|8;vz zNnilq+Z$;|J^XX}c!qqo^%+E$;h=)uuWt{^pInkI0$oAS)pPEY%y-;8r~sbXMKZrU zp8!@o);xvNkH_Iq3TG9XBv)nFFG;s6*79cZ=XFpnBV zqkjN3T*R3?NW?XXKe)_I$$P1-igi0!=CtW2=-#R4snS0{#7{(*xNLK=FW_v(Iia~$ zkSrOprXZX7=qSUnj<;z16$Vwfr8)jKARsxrJbwliNGkj`AydGpe<}4G#P&| z&<_Zq6WD`O{WpydO4lp zuc$aoU{4s&K7BgKxVkgt#AOT@G&}<^3@ZCGu^fuWZ@xsgzGX%cp66ijS_)zvFz(75 zya|x6{QI;%;!FSCJ>mwdS*(Xen9ljSyI2l5Z|^N9Bw1yoP_BGut-ZOIaP;$?boy;# z+M*d_1!5Od-4!R5e}B{%W@Q-k?fce{KpfVS(831yGN@_7Qaoy$fk-jC;RCsol|K=l z%!{FVo^H1=Y3C~A;axL9X<~krVj(MsCR$Dn#qu;*@0S2m3kHJVBhR~p=`>=42PhK? zlTpZWw+~*$&wbuosMk+Rz;r%ErHPBm_gwudvh8 z_!Bh~^2GFVNk^bDvKO}}L7JkC!1t4HuNZ(Y_9^7EI^ZY*3&NG>LJ4!C6R?5=%djLsgPb^189J)*PpsnYr}4_5ierTzbs%Ks%6Us9zR zpiylns?1OwOv7dAY9a+LAFf;hu=udnC+v78%S|IctgqBe2rS?1t{HPC zNYZ(_UKJFb3Pod;Boag4lRFau@pDm#os-J!`JRTSuld#AN(RaZ!BWjsd}+yIRz4{z ze89t6?v%zX!NvDsGZ2w)=ILlq?UTh3)rt^l{B|0h0iO+<8ouQJ6jz`5^t9&K(55ij z$L_s;hX1{S1v9(e+jOg!=#sEJ35AGP4~tw{JxjH_?jznuh0TP&uaGi-wWjw`_xGe* zOy&LsLcpE$mvUZcV@MIuv$0>b6?K2!^B%W3@?nCDL9_SGqBb^|%V$bL?5_OFL28wL z$td$p=9dn_ll8eEyq>NZB#a+pp@@hW6GCgVf!;HtzEN=xbM{BP^y&a zgEnVxHmqDLSqnF1p#$-cRg`&QLs@47n~x>N+asW4_ttkTBTI}t<l>+ZuxY_7pE2Bo!|rK6@MG=L^|Xuf)fHK$rXw3n$Wv58%IB5_ z4yz1F!_GzZ28~th8EUzMD2C~9c|SRPZt))4rm${lOnj?=PDAT*wJu;yDf#U!nk|9B zjfeC|mNWCpt2+tHc#6Ayd=#t7JnHzrGUOfeSrbB1Afd`&?Gu<@Ezn0sMjoF0{r=;) z`XR%sK3o6#5N(Un6d>u_JPtfFJNEsWC2SA6UQv?h(%ZQ+LVysqiSc*MIv>o@`QgU> z+%i6;B3f!kgB-5h7LtSQ5A z&i(u6kz|cP1RBgU$+`X>$<6lgcN&43y+Qbk@|owXwPT^z)DX$`c)RPuBuIrUm1Jwh zr?x@n<;E&p>1|DMZx;(xOQXwk&!bPIp4*g4LNG$pXqb$W2on@2Ns^cl=jcN@4BmoQ z`t<7P_k$;_H|(hYrO|oU*i%^b&)U&y--g|#j_!vLGVUJs{NTTrK+jtxFlWcXzf@!@ z+8eppO;n)35HiUsNz+sfwcYW$g)-+&ctXDSP`xkFE#Ui60t&8~iRUublssw%!h>to zj#}?%v0NZiPxx4WCtSia?q79#9<8|J`1jMDUrE=$j{N=p^p7I~J#XQo zfqYvzBn>k``~siy92``3&X_r;3Cuv$uz9ozdJsUe4%Dc^J80Pk*yq241<4(HicR*g zLPPbUyZZBZ(cTlr`Zm~yrj{+PS4^5M_hHM!<`qkhX?3L3%P zKChp#ZmX~`v?`R~Q++@!tozc;;iG2M47#P77&K7x$kU@w79v}&#I$L?c{-~1czS?e zfuR-;xFqtU2-8RYSd__u8;?#|E248P_w5d0sC*1nfj&D2YTFnRbT8)r-B8WCuUO^l z?xKqV#ud~7z6>#f1#g`LS`X$4`?ryqk;c5Y$GYm79idHk^+mWPg4Fo)<)BULDv@3A z1ilWVGSE(x2DF6F6QfjM98f_2y5lfP`hkeOHwN=8rBLZOUm<%fP2h?8k%&7X&TCZO zI_-dcGo9tbpdj#y*}{DV5&gUA_E-<+iEzKfd)Bryjq}O83?a00=LSKTFo zF^7>+;f!RCDlE`ug8G|yG;A!fdFMC!wTCR%oRHA-c+}v=(o(>6ZFzBj*xIM);AW99 z(~(;EX=5cp?a^N8mEWn(i2a~B(zV;~Hv+nzEEcB~3H}an?%Xw;TxZU}Ul|DcGBtZs z=fC}o8E$_07<>=A1T_5plY?@X8_OLLsXV_A9AIBx^}cS+6)Sz(Rs<$Sp)0G(_kwPf z%ffSPc;z|vp_0t0o1jTfo5}+1X>l9}L_rW9xf%RQm>(o|2CSXFfg0jhD-y29jX~ZT!6)7Ba#p3gfkrf8+$*t3+UOr z4Jt`^?eh2jynE|5q@wa&w*8f8ulF+w^;l43<+*(X{z4+%98(Np>4n5QYY%5nNoXmk z`=>VW+#VaK@hM-zVQbow#|qqqeGXn7CJa6#{g_9&uK$NCrt%=Vvb>LTNk9h~W^}Ps zFCAZ-#&=Z`8|~C!iUu?w>yv*F5BslHYjFSR$1J#1VhuT%asO^1%9|jZtOHBw zaP`^gCU)Kbtj&dmi)anpaD^S|0loPO zm4lp1%O?or)lQ`G!U;8U+1c>J1})l{fn%&NkpEB?2)nrlBg5`VU^+GG6-~rVR!7$- zVi*(OSNru8rO`7ESKX_F1KY37;*>*FU~yx3rOlY#YiRK-tJMWe^Yf}-_nU}9pB>PgklpA0Q6(8+e7BRbvF%b!5gQ5 zNYFV?3?(7BDACzGxkkuLT?UqvjDTz{6d7oU2N(-Q{u!7uy}1N{2@1r{tNHcCuz#K} zxPdlz$R|;=Q*XejaA~Nq%d#=7w$2tWLN;O@MJjUhDKyz5K7i7eEn7bG*g5!k!b>L8 z&iM)!Gt{W~5A&fTQg+dBa4r9b`%RJ|wdMNihsHR~e zVs4Z`6(uSv5Sx$PsvDE^?Q2jWYoH68rln4cc87fkKj20M*2V(ge6r+-YBWwfTQ^MV z=hT*uWXVjzEUzovIE_Y`>!`=(gIlWng`lYv=F5w9LKlLnupbcRSRx0AxNPxkmk zA*~BIjdLzVIAjnW(E^OV<@BWTATG)G-EZ(sT{!`UTJOn09C4S>Lpr~0hT;*=^p)@1 z;y6{=DUX+41q~O}y}+LAzwezN_Tdmdz|N{NV6RsN4F{eQ#^77CEvuo=Sy7W5XBs{7 zapJ~keX)M_MaC$E;)eVtXL@X65E3Bmn+5ySt=w%>rSzMe;&_@RGtM5XSrjxEPaRu* zCpX~q;dRl{<1t<-KI!R{vuZNr6GK|JY=^>k$1jAZVVSK=W8NML7-O0sN+LStMCHjH zvZLXCGBo=%cSK|c^@=wAG$zcmyE)M4K524mNawXq)!Hj@6Fk><-(;2UpV)3K>wiPI zz1%9bGTH!u93#k&Mo~1ZJ;0T>_G;<-g;hny!6d`Sx_^5tT|iBJtIZba2@4qthl43R z6f38!m-%z~z@BBaK&RYr{g}S8Sgfe&h{C^-98zMXHNr0O%pJ8uF2wVri9tQ!@LvKw z7?EVngj`RL;-s`Y;vzm3|6}uM(0*M&){*jSh~gF-SHLoRvX+ zk%I1d+VvHqOSRSar$ZJ=!q`CS=8#?6^tYI7h^a{i>s<>s$2cr-XW~6!<76;QIq2l4 zYdSLgO1nQ^6k*&~kiQ-%vzc}*zsMexy)R$mqRv-c^9ruhytR?W1xcfG%?xkyl?p2q zJGbfFatXc6VChv?84dl|$-aT=2a_}Dq%@5{xVc$>_|cE3;i%PJ!m$tWxvQg{+O7Jx zilz`UxAI8uIsmsq{b>z9#|PbCKet2sFdP6zQn_xE);1w2OF`J~jZ5y&Zas^&oqL@# zy6taiG08hqA?niYgcX!=zy@(_;6A4wD~`gyR$oIVuW=AN9S;!RIlVHbLq zKiRWy==@u$eMfoc3&yMoSOVzVKzR%*JLtsy;+q*R`=qjFk*8e>LxjDpuz51(8Wr3i zI+ke(*9;|NrZD?Pm@(}hL{xw!yAy`HrMLhKYX4Ys(QtzIwGXr@$gYQ z{1u>C5ad8*bWokdzaH!huNd+OA$east2N74P=~_WD4N0fnCawWGMW?!)cp<{GsJhc zHPe+G^7C{L0atK$^DDduM}@yLoM+y>dS;~QI`E(a zdH8%lQ0eaY3k3~3JQf4zH9Y$5GdYQfgELg{P8Q*x-h!R3hbgg>1z1rNIJ`H1ohNtl zVbQo;333bXQ6i@H9gYr+(zC8XE5*wsK>E5t2Mttrz^FQRHzgqc)t5oT!Beo9>Cf{G1)Qh}Sn3%)~M@*=Sq+dvb|Emz;O zZ}u9&O+>wy)1qU-uf(+0UT&Xc?io~lx`ayU*o>v-X5 z6a(q`&)nkS^*=j6Gyou!_jgtg8~SGy;nOKB)h(RNQT`I&BD~mCf2H`@0UOv!9FIv z(^fwdR6W5wi+x}57*(%=RV}00Y|1KM1;a|>P@`tAbAL=WZcy@cDK*RXGw8uj<3DOE z7ZxF~>{iN^JpzKUP_TgS9WVHcUtlZ&WNG z^Q8l`^H*%;ic+;4G7ObpNh@Z@1-n-U*x{};>m(Q1BTdQf1_GaS)cccOYCn7yfe#!k zu*`YvxD8+&=XBL%&<)n6vpvW@=LnK_n-P7ta)?8C}*Xr--fOpui(i{e`3$5vK;=ovWkyqQ@AfBW3#91Bpbu z1%t+eQ-sB>JBiMth7>#FmX#RN5a=cVM9l_jDE=V7V%P4tJrPx}W?5Oc;a@<#M8Jzr!jHgt%|XXruersjz1bjzFM~=T`FmUx5=bF` zf6F7Y&{C-%KMJ}Ht!>=mPTRRpDTx1!zE4~=%)r?wh%oDMWhYMT0>5|s%FngdbztKP zTzq?vfL5ljc#2Bx-E^K4_sFRYHSTNIiJ|{2vIP+!vR)clyT`vgHf_ZERwUYS{2JNio219NkO13 zOy27a6Z#rH|C(BpJ0y=L3L+TK8&n1htl(}Mf*sn*m%q*52o9B*oIg?dvhQ9ZCDoE({-b^}6Yu|F5$^t02* zBy9^gpW_z448g9bOs^Xa5e>84Ap;GWbr?z4dHQ$?^3J?GhZaz@8*N*7T+$RSX zgB)~8ZbpHevmWy0J7MK^N#j?-MVIx{v35xp;PVyV{9coOv@aJlJ#*4)n5>Kvm$w@p zqif34k6}L^V5i4y4&8J#*q@HkAC*1=6M>Q1yDXXS=4UHxu09sK`ZIEf`gO=7O`Kqn!YhMtqN;@ZB4P&*%mbyw z_Hln$zIPC4?c3pP(yfTGnhk(4=BzsVR?_D3XDc`-;P_M@^t&URG}e)y-L2P;g+r&| zCMiO;_}Ncq@XD+j;s-kbQd^QdL3-4Zk>%c{M)8(Bp@CT%+ zw2J$uL8}FOcR6NIcJ)#Y>F!apq2Nbe-d2GMgzPE9DEecR#5jP8{K%at{pmj|1G*4h z9THegHHLkhscYiV{fDj3Cd$N47d&#Q#pfUJT4c_2)@Q$eA^-V=gF(oDhv=t5155V< z$}T*A%$VfG*$((EpjjvJ80iU&VY~iu+#+#ZfqHFMzsi<5IAY^Hkg(1`y#@Ndo2I?@ z5f}XZ1wGrg|50I-Zf&THeA<1>0xo+_PEJ0J>{{jUV$3)-_^prj8h#^5LOr!_{!aA5 z5$c%IZ{?~sGO+pkJM8t%Rbr~qwQc)hICHRK3jFX@z+nR|W`M!pC#4ry5*=JZuH7E+ z(XR1lZ~u-;N=Dy$=>xcHYxv|S>Q;l6{d%f-A#gDPJXj#!B68H`jq@;&hw$_Bt6(yT z%+)eLMB% zlhZYLIy|4joV9Z+9m>xmbO?mL^YU}$5hdR0a=y~QznrCnrEMX{NTKtn=gI4^ftLA3B9!`^lU zgRcx(d2$K#CQG~=0+}4f#vF^ysq;JF4?OrEZH$Vqw3c|S1Cz_VKoyH^~08j<^u&MwjHYx34dz*8h*wqYAj}h6d3+T}` zx&6KJ%LEK>-56YkU^Rv#(w5)U1RZ_Muus=JM;9!)a$IS@h%tXMQ8kdqPF$=F6+yp5 zu@$fv?J2}i7Z!(Rrh`cCVNSc;SAPmVR{MX3($@+D=qIb-qPy`4>P_K(n0lOOXYIJIkr~sd%eonMW1TEUVYA1p_BN$*7JMq<+5oQu^(p8V0u55YSRWE)7Rxq*aa_S69-IHa)fr8DN{-b%jw zt!vRbqlXIqRyqEeJN)KXw2I}9iQVHl(s_*P`#;-z{yslX*tJ!f_78KylGw7>KfnBm zn85jh3fk7KhK!M+P2`<0zI$PU*~%?3UzCRoR@)j0N0<{h*Q{;B^x$}nXUMb3NQrawBNUA;C*tTArp-*wB|}xStY8KQCtmq> zUciqA9`~#uDBoHWYFT>KPRemQn_l$hUrY9knu))cEic zvrakmE%%B#Gy*38C!>iB9rEcK3HFJQKqr@F&j^qM4Se&I1JCQ$5Qx+K^o`Jtp zAjSE?R&-f@KJfym_mIwT3xoNVVA%hJ4szcnujbn4~Y0ij$m%HLe7-F8K&1U>*SNNZ$O)_T5+>!T{?U2^I>N zqn~Z*$25T()Gd2SA9taPF4~ydG~QbXfC?VlJ2To?kNb_m#rLux&OELW9!Rk7-dGkc zQB&4CDpdnU+n;B* zMIQeC?7inkC)Ym1FM#U)ckA-st(@g(6F3cLW><%20tP1#!P8F-UU+^FkkJ)4g8(`TY~n{K0b_RDVh+pM0;^4oZxc88t{cb)H6}X^}- zxl>V9b-p!>`O43&CFCPmzV`J~z%|YUq0y^$z8$<1pu`St0Jp;} z@83U70wyx>QUy23u-u(bf3H0~g%q&KHQf1JJJM3=LUg)wa@Qxr%goNt-+$6q2ma1d z{fQ{dB&+!6_WF4|Uh9<66N#p| zJ~X;+r)Z`aF-!ZqXz3U;oBeRb;psL66;0_}_-p>RYqo9ou^UF>3g@4beF_&xoKqfgmW=EuRb#hOYx2HDA)2KH3>50LVz#am} z@}S|>z#sq=p9iCKFlbj-RRw3(@kv%3qO2n7l4I|Dke5(HbMwcRwf^*@%Kl!#!6ImY z0aJuJmQVfN0fHv4)<9ZJ%=$;j20I%_o2`cZ9L+a1vvW@dA}t6sa4|(db3kDt%ccZ@ z?h85;^;r|@0E!K)k0E;xp0Ec7+hsrv@%ZuO2)8O$t$!!s&zur9c%LD2Fq75z| z9mYR?`~0eQb8{0n0CIQmZ-rF_R4q3rO3&ull(B zN#k?KSOij#Ko08ydn=w_;-84*hfx{DMerG1sbf-768Nqlsq0^p&eI=0d>Z-Pm*J9P zUI@yMu-2IZlu|^c7I_^1?~OY59R3#b7o;Jygvo$L5KXd%ikBmPUwg_Ira*bL8=xnp z08|q0ToZ)?fx5?VMEPVHehwJVgU11=`X0##PlTZ0q`&3-|H>W zXJ0ASUVIJTdqNq^IRc0$#u(r4OTQPqw>Cragi_@1g-A9K`T%^nahK@TL#sC!THS0C*Y zfpf+no)vs}e(wn7u3oD5+ptgemT$dpxGy^l0mGKMRfYdkM1dG}tQAE@ z*}R5AIxpq%On@>#3`sJm!Ikxu(O=)bIKFUEi>R9o#U446N;ity0z1lAHydaGqyPcs zc_5={r+Du79|(eX+X7Ugc&`&F=JRwl&}ZKI`8%L7>ceRM$B@^6oBfT(mE+}n&b4j| ztV5OXcI1;Ad-ge;(RR4wfIYWEVk$paUG}t_-*I51qT2Hi6)=CdI7c|}A>b3Us+9G!; zH4VoJB^Ep~&s_i6;J+MklAn6%QcAellQKg$@T%YDZtN>2;+^5I8(amKuwy{}k&)>J zR3Rt)MUpzdgw{N@#X1Q8{<-0~x5G93O!NDyL(|)F^f$3n#>=?-%r=k={38Le)!fgu z4tVyZ#w>?lTQ<^7p^jxAzYMOLXm(=^`>G*$V&F>ivl9&u)lV*Xw4q-9zTFYc01xX8&P;SHy-F&km1?X zvKQM5VXnsoOp}Ae{K^xZF24U4CL_?;_m5I>Rd81v=dh zm~;i+N6)xkc+!5XDr|9cbVCmjBy(e!kX0%d*HC1*j2@bJ{rh6a#?goAkeAnId9 zan$!C?1q)^yE!>Hoqw*vHgvt00gYglim0#W8?U#zn9#f#GQPD-_!sU!=AwUDAN(KH7AzPPTjdIz zt-qE zF(aVblVjrzWcZha!hjUtC#NTuYQcT5v9SRR7(guC{rv|NaG|+#pl|Z&d!*NJFstTV zfq914WF-=M3}m%iTV4A9n=HIM(-Qiwi<=et^Y@?beWwVoM9%4J08wq$f^?0Bf`T}HM<@Z0#1Z?-F95`mZ3^qFOJ-S_fY`^!txp)LER_7Qu83ClVa`l5R z0aXQHDNH2t%Gh&1-VvGM;-~B`}DnKrG;`=sbQC4nF&;V#y*C)MSty4 zVsrgpZ{NYyWYcyV1tAIwDk{9b8NR63O$yb*c*GY zcUK8)Wu%agSy{lH9oV8oIRX4;64jJ2-S1-buNJJlZgHFQ4f;$%bdCrX+i#Q=T<>_Nd6 zx6v$}qGDH(I$Arl{xMvbw$Z7^{x9>t9UNrM$j8s0WO+OM1-mV%?)nZEBzBJc5|vX|eL;LU6m0e1Oq;++*EKd~u3~s~o)_yJ z1TwRhS5#bpta;TLY=Y@q(R&bJYikQeniZgO1+omN|ggE7iUWCJxv;0U%!GH2sQtXSNBZs>Rwe-yBg0*#M5(?x5nR zo8}Mw&iSsjwRK~vL6DN?VxQ-qTT&~8_D@Y2-5|k(%C^$6YH}N{RFn#=dRU+Z zOzGM+QVQT}e-C)or0S4IQeTDr81PZ}QC6mPgaHz*+R7gj3Zd zxGa!1>_)=_OPHQw0!8z8O@I4})XiG$u_mRAs|p|QDkWJS`7QQ-3>*O(x?)k}9?Q`K!UE@hagg-82N3>iv%1K;)AD>uL_A zqJ&G$wYMh$5^S8eulpuD-%&bqHRZ+|ApRc9&GBZ@d(2L@eLv)!e|imft3_parpi4? zFepP~Q}zl!iURY&D`Ew&q^5y-8n_9*D43bQt#1#o8RHHM^wWctNKvrz?6}1&cBi6? z*_Gn5Ii)gsbtjDUleFPe(&N3(f6u~8cnW@00oN{CN*!Iac2GqXJMiZBilJYP`*BS{ z4_SY(HQaFNe4CHeIJzR?J@tY_C3Pw1yN0X;IH^Ea95^oUb!ozFZ+-~3bF1@N-)QX` zIw%$jyxYX%DIxtf zqR03e-IZ#-;DMP7r<=$W5;mKy&qckAB7 zeYaSvaevp9XdlDO65SK=@rf$qi?2St6GEUKSF0ii4Rpm^(N4&QX+Oj59fLo02JKrt ztGzQ#e6Rj)&G=UZ&o|stXi$}$tPBJPK@beJp3P}kJPP1I#N5x2Y|hH6DxS3VP9vmL z<0m?fL-ldFaewAXDBO(! z5#i_U?Q(R=Lk($Yzh{pUt@h5G^4G8zq<1ajfj)ph7O^1@&y?GG)9cp$9#5iQ-+m>S zxWyJdk|lTCelD0PSCfb0Lsku*oPZq_xxBD3_Ht4;u{f$sFmwbYwaY_I6C3&qCsF`L z4!Pg&5-Umu2~HeOSfUDxL|E;GaI}{M_^sAQ>ZdI=lyJ*B|9^u2L6dYie4wC-}K466<*N^o{IC6O~3t0 zJ6BZP$%rt(j2oE6WFSl#S)Ku#%AZutyKgx%iDp^qw2NKI0)*2N!l{LD{=yD;OC5^e z-@Wo4r2KMOZGQsy-3VDc(@o|#ZP**ENiH#|yG@$8_*Vgr6#b=V--Ah38=O;k9=Ex5 z2l7ekl!T2FnQD9vNX)J0DP~jj0hLnz?fUBc>0tAb;^km613P4S8$^g*>z61*bMv|2 z=NsYF5W!{kv1^bFo=l|}NGN0_u06Y`Q9!su6r#YB`=!6&TII($_@>c!^9QJiW5l6) zH)D@}TcQGW?yKc@B$um`v@}u${;_rgy8ht}C2?*c}s z{B%<%)}Donzt&9E+$TrulBcE76BbOdSPr?pp5}QQkd-Bzf-giImBiik#Vk|6jYwyAaSMPapl5{R*J*OmQQo4Q|W(U}uCK<)fTL zQ`1F6RxX!_-*)4d${76}fQHHLJ!i;Eh#Rn}t(Yc<0DNij36(t(5(+l2iK(ekx*|pc z_E4X1lMbU*-edsS27NijvCPtgPWJ^B7CR2NFv{?CHhc1$EB=Iq*@2Q znH-PDH4#MQF9i!eSbExzX>V_GY$MyJQyGP@UG$hes(^!aUW4Ugk32KLPs_MB~vVREHW=>Z`8dhU~}BrR06|B z-ncjGlp&kEh3%tx_)74heZzM_DhReF>n!h|F$IVn@T3H1|GkAS>faJ-nF0G9^Dfn3 zPhtW`eXQ%4Cj#4!z=8#B%15Tg(;iqX0l<#vhUuG0SP;N3G-1REZKXv)O#~{!JPg|m z{*+q4j?H0S%bm->$T!E8fp~u>DZDlp#iCR7n4Nn@b&|On}7Xkyr$HF=! z5esiQh#=kt?m+_z){Cq|wui4_C$lnpEeQM?fEfIn(EbQq3BgAHCi+djwIFA4?e>Z& zP_=k7&;ucH0yQP;TvS$_%Q{u2AASm>+#bVF6mg&bQGK}c`a95M z(Hn+sXgc_SB6LSIq1k#yA_qnz_g1|5(y?!w5lbAXGxJ$7@57G09{P?QsvW{w9~0p^ zTP7i=P(A%?5AKvL%fOSKwGy%T0fir;;Ba1vCo~{zNAf4%FkGxIClgd}IoXw#l}5NPZ(#Uz?ylo9I!_{66Dhn0?m>87(;^C)JYxsX|G z&ny-^)f^b3Vn>~Eu#%FJhi?LA2}IjKVq0X;hE^6`EV#`VVfX_W;Q$=e047bR%1YYy zWoZ4Ru}{)EN{LHy6|BLtN5ma4PG}E&m4}$W74!nY0n6JV-|n-1bl;l<*3DqQCG_F9 zV0eE73N^Cn$+cW492%FoD!bQ$%93Uz}D61nq!Z z4D%VsjN3s*;4wUX*w;Mjub$0P;9zA2%nFs|3Y*^34bu)|J2$P$K z$K;t{E1VH{f*~i!!8Ev~$@1o7lBWHoJ?0$~Eb})k6X~2cd2-`&N7h-2eHLM!WOj{Z zAG3Ewf&K~$m zC_@|*@P~O|<;}tugMAJRF9QDrargvSO9w3d0dh7==ZbbHg~FQxcPiokG5F5?0caf4<)DN>7$#FUJwe~}Z$o;+k8|F_cMP>N z?8kTKVJD`_T!Nw0zsWq$EU|}6i6{e!ix3|-?7Pv<*5OyQ3cJ=f)$n1<5o{(hvc8rr z2RHJ7XWfVBFH5pjOcZSuMU-`4M&St%2<2xo$en9u%I7YhBmF4#x8Q}c9qw7R&qj;) zQ)FgFk!fH1l{=ScFZ{P{y!b!;e%zX!2F1t9O^3<=2I{UtoK@XY{L$FRMEU?Y>A%Vc zZisu}`W*roV1K{s)@zoNB@UP`9$T~SVPLBsv-rjKbi6M}Of_CBkEvj66JTbiU~@dE z=?moV+V3RU`yF~TQ_GJ1%fUO3cOJU8wan)a96+ox~ zc|$sI`g2__x&>MnD!shG@P02XcfqwuwL#_oI*rEjrloBf`CvM`Zk2=`%{4ZYmyhm_ zJiPeJ-kK5;Dz7ihrJV<8w20wsTJz%P`bqjG0rM%hfLABjc7NHE+tfeQCXu(XWZi#z z{2=#B*sjiPglj2Y&-@6*h)FU8V|2pIc(x=GZ+L4lV-w>uaFH;I?88k)3y2lX5r!o_ z#3sg9?+ZSfosiwPv7cS`IEn5F%wjgpdUyW3i|87IV-Y^3l{{33&gQZx`K5E!TfXB=2a ze+yt=Emi~A36!;ntB}jAETLN?Y2~9ymL=c5iP2p~9#Ua{noiBk;Ijy;Sr}p=_18R> z3t2w;f_7)JN}bDd?kB#9fF}F`JFEb9xe1$mRRNF_^o{fM+#^42^up_2QFoRufNhQ{ zz1sQ3H+pv4va7W*I80t9^Og4Y9m4~esu73kZ?9Q~A!jvVpO3m2>LN%h4^8XIwbTbjk~(}< zZ&gJuTvB+N->P?8QKMwcJnAB=V4LKp7s!4@6op-tqkH$v6R`PBx zAPa8^nAUBJ2G=3b5C2F~HBamh)&oo{)zM*l$@c+N@;hSOeLA|wF+YLYjjnatE1u%E z;zh_OjgkjYj-jVq+JzT*ln*K$QCn9ky-&4o>m=Qh8B>aL@D@~wE1$N5v*!(+%#c&a zLvOEkMIKPWd|-=M?UKjE3LAmw8q>3}Nr8&5-c+}t=YLkDR@u+?Pmx3V<#m}0B0J3P z?8jDJakw#npfhQu#trluwC^Jj9ymWK8J*2Q`_(_GvyGY2S*i7I_hxhqmf=Cwjo6;; z*O6)`lXU`wUu%y)w=71Fz-FY~#AbY*YWMU4F^}bff@<^24Lp8FY(~me_O?prU3 z2zevgp`WQ2MvHZSf1g?R=oW4Ug{}S2hpWW?wXsPZ0@L}MK75}%k&l(QkO?C*hFoFP z1*AotRE3)73cRhqY5Eh26kfXa3bpyWlR==Y1(b!fkG$1d%Y5{-LoTu=ii*|A`D-j| zX}3ITI;~|KG5rvH!K6YwLE%rQ?0X>iDN?H%mS+oPflw<%x!(U;5WD2*Wg4rN@xCDsnDOi z(sH1|mb=(XYzq2oPWf~zdD?`lxYBXjRltWlh&kVSVjULIUm@1bP((b}HgY`Y7VaIfK!9EE z{qXOK4f-DvbXIQt74Oz3X&qMaLo~Mf5ss|&sD$Yv2pd@-;so-r?Kb9zf zz_nF1#p}#yQ=_FzPz}{%QDCFr(W!^0`!(>`IaAc%G!vwR^fVSEB%wTMHYem4`m&{HD;Yv!A;JqnmfA@YL!U0M~b z1iKvzuG?a(|NWJx;=Nx+`%k{ub@lBf`hA z3FWWwW;_mZPLKP?N<|QE32_9Q@>x)r(48Z(%nfCcy3tL@z|Gd1T}WB_Qxx`}C3UC6 zkj2cl7*#CaFjI%Bbvg=XC&=$Rjhx~JCrq<)h@}5)2;^x#Z={^d!q@lvxaGLV&{4WA z?GkS4(6?soxS5hELf-WbIUnPj)XFxATsbEH9hA3Ic;I z<`wvY0fS^qUz8Jkh(=+}cgC@g@7?cGUI;`1RUA(mgNqf24PQl&7OFU0Sd-Kcsy-;23t4=>OVm0L_a2(>aokM)uaKv zbZA^1FMUNL2!%8g^jZ4w1j?#?s%zq_NZj>9CUpkA`PycxOi-t?PBq0c09d}s{YfjVJhw}Yv@!y`?L-vmt|`13))xWj&>f_7kK(;pdf8y9ANSK`s3r{k^rL2 z2rLKc0#GqNjJQ(s4fx>~_MQh6V*tqO1TqSsAx;Mj>gB8;)L>(ZXi4&ZcKlCn|jtk zB2CIlDfmT?Hndq=W=ge71WX$l+J($jphEuk0LR%p;;!1zFT=?L{5-{4vilO5iO55q zcMFw6uBzIU6U-~fWQPvTY%Uj$}^D3n!;^XN41XDnly^qTF7BQ@_}T6F9}sg z=R?28Q54eWa7y_fJ7_O&wCUe3r02;g7{8vVS$1hgKOD#@3rD^rfZ)x z>&GV`kZ0=$ z@6Fn*NF2Pq(I9}M>N0J!KPmG0(TgT~RJYvA+|U2H#lS!FuiFjh6m0ZIQTQE{LDCUW zJ57y9fRmi@Ed{eE*3ctAS&$UO=9f38`Icqc$=lz*PWY{q+Y(e^?ROYOV*c>k>l`wD z&9p%C5l?+haR6kCV6g#7mS6%EGkI+N6$16GxA%bzF{D#pKP9pF6yfPP;+me9736pC z6_izCHTp4|Qe3iWkiLC!g5O*SxzShyNEJjT_w@BKOSsH|#8JSJRXWDDFM`7dzkopO z;~dc3WLJDuflBU#1$l_Y94N(cA_2_--WznApPtrIF94}|!XS{PoCn(`fEmgi{YB}* zUVOA1cab3GzFeNH6fmj(nqXjLwB=O?qDH>)j>Ry^PB@5}0zFBm`rW>R2qwR$_5hBG zzY7EdfKzc604goO$5KG%YIneg2CVduXoKGosBosM{!x2r0~z!Z5ZGAi1Co^t(r;IB zygi8qP?r){AxEV_u|!N?hV&EYRRsoEP<^QeIMf2A3z#}?Bn(#~fUfOZ{Uh}Zsrwmf zIh@xNp0DWIrkVg9N&y~d`SkUbh7M1UHo->;SVtx&Xj^dY(!E5B#oC=b7|s^H4wO*VV$%N6Hq;7=8ZswyuK~Mw zsd<+TC!<1xQJ-p!tC$i`(Xv zj#)CkZo(-*(Na0KX~7Ym%x0rJU#*mJ79`Y+~Nl+72^g!AV+2fLN5shJ7Sl^ zf<(~az>_d)BGXQMy7Cb?IBCGMfGA8Y&mAOB4cL51C}5hls|c7y9r+Qt#fG4!EX#4w zP;|=n!m4;HU!xRDemg zwS6med^n^IxbdcqRG?Hv$S@)1DNH{lfOZj^VrFo?OL?d@O)B0yhEb;3RfKLzAb;tlvZAd#(`|L!t9uvdlZ>+Aah1&{Hzlsljp@po*i+YBHYnxMiicB-fn>DKGt8{3Ti zmg5t1i4jN}8+MlDLjDwxVCriMf^C2$*aNvn#4NvWo0@RKA7%_&(}8qTSzR3u^6paC zJD~$m40%W~!flf*X%|gO$RS{L`Z=TyyUf}`3)D^joY58~5`Xl88w~;h$8ciskypL8 zoOd*h$+&w@rW{l3ZYA?2C6}xbLJ|T2I&~is|rA`s_+%CzqQV}msUXpm);#nXdrG2Z$|~mc z)NdK1r{4x7K`FM{`bmkm4YVy;TQ)YhA!ZKaX|pi={qPN z#Rb7AndaNoKr>CdEU96uZ1O{z%6C(x^a)C6?E<}Ri%4?SIj&;|jfqvUYbfU`H zG;>uy3DYP55&(hNkMmvmX!PTU@NpCxl%`M+9y{+s+r9^;7GQ}rUyV`L6-cORD;?hh z5yq{w#0e%EwS)dLxnhg9=? ze`%3Hv6rE$P@AF#aAaxw3tl^vI%mMf$tgxijFuAQ4#oLqMo`mTiAdzHEGv60l$)UQ zp@yLyRMCl6c?lcnGtm9w_P%IrQ$NX4f7A}7c~x5FU6zW`nqCGH7hfV4p||`@(j4Th zWpie8f`Sz2N)pE`nr6{5eQ4aCj|y~81_cC5pvZu}>aZBxJPL0H)TcW`-bNi!(4ygK zhSV&mMcREGcLDb4-!@MhvvPs-^(e5Z767A`o4O@q)=povjh)<^Me8@8Pk&hTMM~Ng zlfQm&^dkS%$;pYI@?kG#Wn~4l%s5a^u~qSQy~~EaqEqN!(*-l5p;EWI^f}k^S%~dr zqKU}IgkVi6rI%f$4DBNZ=iCDdzfx&KXk3sgUQtxq0Ndq&aznxZ?d1K5{3*lPrm8aWPXI3@?h> zc>6XC6{Mqq*TK6htjKh#jz!J7R| zX^hW_yroxMDZ+v+;z+jMaOrjgt{6y%G(S^hDddZAM{w404gBWNJf;%Z{5BjOUcnl0 z^|#}*$0EXW--R^qJi1co_DMLP6F>i1_mzz(9Xx;V*N4|kCfaq0+s9%fvmRDj~rh6LF^Fr_%K zh4we02zRiHmsvwWBj!JUDT)}ak^ir6@AI1d&v&1{hF{?QzfWbQ8TlR@_TL|54b7&z z`Tc*sd-lBf{LTNoD0qc;c6vYm=Q{e>n z9k%KIJbi!xy69PT=-an%2ka^&#E9>8`Z>@P$(-DE<5HMJOk`~AsxK47LVss_+cL^l zl&)-?K~`1qpXZ@>F#%vIpLIH_cAzDJ=2EohCzF!Z>bC-TMXqR>bfH{Pw>giffj65Eq2krP zYfd@FO9NhJQNqGr_P3Fg!7fw%=53AYU9=bG!sVs~b($aJ-e_Ke+r4Z5%2T9wp?Jofjn}R_@JqCcRXfA#pW_Cfr!jva5eDG(L(2T&%CldpM{7QmtI8JoTu0}c!Z2}Vycx+ZE$#n%+gpNx zpSjIk>Tr>IEu6R*iTZXo#s$MfuLm-9KpJRrh*aA*i<3)*iuGT56P9Ue+# z^e!zey|}>J_Ho*&N{<()tBj8BSz4#{+?(HXiV77$+= zbw46fbHlA>VR4b;S@X*%A(rM#{PXX~ETh;ghF_{KIdS-!xc$8nrNP25ueM(=U+vLK zOG(P8V@h8NLt3m^bJ+e}QBX)$;3&~Y8a%`SuVcX<|S zN-|HWX&;f4T@)*T1kBr_K7OQya0&?AQv3jF;0#%RF$gdj>Fn&hNb^0R&8J}oBtIAH zOioW{Bqz5YV85G4WT=&l8S3f1(JCk27c|FD5?<|eyz{j?rLHkVK|Ds*Y-rmB0-ayRJO->r#nj2 zNicNMsnEejfIG$d{#UZ_>rhsC=w4kN6=S}7SH_1F8tGkalCOml{ZB5tE9Y`wO=u$_ zQFt@0GuA&N9(X(@!6zd+uX`~3$zBuJi#sSA z4A<@e*r%)*bBZO7(cbuxM**e}1;+vCurnsF(;xVvBSN6ZxWk45geh zUJ5((!1!gVnU zl%YGjd`8u~!-cX$X}$Vh%aw?SG_8k+hcB3w;)W+C=$M(AK}Xi2KFJ6QP3#t5Ugq-V zN~He+WTY0t8gSoC@e~zDN5`9ymPWQIvnbvC=kQYrZVejqmoM9HXw|fQ8=Zj35{4t~ zVIKnr28Z=7Q)|{8G!slPcDmlHPv~d0@_PQLM^-V8<5mbxMFshVg|hpj6P^?4VIrm1 z4LU1l2!AQIjgM8C10-xWes`0SAW7II;asdnx+A^*%zmSzmV_l7)xyDwB}WP;$?D#Q zZWhArC>7D$bhvRh{@vr}dz=?y-c#ZCyY1<{y_k5;ELH~;Bm3UY)5c?whBF{su|1V! zH1Z~mNJ^XwTvhL({(5|@+~KdP%ewpx&nbjir{)F3Wez$G8MpaRh?DW9+Xq^Uo`NT# zM$SWce-xtVM}z;t6I9`l)r~K$Bo`ZNHa=9z!E6dok0VP*TO(Hsg0KDNvP6=k%?1P< zO-^T2Ml(C71s%UERJ-BkEwi6C%kb54`k=Ml(e3xt0#AiD+M9t z%$SYF2*j8j3h%S{Wb_qAKb}%POgGEVWvEKqle}>vu)D$Mmx@CE{^fYCWoZ8C{&pJ5 zH)tSkIi6AmlK;2Ev5fP7;E2@!`q>BvjV?Y zlw*C|i;eqvdAS(OrD(xsz^R&*bBW#O5e)MFz5VuSY)KQ!TcF5qe(y#y;w09S>GuALF@BJlaQGW!6?>xk zM1Qc;hx=@NJH!lmX%(wf9dkwi5=j>O4sI}I^%X!{iO*fV`I$t<Z7G;k-sjWr(GoOat8rEijg9m+NSE*HG8j&vx;U-M&Z!a?~{S7 z5a-QWbfe!~*qE*?hG0}tHxuA2OmO&Aw#3QhvXt}dIDZ`tAw3mfVmE0cc%?U@dNo3UpMqp&9Y zYGNPu7|DUatmD!%k)$0Yg#u-U~3} zhSS8^si4ac$S?Y7D67X~;PbNXk(8LYCxaN(w@&G?vV71cavmwl&+pWmF3Fu(P+xDP zPfO8G^Uqv+nm_O1;qfM0@A-34`%&JeSp@dP6+pej98G^KD=Ry@0^TmL!-8f*%{yr3 zEp-;^XQK_51SnAlm{-489wiil(=5Oue_BdqFg~tP35C387cbUpJiPP%;UnJw6_1*? zuD{Y5Ab+y%OE1~jZRSC!<18RPs2*{VoBTZ$Eu>qQ9a#@b6(tF$TMa46({T)TwmUft zC7{WF)%QkU$G4T^preZ9n~8`YtmuU-wR^IvG1L&mQL(p?_sE5G-pdKli}MoTxruN^ zNDR|NmYBb5oY>*aalxc>-MZem3FJ**y6+fc)s4IKmXf8hp^)GY1~_3 z9*T0b9R7?VlH*+PEK2-iH{|7X?^sB=TdJ? z_aB+8EmZyD3i8I;6-fUo)5+1w!;GC6DMY}0J!SI~dRW6s(v4Te>F1x1l^eVE+-I5$ zxe_>_?LKtO>_o-_$K>z#|9a)l&kXe(8qBebVEH?;B4p(K0w23IV4!HlXDq*$SnPASyTqMX8gq4(NhT!I&bRC_Qti5riWp{ z)PL)6?#)Xn7doV1BrKE?S3X*|&zM(D{rLTp1acSTOWH2NazAP}?EL)gF0fWSJ+wyY z${@NkJFkiC6;Qc~rIELa(rBYaMhS6uoxrXtKr?Dm)6yKp48ge-oLtjwQAb+)>iPwwEtoE-WlC zTxriAwyP$XsaxnX9e>IKMfMr)gW+@T5g+GhAZV>H2#jM?xfj15w7FNAdM+m}`A5h* z>K3bBRb3o@eZ(@N)6{o;S&Wrb_=1^C-`NBe>g(Q2DfU{M{Gp%dkG;~#A7usJn0=(m z@dfsY=ReM-FQXc&=BJ%w|1OyY+wit*PH4<{ZtriZJjl|8LZ~lAh1b&rpCf_XJ4Rr} z!N~a#rlZpsg6OD26TT=pCe|$_NG1#Ly43lf3tkP;@hOtyO1OxsY@Nm`W~m*eV1gqviPu`z3X5>l&MM@ z8+}+C*1@5G^smR2j`gVMthffbe_1`v;?*K2G5LTAllObM^_g)}c*Xb!%k0GJjuxHT zZNcDaQ$)Pn_Evp9#n3R$pL>Efc<$(Ccau-)MC0N!`g5+zwnxj#$zLCyr{{#6Cp{%i z__LK+H1AtgGX*W5k#8?3a`$Cmyca1*&oA|Igr8Nyr@3;dIvf(oKQBDD89Sihq65{UIRctj4GB z5N&P}(n_cOU|OE6>Sw>PJ46LDZK`^yV!kMc9sh)4eiIWKQ^{aX9zB8_2EL-vOXp` z+`p!+@+xK$gUJ)pKNu_uH_0*+%|H6#dS!B;gsRfS3!g3*@vtzGB)ELbC?|YuEvi7C zke7wHkN(|t7-wk3eR+4K`r|v<$(V?^6${mkt?i-qwqL*;`>}gHNV-bUVl-bk3iqiv zY+X^0&&$f9esX&LL44-gVz;9wIeOHe-{k$QIna`loBUAniFx+fqc5Z&FfA)^C8aam zTv{1@)$Gv~xX?vi6Tvf6WZgt;&}3eZgtJ$rE_9a!ccK5IC#!R;;ayURXjiz&uYnBa z?V=6A+oCfp2uxz~FeC5w`qvUq7hNO4C%Y#SR|t3Ip`V#AvZFbq{YO%j9#!SZESBdZfZTC zo^nF3QwHx3%0lMk0~RaBD3YfRB5q~&j2zWetfyFA35IS*cnzOA83eJKCkXf7yVZu) zZfeTgPU^B63#yPb`;$&X5=QZ@`s|xDs(y1vd1d)E7kg%<(_a(+vylVGI&pOST?@nK zKP`lPq?X?*syR0tub5&tVoYang_CiY1Mo#Pgo(o>LCls>*{@o5mAQx92&dz-sW_;CNQFM%R5ydGqe1ttj}DF0{U~<=wPCPK6%DZ zq?*1nrq22%h;{w5cqAMg=j$$K+GHj89Fvf*_(@fSd{aEYyzZz#vb}MJVfY?L!Hd(w zeCCE9_s+8E7kvnoB%WH;3g?kOQ*QQk`T6&xvb^USS~c;bfz z(`5S0PEYeee^)16)xH9}N`NoeFSBPk1DvQl3K%YrcXMaY2RIt3dB5vOxvLZy82C5` zL|1@j(Gj~u?LN@Ee^=@Av!^M?0QX5)uDU0GDJ+3LBWh?Bxyej8D)yDU?a0`G{uZFt z3VBmM&%8+?U?eyGMipovHYnef`0YTjtPscDkU>*hSB=*^{z& z$+LkAd+m01vD3$MsPa$^seW@V8oICRT#y=~6mbx(=%M@Nt)4RL9o*5wt$=}ku zXcv9XL!vt49<`rd3nM#!Ee4}g5@3VSHJV7Os*+~NZv|#bZ9H-T+k3CeEshA-fOYkt z#!r|0lvvFCR zSO1_@JXTgy$9fZCH$oEiGCNq1=o* zhdndS=5m6Zc$UyfhpM;$9N=9E_uyx+n^&eLLS$A zp<}x#-w^XhKnE+u!S}hY!m@LL{-zb}2EF3F3$!lDnZ}2s@yt``!Ivw}@fLLaxtjWM z7WGZQ?9f^Up8q(9i=TgIZB4>$>8XLi#Gf&qCwwuB28M={6BDJSr3v12z-o%o8_YMT zA}ccc1AwLg9Z5;X!^4wkr|#`tRoYNca0$YDQyDO#9mye5R}Y@w>x(cQG%_~ccB=v4 z9blZm%xPbg^x6W5U6!b=CSZtk3K>yu*VOXL3JZB)FocCWFj8@F=q@T(B#YggZKK)uOGu5h#v zTo76LqDL8Zj6)_4cO3Rh#ce z4?dUW2}s`8R)pxS`9!Mjk}D#DH44NzZmOkhZGHEBD(k{L0)*eXl?%7^ed7BwTA;_38P=J(+O(V1I~@6AphFFog3Y}B;xywUqc1_ch zFnbTI9)PCO=Mk-^sVyaa4M%c=U9PxTRaLb@kN3JEZDDl#rlOw8x)Z2r%yN-@n(^8YC_M{;iOqc5D2@@v#>;)&sYpd*3#Q z(En%Vzk0!<9Qmx1R2WRp&N_>wCF*VlIA16dgxi8LTTAOYYm0-}bUNao0kWm&S-Jasr-uN?%lglr}>=k zmoZ}=vX#WD^m2JaoY9Fl~a%nqQ8W6wa;duXnu5wr1h-0NLc7+ ziaAbjeQOdN{Pmg~(2t3di=gZISyFBR=<=8L=7;#mWCGF6^6Euu($A%(ajTe?mX_}B z?muljFmMh%2>>m$GXD}LoKo>Zc|+rW|AKb6!`;-Kcc7I&Lrqst&xhN#TjG-5yi45O z`=X+Mw~xWkM@2%=i*Yz8BpQ95r5Zq$!~W)Rw70^V`F5_ z=*uoFzm&B!KxF^z6H9Rxi|#vPw9-*@d{|Z_Id3lkRGE{rzW(0uf7?aCsMe{q6fVu}_+U z-~;GKT5QL84xeDnaJS&dD3;Lv+dQQn! literal 0 HcmV?d00001 diff --git a/tests/assets/waypoint_flight_isaac.jpg b/tests/assets/waypoint_flight_isaac.jpg new file mode 100644 index 0000000000000000000000000000000000000000..16602d3cecbd3c9b361f2e8bd36bfbd604714221 GIT binary patch literal 321155 zcmeFY1yozz_AeUTic7K30zm==io1I#7TimLLV(~-p+NEC+Ts?ZXp!JhC{BUm5?qTH zEl^4y=bZ05|Npt)xp#~=#vAY5H||-V(p=@f=2;M zUA=xcHLK#A?|23x)}g7B>nEP~Gtm8Q?!S*e=Lh_43M_1#d!i^Q0396-0}UG!3lrn- zdfpRB88OIsWiiQ___Qru6F+{(q6qF9mxJc7G4r2bQ-WT&3F!P@06f6BH;fd66d(;) z6Fk9}e~Sx;8OLK5ZRW~UeEbcNu)S5d`(8aEc4Ru=ImCyZMV~M4hBmEDs~EQg)ZQ}b zTX4xc79hiS^A6+mi=fwfl!;G8iqy)>Iyj`rvaOSBpWbe*06v+ioVamNk(S&x${Zq` z5l_cqtHsxi zA&OYk3Ui1@FKj*%|kYOb~%AN+DmhF{G}~A2hAMc zP0X58M2b-mx@}!hDMf58f-P=UWTqp0bLg{wo$Q-Zt72f^KECz-tNLYt?@Wf=5YUO* zZmZVhacU(fCb2tcqrPgI7_!TyW3%~Io!3UM)+^s~dZ4AieB#ry)Cj)iQF>^|(Wpn*BxZh1i*bggiU)~S!C%Isi_b)M7}(dr9Ly^ml8YOj?3aIZ zqH>t}fpU`h5mvuH3K&x9MuWQPw2>P+n&y1Ve!iU8hO#o}68iZj9i$oB9F?@!BP73k zz9!S1mRuKti{q0gvfcgJvO5fuJg9=6d_LpaYYc#8q7ya0_VPpzNQ|&F&+8D5eh68pkUXz5c z3#AIdJzkM2OTZR25F5~g4j?Rn9qN3$YpkIyV=@%$R zE%EW$Tt?kxT=#B@xTsNb$%%>exC4i023FaF%P}way~j!CTL~`eG<_VG)7hzypFEA2 zTH&o*ua~=owpVO=uQxi)ECIcy#XMszB%AhV8K&T01~Z<2Wc-!o?lgdFARy!>+lRNP zk|*NH#~%|D_aS5Y()$VR_KvOBG}DxoWDTQA!g~U{=OhGWfF~rf_!yX^`s&aj>05dM z^27QBgnBEokt{K(%C#URdB*kABSkbBOwt6|Gulf!m+!F8>xW0Gr~a>&E^e6h!JvN@ zAzVcAkS=@(TbQlwD?Ots)U`e-D<&q!S_>ijI3Q0U}|ps{Q!rjfq*-wza8&o)1c{ zIs1I_Yu2Vm*&Kr?<3fq}`+?IhHg}RuhzgJkFYFBqC|dX3%p27nCPZYx>&uA*NrWov z!x&{19W<0&@2kOi)kmAJV#XWAaN4!)?O3tiXw%F|l?1OzF;QGXMxPV;51wHlFG~GQ z4UKw-MhCA5&&;yU(aDf29+fmPekba^1iEb@?|O7E{L8xRUxs-iOPm#k&61OiT;78X zi0$Hc3*o(M;4b2Xf>q)qE!XuTch#kx!-@$@H>YfzL3aa`+9k3oGGVLxj6qu`QjUc} zM9rT;%H^r%W|s&~1(E!-AQ_PLcVL6(7M+Gn%^0%NwxN+>UNo$g*4m+|xj%!fHVL^n zr>Y^0#VG%HYK%3(@$8y=i>7>Qz=SChd1{OM68c_}$BP7)O+0*E9$t(~ljrT3_`UanAiF z(`j=C<$+uQHtn+?J^m6az}OVWa#5d7HZP0nBhyWi0g(61GN*LCmAv|3tZ(EArz{CU z>MAL__M%0nAOHxk3>cz?4^2V;Kj{k^X0}5ySxwQhF(!%ZoCWEfak_u%+^JxH>%#^^ zvoi#%vj!Ki$`?dYO-Cn+#avF;q-)TuCOybABaQE$Qg@s5%j$b6)X22DzBESZhk3m) z!|K*_4jf-DGi6(784@CkFAm=S)VInhRJc{TH#4tPFg=h(W1OXAg?;P!0IY0Tf5^TO zU29Lgh{)Yc-+vS)Ib?LF8u;-_i!8%EwJ8Jgy}USQny@>q*Z`=3Ejm&>8(Wl8iAmxj z_CfE_5c>LZf#1~(QeMX*hOw=94H+u-Ra}<9POzqeJS$~K(o_Ej*!G&`rhm=YTT;?7 z3Xi*Xe1SWkL5OJG#qUExgT_|#e&*y-)4lwCZKH}?D2CpM`pVqVSp;ehmtD#tW!^WT)k0PXlfFP^%r@6 z!;RQH{vJK1EsW$Yfr5D#t9KfQjjTeDF}tNEJy;=cQaUE$O{!0Z@ptYFe|_#Hp65t2 z8_tsiAB0R%80eBj(RtC=#qXds zFe&0@*y9_m@Gkr7ibf+LQ^&Rz$mEQo(ZS8U_G)TbPyZ7f=t9a<1T(yWf_z3NX2!vm zXI5M?jjMRR+AN-kiDHrvk)UR$@0E$qmFfsAT(`Al6ZxPlGv28x|I4PugOKh#MS8|0 zk4yX7PFJ0mTvKVrNe~`-l}wp}^eSwktfyGG6dK=nu*&}6xHn+xRRho%#Xe=+O#X&b zC-TelmAL#c_5-181ih=ix}F$z><#46KG$TP-AYmZUF{N@u+5g(Y>6kP+Ey}@m9m>I zYoCk`u7Io{&7w%~ChyT-3^b&R!v8@gRc+$9qQq^7tG#P3Y7v{_bP^^0v7cU8&-PDI zf6D$!Rhm_Ijs_6nsF z5I$;7&Py{|m~eijC*yPA*2kr1=fXT0<8EIB)2&J}wkIYAF)LVbRTe<#dNEnfg3v#` zrSYfxu!fYmCvBE^8fj>^riVJ8GA%A~Js+|Iplgr@J(VJoI@8Pb%COiCZx8zoD6YTx zD~QY~DAP)D{LjqV`NJM;W_}WG6G#)2rX5^yQPFu#G9TUgWwV!y9;K7+4X9(3XuUu& z#hcW8205yl*an0MH6%sm(~|DdpX&UnpNy&5)`ZoK+T!Ph!;6LHlz!eJBHalMc}0IZ zn;M_T<+d#ypSNb7(Uhh?GHe_zAQ##^`$)3qCvK7KYy&5eb8~UcxheiDjOIH-k_8t& ziJq{`UsJ#2`_sqjnfS_)ow3_oSES;npupY=lrA0Sr}-RoN!r={%&f? z3#~tO`!DROW`}jsOwRYGTa8p(qns3d6{0 zIr$Bs13siPy}T%|RBZrH%X+2_+sB!?G;NO#VQ+yUAo|!%mnA)^zo2QhxOL;+xLwWd zAf(4zOi79VjnEN$Di6VyxsAjHp}S>m>ay&l{B$a4YxQd13(VyGlu|ZJQq97Qr-JsF zlE}@HaqmmRrZT$;>om5MBhXcg#VP=U~?H z)_~9I+~rNexF{Fa0X%&03e7}vTH~*N+n&75bD9yCP}3{1Xr}^6<^$_%DU2ae9CErQ z9Jrt)?d;@v>RSyu|{52h{I!~MF zbcZZ|UVeDipzK&bBpL@pCTXb1k%S^d#YeHv`Wn_wGaZ6oHXG;fDF_m6rj7F^Ohoz! zwb!FV+@4nc{HXOO3cLTrC8)N4E1=2s;Gk!QNLl^;?4w31Xb}g;q84;26>|p^+>?=* zJiNKdLvzYv|u8L)@3~^JoCbx5z z2;@|!!}VGgZ&guGR>4kE>)jdJs?dN#OBU66Nxf0Y1=#Ci+RKM`8IylO6MLsso#*_s zdG~L?q~`y!Rx}??s{ML8ThnOV2~pDh0Sdi6g^@;_hH{U=HO z_ehd}_g1w9J?B>@JeY+r0!UJQ_SGKU&0YfPpKTHL&z3qveWQWCj%=hCw^z}gmve2i zt*!l;fV}mV3=O2fS^2NV{NK8!?+-@!wesH>{`mo?`u{ObF}X!0vh}XLA<`=?hW3B1 zApWbn3!*X1eJ(RH))LaySn4uynl;RT?76t2mLVG>26uI?PeRlCc{9ogsSMf2UJy zXIxXY2SZ+r!~cyP(`p$Go5q(|qQxDG)z6{t>P`NwFQESrqVW%j#avYXKEiaH|1L)7 zA2Pvj75}5y|6z!YFMIF}y5u{o#5}g1X$;jlQ|l8EvwI?SCpuiw!Ey1<{n30W z&Ob;JOt~4Vm*6`0>o2cgW(rGZ^8AP!DAY;Qey@5(hKCts~#m18XNEdJqG%ZFk9E#2sqX7&{c z?>P!Fh?MmGJLMcVAmi$ykjpm7h*3|SQ4lOb5@@wHqQ+KPUFyUpfH0--ynlhv&j zT;<@-oOGSoA;ZCbx5k%MD=&<${q}7GXX?w;i0a!1T^=|eFcsRLe1NV5)6cW0sukKO zJl|jol$chos8gL!6R<@LDR`0BH)Nsd>z}{%XZR_oXrYbq$hQ*6f!4vE*o7e7r<|(S zHRL<~A?l4A^`3SFZZ_yj;R1TX&Vnm)C8|Hs8qQIqH0pn(!7VN6K2Pz3g>EPACwj}9 zSBMo%SiGq%Dkqj6o=KmnrV@Q-1VA*D^g)_7M*0O2m4_q;88ZsvZu*gL;bmL`b*TdB zCIoYA)ik|kMRhs;NwH`B0l}JX+8sW}H5|SAC~sD=z2g$*6}T7Ha0CmYt;=4XWmYRh zF4`D{(#O}tynrSk-HHFf4?cnjj{z^SJRO`_Xitx=>`$EQ6}E9>WCigS+m z0sbFq=_}LeXjUQ)QW+K5Y-HshZ@AJ$9#gWFcueo>EvtX%!c|M7G1!f5YC0X~@!X3B zCV3bEls;*aE^3~d8_cLIDbJb`RvYyPLXv|@*Bc$5s#3i-f9TAyHj`bAbS&R6x3gL9 z$GE9D(Am*(N?j)bWV_>T38(c@O0LMPOWOc2fsOwkh@7^AX8PMl3=L*v`VY&gp%AZ- z31vI+Qi{|E2wC}h`iXn5yiZtu!>gUEL6P(EQ{1>AM^`x+)?k>GuY>Pz0PlxiHkk@x zRTIC)x3H(^;0!vU)d7-EqdF(!*bO9>C$7t;6)Uyz!8b=-q zvbwgJ=AC;D$3c7|5)wqpRPb=DJvV}?SYFk&!xEat#2M7MAD6x7O{2FtC*5L{k33oz zp@mwpZ>At#MkbWM!P6Sx29fSkCMz6pacq2?IB3Qy$j^a;2N~Q=D!$bZ`tZ}tYl`?V z7JE$vU<7zOsR$5d&WJ%g_f8|ME7aVW9R+rk`N(i$Tzi#8X_kWZN0s472gx(T`2vVS zW)|m%cii;u?bIN5pQF(^k)?X!3;n$q6KSq$A5Sf-eRJmW98s^aZQxd+DZH#Sy_&OhkW;&i$&@f=%2< zXuefQ7)-BDzsQ4nsPFq4&AFaN?FcJeCg@k+^SDod6KFhU+(J!>N?oncc9Y(0ZE!pa zdK(9uEcaCnE0G(9y*q{O7ru2Y0z0vLAC!3eQo6V8nmdi~=*~}Pq8}Zh9_Jp6D(u9c%=^SNBT$}TU z!dGPT*F7ObxLpV*vv;Nr=X0ZlmT;+hurx&&i6U&4|Hc0scLC{;!&W~;6^kPwP`-v7 zJW2_HWIAShhtP{PjFajXRpz`wec^P#SV7f|`DnIs)j^A=Td1~M#-hfnnDEQG4Xf&i zYDyFBw<%3ZCf@(<_`FSJL&YUPIN7||&IYw3j_-%)O`1TO+H8AK zy_u6;=DaXqUCcNi&Ki*|WKJQNn-?KN3oZ1eUNqMx|vIS7xStpyces4ibB%YCJu3hN$mZ;(NbaHtj17MtzS=ZFatbh(3*u1yq=kbeA-m`x-~hVzQ`#sq+9qsS{>fV; zVumK`40i$lN1mny{w?9n!4{EmJO~ zWCwX?%q}R+j4pV!@^{=fu?Ww)P6t;O2$8@aYn>MLBPa%-%cR(f4O1j=o)?VvW7Xl&lG zoQE57;G^8(j2zL-@Wbl_?iv>pV)#@r&aLOEf&b=13p_v%Y_)@%R)|6=a z4uUHXB2CAezzy$6$977Grm~0L1w-=X+R(2|d6pBnDysT&5%l=zcvUJ@N4p@i(D&%_ zX&WB$O8D1tcAyGID1R*wOf%mn@?g7pK$M1_(3oW(v{>;(wgtBDvFloZ%X{R+H6;C>4Xr0MojEc_2qP$Qdu$wd~j!7*~9>wWO)3vGC?aLg^Mjx zYkoj-)8#`~yW7{gN~EZcsc-jzKJ@2+t_@3ja!jb0H&~vT8RLFc>DvUuW~58;s)lz8 zVY+e+tcxWPSc9^7L>AEGy;rnnC1tqB4JP3()1q)h!zd(;sxkrF%M0mM17M>(QYuMi z+IP<>{erc~|p^pIP{6xvrtcocRf9KKR;_{`j%ot7t;>c@F}QX4@fF4% z!FwD--^8raqqPxURT{MfnZnn~kWJ2NYIt#Ma4v4gB1=#ar*$deJX6qx-^bp%r=RrZ ziHCiWxujo*!iT7WD$0yLiJ!!c{A|)Kc~dzw&NYlQDqNg-*XwSv%Sqhp&h}b}I02i; zMRuR5{D(t%y6-^B85~*kbQ175i(~;^c4bDUa-b!a*ws0)x1;T}>$6jKWhQZ^0X2^)7xzc7N$mPMm9Npd4LEOhgpuK1d&1HkdPpNEoE)2>5UUf zNu~#jYFxjfYq!DeT9tF=yw)gAv`-8z#tYV*mpf#Gw4h>n_bTE(1%wj2M<@bBdDZXr z>vMVFnK|u*O4d82X&-y7MBzLU{h=M!NkjlwF(b0@Ntpn~dfR(F{X`SHA}q&OVD9-_QHsid~HJOCw{_WL`>gc0Xn1YSU}d z!1>CO9D8wm->pzaW)`Iaq7q~nY3=p#+@n}`0jDeOM?Dru*dL5M9+$_-i3jn0IlG4| zY|QhIVxMTY{z2pX^0Lq{B`p}xHZLl=pEpglxbZ@{R+`pqif4? zWAx8U_NlZ4^<9H2W=_4Q7(a_P?e7jz{+~_s9&Gstwp6T<))Z4dw|e>C7fS! zB)>fz*He3%@iums*%zYZ=3^af=J6W<8dbQd5SIQ8C~GyHZ^{kvC^=vF4d^mBr1WVp z-+{q(9%`YGR7D9rkw?!9Eth*(!4}g?X89EB<`I_5a0xy3Yejh)8qAlSQgF5nVzXB#*zO-Y;|b4k z6eov%%Gv)x1F_h{0~T@cX!D3My zn0ZWr=+n?DPdbrawyP@zDs6urX3q|WM)wV>iboyv-3`?nHIP3sKVQy;K8Z7?|7ynZ z%zNW$eLJ{ml&f|wPSvozJ-(2D01KB^lEOb!3uISu?sNLrRH%>bpwmX}U-tCiE5Wgyl_u^I8RR54Kc zR-vP#dp0FlWquir8N{%+9muMB;(a3CI9p7^-9#HRXguZ3m;xtz^*I(zd%AOi^f%ys zkHe9mdG+}0k>V@5k_xvAN4!LCox&Lm zPi&C*kJ!8DV}`*pkz-SOmW4Gu_^YfHNWv2b4*h|hIK7w+mld8+HjAN)WZhMPr=^Ql?(GJifTSlxUx8@Rh;C zsgh?;-_5kM=lFqyAiyS{PwF1W(21m1x$27dTKkocCq(N958BCWeN-OMMSnITZ{+ihCG=GyfV-|%fNd>mmR zsGBZ2KH^S@{q^)~7#xcX0V2U~1Q6@rE$p*z?#nd@Jad4cG86bC5>i?Gudcd`s>c>I zbyj?zOf`nz0;#OrAsCUq_A>ZNGEl07`+w6I5dQ|?is#<%EHUXcyWdbU20A##F8#R2 zt?Jkqs127}Mp1pEpIB)Un#{Sn8=4>^Nm8X5#J2$O&M3N|em>}%PFER6sW~?AZhsB^ z4v`xG_+c4dNP`<&FF)4Vi>TO7vQAOZsBv-`&T?MW7{6OiD2fh3)5jSK%nI>x6JZ#-QnRPyVAnbT z(PFecn}QoNxy1({4G+^C>NJf--yL%ZbPxB#x!aZpwl7Cj#lhW>xK*V?lh#S@&cVT* zmt~bw?=CmF7#kVeY0n$P8xj(a4^_n#%FFQ{AH+`g$jJ1eV~bePm9MEOl)Is);DW3K zaCURFOp<9v+@_B|!FT1Y?NIY}N@~lbTzpzuDL=n?JJ^+Gqa^1_4?TtY;?$i84G)+C z)7_?lPpunWLeebT(D~MBamhr#x7QSfPtP15$iu?C;H{*EoWql+TwCCu$yW8))*CAXk$jS7?fI>iq?*i>-tG)qjpK@ zwEt=KGY^&e`8IiG{6=ow!aRIb0`>JyPcIZwWOEtODnl zPi`LqVX_RhS$_7Xgw>ai#!d6kq(xBHZiYTo3x?rDK6YKp(q)-VrlvJs zh?)U!T@Sc&Hw|->A?3UgFauzo_tAXp;>9$p{vtX$86GJaT9;h zXiAf|^}gAl^=&056P+#GbRoX$aVa{nnX6>}NkmmieQc(lMFSs~v1HkBlLlOCdR(^% zq0U1m+e_F*7c9Ee(yWMZI4)}yHUE&Zc5F*_{ZY~IaFyC zDafpOSA77A>CvH+oJFRqgnjqnP4Yl*ED`oXFc;*&URg9(&nw<)Dycy^`*K92#VhbN zA}~}FBqlbO-P&XH!6!(ES_Qlq$lJGfB`UmmeVRg`JA>ay7^UWtyvR}Yx%3y_^`W%; z*%s&d$4wJClPF=!$09{=K?+qU)$3=+^Y<6YY!g8&?Db-`?(NMV^rs(MPQ4O%q@8o40`&iFE?>&SN?1v zaA(KQU^nHIbMWA0X0n@~39bv-qAk!EXkaPfr5s?#nRv*p^X_uuinCkI8w;%5O4n)g z@I~g{o5Py>`DRjYo}7v!^6NoMaPoH!t4I(J{B_WArH7mM;1RCZAg}l-Zbsd=@@t@S zFYy?x%JKFZ@{99PVkID4+ogro(eUkZ_}d{O@f+s4ecA~F7BRtosuw2s<4MqD7i}$I zMEK3UniqdlGv@Xf@{4MWc|o1)ygaPd+Git(xT5UyT;i=mx3A>p)?>tysEBPySeGw@ zYjo$mZ3?%SxhB%$6P@acn-0m|8^s$u-6V{*KMg2!cie=J&%Rf(HG{^s1!3q?}SOg*fzPqj%6fbc6dbbMSupr^^(x^TjVfOeuNZ~EwMOL@JHzJjvIBjbny<9y zP8$&G)8O)oN=AnZ+8&K$sx))ifwW}FQJ@DRFd}~7=)JFEugNB3=&?=+BvSHo{cP7@ zUofnf#n{V!JOMo`V_k3Mv!}(to<~8Tzvg&u4$J~!=_Wt8^hJ~{Z!u57!9Si`E-my2 z#*70-Tk@&b`s`OBByn^whw;}Me{ZLI>*Vf@%gW7ySritYGL~$j?^%kDCj^|Fe_;Xt z1_bWSzD^ne(wamH(QH-td~j2Xk_wo9&0P9DgKek&rZ~_S5g3PblP{77ZebtYqW7E3>iNOs*!9**9ZTtRC+NJZA+!Jy+8i+4(ObyQ z!5GsyHBOg5Yw@(*{$TK}Pn2>}=Vh?~HfC5U+$ULYJ+9XMGv&Gg$PdE5eD?A^W6@Q# z%b@vC>TF$fpwg@RdBe>3&l)(-1$@hDj$SZ3x+hpK>w^5H*$x?k5VHi zE80*_gSxF5d&U7{pEe4lzg~%=DF<-zQH#EH><)- z{mGj^@t+ISzq`#TxBg-j`nuqX7`_n+1 z`*P}^w5*w?+cXp)DxR=WPJKReo1-X@qEZ1EOysi4G684gk;FY_IMW>1^UJ?pl2!_I zyg$hRv;eCHE=fNvIzF+k2Keql)&_}!kaMf;e6x~er<#&qXdvA*du-JlzfuAsbf&RT z9xFy4%E4&xb0?P0#8z?M)jyGm%km9}DDy6&NO) zd}Ed8JRjznwW{CbzK2;9JRFQ*J$>Y=E-1tih);1kCFwf?CsGoS^EfR5Lv%))KLR__ ziVi`(5a}B2OQ7Ni7OMVUT{t2z(mnB)2#B3^XzvpZlOE*)PlmB^4d9LB1?me~vG}pDMH&~r!~>G20iM!w)~P3(oH=!>>{KqRzt2*OAgT9zYyt;Xu9f-ULafOX)Hy?eI(eSs{VPkXo?0mu}VUA@W z`Vib@+{0?2j+lZh1|abK!m6}U;aB0nbXTzeW&nX8lThBI45~y%f9Gi`Fe1dKj0hBl zMK}-AH#aB|x(rS(Sbq%Zuz$?;HLt`!dO-{25yyzD0*#J4?Cu1^mdw8ds8as2WYEuV zM25^QyhCm)j(#r&VIYN1Wc3GeN>^tB)GFU4mNL9n>`yOQtE`zwpP1PkWRBu))74dy z>6ZP}Z?t0Z{`H&TGs&%=KP^VDcg<&~F1?rW1)60VImU;o3Xo;1Cofh{eY?H~>&u?4 z#;Y4}?k(!Xf&1w>jiaC`2zM1@N5q3MrgpN2_n{KBR1 zrEU2cZE9H51S8ix$xP~-&^{T$n%W>ay@nVs{lj@Rr2P`_%h@NR>6T8v0bDyyv+U0l zBM}s&RHC>XbhugzL*JkJJg2=~D{)TtK$e8PuBW1VPdMsYfb33gBasC@50O2aibWYo zyG|B&5sll_n$Rb_cUD3Q)Z;8cAW%eEZ>6#|*pLQbeN-mXeA){BQHj*&PQzJW%=A2U z;;SMlYG*903nit*qr(%QKoH?ScByVDfmnDx0<}=>+Qc&G+R=JWLA2ftQA~tJq{+&q zQD!F*0je|-7Tjw{a+b32ub_ka<7$d#MGvNNcB!AXu~bxl**$$Xi@D3mpIGI>+`u?6 zgli{~!l(%5Fa)t5Vi6{<7`Id#T2y)kY99@T9wu!&35CkVyIAJ<7Qb;OawS~NgFEZ% zBxKj3RS8;7SI-?JACxy3J0S;0kw3p45rS2AOcrnr*^Mc1@wGPIw5bwQ0K3%NRDPXC3Dt4^QHq+T29nn zLKE)`wqp{+fAnD5S^qu1FBW`0Yc%JX%6F9g;$+RHVheqc>GQ&sOz*V{GTD_R5d%`j zg6t%^etP;BQEHY(DoW$~0PBfh^0Ll6$~&glu|`jFPXnsIr~d}D*e_Xcez4h3E)TDm z5^8=npXvK1V@fr(|xpF%^oBj#D{_d6G-8oY~Z!w#xa9~lJ5ut9*Pgr5!@@17PYzEIe(s| zReM+yADOFF+%3fvpr`tM3$;^zsdJ_CIo&y@dR}u4LS=>|Jt5!w|C(OiihC7xp~w8E zUYbN^9e$lQ(qZgI-t(HhnjbnGnb+B-b#wnbE+Oi5z4t5Sjw0U%W%-Cy>cjt>K8KuN zUm(6#e09HNQox_7%=}Mq{)7}PH(V_+J6JgXx1E{);kf<(FCHIeG8_67_r!T1P!u#0 zszUdJVXv`%GQK0+2(BU0fSHd=!vR_ah>gg-^DXXBXBa5?yWc6rkEHc}1J0LbFKhxS z5iWI0m9lm2NVSMjTdN`g?S?p!F$Sif_gBX&LbrLmL)IB&IkgdJ zuX;xE)Q*q=R-LR|3my?XwYyYQ;U4pWH;8N%tcHRp!H}3$08y=LOMNMBrNZlT${$Vf zDE*%u#Y%6#W?*xbDA`@dnO1-^8Goj}Z`KssP1xf{YG`b-dFos|I>-MF=utkbSU}(5 zeV8g%#^LE|6PXX&G_y3 zyvEF}P#m7-^j@XfE1FrFfq19TgZ@(rT6k^SM!lEse!B(RhpqgVAFLb?s7J|CK@^7o z7w_+)l!lw9-+r~K^~&Sf<@YkUvM_@V89>fsISotOaRH|&>zv?ehv`>&K(G20o z5AYB@AD=skf75X_HCEAgOQJ|{H;p4@Qx;FsJPVIs`l`w+MRmtL`(YZ{m-3kTk38l!@w~$dr#PiC@qhQ4W4ckRC^S&Or8;} zA+v}(C|O@HKO=!>eM2>WlWDPAT`+8eN%|5QA{@dbe3xcYzrw82lTs zMNmDbifS@GmBI~)xm%c-v2~L+vR-|VMfgcScH+Xmwr5w^=s>|iWWm#LuzC>MnPLET z<||&vbhhE3CSoN+kH{z^_nf}llGLfAM&f@L6_EDzBY%= zJ_FL>bh~*WOttX#mS*N{G7c=XPg3M8#_B5pPmiQ3fPw?k=3!6vw4ja3weAIJv zL28zd^;KHBf58*dH-^!AvVu&oq#PLp4<~cta#I;3Rrc+D-}}UhUX$KFA{)muk*AXr zllxV`Ik92HtmOCwG|0i&XDCNBu(uLj+&n~7l#t3K8?tYmq-6yHXujmt-qBXu;Zu{j zWc(Pt*Q%78KYV(2!LtVs>xGxwKej$%d;(Y->jMQ2eSb3CfZ*u}|JvXJ3uS>YkHz2@ z{VZvh7)n@-{3a{Pg~-e3%lx$zJc>AXzKW#x-@CJYmYX)k<64Ce>lNtDAV3%`A6Taz zANTFe>2-9`AU>So^4%#7^G53y<}A%os#vhoW^dF@Fo&juQbg{e$#>LePc`CD>3*aE zoe&COgRf(yTQo3^3$8JEoZKX1iFw2wdcgI zEjd?~f=!~Mqq^SzDEI83*FLtfgRw(D&ejSi>l$XvY8xkg#OpKRwp!62(#_vy3$xU( z?hw;&=7>Jc8swJx1g%@N546%O;Mv-lPVaDkCUp6Qz{i%E{24PuZSDDzkduJbvSQ zC^nvsh9i>r-AQ3=gP%B;=~O}z#Ms1yt7Ig$yUc6Jn7&D1WE&GkSbDG(dIrJ#J}OMA zKk8eakxV4+X0+0r2Bo;aXQC-QO7H0UV#MTcfFjVJ?RI!olH$GZludnq41E+y zYzi;+7RWn?wPCpL*?0RifdfhmDGJ|O;jMR{7;zcnpNsh_TNhWAk$tIXfY@}Wh zcOa9W^vBhkGxm5Y1Q|nx%+5Als+qPN{G^qhi9YRGOr(@ay*_K3*4~%xbVq;UGUsmH zN+v)5_~|H_>>eo5f_%-u1`#R)-?9b?y!&P-a`>r?$DQi(y+YY+k-TGURootKfcbL~ z-w_*wu*~S(CwVS`u$b$UWb@1y>%$z0Ztw^7L>CZe4x^*P3Z-2%PAHS?HXWOPsX+(n zEk`M1^ZSRzhrEjt7u~coBFfv{`uf*f4S`lM=6X4X0(PpXBi0H?|#68&ZjII}deUEBxBcN5}x zQp8iGgijP$N*AX1$bm?66;|Bkq{ea$HH^-}Voax{bGUFhpHY9WVkK1kF0NbRxzRk4 z?^(Z;{Rr6-{wZoq;Qcpa30npueAvh3LX+5@T9f=ruibHO@XMJ5#ADtkof?e! z9|YB+xrk8Y3VZq{GPb^W%U7V7afwEtW6gq7Jr)g@o8M|RE4vhvt8(gA;(t%LSw0dc zpx3Mz^42&{V&`{9xe3+D^@BoRx5HQID*ss44N5uE!sKqTYKeCFeD z7vJL4xdMp-@eiGDJXYqwv{i#fp(l19T~noQJt{~qWM~WukmRMP2BA%H0lhBR>r_t! z7}4bKLIkKFv$iN6gHNgRG$oKNFju?bJlC{&$zSw2z{%o0KL~BT-{g6azi$3dPH~;h z#&33lq;He=12iQ*TOV>OylG6Haqr2PSF8N|#o`)=B5 z+JunS7exxp>oAo}sqDx-`qsfr5(;YtEqs?GVKQa3A2XTNyks2@NUZkQwz06nkP>BN zxF#?L_x=$RGUu;{6I4AW)~W=j1_HIl&`z0GkECg5vdZe3-_=w-ou>Jiwoz02f3f#f zQEheKw@87aEm|m2TtY(&1&SBfV!;W;9RdWmwzxY&W7tVzf>nsk!Bh=uVXJU+1i{p!wn z-fjE2%Sf8&Vty@baPb5x=R1MXY55P%>Lwx4{?nAiD?g19k18^V^>u2-k1;TCSlba~ zhIk*cDWUkIF!~<(I`b$>#T@)t`k4pE+UGl+4fadST&~fqc!FTSvp$Q?5f!-wNZq1$ zr}>(3pj}8H$;a5EE38N%_Z-he@>1`ZcKi~CfbR~RN93{DqraCY(}^Gn*W7-pr|Y~% zL;A`;;RKq?En)~^({x|~&7QrAhPTdb#zdr3Yty;{TFuoi+TkL2)uX#Ea-7_E>-Dee zN#Hp2%NE|=Dsv7I!Nd8ERh^lf5Mv8__C9k&Pj%(P(712G%A9R7%Hzf2Q zBE%86Za&V-XEw}@Plu8b>7Mnu=WoiOlt;$m@BlvePbOc3lYCjjndC=fQ) z_HU~#_dQWv`%zs8`#D=M}a=mNKY z-JdCc&9&WMQ_n>T{*QB_@y{1Z>$?Gca8Ru3386jh7R5~j1s zGbZJ#wMxust9#jhDP}TT+>;n*MDp#5!qID+KEUojg?gNT`sEU9?!EV{%i`-exWYUAK(A z8Gw`!nWMr?N^s{>vu78HXE)4aZ#oF5e+JfI9X1lDDM3> z1!a}8khlLGSI*V{pM`~*?a(WG9p{$EFT*DAKNgF^Sfp~eYW2CvH&&RSb0ak2yE`Il3jg2-c(Ds7O4za<_uQp0wbgBK)b51@7*2gNeECxH7 z+gU(BPNA$(ol#&6QK;M4N%Y z%fE5H`UP(W+k#7F$8P^5e4Gk@YA)*>rx9{)^P{Q3S)tTXE>d(gguqC2odBX{QjKu3 zX+Dv(0_XF77~f;_3w)6F4SUuR-?)*;VkfXZ=IyUb%Y!na6JUi~{K;o?v~0zHALipV zjU`uK;_11F<=2M=E94;*ijO)lNs4-b^P-QA<4I1JPE58N8r(()fpJLP{CS2$9eP2z zvZM!WGD&V=cFJ&;I-!eahJGRZJ!kS)^pUAgY$Ml?fC`ks&C% z(f0uUYSCf+!MMZiVeVlgBJtj;xafMh0;E$&l>vEOg(wa9jQupBB`NB_or{k#Yw1Q}p}q;Af%DEGIW8+*r7Zg2oOyXe}7Saoi2`&U$;_XeE#)Ejvf zeX3N(e_q9$-zRepL2tK}$(mHBi@8L#x4MfeAG5B^Eox1;bj8o7-Ro11LL=U_;zpMt zW}M$YX8Y!38f6dMH=EMy3EEIu2^Oyy_A0N5VpJ_fxoiZoYzSgzMf@(nmt1Bp#9nhT z)*jpGf{dS~)IL?Ery8JV@cqm1}H zj~U>4s=WS5^mrXW3aWN4pB|28QZb7w76~fMS&q5&Zds~v&(7pY+ZxoyCT&P?K#_Ja z8)VuPaCGe;P7f?fbjHO#;fW}3g)SMXwt)m!Ht6-|fait8W5eSrPYMK{7bG=aBfn~u zmA-IE1!`2wN5C3wRJE2ieP)|mF>G_WQvmK4|?&5cx$C(Z{GnF zKD^^~8jj|5woN)pQ)?`jWp^VRRKXsQBKj<$t|;{&DxiCmzl$i)g|?A>2UvP8d_+KI zN0M~NK{DEAJ1@?o31*BPypQ$6_;4xE)6ZduE&%)a6Ft+SFA+CcVLej@ZrgfaTCHX`oRKFWCRg4CjavE86NAr59WlM7BI$0~&zpxeygKgj@T+I6F zxBa*TL118I1VBe{|C5GQe|tE#Laq04Z3R>j)4qEXI6 zGRzL4NP(Z8Q~_KD&&x*aP~B~&--2K69|3s)(Cd~ETu70Pl8?8O%dLRkuUtmlL0O#ndK+{tz#f|FobTa3Oc|3d3ZsHyjY62`p_}=t1gAp!Mz8SO^k2d`zD=z9D*IWWN*S%U zDk_Z=929>)M^x77IS>rSv|;27)8<((yXHS^dh8YE~TAE)3%M zniY>^o$0M55VNv#GMr|4Rnz4z$03lo_Y0@&^_Yb^K*A3YXu)Mpx!jNd3=F=-$(i8IEJ~!hreou`X#>Ix34W=mSINEQi-JCAiSTY^b z7XR_G$^AEFel-5HboWl}ZU79>QwBL_1y||guY^Uv7JU30^?wII$JQwminitqJ$G6l z1E5+ifvMzle7Qq<_9x5`#;uuyvZJED_ zGGagcu9QyV&n4#;Z`Wh-44EdTdP+9Gq;?8X@sr8VVjN-+e?J{adv3cz8wZz|fLgF$ zd>_9$tQv{i8E+qi^^MpNn)aA%Mu`(H=Pu~=8Bt}EGmI6hEb!}2-aW`C-vfF8jA;BI z%z(8(v`(&WBo$gG$3Z+v#APkj*x6SEQ#eqPo4y2OMtIojqWsG#7Xz)9B-ZSCk z)E-nuv<{+_L1Wa9y#KAY0j3l4S=FrNHFD)r&l1QmZTJN>M(%lEJP| z`TPA5DDmkin0Yyt9E0zdW?y@pK{gNRi zE25f5%iwwOm*we8syjbH#ACj9!`q+kJ3WjdyRauwPo+}3?7g=>t+ZmdQSx(hH?YI^ z>lpad#A3Qflh`l&@v0^l9&uoiO+Z~7Q3AA|VzLKaI~60U+6~L{7dvuXW_?el3zy`` zA46OQk?HzA6XIZ?5b(BuyJhmMPnX)z1k-*!h_WMWgYDKZ$Bx z`p%$14|FFoBsC%*$+=4GJnCb zcglZz^!!nd&kmQ(CY4TKBIkybX4vNwafIs1G59Nfn&mPjVEw`Gn<|x$q2Q9G#Gt}w z#&c=8%;|74Hc78TiTxN%33dOw1oUD|23VC4^sd)+S;01w^T~*ueS9NrA3H~QcC{Pj z4(`&DE;&TFG*?c!e{!1QU8|o+RkYS(!kNG{YS8yhrj7$d`Y|>as+?ExV{zivP}Id& zW^5|2^Nbn!`N``_bT@nPjR%%+ySGh(+BJEtz_peweuwi7J?ZUi{h2eJIv`f0(oV1ww<%S-uWSq?Y5eo z{29`TEmXbuB864EYo9W2eKmDjI~B4fKeTC_}FTU^zhl1Qzx65Kgl2DWyRJ-NzmHlZ0A zQ<5YP6D)p_-G5w0!ogM};YgKOom^7$lhwfxnzduOXA7c`2^wYHG`fiXuK#{7v81{4 zlktf}fg>209&1ABvVV*N^QcW~G*9s+?fHr+qFeAlQ^;|6=7C{KNVIqB}j^ zWD_|%b@6A~qq3<81}SBSuwTc82#Aw{y`KMzHDR#4`2hkxGd2=ilisJJX*Vm}qW>KH z)_S0MH6!LqBl_9vqI~f0*Zzmp6Yr0@>4d34M`$Kl_wEy$KcWGR#)A1}`v?*)GcY;W0ZEi_0I#|0b zLV8cNjwMLd99^_sUM5Vxj+3`#pmyc}+c`LO(Q=qjJSMW|%+gO7V=*Hav_}VH$BkcJ z>jMqSZ#~5|tv~kq$32r5M`%=7Ud^*^+GO1M#_RPIh0$#PxG>vThEBClAHRs?q3- z9F#X^YJSgvUSV-bq_0ILl}u4MR=`+?7Z+nA{c@T`oq-DZ!O*Q5S?t+S)il~_CX_`` z@|%;Ghx}~zz(ACqtO@kf_N}pQ(e@kvDCMCeBwz7Vi-GdWAJz0%D$?J8TFzI@y;+gO z{MpG3p4dW>HAKMJ5mC*CHRur~Cyozt236 zz;Qi6@v(@>wtRJ%1+({Aag2xzF_g?2H&swz;qJz#maXEGG$}2?oQSCwHH`uP zPt|K)aA8YRP>zB8o{8cx6qz9#qlYP_0sk-L&j)Wos0Wo zBxo{ocV-mm#4Tb(H@HYELRF_E>NxcUZue;{R;cU9!}>%`UauUsm0*Hv?SG^kAx&>T zu+K2(Z1FPEz+Lg10NqT6-aTg~6r!Re600!GitR>#zaB|L9h9d_15E~D7++ZISuZhc zpHlJi{H$6f%ZHSL(8W)q4Yhon%xy-QG4VU{7lL(4dzYY2YBUYE5sBmVnqB80_9T4v%kT)! zmZ3N&ILwdQY^4wo2skpZCOr!hZ#nuEfcEx0Yju)Lm8+u}qnB?v{;Y^t;rqz<=w`_*epw6DzoClIf)^IC7xtxwQ>b_h53VfeiIZwG0WsLN z=k-93R9x50gB3=b?3&ndD?7=BN`Bx}wGKJW6T9Lew+)iFi&;>+W}YV*f5~#60_0p0 zG6ZS2kBAK7C%jwu2Y+Xssr-v2JfrO=S??%rn?8G@815Eb#&OWk4wijhr0gxWE&K`0 z?AuE-Ky}6@^co_%4W|A_Z*0k{W56?2DDJ9mF=i_P6B>#NBBW)9Qb$vd|Ey5Yd7>_1)ZjLuAN>zVo@27mvX(LwO96pC-u>j#T+(I zAzb43P8sfXwhD=`QkDk5=B*@!Acn3*@`uV}BM+k2Vx3_{ZigcM(gofx3(_$RZ#m@95bE88%MEk?U_NkIi$A4e}PX z0$dH#gLqXsAAHwv;y+(w-WUwb9x&3dY@WNgBZ&MFb^xxx4cE|d$ zmBB;rb)A<0og8l~-^Ei+BcbE9s-zVVgl77E(ueHaVY?b@H|h<8cuh|K*zpe>HX8tsw_B4at%g)=#TiAWC&THC$J%mkxdZ~qE1&21bFV37h>g~(rl-Hn!=Wk1?^`ig_xF&ueRWS8n6JA=9< zRNGktSj9G@DoLL3lgo2Wiv7sxptos8Og@xXzs*{&MQ!w~1%KE7N@l#YfIq@`JNec+ zU*XAZaCco#IHY}dVTVwQFSlrzY!ZtC}m+iaC{}|o~D02_?Jae43~j; z+`|5c`H|4h!qZUZ-;fB~!G$MEm$I63Y?a>ng{f8=euT+%)6AqW;qH2ZZ+dF?%4_8v zY`wd%v4kyZ`Zl|5?)(`avV1cjT3w0NVm~ZmV26Xp&jAa{K36nfl{8`2uI?`oblarn zN>=&THAei1as1Py4yiy--D1QuD}8>y1sl15|$Kx%7XzCnJ#L-(q5Nqh!ge@BI1X^L6ynb$fpr zBaQG&h@5{qRM|Mufh{qk!ZhlG+md$~FaGQAzw!c`N$(dW<~N;Q(zc0v0>U{!+m$Df zx4SA;nUIPpFX!e->!l2h>p@>UL2+pbmD8%V-7)Z|$db}t=hx2doT`5?tBj%IL8nwJ zX3yc50V!8#Q;1K&%Bru%eyqllv)xwm4sT0=omsL^#;X?a#8GAnF%h@OA(k6c#UHYn zatXV9#UymsMYjD(kx%M^=}OBxtS7~Y9W^ch>?(leQOT>nKy9hv;yCpTGldV2zfueDqW1b$`B zn7!?K7D?=geBpwU&0-F0F4RrVmEp>HZ*v8Li%#?P2{r2GUo7c=s5#ROz&b2G+bzGf zE-X6c^(Rh1b+E*aoU_+v3SG3lv;vj@r@=2&x~F`2cqhDd%ojozx0X_au{dhlRzU)h zGOAgU6_TF*7c@ry1UQ%nZ(k7uo&ADVK(CD|b4Xpzpze?k3IxTHYWM8FSS~+s6Yi>J zt>tT;_Bmk8Oar4lMzkjgMA^j=ON=>Ry1)iN`3LJ}*U(oEDZ-|3mY_6iD^vpEX{C=s z$*)7U#l7$G*eU)Jvh_b14}VZIEq3Aj_-d?rlO?Zv242{s44!|#eXIZWip|I{Iz%9E zd7D!Vg4bxpp}oq??v>iu-B7her9TQb8F+q)5#_(R+E)~L^v4?E%l7l7s71E;VPeJZ z70C*=8@J*PT!ZMgX?ryr9(oOtG=U|R-@{x%RrhS#P>)TuXsaxLgil+%12RhWv8Z8qbqgT{i^gkd9nPH-|A7jN;p-S}5RVtvw%rAGd zX5YtO^q!ZbvRq5?si#r7t6qjU_7DMwr(7scUB%9&1byiPY^J7*$%;Pjb_^tz3f}a# zrh-U^RvfJLw@M23}~6`b(pkv zC)@>Lvp}e>9DbJ%U>mUtS$nfIndD1S2>37y?D}1k@hE@rJFIbD;`_-$=OMaN*P3bb zM8s6mY^Q7iBkHQmxrLVk5(Yg+g|*`N+$p}%z$-ReC-hP!4EP?K6crD9yCu((KmT-V zuL`LbVI4;3(1MD3s!w&vxRx6O%W1iCK*==8xF3jO%)x9~k}QD5y=T4uRSN9!ZjxwkIlz*kyk5?@41Ji&EobeU7O|Hee?h z51k0x`VNbjn$>V(H+=E_?Kd46dxmoL_RT!+WH3f;Y*b$OW(g4UmeFfgacZFLnryhu zf-?x+k%s-9sQ?=m!BWLIw%BKsjq)>7BFS%ztt_1LYfY?)aM z(59~y`q6~*oUN(I>WJa)P+zx|v>fvq-3+2No@x+)N*iepZ2wFk{9*P<@rXF3a_8Kx z$)t3VmL`({KH~RCivF@kb?ZnzAACFG^={~M(Qt!e0IoSRdmeI9Y=5OOjnRST9%a+C z_vD0TI3QZ+ZIc1y13B$zY|1SM4@(!r*NOsFkwg9EdWROIUZ+j?+k1hT<{flihdopN zn)XpF#1ORgN%*N0z*I&>D%p(bi!`)%EF@(^U%xo7*r>coinu}1 zVlsmk3!$p2Pr3XZF)9AfVUN_&mt0ZJHotQT#4rd|%obFc7Eo1iTv8xV{v2{ap-l}d z;`S;3w3py06l?q{X-SK3l)5Hj0eJjpsIulX*{n!?P=TUYbb}sDXu$@V^FeD7X!RfG zR164sWo$`!#HmD*qK*ZYM6{JpM2{rB*nG~zeCzVSm-mFLrNP8Hr%Y18Gg^~kK2l+7 zXvVg%uqy>TrTU6-<%>|u1?v~FBl`F959OmN7z(ZDk2-~#ULQ?&KjLLdM#!mRPnY&UE(eU(w{OLTLwzw0(2- z24p*ls!Qe+S{mKt{b&ldkRqhzoxG3uBp>%DZ0jZDNh#DU&ShccpF<_Hc)hvdHjU}! z2*z9vdh!XAhm2>`XGh0nbET%bi}DDHJIdKD=DWcXFc(ogH7gE6G%`m<0%xjIDh91= z691-@H0L#nxafOYaWkg92r6eK@yw`FU%}WA)mm1KK!C~N4b;H0P?l5Xtq96Ftwd$C z341HRIXZsOOTkfF*^qtuN-U7|k3lQh2VY~HRS3oN_dE7P*U2F6tTFT;XR|XTy}nCM zp;o7U4ySI_SD#)eMPi6-=tTs5G%W!WF9#9N_;&E3FKGSb#%pNE;VBgBIG>cNoLf^1 z2U)Vut2(&(dVEK&uPCDTRo%h4?OhBttlb_^sBdHtK?YZddc9EY_=I%dKso z5wJ+HHQWW7(5cPgP--HYpMI}qjf^o@g!F>*O25S+W^$N7V>gv${x zRDhhzLB1@KqN*@QMEuBmflSWhh46usCO@IX;iPs*!@15;nWs|JQJEYC49>j{DI}1T z^U^e7NOO%E%USW@-F@(5M-uVnuzh_lqL)^{dd*@zX6c+}J?1dQ(MJuG0k`XQS~T>u zuBLQzw2az5icoH*hD4;cEBPR&3=Q~>->bwdhoOWMmSp4@w{=G&3k`qiyq3fRj<;Fd zhN|%Zl?eJ2#n4T6<(!j6hDh%*`o6^tXeVIjr9TeV?z{u#1YaEyS($G8if81xQAg&4@~eK~*?IvS}#Pl*{5N)6kv{_Zbm8JCTLAcar&isnD{QPqsqdsVv~v+{2)#~ zyhg?`0vea^r&~XBE)7lZrO^_crsz;{5Heq>{~i`^)xgtNtA&LV&_ap(*WuTH;@J=j zHet!4j+hm*_O6JWgvV~Y>mX=-2mS5Vm~ons08`s=x;wfCGDkRsCvv!QSczaEjG80- zE7Wgl%TKayep0sXdOe%+LbCsSWyFDPy=wV$f6cYwzHeNV`wsfMTKrY)Q(6}AP@m@E z-d1O&wGf#Fq^w+CfBalRJ~%7kVHU!4d>~fxLOpJzZsSD7Xm_d$*U68`F*MmTR&?mu z2S-^GzZubSbR&CQ*7=57bIf_pXln+TKUwq{rZ+w>|8~0gRQ8+W3gJkA220rxsQwwO zYA>>&fVIIfjicnb4Qg|i0w3olS}JC_RWPk4MV0Ra54|S{d&9Gwz3i*!Wj}78$cqUp z6L*CYOAwhbKa(jR8VVbas+Tswu%Yi(5E}hIx!e_{4_CA~VpIN1rWqw+UBJ>P+E@MI zGZyONw$VRhCi|LetCFNSo)4C%f?0Px86V90fV{8L__!1zUzj>4oy;2 zRmiyv>h``_tu2sNIbv{Rbc`|9;zul{Z(*AJNnM*=9EDp@;~~;v_pNS*=f0J2EJd!l zf8>olP0}wSISwK)(5iXm`Y%>hpr)PqzgTq-Pcuh-whw-f4cNGpQBlB1vYclEy54{I zrhMagxK!1M2N%gpu1!weE&UjMxnGl%ejQ^F;_~%v^6}LF7OpS3qTpk8MFHZu6pP znJJpd0`FYLCAstR3a#fIyTCObw#CQpoTK0lkP_+x5eo1Vsvi8JkWP_k<#lB6AQ7Xy zmyh>syd`DWnN!Qa(Y}iojMp7GM0{$xlneY9%l->hYo8@k7KxJLFm9P-QFb+FNyQtj zud=xp;J=59Z$}`}{Hi;As@^xC>@aHQa9*zHijImsII<*_5#e6+JNY4aGlAM^s&)l& zTrk$IqdTv%J6Y+KBUw16$Ef^D4DI;QC84$sk&>O1OXbq{8Sh^#`7a-auGaou{{H&J z5R{bl;@Q5zs&85FSkXUGU3prn^#{bKThf;r+lA(zHH3PPKhK*Z0iX;5i~{;MvRPMI>QgGvl#YHnQboW7DWjPsE&0XE?2!A zSryDlsM^9gMAw2;vT>=!F_wOocUG_&F1L8nrio;z7k0KswacijWvPnJXjpM+==iqQBr4{jBN-Z4gm?fln4mj(DVvBTT$8J{)(RcXj*K z7Z3Z;ZPMK`;p`4m>xI7JrK~+>cHcUZa75}qACLdeNFuL`vh_rZ;iDM2kKD;G@C0C6 z8axr8IT6s?tn92a`c`#4zX=W80Jb^?hcT{U&<*xR_V#qX?HwOR7yQB@()(K!8AAnk z3JO)Y8h4zl=d0fRYo&5Yg2zoB$Gt2%rNcf)6d2lYt6xIa1ez}cUISmJw(T-H86T@!k%!rH0gqpo7nBj zyvT6c(k6q$^7$D4I8t9Zyzx^uowFAY$FAiAX*&t`pS~ke@9-$(*|`!oqt6^454e#3 z(Mni3?zr<>*@A_nBbr!sQJEGSK){3Lh8ynW0U;y9LvEh;1%h4e@Wfjwwdm{4-hZQ9 z$)>n)w*@!vy|UGbGQJvcFyApvHLg}o=9vNRyg)Rdin(`GxuUtEC9kin(aAqvCzgrK z^-1XpgTjg>qcUBHGHhc~Plmfwui8g=;gWh$*DT#Ddnb zN}e++$&?p&ldR?W5cS-rTt*H;pEaG$Z`|K7d5Wd`q?>B(+7pNphD9k zRtcvGhq~sbvv*^~j7v>_j)n!dzYX(A9Ws3%bfnBzllszS=)JL5Xtd@~s}tVNY~lFp z+91mR%lX~N&+l?_{>21u!xARHh6jk4{M_0(=UAyBpZ?Z`$wy^(6|?_dE7E+qU`gx# zHq$@xCA+0>F7@08g%AB`hR%Y~@OOB(E&}*l07tUn4>DXg89()Qe9}9ARJ|Tg+Mt6z z$v3<@r+#N*-gGTY>D!1}s;h#uJKFSaxLRdHeK$JG{E#x4Y^gXomR5L4fp$6KuQ2hh z2+u-+)NyfO@%o8irAWlgsuH>3D*A)>*FRO03=O#fHe4Ck>@%8E*!MZIyNgNMj`HxD zXjqCeTeHv&_3*t%5-Wl(M`jq#k-wR6or8ncF6A=Hq^QwQjU#f8s-s44KKx!zHg>-j z>Kg2)bZ%hc6!uGS^^bH+q;Rw}v4)NlVt-Iu`UG(?0%yBvcuz0ooQG!5zaqZThlREaZ@8~KsZZFQ{OVBXS8b@q6hX6zUe!4e`IIcn6Z9%+7(GRVr^oM9!58hU*~YvR{Rm$6MvUK(A?1hFqZH* zphjtAMso?QOQlQrm1Gk;G|c3z2?3gA|B|vBZMBbytqh_1wXrqYLPEsd#Eu-CntIg` z_p&Bxj+e_e0jfR>WT#AyeLw^s|4+7_%JQF^1#Iy!=eTo5J?4IRszX9%_{Pq=#*k5E zPb&e^aBf5olGgLMu@#V73mV=&qxYi6%bIWV-Rl& zBx$PB9MaXh9XXoYKd~_EYyV~GK!fXr$Y(s<^EDC4n`kCN)n^VrU;#)Y9pGnbZRV@m z3C*|zjm501octBT{DVI)=&+Gi$5Qy$B9-~Tgc-{2^uq&6Mp4OOjlw<9)yarkYFLu+ z)Glq+<`Ow;flG-R$Jue!zO-6`Ph}$e=UucI_zvs>4Bz7%Tv#yK4gsr3$T|GOSzz40 ztlubbPcjCM^i3LH9J;zFPlt9v`260PoP9LSaVqNPH*|Q!Q`2KV#cjoO)D=t-yl!XR zM6*^}180>%ol2M=;H=Z^XuQTTbR-vLj^n!`4%|kgWlb-~A>pMMe>n_h$&&;>*NI2@ zkdQ6h!HW24qNv}q*r0!$9QAwke1}}#{Q;aAe1?D@PZM%p4fRp4Nm_o*c0oE-=zq9X zP6$TF%!SnmEV_tkkF)afJ?Uht#R^OcaMIX=cEZqbMK~GJ`vwMf#P^dk$=wD=Lxrkl z+n$VwDGITCFp#^upD6{-XH^Q{r$@zJk*Yy8PKhBVr*mZ)JufFRV>e>0i`v?#shS>%TZ@`l09U}& zgdY-Tj&HN*{-i5X01;_ytm*35uIo};KMdd5pv@w8hK9C8h*R>ZYPNkR#1gvBW~({8 zJA7#~gv{)_57m5uX=S}V94n1TcWN#9StY66q>KpEgxRgVr00KkZ*WWW8+QLK?w*E_WVpGUgMt&ayGvk9&uDk# zUqPbTDK`Xlf6Db$BEgOFpgjFSs`&EZ;~TYjo*7Bbci}Tx<-|ya#?4Dr%K?TZk{&l6Qm!W4wws)m z5r^N}Mab8tLWV6Z5loLPTwL+OUS}BofXgXB(zemiat!KQ{OyX>xs=y*0?K{HRb~*i z-_p+Az>ysGdmNxB^U>jH756Nw4f=}BQq6nM664bN ziB`>MB*qV~@v&!k057Gt(+0G}qHOTZYYxJMebb*;`i0Pqk7=WdCNLgr{E3pAhQsm> z2$Do;)3~^HXSCGjkBaFJ!DruHQ8yXyF;B})jZSym;Fbv|?M0DxBH7U%qMdB_s8@@Q zt&`?W>wCa^EG|2e%8*B>{DK^2-%phxxqg*KrYD+iEvIG0akI&p4#WL??ei3BQh)aG z^RB#A5e+fTZy)HVbV6}ZJT@kqDFo*4-h{p~OlX)Fs}L%yf0e`-50zat0s!_@sdFaP zZQZ>pz-N+tR4|E6bV#~~{WF;@kasL~jE=@i$#RzC5~DO#%PQ4nOHzv1{_1iq;+V~4 zqHmnKZe-`rtw|8v;@e2~4F7nZ*o~4=SwKR5ntR64ty-V~gVqABd0{4_470cX%z^=U zW>{rPe%QSil1U9aP6b~ud(_(Xv#lSn12^b78$hw}LX>6q`89hK?*!p9`C5qbI4i05yj)a21LX@eQp=SwI zCU0xkaWRtclzZq?zgDnzq-=~E(GoWwvdXr~Q{J?)5Bg=r=iBxzUo7N0drV}UB1r^G z!LH$;HGWacaSdwTnAKJ!-i~_qsHm3*c$I~zVx}F>H`-jh3I6r6Vcc=8)bo2o-FwRu zA(kN|&qHP)XjZ9mQoXp5MsTHpDZkmoTG>UA7Eib=dbcp|J^Ja9(V4-^8sS66KtZIt z{^p=6D6VLGR-d*>SU6vupt-c}R(A#cd3?z(**k?ek^4#;gqFR5pTjiu3Dz_LMVjx> zj-Mo6kO??PJ)XN4(c6_43K~{JS41%+155w7O7aXC*T;+^W64AwLkEGz%I$Ihw`jS# zq;u`^sf!9C8d_DGc|egMdlMv{^2~3=V6ZCjU6C^L@~@i|qK}a#t)D1i7Jje$nZ7i> zYC$H9XNLV6BoV>t*B^gYslT#u<7!aqLfBjn7Yo$S!edxe+%EFxr5DbeLPDBcIQ}+< z6$$Pe4xLu^u~%Ih39qoZPqEH0vE8N%pHD7m$$RE25gTm<1>oPsdKy~~LRYgnih271 ze(hC-d(>QC(|FQx7Fpm*F~WqT3J11xUmcewwIxh}1=FI7$PkpW$)u|}kmYV{Sb?V5De zZn>yxITts}g7DRB2>#QYk*!-uj$CxK;X@|v_$L(&Nj>|P>J7We+50CB?8ESf8@t3U zhJVxvBxS_^3Kf^;FI3+J2R$hC$5R8~K$G1`HT$)OPSm=;lg1Gz_2C8UZfms7-ZEf! z#d!m_3`Z=evTZn5PK~@%hdkHg2|$XPl1Wxn zCQGQdL0ZBWUF#fr6*BBYBz_7;!S;yrdV@17-q`|#znRqKhueyI(4-+I2UZnx!Cg!{E+rnQZig%WoyRQPb1vwU)z%1|a=7iU0v# zvE5NMY>K?vO@G*y$RpKg(J1rNoL-luLTkEBf!XK&Ws!NMxB`!500#X5hod=DtXKz_ zMhr~&~#nW5NDIuJNOvTTo zsWgcSi>t3+yw12=%3%KwXX@N7`zKs!m(^hLviy85B=3&Yf{USdkRxNfb*B^C$K)v8qwEq6vA9^K3U zdo1T|VeO3CVU&u;#|FdAf;H>u7b27sM>t{@EIT2qR0qdbxLS-a52zveky^jChUJ`8 z|2fQVqOovScS@iK7VPj30SH$M=;@iy4u^ycbjZEEGhm>~b7TH{fSm9Mmu~dZJ0JXO zH4q0lC1v6bH{u3b|87QWe8}=6`~yei^HM7PdN_1weBIG_5*epv{pVu$Dq2orb>|O< z5~Eka@ls(G$HB+4sdEl73ru_aBfF^vKa&;9UntwYss-rKuGS`An`*?y3xC1I7Tp!igq zNFQB&%$`p#4FsrYA5)8d4fXx93rwbZ>?1m&%1y+e`Yua*Ca*f(B*~&$(F(n#?*00N zT2eH zMmhKU>)*t6r_q0+5_gqOr|q}Ew=Xg+M;Kj@){MbfpN%8W!!+TK0%mgqQgusg?vRpG zjr~%{*pPuptiv=v@i2fJf&DS!2UQ*tm?V)}?|)e3vVAgAo!GKQ`(?X(gCAf_aorYG zjUL7nDKD+ozl&p!i=)hlYPduO);W%R$;YV0M7x>4cb0v@P+m$a>kUdH&!s&j5XS`h zShjP)!4=NY|Bbo7ifXG3|8`N@LMc!vUMv(XPH`<#oZtb11Szh;UE1RA?v~&|0t9y} zP@s5m2(HDUrR}@&|Gu^M!8+J`kFieHac1V2BhO5p%=^B7*M&-7S!#N6Vwm1$l+4tj zr}y2=myL_%qg|vbZ6yKslzOem!;-;UCj-(>1+iAV$lA;j4VQS{^T*^;wIq@pzT(&ON5REcCXhgf{qN}HKslhb&2H6GrR|S3->36PfJi25o4d$&Th&>I~5gI~~ ztq%-vi)Ywjy=+Vfe!EWFoFp2TI??8s7jupDoX{v0wdw0Ii!OH8AFJTSb4Ia~8bKO6 zMcD-jR|QiZQR=k1%aV4c^KY}uiMy}}=3Hww8&#!Q&z5e;Wo+@Xa5$|~Ryn)@5aS!Z ztGxWtGZ4E{Y-OhNnZ*!nr-=zx?9u4gIuIH;XCDCVE=+oUz=Y*tqVRH?r|lE3<`_fz zEr~XzZM$@YNWWEL)ysbWR##qAURAjQ)@&!)uS!8yp>nb#KCQXYiE0$fNBvu`CAB)g+8F}6RHM}~KEqq8k{bdAgAY$)*`kF8(z&iN^Yb*z>q>b6tmQ9l5#Ok0jKXMhn;2Xq9u$~bEkKA^S^IDz{7(!Uxz`(wK>mTU(~h?+yyh16-I(BFr5q{WQ7k(5 zh>((?io<~XjLojgwsd39c8~K!!t1LKhZ%{yWsoo8QT>}S(X5#}^~Ryh3zDJukGb)# z>9nc-AMWjUM~XTshD_&u z5AR4rJx9aHow;hs=~qU*c0&$%#8AAm>18xd@q)a#%HH>u#bPixt>YzSD4aDY2!#Kk zI=D4`b-hAKu$*Bnd5!&gwO!d!`C0?AkVG}Dz?v1YuEH;l9nJsji;+0L);>N z!=Uo5*Y?RRFS!adgFbo$>4bEh7jxf#(y%%*OY`n9i5OUY`rcq2ikb*!-0Ly+Vb=zO zbQ^vqO0$bmS7$afmT8=TbWr`jw_ldwPF@d1T>oLkRA|Lq^*;PhdkXQtUn}_EN7@tSf?NOhYyanw z#UlUjI*sEV?DP(&Fx{qiBeC{XH$Vu^FaoU!d3jM~S$B|{Z*Xf2LYJP51( zxj!q)UQtQ6OC&B{r{Nh|W3#W{?ozU#?1cg+;*XyG1Sm7L?p}S_8CPpub$lOQVMCg} zGYhc)>{FqK2-^_7U+~h5Z}|ckk2%5hFBai^EM1-j-Fzxd^Pe>%&aw+-W4-hU{_=0` z3`Gnp^N=Nw9J!7?kow;X;Hb1(o#lvHFCOx)b3CF_NI@;@{2*fOM+PL$lJ$N%{r;Nu z@a>x~uHV*v#R+Cw`KX1IB3HZa`Iub6x417fXCy}x601`S#%MIoy)y@^bUfN2=QC%Z zl>nx)64u*vMhCN5)EDi6$mbGB&hB?6D^?z=t}~e@u$-XX|FG~hASrRGcEebF3A?p8 zQ0^KxHYz$tWuvQs+c|@EmZR1(+3?$i*OYL6g6uR*>b3Ga z9>{oB*X}{($Cm{QB>#`@R444_vjJnlgjzPK90r5JBC@^Dm%`g)DQmJ&A{#~3zW#@S zFMl%}X79Lo3mhki7G5esF*;IpxXJ$WtF9!fw6tDSub3l&5Rc5bEv$Lfm{7qHL_TK~ z`gE0G88GNX@jL%^X+HrYhZet8emMPnIU}#Y$^m@V>fu21enah>s&mOcAb6LjB|hS1 z!bl~}roz`bK?yzWR)k9pVViGe(vZQ0)Te;Mpv)@v-E9KnP*Bb*Qq{g^%>GqUrjw$8L;yyA{De~ypO>(;hUjOEvr9v#u5Y`=>?-p+viD}$3BEaa}Q?u zEoMtp)maszKmOe61cd!2{|_s6^QYeKHTK%3NZ&&wSdH-yH2n5iVn^8go9`8W!2e+d z^j640QoX^So;&6#mXGiH0D^xJik**4jca!o{E{34=wl>M9J<~Lqk2ItugG1W4L^I( zFCS(rrFJ>{$;k1dk1=qXIH0EiU7);*#$M4VQv;&NYZRTCn zg%X8mJWuf}N5bIyvi2+y0ns!3Uy~y5D?}utH_cVqHK}DcFPc)FVt#lj`=*6Tewb8K zj3DKYo@fN*xn+}EHe;Sv3=WS zYqeCv2aSRluXtF3&{+0tRLt&x9K;bT$FNr1*n}}CF&|!Drz-JqBV4!G87L9gk7ui3 zN5{ywC;i~vj(rX^p=HkFlnkM!9&yiKaE@GcSM`SDz><|M07?i6xX9zebYWC5(plt( zth+AC66 z8OCCxY0oIBX?_2;o8y&pJ_~tLOi$w%P>>Dzz4sDxdq%6NZlw+W{y;ieXBQgfqVt~C zI|Xln@!Y}iK*UFNy^ub4DeP-!b3J&96NfpUUi6ouCj8K{g_8{W zEqzz7qnv70&UB-ov>8S!ahdyTcCm|x55Yp;M2`OpOC%3EoyQ0~Jld$g#1u#5r|6RP zH^dA|CP(;nM*lU90mP;}Jq(JLNeS9AeF;q4YIdl~D0T*}0Ah!f3;yWXd3)~nk$Iqe zZv|F%%|B@ipl`;kGTsW*ujEvVfU8<=_gZ>+-`C2giXDR0xd`DG>;Q>4QZ=Gvj|)%=T3rV*tNte6Ov{F-6pmBi!itWZX@| z?K9q0&nK$PWo&p_20f+C&upl8NfWn6x`{mEEz9mR{h8s^Q!c>kM`pkEUAyNvZQf62 z6oW77aC;yK$8!=TrwNj`V-eFO&O&`ga(a(dN-JN+p1s%P@~0G@L_s!VS5-~I>s_JW zQ{gzoPoJROxZD$)+a4dAGGU!U0M$CD?a1pZp+CI%B0;n+TDg3iW*siXjE|H-W3}X7|R2A^^e^gXw zHBCmNzMGUb$38Yv33h*%7G&0zOlRz+Z=FyupBw8`r?rYS-k~Abu&v}+uzGECL3*uo z#8X|&!rZATP*@jEVK`{sj&mM;fORixYDw9}pua-*_1a2Rcj8wVir)Ck)s8~w&9a)|iA zihdJ~%5i`&R>s`J=(mbijW2Hg97VqqoqQ|cTP>Fn9`dMR<8t^ngO<{lRj8D&b#0Ae zBW1yC+;_5REEb(|R>3;rEaj7eoNW_8%jKIFU(^$A7K}+7C>IwAtx&9Xs z=AO`5-L=B%U+175>K4`ffjD+7v_R;5-8ZWUwO+wD*p7-CzsO|cZCP)Ba^g!P136X9 zb#V|ePgxgr*8CUy#?q^Fc*oq7E@=9q^53QG9x;qtqY@HM?6daI-T z*&=>AEpLZ9q(G5Mp-GWLn@rB)GDk2ZWyWOfm4}9_-}5n76`!BpJYBlYaRF%&g?-5P zxfeCvk!veEhRgo1$a;3SR?H-c|L6Kk-_yU9$!>kV+l7wd@WqsKSmvkuY5v3F&S)R) zF`CL_v@PS0*hGKBYQ{la< zHI-BqPf8SeN28?_ckuzjPr2ODFK2__$FliJb zfL{5S`eWMBq^1dKA#%@bQj;J))9ksrPQP3ykIv!%TWsxcaQe3Id+k5 z+H+yh(Oy{}-&`%@gMN*k35jRmn;MSzjB6*OhY7Zr%Bnf$^@AMHIq2C0+D!91U=X06 z(uog|%@9!IwErp3Q*SdGjQa%+ww?$km2R|{e>tHd&pRT!VU9meD(NA%)bz=NtVThR zb;%d*QK%f>xcJ-;zn|B8yQM78QGt?nV@@LtiAiMUrZlT|t*=976_go~xH6nh@_Dk? zmuzjWW$3;B*Fw>4$4L?f{9Evj>nIP)%7X=6_9TnB*?yDEy|n=kOc2j`#K5v~k3FD< z`FCbNHBmpxXaP_mDlQ-pYie{pSd}aI3TUV2On^%YO!se`yy<;P)Z216;yey;UNqm;>%aM+0)xP}# z%|WZ?XWdGr`KDLQ)w%;tz%5|NXl{Mpi+r0r(iZNVF*SiwIrnh?mzeA!y|ee?Sd#j)Ywh^W@cYiBGWG#EHL>JvOW3H$5i^r;a(Lr!Q@ai%hoDiorO_%5{@TfD8$sf) zQDVvJdLpK;>w7s%65b(n-(ThYSbnFX1p0ca6!3V=t|ec$uTJ>})K(=9OxH3ye~Yr5 zoF^2^PFdmsx@`%-G9m&94aovRD!ymgKoCo``yTiZJQwz#%WLp|<4J?FKa5UP=4}ZR zrv~iInR{}S14Z@#wRap-4Ta}#%j#Ua19NRIt&CJrQ^MCKE zy{3HQ4gtEczsHjfqFIgmh^lJVO!2cS1DrKnaJSvNO+VRGR1?_k+IDl2N15!T@U{K% zs}g%B(1j7DVTi6xEgHiIpz%Dp($+RX)&^wRWx8M5rkM+|>($!93#cn(T@p6;blGcZE9{tCLBT7xx*}L+< zje_&fnXs$Hs#N6GA5eoXCmMA((W`^)`r>cP8_yod_Z`%UxtP^EoY_IL0{5y{ z8f?Q3xX_pPHJBe^O9j8o%os~`BLh|!+Q76oy`Rwa%Q|b5`($fq3RX@wR;ZYY&e^(w zz)7fum$bNN=FYguyfCAqxK1!V|IPeD$EJg6=b&$!i()7dtvs*ate!OfhvPHhP6zM9 zy9~1y=1>sPapqoheTsuJ zcfU_tQ$TW*fnYYYvEiHC@(UZOus5xBjs1<2SErgh(XR!kw&65d387B|KudA z24M9~zWul{6=}?qVcZ%r(%4>_ez?Cz?tUZ=7H3H={J`DRnJwP{T`dW>WQgSxUmLBj z|KK`ZtT16~7gI?@qdq)|dMfunHl8-{SxSNG+vCWb3`R`hO3nK}ENa5mMNMBT@*UnN zG38YZKXSFk!`a7TwUmKVZ$7mk#x5!q+Q=k!zLTROAHFJ=15*{xulM zsHvd6<)!rOiYqAOaAp}CAZhog&vIK7$|*fTWRtsgUeYUYT-wl`Jp<4TP+K?OQ$;)W z^4^^d1YYyH=OkZ`a)NKt@7d^Zlv+DXD5PRH-x~ZO;FM2%>ota$Sg2bTvNZ zC%-!Q+TL*U;(&8MQ@cV7L{_+x5JrPD0LW8I?6B zRD+xmgNtyIq1-vw@tpK5?rArglw}un6W`QDH0c-eaJDbBoi%3j-%eWADR&2K4glhy zd{?`?XR$GKShOcPBrX~jBn7&8@(jwp;y2I0tMH9W*v(eMYq>hmDt_jrEGWl0Ve;9A z*t|=Iaw^=+)EXR~Pk` zdn9hNgNuISASLm|dF`@;syHtQ0cA*V>RV-q)3OD@6MJu~rg~qRn!cEQt5UVb4|?p0 zA)e_)L5$xh^U$P%eYpvGkH3j5Yw$X0MFWvc3C2;_I+ztdGDbrN~$D#`nAwuMR_xDA7@v_5_*`g_QtL^xwsKwdrS0Iz z%VYgz0w5r|l^lts7Jp!0@MBx|cc=-3rz(Ma%`3};#l*j<=_ky^nSLm#2YFQV!VrOM zQK-m-jow$Pb6>~!Uy1EBGJmeFspj;Y zIg`Kt?&BMN#Hr0`5|j7s43bkqHL!8@)DuO8$-t>lNVNrbPl$-1^m1Haak|Utn zI`c;oue?h~LpbY5Kkc-AEq*rIBZqI9<@_@52NS%uCWMqc?lA=%+P6GYQFsuR^Y@@JTcYkl^&d|t_TP{F$)t-@!8CxhwJ!095fM+arQ=^Nk zz#oT}Mft#JmR#Noo1)8($PLCF!g$OF)D| z_5Q%TgiSefHX4+=EM1+8ku1XVT2+!81)mDQ%u=|~q(KT@@ixOQS~dPR;7}bY-Yn)d zQgqDJtXV(XB?@#jeaP76ust5k?P$y@h#Ty5C1)D{E(a5Utzl4%H z4@FOF>u@-RCKKJ;SR!Bn_wx(0boMKb%Z@IGgDxCW!n{gtt@)lKY*Oz8ifmYUHP^x;s2z9VgtX^?V!5Xp>SSC&_^k&Kaxo-UrxNR=uC zmxgAEH9q z&<~#;$`Tbw-i`>hmsSx;!6g=S8T1WS1fEhtjD?2a*HJ0FTZ1DVL?aj@Ud2YpCO zm1v!}0^equ`p3cKXvFajfO_?O!tGvu!WJlqwc`Grb`>6oqL)CbFg6x16ei|>4=UMZ zRhe4KPo%I^J~Z~qJ-=`jfLGx&Mr(F?hHB2L(Eb%RR7hul^%BC93-akIKD#(hRLsqI z*_QjavlE#3{dD+2Kz&UU_ifo6kBa#%UCc|qV^dd$ zK5|&H(yNb%&i%m-B#?}Skr9GtJhp`e&`+Iam#o@u6`^$^NKHk@if94;aDn%j=nx;6 zWF4SNy5pA+aj|K@?dQsZS1-Fo$kFq7+3r?WQ204Gph;iDibh?mm3?*=bVV!@BF-$y z_OOr(ag#9Cc@C*@Fvr`!0|q@*_tZIMhqybz82_ff3Uc1#+#cwX>iyco*4f~{*%vYY zW}&SYmb4};RnpF%R3kKAPkR1SLDgS_5oHYQ%C}C{a;OCSUo24rLIP2_$=$O~>1qtZ zps`%y3gUk&SkS=tL6v&##d&=e+2OHBCLy8gm{%|tT;D&&?~}HCB^252!LP*V5u@G^@8(IIXK6V;2p+e@>uVNdi^#-@-$RtR zq{5cfQf`@HS*IXy!%|6E?DvEX4~CY4$3_7kiwpFaNGzRO!GvrX;YRgms*xE_=SdZ$ z<0YbjLHbWFBY)?)Oyh}U@<^c1Xl{A-o|jQnSd5Gv%NH8|m8^!3HKwocJ*4{2N2bRM z?FY&EG=1@Nt3(LX9C+-#eSa|@!*1!aI!lpCoLn}ne5ZfU)K&34)@9`M%ziI)4Tk$N zirTIa{A8K4kBv@~|De`#&|N}tU3IdoxRpT(tiN++B|d8Rq;QstJLaVI;eu*~et+r!?oy?WS720ijr2zI#Pc{I-YYQa60*%@H+Qvoo9`RzhTEx{gvZq2r?et|j(Y&y0ak2m zl8F?XKzUD&rdnk2XiHluKz~F=1zJ8^+Wc^=G)A@Ikv5P;z?}yzUzd>kLY8`@rpU_W znrGIJk6SP7j@sn;YMM>k$-npT4;Xu3V%w7$d zX6-lK)25&7mx`{7^gbt!{)Yvi_-*ok>v9~-;!0;K@YdfbBv86O?4B=?Hty2 zhS|{P+e=Gej*i83_uUeNjP(1cI*srWBCMyn4MF;8OX%}q9S3t6$VJa&^$r`2d*&aj z;qySR|FDRc=0LF-6@c~|ir1AgC0Y-Nf44wAv*LddGU(RFg&Uo1n?cRGZ!7zCR}<$j z5%F{ZLO)l!lzk8xK-A_*^7x)i|JVR41@Ig08?iYwx>U z*AE6RDJ}>aL2I2WU#3zNk-L7a_b|Uu{yBfqifIP){CzPm8PRp`x7nLaIDk zEml%CL#uZbwU%_T#Nm0c5nu8N8+!^WSoZbIhT44)k_I@Lak#BEmHpPp>Rl%VSg zHyk=*yfrDqYC;WU5i@%&#)7J;2hv?>G$$xwwqfA@V}pOC2S8`oH3Q}u3!<1~YEPRz zr`SfO{f?^u>NCCuQ+RHql%iZdnCouLv)8ir3{lS4JO@~|-fD??Tc3^6T16ek7HD|! z_gj~)dVwN0uf5JSFMsKI&pWt(QekDTVuJB#Hd3i7T%9whxd%v#_rbN)RQh=>2k$@;y=AD!M2s5_;Q_7~ z;?5spT?$LtZ|dzION- z20+tFcmd4a|7AWlZO}0f9eZDmHVF!kvg)>0?BVc&w-b1Y{d@jF<(b&Yn$zTpuC9xV znhpFEg9S;L2PI2%EgZQ+BR&ZojRNGD+#hcaQ9g4SYx`_6SW46~zveT4aYDV0J3hgq ze4oS8x^99UdJ2a;z()qLaJvS*(oC{)QfpQ&DZX=B3|7`@e8HY$Sz*~;%S+*4sa4fw zHN(-CoP^n0q9WgaH+(ml?8_4=c@)f|MPy}Xfu^whm!hI{FwO7WnGLx;WLRd}6dRs$ zzFs8!V~zflxmbGW)|%#jSS)(Z)@2oflkDu*F&l_=%<4!wnbjABX9RAIBDtpL@4Wcs zQzd zUQOnjs%wFeDUW$&_!-)z=3 z7M7~MZ9MPRy}c8g&{6_-*EKR;r^&Ew-L{Zd=#)dEja#;vkMoR-zrM5t8lr%`~d>Rm0OpQ^H8f5Bq&!G(IpiZ+g~_c=6Okbey06QHHMPzd;lVe>KAKpjA@?Dy{gE#s8$3J6kb3 zb9V>Yg!2QaxiYU%G_rs|^9Y~Q#Pouh#D#G^s}8{coB7fMz}*)rxu-$el8qg((%}@E zjkSaiZhD5C8`*96{_}rz<|QZn?Q{Ur&2NVpT_;>U;9oCOC6gmuM*VA{j(7%=vxS`y zs_yX=G**M$^+Ho3#c5#f4sHsx$HO|SnkGP^co5b4{E69&8m`T7;xh+RZ`dRX(2ws! zvl+i4!@je_uJxArGFa( zA9<#66-+{9(@Nc3Ep)Gf_0_mR`4IMRC644gCOf^P-QpIoioN^0TE76`r*8hj#Hx<4 z>tl`r)cQ^A^J`^qPQ;iGQ32#~hwF)!DppsQ?;nyi&WgY7%;xK%vYI|$2gri+Cuh(J z&5(OBVl5)~&wb$VGo|4f6itWmERBsMyMp4i!?VeOX67j$lhQ22sTwFF15l85z~Y9u z0Z2WS`~K+YquGyDVzQ%A^MEd&F9a*?KWzQxzEjqoZ!uwAEp6Y;fm+wJ+R+8w9z8lQ zQ6Ku~58c^!DDCTL}0d4-9iX9GTRr^dg)py28?$9;% zu?ilYi{#Zg*CoGsaN(7Y3QQ352(PC`%PH_MIM1;7y9a&f*qErn3Zf%nH>r)!r z!IUMI?e9f=r#}%EHv0XyoSS}8{Cfps?uta*zmmu>V;>5@zgna}+P*ns^F$fi`ym<{ zkwuEmD^O88kaaKrs(yD7wTC$i+qd*T%Cb|T024%E*2SRt&tvweP5v&;mXqVy`*_!=CYx+(1CFokM3#o_F?72 z#)}S<)G_3kgVS;NzWZjFIRj?*)yZ>jZLmm~2)qjXS0`3@eCHyP&iVs&@K)9W0|I*Q zE&1CttWAfdk({Rs1!W>Ch-{r5p5axe&DDbuuSl8b-|Knh@u1aW)0*dJeKFO3Bk4rp zggCA_a3o?jJNr5c&qf@Gc~so|;HnIJzk^rgca<{vaH=F#K@WhuGW1I&gKW=GKV^5W-L<#;@Gg%S)h&Mpo=Wb<>e`2~`5gSaCFLW5!^1bC~8;z81 zyE67rQJ+5BIA14Y+#_7vyGS}JABB{N4#_7EOo@|?YfR<*pP&d>K0KX8Vk1!NS+3k{ z{W=)7!kf$*Ifpu*jWe13B!KFuM)lTUV=%>9J10=t zf#ICI)uE!!3AAIvbffg;-ME={W>og2)Ih7NU}1y{WrECeJuVO5P8L@W;~s%|F`}Uh z3Q>bd?kct%YFl&ta>SI8^K{2Q1D)GZ-|^p{-}uT~XMH(yPuS%GRq0eu-j*?IwfCScOH@-@+8B3nf)66def?lccMq1%qVBeV2_ruJ z6ieiY@r2C9dhl^(@FW-1!rh{sC+7U6B*T|uLx8=eSmB(pfQ!h9m*-O!@Om=rbS$@@ z<3B7W$xba(#c<>NsfjD*J6vLNDNBSo^0hgcT|akEXj@kFyFyk19q1>x-g=X!3Uc&f z{fcWZeQPv&;HnD$n2nBX(~N4){q!!=)z^+H`DDkDuv3yV@T8U=?{CX1{xexyKRIO} z)M+CEr2>@QdM@?^wAN5a02Fs(>ntzrTyt?sb2kPi#b)Na4nt)q7%rZ6%R)PA;jG|0 zh#Xn-&LjrP9I_ZtBaGVq)IPRP7o4^1!0N*5qzZy)8L?5=So<2A1f=u@`j`pd=wx_vd243eRW9 z+$@O9oVlwk@xEs+)a*Chn zwc|^+Z!Hb2lR()mTzMa^1d1alnDDiH37hWq-5d#`U9(e}ftriUq=xu9QpY>yZ=x!h zR3J%$;zaL8bi>$OrK}T$TOIW(R#OD_cY>ebEqjly;sB*ol^3g1fgAGB4nu){%BU^c z1oRYDY!q&fvRmloLyYR*ZvDhP&bi=1dcFHC77Idw6NripCiS@3{P-;}2Ko*nuZ}E; zq_yxPOyWleIt}@e@ajThoISLpkMQI*nOTa&j!P44w#*frSDW)sh9~(>XDv01GvZ4E zB{@!XaK8XrID$-SY#IXshFCfY;g;6>{o2V}=w#(SfE;p4-bYfm!&B?4#jmM^y!72l zoHyqX8FKH!4n&E;getquSfd=D3RKzJ|FQ#F=duaFKgRFK&hSbFCqJr75Ep#OXnu~P zu2As&+l1#D3NDIM97!6nEBbdEH{_I}iFBJ&IC=eNWddsp*RPsaNRi!lwW*@vu~DDS zT-u~h?rBvI*34!+1Utn5Uuc%{C_=eyCvD1KM&Q17bLXw+bn9A{7R`6BLWsjA`{z8; zgv)vOqr{oiF!GA6klTy}Lxc}4^q+B=@&1^__71tKD~B9zI-J`D3%s`;p&pv2AL(p> zaGS>hn!Lc^AGsKIWCmdHzZ^4GZ;tKfIyc^dT%PK3q|wt5eY4eyPlvykj5!mA<#mnC zrcPX-f$qZzW@+dGOU{*ZW-~RRNKM>U=3sO`M%LDK za9*Rz%>)U`do7s|DWte!xk^5e6~=LRelYS7_8!DZt>c`Cmn4SB(U$7RNo@qasRMgvhTAUEhUPk2q9>Nl$HXp5yP zo7?!eubnO7ZHZDpT;yI-LGh=?169t1Ieh2I5R1)6!{yA0SQ3`M8t{{Jt&`%kM{ElA zX8}!B86n;71tY=FQ(AM0GP(Ph<-INV_x;Aj@Vu$sBC@Dvn|t!X1);xRd*(#(64-pB zoVhUi_qNB7(IJ|0*f9sh%9W$z!fiz8S0|G&lrqem@Il5tPkdq8F-SGBtF#4MDq{DZ zCaZl(!FZ*yO zuPkd#M<=ZgWQ*8J=*Yhvh{(DW(JxP1oVw$T_s5OMYeiNLw0z)osjg%a95XR1cfMwZ}zm8nJ!2IJBc3o?s zi%S?jA}v+);@*DY-F>LCCF%>t9&dV|UNWIJDdr1R@*S6gBp8;rD?p|3pllAPULRAm zbvo}RR9VH3hid%aQz_12<;hR??wwydUN7!U*S3|;b$1)q;QVH-o31EvP^y!nS7kD} z<-ra`N5zIE#mYdzfAFtD4Tzk@S@N;wIyrCYz#q@fZDCbVWz@OvMsPWuYBHiB9reig zvGTls5@>$j@V(qQ4y6cj`_s};no#j7BT4h8Vqnuq*K*mnt_k@ubv?ACljaXd_1cT$ zH37%Any@&3VCvjE9^i_OI~B`09cmFC}X2HrAG9f;CGue$SQ8H@(&Nu*!S!LGl9+Tg){X*g#KZ22bR0 zVPNY|4&Qfp8?sm%5l>VSI~?cnb>oq(-id%BYmmTuvdWF&t5l%#X_4Wurh^!Ypr^`0 z5Sk=3h&3M6e5XyT9PjbUp|5&l!Hab5_*-r!6Ax-aGc)lwt&gNBuk8xTr_j^ymd#oZ zA7?LuoS$(IJCRVbnSXy0YHHeQR(EyoVw%1>3jp1{u`d`prw05$=-Q=5{`!hU_q_Nw ze9ATN<|vP!oz@!axa3(tIVXM1gIaSBkB)~se*tFgS>ItFENO>A8;=-05Dd)QFV53{ zS6kjqu1s3DE(7FC1`@N_d^YrYS>6b^pH%Qvh-c_+1yl?q=gH9pB{Nhe__Gm7N<9z)fzGm*_Vz~9nxZ{dL> z0v%pS0&{+xW(BQS@*)z+e^=GWJ<|6g;E9V3KOSuiD$~#5o9o#o82Z*ivBCzCb9m zo_RvMDUu;;ZUv!1w@g(mEY~*aO;758vA^Y3k6yoe%`s*+Am)!$lpR5h1vDl?{ra|f5J`htFyUJFQ*B-|=OwwlLt>~}a;R+w9#*GRN*mv6eOd-G^u6Q(wX z`g!X&j(tt>QIFxv2I>8Ohgm(!yccO49I$Ya1hI zj(1%AdDSf1;m?Lai31NV*dbkd!K->3yl0t!Amy0+=#(gtA2&qOxe;)Ptsdz%(K94$ zWo~KotUM6h&Xp)lrF11Ji&pldV1=0_k*cL`gwTVMi^-8nW4WD^^b9_6DM_`0G7z3RPii9?r`*3jG-#k=aN#N&4VA^&e+F;m9lSqAUSg zM3v6h+DDJML+;`0v@rF2cAdI>ysCV(J8J561IVp3|0kV;yaz*-u1+&z-$aD9r0uNX z#v4e~%}59*#5sl5rlCU-gYTWtwJt4~TEh;`5_-Z`RFYnfasy26hBam+M#Zq@eF>+W zAp{nADsASrHM#OA=(Xy>vJW#?GG~B38BR76i&c;GnYV8x2&JAd0`J+5acqt^Q z!v3XbSg7yr-IIazkzFu{F7yZC><+sjIeJ?esJ9!JqTqs=D=}ijI&eEXtxpkxqaDlz zvOnT})8iO}7E2$ZBV(i={M^pSOWKK>tA5Z(Ke|V+Wl|@uy7aVt6|gQzHC4rGjUd@^ zUE>;(;tI$$d0`QW&qDPDm>db*po1vQ&;ri^rRkzflo>~hYc{vYWAZ5C(g;R*92Poa zr%Ad0ubrm<|C>vljiqOxvcdTI$5dVcs2Lf7?hF1@5xvq3b^z=b+h;#GNe2gcjdcPU z|D$3#v;@y<0u_@ebQ;b&T+hix$gow3_2i@y=1|j_RyM4rK_T<10@dT?OeqJ99dCPy zou9bAVDF-!$Z4sR+y22OsyzDZ0fE9OY2Lwr$;cSb0W-rTbA2#W3g&{DI8E6ebrG@` zi1+68LVZu?Vtc_$#|m6u0Me$I!qwhsnyfbYX>3H!amuYhd4W(~iFDEoju&j4)Y#VU zRkZtsErw^@uWi-=xqp0ih0s=PrIM&89Kh?eFK$})+nAFW!ZpTceZMA-TI^E^z{E0B z2-{ykwt>_tx*zu>JN>+u=3uVD(&S2cag3hydOij;Ab@j6N@^%9^=Xs^oInA<>8P33 zqo|-yg^-yoIz&WIm!Ro?2|K4D>vjxzeP|QPB|TVf?5H8J_L7uxZ)DxyYXemW{?}V= z{;xKjLG0c10nex0&-RtN_w6V3lQh@=JdcwufBc8FQ`)Om*zsqkUgkg=qX`5g{gdE& z8{55TxwVp?`~$}dPt(u3P*_*Arb_&1DoF(QqK3ZgUx@ib^xqZ@v>T^DS>T|kc~SSt zn~3*<-qQ!u!J83Uhep3$CTeIt>neC_GJX%|SbJ_o-%<-WW^SVGy6qj;KTtJ)H~wj0 zHj6`&(UUp-THph~${n9ha4AWH>$R<~Uv+F!q)V@~@A)52jj{9T*;vQ}FOwXx-gCiT zhWV%8QER#`j4H;O;}~n+D+>w^+TS(I|1eHJ#j%S1nbYs)1+!s5IT|SAC>Y%k9-opL z{GPM^GfAnVzkM7t9J>X*jTa zT!fFA(=1f(Q;5aaK9itKl}G ztkgvJ#@0~HWleNGd3;Dui~D~H-WHR!1?Orn>9t(nDT`V*I)^iCn?_bkChrAt$Px>&xudJkG( zyZlyRyNk}s5An7wKl8sQ76bO>Bm6F72YfUGYmDt!?gDK#4w!bTScN~f8dG;l8#-i_ zvvm9v$UX+r@kEI{6x3Sb)@8=2??sg!;I9BX1~g?$plXubZ4pHMox-2@H0SKYM`A8W zfZLMg*}kysBa6{7|7{dnDJTdFZ)|&%sxA-}`5)HX{G#~!Z3X2pA;T#=4UXwR6?e($ ze7#Kg=iG&~L5^h|!*R?oL#UXZi9zAfq=h3(TmJn|3XZD99}AxH32gCrD&1!?eT<)cZm_6i64f!^#QN>9t{VV%Vj* zmQ-H-*>D5PA^r2(C9EX3%amm}F{A53?VOoWGIR};9+L4!20znSoXTIL!X^|()KsvK z!OUxpLAe01&ooDIyHnGgFN*BCHL_^DBTdosQ9F#SYd)M8H@+ZpB_FwT^0eNkt*$}I zvMnMv6|Pt6FML$VD%v7-e+uUC6c5whhqmTYOI~Q6%{K$>*j5r>T#{!53V6TSKz!^I=trhEmh`c$mjq&m?9LQ#%zC(Gl$wo>$&R(U zrqlm}uD6bA@_*z15kx`>DUt5cD%~Q@kRG)GN-8m6Afu7)9HUEObjL+`d z2KxT)bAIRd`{#Rpf9{|AoZaWTckgvw*Yol4k$x*mA19=*MNuSs2y_~uK}q@AJmm<_ z315&Bc}kMX3J7>>H6Pf1JtPnyawW9Cp+au9>%hq^FV77CxKw2bAeE0fLL%tA>0vhp zFJ|PMCi1qArgz?F4R|8AfcoyVnsj)f4>l!r@Pe=M(d^2j2?uw{8JocZC;+x0O>4z; z5ZpEc6KOlww5gp^a}iLrn_3E(@{)1bmQ5LtF_X!AyBqwaZn9gMQ^nGvBGLx|A``N= zDV9kr>RQG$*9?;aY!p zMrJNaQto9+YE5;QH9h4R|0GPg#_JPG&3nHzv1-#HiYH_0Y-@EQ#z{6|#Br|Y@!7V8 z=&5yySf_Wv+=)3En^=iZKBS~oN8kM&dF_CDpOIgd`u>uS(EU;>MRDPc2i)^~e$s1Q zvH6^T!Vf*z={dFqPVOlXRGNw!=Q+{TpQ^*Tl6L-Z=XnHKzFq3feBpMF1@A9mMz;@# zs@lWZIu=qrLKSzef6j2u|1Ht84*>i$ss1L~k$Y}+Lc_`Ex`u*S7&W0Z%`M*Kv9DTl z9il~>+T)2=D@kH$jHg`j(>7Dz-`3<@T)V!VS@C?b#S}do zG3W(%AUAcZ>`g{dSI*RRxWuvr^HFozkh{kK#@BX7X3G95LGv%QLdk}6%gyj!dsQ%K zj87%$Mn*44qvw_Sm{=Hx$PY^vEkk8}KSdL{T1~+vzX+{FA0I}ey#~4n_A1{(MNm#~ z&4g3Es)eYw+d`}vXfP0ft{_`B~>1q#@^U_XArwjhEv-pq3!<^7%*l9dF)*jYmOZ{>XT5O20iXadOlW3Z(un%=X03sddIZ zN151dj0Fo}J%|K~@Hg`Ax2$mmeC2)E;TrjPZK{0h*U!0AS_3^yBg1`}akTU=K9MtC zKgH@}!K+2T7-^pAeP)NBXUY6EboYJI7S(qqY<~J`E|stEj7@l=S}qjCJoR(R!5yN) zW^4--8aX@B-Rj<-O!b|SGL%RCUoYTJWa>@+qBMjad*xta0mYb^(~6<#NKT-q!{VDDV zBz*fWRI$Hla30OaRfd03kunk9k`73p>H|DC<7ON z2^D?2>XFnpPA$PWw{uY&m(dl&t)UG@GRhW0p#rHCvoXvjMs3339k2&cAOe)j{- zJCT{-Dwdyy}oqU@{?F5#Y?Ucgr7tp6Z5weWrUw73Nah83i z;jLSdhU+p3O_WA)KgZ2oc%>6Zp4+Zp!t+1;s7NB6DMM9!t%>vOx3{W)J=m@;P3!Y5 zV=0L}m6urt-?Se#%R#$~jy*SYY*bM-{UVjDX%0b94SQT%d(bBxFkJ-uiYmkh#!OLU z*lB#=OHzDPp-ZsB2y`JjX{0ix{R}xFgFQR^7HTsjPdIfWcQ_!az+r z6S<{<+&5&c4AVjMWf+GNfzy7+HOxbvuj!L5@=q751>xKytp5a?|2SLYP$y!l(gK)~ zEi9-8s`Jjq{qc7LjaOGUMv{mDrx@I(LFH0>=MI>@+YK%ng358==nht z{b0hC*XDoWE@XbMsp!QMtwmG&T7-X?;@LiW8^Wb&Ri)&wpyO2u6JUP|`GG81(lyJ9 z_jU8MZCF6tv-aJm`>p3=+oaxP^~T4kqOK{l+Y>|zZN!4)s&sKCVw>!JKXl?J>>tT< zey1x>eoGQ2U#%^H9^v`0arpLU=kbz9#q4^blm(+FIF}^%6iCG`djnD_M0cjYBpwFo z0atupdVXe_7X5+)l$&oW=nuTFO#is!;XFZuD?O>Oy$G}1$xyCc)u!tQO!U-MPN}H0m2~OcMSk^T6TU$EK*YfhU+W?i`!ZlO$-&ECV^C zzOC`d{IZ2;jqbt|Ud!}g^i0rv$fo?108yoYN(cdEAGl&}c~&~d@<|B_ADc?u;e5$h z;Wotedu&f)4uxTRNXB zS;kXxLiOwwTvf~u%_6}&=P9P4@*0C(>pIV|l;1ijr_H&mGq$*q-jmZBn9a(ELDPDl zm^xqeh10&)4tT#jt50VmAkplAOuDnWd9zD1Q{XCH;OhxZ0D*kSpK3N&fYxSafo^hz=*aydBkOd{DwL~~k9#DgYs zZEeR|8Jkj{R1?%+(xKk|mEg1+O566qNA7MI9r;ts$L;Go%)|^mFL63M@P%o`>0zR1 zlqiN#f4Ea2v8^HYAmXjBLWBPwn!s!5D--)lHy>X~NF;QQ0>gVS=bkL}gf&(R_lP~J zLFvics?G+$Ys5)`cl&g$x(iH{SU~LpP=_gntaRx3+c9PEWa2E0I&6g7n*49iN)#o~Pe>TozIRBPJ04nTi3X0x1 z9<9!0l+jJT+vag?nT!hM2(;BuaQpoW{&{Rww+U;nhHl7RA3$%c7*s92gx1g2^GmJS`Ym0W(ytr7h-;=9&sp$`%@HlJ;aG7)w^d>s{ zD$emLyEO!uvugIKtd=a-VGe#^wsm0l=AC5dRK5Erblv2gn4(H)!J@W(Bcy4y#3LB@ zu2p+pY9d(-wKat>MqKvjaEEEW2{N5RjgWW9HeTaz!=atX5PO0uOA{%K+y%k%p%ycJtoyMQbz1XjTetSQ5PGNx(6R%Akx-X=N-~r#PgAntMv;U>FQ&1UbaA{3OSS^dvuzxWg9|L! zNeVUDwQ5l9XWyKvO#?)IliLXU#;b9Ojvz-x}+HLOI zq1|Zj_ZF_KVqOUL75`1Dhbg-{!(WcH6wA6^cUq^xgCTBbs!nZ%$h!>0YB_fSF;`hdWZtaWQlvD$46TX+kGgz@f1lD0{XIhQ4_xB+|9u zl575R2ABd3N2bKR0u@+cH&(RHQn+d1GcUfE`B;Ip#p9Mk!kV%L*BPIb!xtKEv~ zA0YnEGy4N#v`j$dVfOd5$GaAFWS?{6=L_OR@>|o8JV+jIf&gM?zU>!!Q5jKtBAWJ` z!+ksjMVqECx*ERD(*B0VOCD`u+%oK6kkdSh)z|bWbxk7MTq@~kb@yQwE$&wI1XVt# z>*6hAad$%J5|L>Q-`z7ZTYc^-#rGjAI#!5FcgnTi&x=hc*gpKYDqFpM$PIZV`%4Xa z@2F-7g1eLi=s2N;HGm=EuuuWbt!OFw+lAV3oPXot52!5*vg?ov2xEKBPS4yCvI$1J zh;zE(HvNUOU=NmtH^yM^#`Noh-ph5ac`ox*fmy(D=v+e%6AZwtPsY^7Wc)3!m=wzq z=l(5zwA@V4E^che!TUmlpOVt`403Sk)VKTjV{wtiXRGirRaCq4G?5V#ITlbIPkYzx+x<4+sXe3kyyDrjPaH!U4MFK&-jWBmU7`;aZRs+>j(?h)_fBT>^s-NA7S z=#JrZn+EMjbE4CPvsg)c);Sb+YPh}1S!eFX1;_DQe|g5vpWsbNPm_B(C9se3GNQe( zZz8+7sILcm-}gaYB=Q1{Vh*fn!{=peqx<1e^$%SC`7d7{TlwAyXa-pOWo4J z)%3@7zxx-oqTK1`+~V5f+QdHBc8t>~OEK#qVKZ^?Gh<`#?Q1D;Lo6cOcGEkW)2)Vs z07xgk9B$N+#JgqfufRA*Ps6O8Fqb8-jNI--nK9ROxvQZk+Ann?jVn1EgzSE1SeUbm zBiYAE+kfv-=&9_}EGFY45 zn%y>*<)z4d)kNi$l;Yi3Co&j`XKe3|{84E|(u93`#!tBi8|!N-gmvdt0f`z4D-n{K!^TqHB&SXQ{K+%G**fKOI9e?I= zI2d2Tiy>-1Y0Cm&m5@O5)RxFO7h1z?pcqifwc>_MWwlm`AJEs2Wtdom+ zYj9Sc)Qexnqx-z4EN-jb-MeSi4=RExuCO*apv zqiiL1w-0&5XfENGba4N>8=-2q?|85v@z_JOMn2OE>QzdbJl>XwJ!;=Qrim-w-|2o3 zAt8pFrVwmt%YL9|16`hvZP|xwY3|`{zi3(F(Me&t5&`1?YktES)8C14oy^Il`p#Yy zd8Z6?PxgCXCcXxro;-W4538<*J|64}f-}qQoT5=oJeiZC z`rdG63Pqqv|ITimya4Boa9FF-(kx)cLOsV0fm{GQMG~S_RzBEhwy~H_^0^CiU>tq5 zRIGwp6y}D#(kUga9$N_20g7;1lTIndcpsuYZ>#Hl-%d!k_Z#0;OK(=j$%K?b_WXft zGq_xpvFZAke?46t6_Z_vLmQSY4wS&fE5<7jEilAKX=2Ou0;-x^R+dvyJhh%}lMHoW zlG;_GSdH%g!RXzM2um!<|8?aJda2&o8m=K}`C7rDEBltG;LLx{w4J^(}#qR%(4#0TxFVtA2YWib;88Kyv@r zv1Y~}PZ7CeBgon%m?HEL+I;7%A{>v@!F5OseQst0vgioOl3j%2jw;R|b!_y@V8j-+ z;dIpW7~?Rr>qqimn=w{Hg__@gRB3&3raD@GGpBjRRJ~YY@X>VVAL6CQqIHk6QVdD} zL~cyN9I9lauXs=BT|OH>tbD;3{Yj;b6&C0c@V}UZLTtEgQfuAh+MK?oP{9PhdYjKEb7%b}a-yd&!MX-p2k# zZ6q-NeQ_mmV{(ZTT!CI#ALJd-6qi0KwfkH@Ysh5#TjKAqeeWBcx;}q-p|>Gr>*n_a zr+o<2RXgSry6|>*-L^2JQoL3zsl-}nvEwp{S?Ez=tFzf^&+B$;p2@u9@9Z%w;2+DG zkDKz+*?B~9h>=p7{%((<;j*SQ94(H1?ys4ep-0HcG=8-G=Ra{2iGd3u`xH_|A#5=JJ9iVrm<9<5e1WYJ}hNKTatv z=%ELqgrVxGIE{g&PGc%Qo#ereZpDKqO**mQ6wO~<#`(SOl`SqS@wAM_%jOhSd)NDH zQEQ^Yr~yPIYN|JZkS^`GidewTS^RZMyU$aaJlF`sahyl(B$PibubR1eEp7k^k`Ms? z-+8qEFY(4U|EsgGv~bs22>Vgm>pJbuWr(0AA^5}QA^9L(x}PA)q?d*H-Qnjsx(6@% zYkcx8q`ZsQTq32ss7rdC#b@JlMcP^poXkmlWmy6Q+hmL)oF`lrO$env&8Mv=^I%*s z*P6;?QO*b zDWpkgXj&dQTDJnDv#?a&GEYq>`h<)^ra#-?MVD9&NC`s9q_s5o64czYK5%xS_=?~6 zDGf+7_-aM(m9+PE6~?Q*$`1y?y6k-AlcB~=I?RSH@fFY)ErIbm{o67kd$osP%>BY?v86oUDB8yAy>_I`cY0Sha zudEEuVRiw`Rq;Ucm3Z}dlU6(Qunx@hBuezKEyRPm^uai}*Pzfx2fu}m$m$lUUG;XW z&36UQ82HfR)Eg&{yzB(!FVYfmCKmXLF|+-@^MoLY;9EuSUnSNbr-Kat$+O>mFgx^G zev>14aCflCzT5a8!HBy3tlq-MFL(^?V|H|@EJyU)%vzA!IIs*&(XLN$kC%(#HoG6cqBuT=<2y^Hw!!OFtD;4ZXNdXG!|G%jLLvege63j^FYt0 zsc0?j)GTrq|-OjbzhaB@g|5C^a&jttRi@5q<0bI|nuIgu@NjnB>&c4f|2N zzubm-i5#E_dwXXu6HV&Bm~NCJlVE~+UpNEBcfyxVkMG}#jAipyyw);bxu-9UX1>Ji zd^%^S&cD2@Y>Z9kJ%N65lOK8F1FB#H{0MitDN7j+&c1PN3OJ=0fw{98nO4;E| zuT=?6YGVjg5o59f-YEWbga^DiH8kwS8{6a6ACSG(44izqoklM8t4tTS2OK?LOut~= zU3ekNtlhOWp*_{Z6F%OjY3xq#JhZ@GN41$BaKCBajyj1zJzP7&>b0hix+njV1W*}o zR4gt-?L^ma#wv6fspkph4a^^LQt&z(88=Z%LsR#O>iwdy0$P+6LPr;pJWwAR1z*w; zXG^V@X!NJ|6N`&NAjUSJ2B<%!N zrcf2NH=8qu**OdmrUV+U8Et<5W6Vvc*j{U8m(y?B?qbodP;>jBlUgKQy`!p=mQVqp~ zaoR+6je}oB|I#?r?uirSVz7ZEvr8i~3t8EQB3+T}?7W)6XGDx##TXlaCqp~8bz zN8C#%@imZVt7Tr{ncTr>XOJ(8w+woGKH*8O>Yh8C5f~u*{7bNE-kxu~{ z;TNm9KJ7czzIuA*4DEKUa9cGbqCb=~HsIpOXv-S_jDc}Lsn2S(X=d$(pUz8;R4R4A zO~u+Z;Z7|DFng-_!JM@34mfT?F_1i{bnoxYnN)s_#LBp8D+5Vya;RTp!B8rv z{kT>OKY=qMq-H=A7vR~`)_JzO6=lGfvw0x{1COu=&+&mBp;y+~DQFfDaZd=97(g=r zujcSB&K&T*+v;)1NvC{m<3edAFI$6)ke+^o^`&ISD->tfl{RHXAa}1{qzyO&h1fPU z3wqsLdOo88LJpAH$j4WE3+M%i>HQwr3aEOe1M80pM2Wmibm=H80pRPCx7^UOCgUNl zN-pc$ZPw`!A``zB6(N_Y(KizZM@Z+LRWh^lOS>@S!uxUPDLL{ozO6Y?>VaVeWVyY< zhSojv5o(R?j^dmKD-=+b0uo;@ye;7LwxUqdU29f4ON@Gb$=qS+-3AL{#e8J==|zwv z8?s~2r}2!=cBB~u?ldYfwi|S-$LD%vjAlaL zodsoeM^9}vg3D6#4e)<0H87+q^Fg3O^T6*6ubMJ>X!C4He=dwuefoxtFJpgq1Hi`& ze@A!uleD^5-*0+L(0UWvWc}1+`0$=7v%e9jQp*EbbHht2KPGv&YC2ML+}C%?y8y98+hA-JJsQO`VD zWT1oGTBt9ZTh-6*7lA4&vjn41HG;MY%F?v`dZ(nNyu5ny=cs9A;>VsUG{K+Ub6P4L z?w2D8)c80AY0$1@Q$!k zKA?3rr+MGx%y?{!4KL_ZJ`9b^c|GV!#|A%S+@oHYVyv zhU#3tJpXeBh^f90d5*JG#ysu1vDt$?tJ$3*vDxX7gt|FnJV3JIhBCtE-v1_uRFR5P|mV?YHpkN)N=zg`JJp~Eq;yzVryKrzL)OX z{(M^btmXvuI8-I`sFE3&x8TO8(Ra_`b5O9S;+{n4U`660PxUaeW!QhMjCUgDY9KfG z+Y#YH(Gt?m-zdnj31=gmE;Qk*AbIAZQ#;g_l!5z`fwp>7gP%!Ag-P`T7#27(XUjl4tLE_&+MIl{AtpFe(t~J z)qZEB+NN|kCI+&2^AFPV`wz3GHK?^yOl&3U3l^?pmroF6m3K;OzI#}i&=enul}}XO zdFX3YaHjVX>Z}-$7aLxraN1pVE6s{R-oGuF-vrM*c7#o)=2-n!v5R+Tz1?{QwV

U>C!Z72Z^UHzrnz&Hsyq)~SO4=FWbOGrH$VmQz?#2`s zI73wDIG@hg`?VCQ^)%MO4pMev4wiJb@z{#5UD2NX<`SGv@yMY{XDENn3Knj_`5qXA zM|JvH?O5_&yMVGic0Zl5Y?FI+tNt`ws~y)cc28zq`@lfeJz{E994s~iuC}VK9T|Gx zmi@|_Uv(;4d|leHgJ;7cYOY^uZNV$g$XH64InHbEGYeGcrG<>79mQd-JZpUV&g8Do zyFfTc%h{X-s}9)Ym`w7*1%1enl?H2lUi~K)``R&mXHa_!6`qP&A~7*RiPHDmAuQW_ z*gQ@Jj8Z_Lk6LDO#=8%X+}36H*0Lra4h~JpAw?l0LWi{iB#ODJ-1KLn@tuKQ*$U4>B5PWH>+NLblK=U+ zWGYF2m@#0xg@LfIj^11Noo>cgs@$+zyxFQ$h^gB9wub3*V!^Xg> zl2y_#FaN;RW&TTRc^%ju2|~Z*J{5is$;rpfQ^z#E^l8?HN7Jg-DW^HkRLIT#lIPY| zfsMl{79&vUEOcRIPf(8k>KXIckL0($S#9yUJ=47EVSRUH*OfOnH9vgFr{IyWNRqxU zZ?)7USUNe$X0*(lxVMsu44`^BV*+aBAwR23H7gd)gi{whpd|*llojY0X-epKdvvDK zrMPK~S1p4}pw8cO{XtrgI3>5Bz`@II7yvF$VrAQ+naDJDRt-{3I(b{nsYXg_IYMGF z(#z`jN%RRLraCRH&Wg8^l1;=$OY2BW-+(ne3YpIvu6=5{@p>+=(F!FirH%~{!;Pc`@1nlQM;CVw%pvOZDj6msdyI_oTPx&f8w#3KF#pqqM(71Sf&g0nOl-}~K@1gh zZ$YRH@>A-e9BjlG%xNuKq>%ajof;n7+k>6=j%nicdw&t&9LPTf-tJ3hqm=>YYbpI} z{yxTP6;@;j%bBpL3q%r{H0-EE>9a{5R^G}e{a9#!O|ii@OVkOgjw(Q&tbNE`s&wT3 z;k!Z(UIALB9P~lSS-amgCHFQXp3(NjEoCuS$s`&EhO&C*8-8~MB|HJWY%%{C`)A5@ z4;blzXg9Xkd0h@NJx0mDO@rJ*RSEG&bIUbU_dcCR)I2?&H)|JWl(q^U!&sEb(GuQF zRRULaJhA*r2r)r*Q-?A8^M~OqI)ZgZImzz%tY6#nB3Th3a*H{eowZ9Mji(t|(_9Z_ zjrOEG?c1d3^SmjGATz5dtZ*C#JV5J4Gd&o++eT-QR5mAbek^I~DKbp>!^#m9yhE<; z{LHJ9Cu30q?(XB8c%?h-Cvxx}{cTNKaoYseWI6BDKw%Ns4A~QIZ*~K&tr|~L3ibXJ zI6bX^O0t=(9^|+6thDsIwhP>Tp&Sq9(#L~eg{=QpGJUqBIf?gm7AJi)geT!d<+6|z z)-2g$6k9JOpR~Gt4!s2Z5y5)R*O3(rmamJ-9D!Kx7wq=*49w4p&JDU)oyp~a8>03~ z(It0+&m5P~V``b$H%68knl+$H8)?URXNdmun>lCScMs&13L`!`ICYrN3P8|)?S1XO zl2Y-ieCj=dm&d823Ri6#C5ZA^?)M{jNI6VKValy%(LFT0B8J zTTqmAj z{^madh7J2|Gg7yMZJULl70E2XK4L<+@VdNjN3uiV05K@AreFJt-)VSM5YSV4PF?6I zCY(`j%qMIDRY031d2ci);gP&2%=QpnB(C=MoBlcv;kxOSB7!-;r(mE}8x29EkUjJ&Zq-AhYrJFAZkwa+;MM|%5}o7N(wdGl8l z6l}mY@N)X|)V&ehP;2C)-_C9`59oxvIwJI(fZD=?s-wN4kGoRW@sPqUq5IZ=G}~V` zK@}G#aLkUDn(n0(J3-sUj)#9Vo{UVqzp?!=Q4*1sP)b8mYAWuUhmrac(NnSum1xkG z%wfqrre#W|IIQ#=ijWG?#u~~Hv3`hiGFeH(^x4ccS;nm6^u5Vo@o?FcqX(9Vz|e0y z&OLwo6;}1e3khV7Cxjr0ZARMiW7Uog3wp2oMKg6^0Im%wRNJl{)hPGyw7iVJVBI_O zO^+(w{3Y>a`V+fJ3dg4xk~3cxF1GcNo^Zdo`&nJZ|~jhPR8;7wSQ-S5m&-_Pgjj4|H<`zevv?-H_QjzXqxP(7q~@`$My% z4~Z8PQtcH`yJfxc0f_qYO7!rB*;(1N(P8b;nh6gDdHiloIyJ3DUqt-5nbwr2S=p2c zlv3xVtlX*iovA;tb1jr)Z8rgPTNin;>Fpu}sjbzmKzRC^wd6~3*KLyp67>4X{z-JM zdTmDaS=p6s32A%_Lg$CR&-T}uRyw9o(?)*xg#ySn9%woZIW3rETYP|S)WV9f92T6o zB7xyhzw0)?CQCE7fConrR&keOST{i&3w&2t=H-P%&Ul?TE+;#619p>+Om|H!RDAnb z)bO4+x=@*&0_AoSr!LV>uSPEZB$+bfeQ4(oH9_gfHsGdKa$ z8&8<16LD}cA2o`Vl|NdP`>HfH7TH0RTt zYahFs1!>c8Jv{ZM(Nm4?)Ld zJY|TXZE794XTxTc?eUSk_1n{B#$`+);`lG&`aoJfv@Zpe8UZ^sqz5wy;c*_1+xJ2d z6=%d}*G!E=hiEsg1E7aYwA&T!Q~9@hVqdUkpDL0~B$*KPh7aREtC0Diw4Aha30Wcq z)SY44oPGd+Z#}vH@R3h%I=xk`Ji5@Rnyif;({N&57lKdzvn#3l)6Lpm$ zVbM~dJi9WI=<1?~HGn%luzMIUL5i_Tf{DI12)b`yvn@LeYnLS1((YP;%k7M6Z%-;A z>dlx^7J>*Jivovo)}ZJj))o;S0whJ=!K}glDhHa?wdFg2DA$AbXxAbVHCFlcJ6Phw zL!_0%gI&*fk+zh%_BcqT3};I^0aYQr1v~&@FYt@hVUa(bx^y>oZ2O>EG09|lb?od@ zo6<>QI&Y_|$4)G%aU9l=;4kR!%^Ry9kQ1rFGMiCobDIb<(hnW=I~BwCAA!m^(nzup z&Rkbo%&y5S!`Yf{|Kq%}I^58F?uYM`ju9kKUc_ebPWU8=wRgd8Uc{48h4$UI=|~4X zi3^UTsWju9D%PS{SW)rWVJegS>nmrMK@}i8>0&wEy=udV%iJ{#PM(6Ept0)#NkvO^ z@6N26>QXTUfX}d%T7EC)P+Dd(o^;fRdVTUx|4B&v%t+hX*GIb!<;rFiJ2n0qYaX#| zzkCrBr>*2mw^GHPMDffVR2G$$yp4NK5PBFyQn&mjtqfSpSQatO5Id%9 zrn<*0d%x0tuTWC8n8sQs&e`6sLAj!@TBkktW=n5v@AEGah;Fd zZM;+Wl@AySs9(SRL7e;Z-=OKIf3-t@d4jjAy1|kpH*U`kxo;^jW*hp>i$HYLvW5Ya}P3YxAmCN4coavt+{6A28llZ zsV|d~4!EZ-pG>1!P|ST7&{AUw5sw*0Ggx4tq;jBV&i3QC}=4Xt(?)-Ze}B_-(z5n(m7w|{Q} zTP{Ouzh0`t7xlF%N1dD<$|fAR?N4mp6SxnXq!rX1eFaN7reG%0rA^e+U+8{fY=k_8 zF23`%zQB!m>rQ?RFzkjyzol*uVOn>J30&z;e$c~aa)~;6^zp6pU0m{;kMEZ&^>igV zN-IsS^$Le)f}?l4o0zacnd&l5{kJ+LoxWDqW=-`*(Of!LyzPQIRo(|6Qw*IaNZ$&0 zm`mtQG4`{}C|l&YwGLE@#ru+!nWD6k6S$f8E<}>Ms;)!t=3Z`M)%s1(3Fc+q^Ay?Q z7u^=j&3PU~T_EBPFFXip_NpT z6*h3|ACVb5S|avHRvb`_xexVLm5iOu`*^6mdLJe1H&U>)ej9tz(k+YO9->)Vj|p7U z(fe|r)4{^<=`>tfG8AX01gfsiBsFl9_N0i;@+K$McC~5B<7X*_rxkQ9Y0J&Ij$Y;l z6L(Wi#bk+#`L7aeG?FB+_iYbUqLm`!dhMFi-cu67q8e z$RfD$KtoU7gCcN2j`$CK4rpHFfSzJd;ye7MM6^_r*c>|6oCG#XQit*;-slsY=WLYX zu1FZCbp~m>XNJu-QKOoaq20adZ(-Xx6}a=AYZia?oRx*;3>JSckxJ2yx+V?$5!9Ms zij_t_n+j;u^nnh=Pb>hu2Q&)7BToPSKPZ-8`MlJAIb1I%Mo}hKEAFTsD>4yY|2&&k zW8r-g3Xls`-K~TOj$>SRnJ?9!zRYd^t^H3D&r0PzPndNhhgp#wj1JmhASvZTqU7L|ETE@NypGvNL}vexOmb9Lx*@S^j3M5eFz2FlnzBg zFCtA(QsOPMepg-N$|h@RBr97Jqq1JmyRVu_w9Bqg9mwu~_tXe%OzJ4UF$rbr*ou#1 zfBn9jp1OD|O~#$XsVhoH z$inM=1p&22FCzzxo}O8&zg?@P{e4$E7v15`2F5CJN@gO$o;AOiTuYWj$v@G>cs(oC zF<8c-TttMy+(6qD3q%$(P_dj+2UI)B4MW65>RzECMVgrL!`i{5$OlVj+bvBXkk2DK z`DpSsUhStj^HOGv9a^_)+sQ7_R&8(UF)TQbU%-`~u#V{duB^ffh`7Y-2lQeJLCW?z zj+@&Wx!)qmAk3WRM(8e+tpi5xp6Z$no)@kieL7HvvjCQ(7+JmX@!{!z^UFCn)9Qli z_~zHXYiw-`rGMm zame2@|KlqSyrVAfcgY?6<Ra|d;wC0u5H%~ZV{vHOo8DQ>K|+Ar>8LO!8MDsR+4 z-;IjAj1e)X1v2YnwOqXuDl!JNEEKQKqzo>t>m)XFKGY@$A|eS%k4dV!d*Tx+g;IHG zHvHZR!UijWhUb99H%$Thb=ivL%RmMb?#)&;=+kG~8o{b?zho}8nAn6oxg9M5qx9w$ zSZHV-`3LD8C*G@X{Kg3C+O)w(p*^*Jie<$&B-eO`7yajrstm1FuX*u2j8?h`EyF0# zNVcWqy_}~9g7O4bP1p(|<5h7ibSPW=TS|N->a%yV=%b4*=$uHU&wf49uQFa}I{ut& zL*MMrVgl9%%uS}qh!0+D^i7Z|{@kegqnPIYp+0RF-i;|1%gsmu88gW)-B z8>YFl-40e8d3dvuVr{#gd!0aB^n-r7A#@|Pt5DJO?1{xPKsjeiOLEO|BoB=6$Y^Xr`rN+(MVK3% z)_TnY+FhLj+FPnOpZwFCzJgRPi80+ak_%siw%jd+`WAdC>N1M}=xsWNM2V(2$^?27 z%^4mjzBNIoOKKUp&7*BiQ5MEd-vfONuyEkGilv87g@tkZXHHZA4VAvf#5?2pQw=#h%>RkODV}aE}X**GMQQ;3)i$i;4bXcBgezH+q%7 zy!^TkniQVKsUu+aOL;>-2Ss1wpzb;pQ99$ASEW9b(pq!toouD?*(cf)k&fFE(CO9k z6#Jp?6Xyep`!IN4!_8s%;tSZz->z$e&ESM+YRMV+xu$9vIXIY0zprXoYQcw*9J0|L z_|Tyr+;`_}{{C28+{62klb|*^^@f=ycw|UW9^nH1@F%}OtjOulyYONesf3Iezgt?Y zB@2nikO{NRFHgUWHRHO9^BI=gS-#7BE_9~gs%>|^+B#bd)J?gyNDcLv7RhqwkSsLk zOh_l~C}%W1U?ab<8@B=sl9VJjpKtu3JEwD=cdC-KO#x_jvI@3&!{c1;MheXxD%<3f zqugssvQfPoJ3g1RYZvB{3vAw%PIc}J=QQ54BVH9)gR&Q=jm(3mt=4i5t*}%&eh|d2 zi-5ryQFL+H=@P|D|D?M?V5UvnoYX{`rVO?dA7s~ceD}d@N! zxXv|5qnYq^tctv0M!9$wza8sd0sZsI5n*x>aZgV*WZ{@pQyJ~SkqOJfE%(!v+6Tr@ zr^Z#j(4i>tE6Y;fT&)YfHJm;x+V0kS-c>}@YOy8Q`LB-j^It!4twql|aq83GEH3d{ z{{|`T@8O#J(K}d)yKm!#5;G0MraC~d?838l08BDFM$Ug;QKUa-=yJ!0LqWFUq`D>T ztKFgj2J`I7A@gkgs*_C+_yDYYqQx9iBj}r?gf~-IyZ-&1f%no)BEMm51c1DvIQ|>6+l=yQ{yi#_VM_$h%YvjAX4LIpEaH(e#@+q34)i;^!Uhur=17 zQgD01PHPXB!oyRopHbhi_7f@e?&2fX{mbRvDd-E|Rbmmx|DA$1<_W~{T&7}ki%_A!te>o*7H0uOlTg8Z?Av3Wy* z^hwce@wxvr@H@A~+WdPOa2snLtpMHAxFF*G4k>lmsEW^_xsnv<>4_d?P z3#S@MJzq4DnpMSaQmmR@ZB_c(j@LU;%d+%a-MF!%E8C(>r*O+?!DRz<%ra$87QBodxZVK_9kIod8s{}=x4&RuIv zv0Fv%X}}_%xd?epe~fhHHh1P}Dr3!`V>BFdR3VDJei!ME&5o}ap1{HtOmptK$D$uP zxxy=02N$6=lW_%HsxLh8vC5cm5iFOanCD!LSPM4`{Ox8upUUy6fW#oZPBt6Q5(@OTR~AkU#dDxTa&Y zX}r1YU%5PmCb>piD~2oJ)pgG{%H&jG85NDjdzw)DQaen`Qu+n0XnSh5nH02!Pb6{K z;Pa{DDP85eM(zqzjaKzZu?)vz$)Uzxo%h1Oe=d(2m2J2|2fnLH18@fSC@@VWhb#&= za>-tDrh7*qTER19emYgKEmLQYT%5pX|N_0yJn=_L4l&0Nq=ChQ|v~Zhx^8S`2>4v_&sd=B|r&(5q z+cqcfuS7-z1F;iK+71t#a(-QqAC)oD-o+DeD-sL~c>L+QG2wLB9L539^Qe#W7fDPP z8?aeJbX5uwb0+OkHgB3I9(rV3%0E0COsXH3p^s)eR}AmyZmvArF4aVs=Y5f(7`K;f=^LM8q*zUdkeFR^8pXiDH?9-#+xCzy`5@8cufBCrdj{BB8 zMa@cpl1mg^HCf@QH6gX)#?pC{M>48^S}{=+9W3D2t%U_8(^nWViEtLJAf!!MFdo(G zb0*XJPlURgptYUqmCI0rSL4}Wcz9_bZ-qOcGo&KbA!B#cnK|J#c$zxvcs#inos|Ar z@NGO3bxv~^qYGsM{H(sr4jd$eceP3LQe-;FHJ<_~J%pF<>be^6y$%Z)aiIv-n1q`@VkHwLfVBpZ|3$9dLRd%||ZqB<^%2fZv!U= zc=?sKoYQSLYae-+gIm-_rp20RjHK%jONi*Em1Si?y6{PXkx^O&t}rME!v*=#0|f6DC1^u}n~NKwLUs+!&W@fLzsWL^f`?_kLFU^--EAkJszvtBDxB z83)F^08*kwU9*BW)Y$N%_9nqB@l{AP+EJIS&TYcWUYc>g*`mB&tMw}4PbcI93Ybp72J`j~*m=xyF= zzO{BrwgOI`q_i|L*xRSv8#>0E#RZuQB7+rteKvi4w&@f)4MMYY3w!XkFplBwUSr^7 zjy#Sns^LsHtJJ9DU;wpgyvCrQ`m$d#UK5Zj(h`(l@av+;?HF3{!?@^^q52vd@1!G5 zAdo1f0|2kE-eZkiY@7J3wV@BNW`CXJ8z%D_6p=xY|U;( zp}m4{d`Y^IxXc>C;d6LS=}OHj>fcjd*~44NhSbCA%);E36Y#pXr@OtGZ2^R1{%ruT zK&w~*pRuBCtk7 z+-p_OJpF3=ep}~SWXVf%*R&eJ70%@~?X~mCFOWxO-Zqrlmq)fT4O#sWCL{}>FnKZJ1w(yqq{r|Vrwd>!78E46B7zIO}i&pp`iL?Z?SLw ztgNYoMJm&LRvfS4fB>W4>NkZ-zwrv6St8sJ`GHW}H-fZe*3DL`K$eal7)dZJ#Na|R zkkvl>dX{w4a1tTTHtw`ZsDi+e%D-!2Flq}y?M(diQ$pgMM_H4IL2I@YNkymovnF2ZA$5c>09O6rW#4-1na}yr8Pc@3PSs=F0oFK zttoVVk~|9EM}lZH1gr>cp;;*mSa|B9LIPu~nM*}GQ8!P9ZsYV{LhQDvBg-*LxU{G$BP8u-grhXmFf=Z_ujH?YBX;!8 zTJhS>;Los*ESXM=ZSzF<@9=JqkNe>afNml&9o;|8_Pl+)*~68Dfr`~#bEB!~B&1BV zS<)AUkMyC~iAwA%pwnMfS=0(!8Jvx89NGuQ)>_nDxZ?W=u~EFn>Pi{z-JTxha&t%_ zjZ2!^=NwIEY@bXwWUk9+ZI5HqAzh=3Sw~5Q-}ZKS%?dlr{~YpKs&ce$kLk$-fw8Lb zj$HhSC|#apq^IrJXHHl9!56v!)w}nK`zOZ&8P-TT-KD);?VU2)_9m*`g6o}nUd|BK z@Sf{4cpnu4-U!Ipl%Y?3aqy6zxviWf#dgYvArsn!IqxCOt9}Po++T|N&<12z3=bW5 zRLEa)`^~XwVTZzbj_i`Tx~cgb=h^##+h3Yj;}c=<(gttmZi%OZm)nm+eRbmLsszEa z(4SSwU{lxEdegkC90L0nh|d-Cypv^iqm5=@x^CN=R6=U{hEY1(42(OF4^lwQW}-w3y{`7gp3>c52D|zSvr0@YddB zm3UfUv~GLMY?RM*%s_3+gKtcR>R<^7cnnz3s??L%wOo}kMkLe{UBy9ig$CCS1LKjCk|K@ z2vy$2VPIA*dsQ{g-%&DLj4s?O+qYNEoI_$e-Q*J`=T$qb3Z`+Iv-Rgq zOBlzXal&SWhodw`TXkkSvj;-At0-%U5BP{BxswV;)$p6ns@lO%5O2qjLtdoPlJDeF zP2*Qqaz(kCxtHp8#@Q23O;{uA^RP;nK zK~dn2wbCBfGUfdhLiUs|kn5tih!c6+)t5C_(hx)8=L8chmbY76ZfsTXjHWmS&PnQn z11BTaE>bHQVzMVi?*>004<5E}fBB@-*?IEJBDD$VCC`%`p3Hh6($g~t2CW!CBJwe- zMw2{J!Abpovq|GA=xs69K2V?POUZ%IK+4?4biLje>Hciq-hVus%?DJ>s0pJ8j#(V> z2~^_W*elSwGn#4K%tuRP8t4_N5TS41u?(_R$VZ5<+pe2p6sh*s;@L+{Lf?E$W!O0T z{8u?>?63rAsO#$4!O34p8uy`d4Ley$A?~T|2}v_9UqJWf?d-FNv?Fy7nZ+8M-_(58 zDib6o;ChgYqq9YU@jD8jnh=xY-V_z|3@3Fl-{5tfrk;vo0K@86_D84HKrXy8y77il zIqn&AK8OJ|G)6zQ6$!s5%-`U?%3EEp}zaL1ksq$Z$5BB8r5P%d6E}+b^wK&`Ce=B?UrjO{(8>Q*r>W z8R+Rf$izQJPuPVE5#7i`RAscki@vw`G5T0dalyYaOe+(ggx%V|iSnFjp|A091+*Cc z3`REN9Rx&SKGh)hzj}cXq`=?`*&FD0gxug$=oEyG*>fGlf4L12G z6;moBP48wG@VKL++`Q0-F#}dreGL41I}1?_A)C4s{FFKG+#BiCF|nP5N8hgw!Jl!b zpU5Pep|+UqJq2G%ZM;YvQC0UmGF0n;25aCN9Tj75KBPEm${Yzj^@H93YBzI##mzUr z+G01!0nEP2qV-l?l5O2^#*eb6>DRs_S+O2QqCP?wk12{g!+W{CRMliI5O>ft-qg+wXE+zp$xIL z#n$*Qc5z4U=8o*sKNy8tpR~0{^fts7_CG!Qt#4eX{TiiqItBWjteliKYx*Xqg-ELPUl>ZBgQZx5(rbVK>l&Bw%71! zvpKB80g5bUvwL}(hBWHtURwt_!DA-s!&3TqExs=+C->@};FABrR<+CEPD@SltgdGnvU9>>wTwAXmhF0psK0n5bm?i1k*o zm(6d`(Nv%g`r}7QU<1%``QdfJ-h2rb)%*o{zw=jG=reai$BOs<{B@{RcNIbfM_xc@ zkE`f*4BeqoEK)k8y-wK%=Xur}BAFO{3Xd{K9fX@by$L|2?ICfx!7DY0ucMuA)n0E; z{tw3Wz+3N(5n!b^J1&RQbRT{!eX>1w-c+!9>Q|}p_IR7p9xw%VwT=iHQk+sk)M=yr zcM7|zw2EH-KY8W!%dY>Y&rh-PbFQSbh{DEv)C zWL7bO#AnS9qeb6W^Ic9LubKw5U%Mu*n+j zI>iw+v3v}D&kKo#+Q!qGSi-E>S)vqI_Q@pB?!{2Q%2vl+#> z8k=2s!LAe0YQZ7{Nnn=k%zX&s?vl^7g(1;RHVH^o|1A>8D-D!U9X5XNbEvx(&%ygA zu+DKPcoz8Ot&jA)eS=x{P6mG~%@*KG*-s2y@X(*-qp9k8M`;=7jOLhc36XNH*#M10 z_Y3#tDcl!gnD{TsZb+_*yQmC`oxQ{$JQ{i!>{x1Cp;+BKyz;YAP&I%yk-@8X$E-Ez z)RbGnLr~C$LKeX&uY9~VfeTo7jm!`6!^lE<#%`khRsS*ZZ0rnUo}}jGlL|k8jFq|R znhwMdy#6?n8LqXYqID0^Wwyu8V&UD_56sSVA-q9FUAeGxQwuu5&)HVDiOIi}iDNvn9g z9N+e&7guk*Ig+UasMcFjbA@AmaFhuHsycXDtng4=A&Q4m#DpL zWsapUKoP{qz<9x+r+P)o*(@^fOYE+Jsnu+`RRlmeiBRH@p8iZwIlMGuTuJY;w)fU6 zUu@vYGj`ev;1^JR;-e;7k`cUg{$SSSaQp4oX}M!_WI6FOjxv`Vz%r)F)!R+8>?ZSd zb@_^6NMkGWCpFprSZX0j=?>+kR8Q~jxJ2l4buuaam9WkLC)r80jiD@Jowq-&B-P8> z#h@>L5073Fo#ww!xoldbthNwXO33tT1*%WM0}h3 z7+*{M5#ur`ByH+)Z&Od8L@_Spf+;{g$s<35ayd6&^Nwa@HxE8AX2QhJf$`(fwWhs` z8c}=SpkeV!UNM5#_Tg!c*+Rh~KBnxhg`thBu6o0;rUQ7qy%PUYMMaWPcA!u$em>wI z46Gne3IpoptsS_is}Y6atg4J7zJ}n>EQSVc`}t*hjRU#sI_c5XTa>}?9iz^2HM+C~ zUlP_^J8lIlDCf$V2xPz3YR9ObP7b$kKJPQ%gZp#7rM**<%vsEhcN?`ze%{%1y2Z+i z)nylnUoL-4e3;f#WqO}~3M0-y1+O`%wW*s&ZQw+-@i4w)3#F!T9gTEs>Rhs~*#u`n zzX2OfsUE^4EN08z9K59^_|vQ%d?0LJc|a=7DnI?%IDZ1K|-0q_<-M)6DgU<=N)+0(IV|NOp3*%qbeVF#w5O~&%N_#XUp3@ zrnn20wpj<8yySyB$f-X3*rndvsS^5INc5Y}yZzUNExKeX=@+dV>@tz7vl`}qXe1lI z00#cD+=c$DdQa{tW~9%kPwwN+CDR`PihgMDP2YnHxw_S$P$z3EbcFP5Uzl7lSJK;? z@~Sj&94(Nb7Pz3;ZZS-lAwN1fe&;Gi2>%KD9c!JWZ-e)HruD=?$36MxPHCi3;ZMFPah91g4ema3hz-NeTxY#t_X%ZM99v59pH_Iy1>; zo}q9H-4kA&_rcEf%$k;r9~cO-Ooa1|Mrg^z)UKcMwu$eVuxi|!RtxKmw>Ctdy0faO z6Y@yY6R95KcpF_2KcVj@Y7M+DTw7$=$>BvS2U5nFi(b=j2`Bg3-4poaKps zj8a&EoV*(puPjpv%274*C_n!rHIP@s#MfsMB4}i@i+7u`w^R=s=*g=(M$bQTxoc9} zx>*^&H>95@7AA6gj`A>IH9m%0rf2!3+wmMtHj`Fb>etUr!f$U3KFf-NQ--4)RxFy( z+Bb7NlHUJdRE0|1{W9>L_-^>Um6rEMs}$_~W2e+RHy@(31ao4OKNWtrLCdDf~iob<4$DrB$hA^0Q$RZE*f^ z;Wy=k3ikZ-2uSsw|QH9B7$C6bGSS2TXc-vEZ`6PZJ40KS-qoWFD@EMv(?)a(Vwx#23 zkd@Ox@(+d<6G_c33L9_j9}Ed)A0NWiQh|3bPQ(4r8$!d>fHZS4 zDUOWWY&(B&D;%-j<7?CsRr)oV0BtKfYcynLdJUd*9=J0Rs&pSMphN!k)qo^~lTm5C z^$HN~G$^Q&ExGU8S>cx>iI1{ghBy!bRqk{kfl>C!6(Yyn+#OClX;xzUiA?BP+Wi>E z;k3d_Ghg4@XY=R}tfkVfR@Sk)vlN-bKhAkiM5p`C8R*9@?J`o?beb`io9wS&iJ<@w zmlQzv$4}mH8kxN@6RKt2Ot6H`UUWD=cEgML(K765xu9-5OQ)q?&%1Y{mF1c95Px19 z)sRD(vZmU=q(H)d#k@9UuJbhtobJpfgKaQ1$1k-XL81~CTvFN9^qr*pL7@kC$X^t* zJ~=A*P>4EnnNa~hQI*SNEkCHu{l|jlDNp`4w*WG7;Cw6%NOv3INA9zQ(ny}R|C9M# z^+?zFnEtTEG?!Q+ft6(}%coTo$8>vz*>g~x&eh)OY~=Vv)Jmh>Q*|7jl=A0_q|jSY zW`KiSaeZv5)W@buXAce9@MNKK! zaO=8Y()7>#!1PmhJx^E} z2HtkAg03~Eh=?}vg{C;*CK%ffo`x-D%E_Bz?9WCj(X>h4_*1`#$KpD<4`gcg8V)c< zk?vPKIuYKo)fo>pTO2U)XQ2x}+uAs*c`;kw&uLxu=EZNP$##_8^P^WrRbwD)qlml z8_~OTtIgsu(wL<=neR9GbQ!U6M%eY||7{OQ;7MYFN71|5P{9l1GwP zW`+=`_P7yS$*fH}*xu7Id|UplCG1tr(=yDWXRdjA*lmWH%GsT-najjxD4$R5vvg*u zz+4}y`NVGc>j#dmyklmC>4@Wac#q94e+IT64wZEBr?tJI5bCWj!C@vjTC{Ve|~#!UYwS-;-|YTErj3hGF(I z&b5)cbU={SybDZQm+?#T3P3@old`R5tq;n+f~+4BV6JJaU@Y{9!wOY-+O$3kK-?id z?Ze^x$+Y&z1sorc8u%}A%uYS>6m4ziFA!x1x+$!wN;FvJ%mAjGjF|~eapk2-)&^_p z`W3^kuc-s7nyPIqy<&}3Js^l*W(gWx6E$2j<~KvZJi%J0?zbOikVhiT&sdxcE)ygH z;3t(0IsrrP;OAb3ry4!TXoh(48VyqDK$*{}d+xBfqF3F*dO+|W$A#d3O0)i00hn5RZAQ_qI`t4jMdM>$_>CoFSD*HRu zjqPg^`$kb)82dgUqeB$^VnaA^jW&9ImO!lOFwqp0wC%+8rC;8^S>yeXE(1Mdmp8dr zhy1LuG5DFJN2#hMmfN}n8e;PHdg~+JifU~TI2EgL-lJl>GX0_u{l8!gQfhGx);yy`L`cn?K~Y4>^s~O>>guvd-F;^A!rm%b1(Us2v`|UgDXuHyIM^ z7L$4lzfSvp6RqTw;HC1D)nW^w63#TPa<&SX59U4BXmMWFW}TrnG0SC9mcP$)Wl%dg zq=NNYCsVEIwSiF-0`uE2ux86^Nc9H>lOtlGd`3Hx2kfMP!&(oYjqAzLEUL~pLB5X} z?4?ZLUQ_Ubqd0wUuKstdw;E5H)D^fHDBjSG&6jtx5Nf1PKrln&R5eN&^M;)DK4-;U zs7Z9PnTd}5pvpd0^F}q+WkMt;|E1M%Z(G?u-~${Vt0Z--AbVO>lec0Ydqb6NlQ}Y& zx?!a{B&)tYq+_Y1B_#86&*z@SJ>5?&W&z2AOu1E^iK1?P?FsN7t^%g-HJ$U-xZ-$q zK6!gKi+d+mw!4|{%*bH{`zbVR+J#CE#8BXyj;sTbC$FjzJ)%VW1{#XuQl)6Ly^@ zT08E4o0mvK3}3blkEX!$=`eDicM=jE+9YP>KX`&Y$YY9e&(|-^p=@M3=3m-BXlQb^ z7;r7RYoY(FH`Q!8OJFEefUW*29gHW-D)A1@JiV0*q;W51e zo2XS$g&<9$n1Im`xuEk#RUrJ4;!f>tN_ydQGauD3BR5FpgkM}Dp8@Gr?dw~Z@AToi znFH-}aGI_AyF#zIE1lV+9*^~W>HU2)B@GRX34%OCJ|fZ5RAp3uu7%6Dr@PlpA}gJL zOz!Tl`VMOdR0Ker&E$%|dVY*v9lXtB=iqOP!92!bm>UMXPp^Q34h(^aEtlxD>gts5 zlQ3^+N%6cyA$%?`i#k)4{nAg7w0w@-)p(s^4T3Hu{lM;3kO(1*>VTh%5+?|ct?a;Z z-VpxFmiEY%&PYF7VS(O{Eh3JLomrw6(czP(wRJixZE^7g0t;`pLjsbTtu4Z<+{3Tf z0O{AlCI}aW$eS<1wBAFbig+1Ymo?Y0{ZJidS>v7F*2=lBoWfHAb_i`-XZCYRYVj&t z9gES|Imph{ZVLbG-lmiA)IqzXX7{QGz^>B!&QX7;4=(wwzV>?jz=72y&AxZEf2$`{ zw4k>MzHw_cw+=X@%>;>(X^t2;rZdainO}T+r(;IP?%uQJtgk)^q`7M^In6#BYS}J@ z&WGt=kfo#qwibyR?U$=qshKmkWcVs9VHymI60Tip_aR@ zF)USP6~?TnSc#GPsfMdnCzJ9E%sF!Tg4OpQgVXTGd9%Rw506_9c6k z9Bj;(AV+s`WvVwK9rsSlT3^Wa<&gc2>d_KwIe(Uxn&27f9H{W2rh5Z~%)cE7_oa-O zi=Ve;wz1JHoOwo@X(23bq!HBKR9y;8`+?^5OQsFHPkRG0{JdxBudUtEbdjkJDdYZ< z7I14=VeI7R+$L%TUZ(*e=hOOV@WsqMH(Ot$Jq`C6Q79sLleV`;;wfJ)JP&%|3JOGSd__L(F!tqXj_c|l%Im);)wogw)e%haM+siob*Sda7(uFN_|iUl z)pf)Ltcv673M&@uxikrxW|b~t;=5xStH#%8sl`%)b79l#t#EUF^VLHNJPPZP_Hskt zG4S;s{T#HvE#m|{T-l0`wd&#;lcr_HCMDC^ngG~#C7 zCig-}Gl$UZCXN0rS~@w8x;9|0s_T%jh4v;}NHHdU2(4}a`H4Z^WAhm7?i^aFJhKy} z**b=!)*{lN?USeX?$LNEwH|%aQlLddsdS^;;>2L)C<3?md_VE4a#ntFN(LOWal&Gs@e%^mYS$+dN=nP-B%efREr0gQEdJgL;cSY!gubK4VPo#|R!9N`pqFis2a zpofkzg@oX17{>rEhH$IZtxfa5Vyy%8bdV-gBU5&j02h}U(kl!CxL9j6GU;Dh4--o2r zc(Hp=T7Q8B`lKIB*VI%_s_DL%&5d3e;htw#r}B()YeteIQ?T$)>O46&@jQa?} z4^m@!F1|KQCE8}ZLfWd?J4<5^ut~1Q*Et84jxOjZ>xeb=Si{sFH498^R*b`6MKbMD zZDSj#m$y;0d%L|Y-NL(&L;KHmlyhKVs}y=A9@dZ&mi}n}9i0A{x{X_5s%EoIJ4kgx#SM9k8+L~a z381#2f^7`POpoLqQ@xqVX7&*T=+)z%q$V2vjV4BZ3;B8&SAC>Lr?SE2vhn;)Y)t~V z^$m9Va+8~nzUWPA(f83e$hP@1Ofkn^E;id=@ppgP41l+Pl1mrOJpa7$kh5czGV zVh@Hsz!e1}x#pY>+F%vO64*1mRT-`g& zA%!Lc05U`d@P@`dr+x9*Zo}nK!R%6zuCnjezw;rN9BM9a*rV_B=^y9OvErh(sjqhs zc*gam^0LsiVAPHoj&}(hQ?3!td!7Hz>dA{?9`f+O*Ztpo7r2(Zc5@Kx3=93=sx5vd z1znu9tdeECC>#UbzhX^vckB+(i~Oihk!&r=Rlucaw?;!S}DyZ$7kh@F@6C{r-<8BLDTT+^&-A z|87$Fq%EjVXDRmw1WNMXCV&M6q_}fGo9(#-?=1bKa=^d3j#&v*2|mW|M0>eRacq*kTa=*v~h$P|pE zNB1Sn9@~zp8m}sGYPDwwZvluX=cb=Oftk6wae5ScczS>A3vzuQkb>B{qkWdmdQM6X zaXO3MQ*^qh`itGZ4Gu1B2OnQ=gD7t0g<|bIE?xDuIL(19YGhd(C*PAVkxGDtCLeL| zda|L72DP+`9k2diL(W#?AlMZ}0W7JOx>Xt-B*R z9LFHx80S%5Qr{mLZF$*!m9uh)A=~x$qyHAD%=v#(+r6`q)uc(C%?`7)X^7afiafLA z9?3Doz3G#xYu*aQv}VnN@MW0SP}O5mM2&~6=^X9BW3-FiS@!wY=+w`5Bb5Y6hrrX~ zkb^2~MGahf{<=Eq3EM4_U-*0T*roYpt*-k{+w1sy(y+RX8_9Qll4`)mA_# zm%}W#6p9u9sW)5wZ*}zuTpi-;-bro*-Bmh~{m}o*_o>SSyjsAO#ql82ljJ09k zEU2Sl);PuD08*TTtPAoFIZ8DRvYmm~ zs$Zn;6(tcI_fBS!*|8tdkKl2Sn8poI3MgFi@HZR|_pU}}K>8N#+uGeVp35??fBTB! z8p3MM`|itgB!`hjbRn_D`*UM@NWM7&}5{*y;2v_-gEet7;+r6Sm`pz@I| z{bIT$>JXNHsW-a+EUCFR?v1xS*B`vR_wz2YjxCNhq|l_Rgm}aD6-;%EL%#R#vkeSU z5AGcz%VdZr{SGYIpSi4&!`beax$Dh`AZ3Ohnh0@r^6-K3h<+x|PBJYunZNPpwrdDd zrCjP5hGtphB4Zij(kZp)vn`{)_mm_C7OZ#^uJ(hHPAN$Gw@dE()X^zqPt{g|N1bLR z0*;1A0hFcZjJ$@U+KIX!vnnDESq?tg8n3h(wZmW1NZOF^|5Zyaa9GfGWmj~8tAo^}{^FIyPZn4Ch1g8>mRchr5-E{Dp; zB3TnFt#)n4mYPX61D zLymFj8qimYO_~tHPUv)y-W!86&X#3Vd>HvNMzbSicDxZKv%}r^;xE|0;L2v)!t(U= z-ZVi-a)7&*W5Nuj#a6Zlg{fb%_Vy8Cl+Lk>jha%Fx=N|ftnjLQcJfRL{Kocx0Bh~vudv96B)YLG!xGn9*K>E0^ z2Bg>-!$5|?uF^_Abg+i-=z|eu*k;Fu+Iyhm2+&G4&|BhB-pZQ}=m~s~5R*tTr&qWk(5vRD#m-Bb&a$Q8LuYrFUxZp(<$FG^*WC09H70 zhF5*=pGnXuBa?^nHjsuWZ)(@Q8Zdw(EXhnkPF(PP_u4wR7~e^6s}En}`MTRC|FkRi zvi9K}vGR&B$Z%~(kxxHxjt4m2HJKdg{N#^fPiVDL^z)X!jqR$n--XxJ)f0Sj=nq|c z+9Z4%1cDP<&2&jt67r=mL1}zAjO5VRmk4GdXiT5ctFv0+F?W>YC zg~a_PC~&%_UKG0A`ofRsmNYP4LetQ`CHYQ^US}#?%b)g3rH`r7zC&*+<8?mN{TU=; zEgkSvB*;{3sAlHnXNAyaTl)>bWx)tzNC@Jgy$*g3rQvkdOh@+yF%P) z)!@{b+_lU^oBB)1#1srdb$pw*E4Y*%7czLryG*r*iAUS-*Nx~C?PUpd5l*?#d^`#OAi%iQnDb|x8v_9@eW8|w5gBEji;S*!| zg{;9*pc%99U8Z&CpZSEzj0G`nmsWQ}Kcy!%2WBQro`gA?lPCngTT~t!7Nu`iQ}1l{ z@g|DwST${~+s)JNPbHEVv&_A4`<=8>6kfP#=(0V86E4-euBSBW8P=v!uEtj_UYNg= zH%`%iB&nZVc6K%Y{1tQd0T4$BEncIZGzPpZ+SI@Cza0-Q)-PXOO9l^(v3)&@PhEjj zS9f=f&A%3$zXG1-hDh0nH8gS3C8*HfZayZRqg9;N#qh#<`TntR0u$n zFKaCuhj=FFv!t<{^8tkKV@Pg~-??!*31(e})(CRYEKg*=6nMvD<7ukKRSkL*Wn%k7 z-0NXly;A{v&I36S2F3o+vTzTtV&OIo>`&mllTh~R!Bk`u$Rt2)pm(0nX~`c@lXmf{p3tUi+pEa22X zo^k`#6UGp!s{mGxPZ^wp$G4Fii5>i~y+D2ab-G!{*8sM}k)OzgR=*Xn2bU{2%P#=jD(q6C!rOUv@KMFN%M z&9Fo7F>C6fcgTxB`pDD^sLsPOgv-|i8tCj_aFiiGH=v}#-8-lY#=s>%<`)=7O&ehh z1+e|H9uLe2Mwt~=@_d3PViFh`?y?neqGlmDi(><^uHuEX?mQ`883^N|l&207wmH*E*oWZIe_bQ~ucd2&!|%9BGs(0vAHw^|P2 z_&%2g{;ziXO%-9auZjmG!uiOP{x&7Hx!l^7G$9UvPLbo{cr#zYl%u}BV9e=NGxrk! zL$*m~ItjtrQ5pr{smyTPX{}M9R}RWxTky)~IlZeom%RX)#L+RM_e){Z6-WSsKb~RR zb3$;SV_^QUSp#3GN4tttGXG3ou=f+zWqbc$$Qx}yL1WWJ`+RMGRdc7%<=4W2m{-3Y z%$AH15)VhteMnjpt$uBe(t^|FHGv{)Q=Q{#>U7luQ(a2f>9ZU?_=s;FiRMSKoh*sV zkGdml%@&54+qhfG5PR`xXHz)Zw>RYAn#xfZ0jjpn{$cq&cjFiL)SuM(fGVjwj#VuW zN`3FJ$Rtp#l2S2aeJ>BbkF4}jAmHsBUG(P6ck&$FMIBMv1lzFrQltb255Am-@_j}7 zI2>MlV-9*xJpd`{PBGr<{DZUV=Zv>JpO^2`pCs>V^*M?igndB$Y8>RK3d>=kHy-|; zNxS*|X$allwnIj9tze`~*Gt!DhVZM~^z2k*3){n7@*;^6@15u^WammH9|}X*-~6^` zM$sR3p@tQBOAE^fg7sl0?^L1+gPZvrwbc4yToYfMl$T|z|12d0q5%wTFKg|d+iUXQ zo{scv_myn@;@#vSk3s@pw#w6M8eWE}bGZ>p|GjS+v!O|89;rju)OX5qyJ$hSkO5Kz zzvFcIvg@QA8;M0s>N%UP3_W z5)!u4>$Y$A!~LABnZ5R! z*=uIr-#kw_$s#?~M9lEt5q_QTOEQt6hDq7!->}F@l!d_@$EUlKKaC>I+%fJ6WgaDx zTLD;{D+Lcr2kj=tCBv^v;;z(P3&PCJCfzg>@$%Kf;i16^6Ju)%C)#=6`S zi*=-byP$0oZ9ceKHGal*w14xrc`Tl>IqC}|2qWIqEzJb6ru8&&Z?Q=Lw0NB<)Vy3M zT6IF+A%=LWz3PG@M6j7Q_U7(Ll&x^`xCyU+u7Jc`a)#|p5ZVE(eY`a{XQ1gX`(EHD z?;L&1iHHyh^|nBbgB}OVJE|@MciK9EEFNlOhV&6NBBjR0?mLT}pEMg+OL*Lk>cNi! zgx;hVy9f3^!-a;b&TkV>i-*-NPZFjg^ zOJ(s7TorW)IH9G-GvQI|iXFqyXSQ$P&XN}_^QSB1D<0BYGpRpA<#o8WiK3{!CWg`z z3q90m6=~J++o~?UMaA)cK7cT1{pjR`Uh`#j#X|;`5VPAOY~_4FAJ+#czLou&6?xC; zlv02iZc&yK!a|zDl^~Tv5gwt7|DMa0Vk1j`>_btDjDKK`fxh;7*~X9)GGh;W19YxN zN*nFXOti?U4tazATJCOX`RRKoiq@jV&4gWD-m**Iyl}0fYQ+9+*y$(aaS-h;YdB@q9>UFCas%c`4g&E0vSkUhjY8eKD))>YnZt%zk))q z{-!s7u)b!a3n-o!`PzWuu@dnahn7<@8({;7pZcn{VgX;EnMR2yx)=1l+Pz}@RwQ>E zFuo%0;=g06HYe_AxUR+L>e6rs9Yz7F4FlwPpju+zR|u}~8)gIjnJo@o8PA}(MYI4aZaD6VE6R+|o+q@H?(;Ue`&u$8%VGVKPGzSIw3YF0#xp@c>`RV_r(-Tg zw%*fT3Z(ruo2q?oQ6D=~;^NB3_+EmX8ZkeV7I%I3E&B5;e6y_lL~5+2n5m`rVDf96 z7Ti{AMV?7N{F!C>4O5fT6GsT!9~x$CXf!Tu?GL6O3TJ%~xv3(RRgfWko4=3%)eKFV zhDy*gyCpo9CVcl%o1^2VgSuiv%yz|{Di_!!_9YU}m6N+Ax2W^!eo(gO%9rxQ5Tk@U z#&Nva{uttE+vq+Ss%*L~QQhE$2U$I`3Ni@4vny|}#^UIAWK%9?|5d|Mu#h5=-;g)6 z|A$&WE@aDv;ATuFkf**$x+gJ2TanD$UWCzl?r;#lh5zM_2V_U?Pl$IGLNfIv%dLxaq3uPQBP^5bD>up zIUPOG>IW<8wnf!h9+ug!?ygEP8{2l8=_v#xOM1K%%hDQ&w4d>pv1VV5-iAMa=A)E= zP#n^`z$aMb+FX2-TwITM&nQu?cMewpx$5Z7^=~+R_%_ZW^aa-&mG;T^x>S6{o2nIJ zLRGCJ^pV~+TW&?HG!8nWmKcEExk0nnqR|f4^1Mv*1K%@8ASm)c?1fj{pELz!l;BIP zyvPFJWq403`LMjqZ3<{!&3_V{DWWE8{#JZ2Z6^OwENb~1}rX(;nh zwT|d5p{cE*Py1!OA^f(*4ng>VeLBE(GH4H-d^+fXIS14QS01)oX@zT?knO{ zh_bM3J=gc_RI|?FvMhR3hQMP=TTeYtevsB&EX2G85M9wG3A{`>5c6HQzyn&ET5-u- zs>(I#%@zHgkC;J=n}r~>5sh1m2Dp>!&%>gK4`{lPe+18m>w{qCMFy1>D}vE*z4cEr zC*PtoK63q_x{k~LY;LF0j6F)p6+~o?f8bMputvYKJ0K9mzvVUe;)%w_NP_W%&HjCdWL9T0GU@odki*UBpSMdQNR(t4d7`mBnaYU=&67>;9v(Eow?X7Y2C#ZaBG?XL-V-vep;-aQCF_Osc@SS@!4U zxRM&=-1{Ty!%zGHuroM3wwqjf#BP2*D3xo`MAuunQZ)0?zI{c6xt@Zwegr5=A*Ira z@Yw(<1!v*!(+2K){wlUBwg9;Ol)2GkL$De?I??P5 z5TUf<(vA#>-cyQ0Jwi24X~bf5e7+{if39i}h(H1HN{ONnM+(kcWv$vVsrh0SCXyiK zQ8fZp$XbuM#P?(F+T$*6CXDl9S-UKM9J!h~w9<4&T{K9uxuoO0 zA@&-JD4)1?iz}5G)dZaS#KqzkKYV{KD_zAvaGaVE!-q-6T6{U-bU-lHLO<+y`PkLyL6D<3X<3wy5?!? zkYp7UB`3zNWn^8o;`IHC{!^ItEB=z2UHgx!79L~^e%Vwf!V1jkSPfEST1zX*a&VMT zP>x6&SMN{pJm1!reDg$77}8$-8@Ywm*gSSf-M)_n1^3#0rR1JuVvO4sr`M`7q(7Uo zf?J75+>e0Y8hs#R7rAJH&0i&VcOl~*wVc%@oPk~wj}ASVBzhMzN(-z4W?~*%=Gz=( zD!c)#**_~5uizyhVL15HW@Vs9*AS4sW!|o)`a0imPBKYl+PJ)A_x0Y-%o#I?$_~`_ ztrd~!nR=}KcH(#P-T}7(Xw8S0y-Ax!`LL~*)q7hbNq#KWyItw20FJ30Wq-;+ScTUE zyN^1qoUZ5@vJpRlYp}r=-s6w@Xy1{^dn+5LyzRoi+m_#(O7HGV9v%G^Lks;BC&{_T z>d`O(Xjt~pZbO#rKTNG4i+99f9B#@dYWh;I=P$)mo1vl71CK@++jR*BWWE> zvc-82CoO@QNFaLtQI5~{nmb;d$B4H>odFNY<&JljwA2k6hSNEFq7XzNSEkO4q>b14 zN!4vM8L9z=Gg#fThB8C!%=1h;PQQDcZy8N9?&sKBEy~-gPi*=p)n+2qLg}27N(P83 zO65-{8OtT=-n4CVtFc58;tjiOhrdJvPT9UuTkO%aG*aKy`F*-(rp@3P_WnpBR9Nk9eOZI-rjo_gh}7e{x=IsBO_KU{YBL(loUn)Z7?)Ox;@6 z510^sKvOuY7i4qswr8L21VRFVxfs{~GOpsP1ZWt_VF|4O?>L2@Hx{Q1C3?cv*ir5- z-s@vpj_F`rH`dRv{ERsuZc(aop$__j&E!}?_pF9t&_cL}Pc_~DSH)jkRcj%l-N_Mp zRJP(`Kr}Vx&i9b+v92f`2{EUwQ4Osk+mJ{XStonG`NO!V3q`jb!g%Wo&XwATNJKNAW2Wf0*5 zvFG5>@SEBHnl7%?nJMm>$AxL=Hoju~pKU!J!5Y&4H;8rr>o_6@q!IC7H_rdHHD$o# ztY`J#c$WFEEmFHW|Fae@}*g z0c|^zBLFo1ZT=s#Ec9C6-G6mvIRWlDs=6hwmH+>b_vWf{@rU@j;OVE_ssA3{_1=o& zeK%TVXUZsq8bQr7N?ukhEsx_dVgKT8Vj!L|cBy)E^!9A}b;YD*nA2j>MUmm#q@jNV zEVScC3gC>U`w!s>bN$iNu=%?R-uE~9Q=Q0z4F3_d_b6X*|HZsY47p~g4s%$;m7T%c za@L-L@Bc5LH=Jm2QEHw|K-V_cEUh)4mzQc;+4z6rzN^41rTdlamRvGIU%t}_8aQjb z`vRU4_OFm}Xwnu@nh(&D&EY(yERUj;Z!mMgh;K+KChtlRtEzd82!hbjg8_RVWcm)mPsm-`qo0bX(! zReb(aC1BU4{U6c~G3QWC4C%L217zDajMwwWLV8wVqg3_O`YZCyf{fwJwNI=o`6p@@ zhQf1OaDbPqmy0YoXZHU9NJ)u&?HzASFNct*fP z_uCc?bup2gael9(x)YL!iPYZnrXa8mp%0iPP<_ z;CROe`dHc4=lkQ*t(6N`0bt_8yT(J2#k3a+m4UjSE zU*oBByc_B&*Q}-(XcM-ArLT2M1IK;nn{sYhze77vtE8$!nXrD=`>LE@L$FZsxq;^# zSi6%wT5n?F_1hME7P+h3HRa8f3XTrO5um^NHs0qJ_znFxf-+roP>cxxu0Q4+{*23M zmjctwP;0sqfCf)HD<&*F&Ejqf@x%`WV&qbucyPc=;E+CrR4}Be$I6{{R~+6YK~A@a ziSbG@>d1M~H>c|zlX$(LmKm&#Zu`vYbG1v2&*|H%PrlJZjL`jr1O!7QYcjJ8BE96v zO!4P6`oCOc@*!cj?|0C7y2{hzFByz~S#%lHa{{#$-xfW&9Rbbm*3r0~tV}oAe!}f3 zIc-#zgtEK|i)cz)Gm=mY-{Jf7$uqyYBR_2|Hxs#!s_6TU80p6y%bN(VFf9Z>D&&`S zsR#Mk`W@*f1_k}b^=6pv5*0piOAfyH%5urc(Bj+}!|fvx-;mB;z7tIe!G2;mVS?MQ z_{(Ck&|9!lb>`JsBv;7h3VtoyPI#v5RG{dT-vi&xOKM|2e&u1B)EungUq09IS@=t} zAgkKQHN0okIYT%Y4)M;M@x+2s3ke{9K5ck)jcNHGrwEcLcvc|Tdfs1UB+;@jsi9Ei;a075i#*nl;14|fdj4xNs-a5vfP(kqUCAcyiu>=O zArLpeUjiy8W(D(H^NE$fgr}jNes#a^TrU(wN~8+W@y??5gX`vjBgAyy5+C*m8ppn; zIGJ4{a!d>R;QqeLA;&dW0VU(*#ZegiBaA&b`u4(ifgWC0j@W6AMZb&QII?do_k4mk zM!QPLV#H2d?BCPpGx0t*1pDAxK8x||iYT`>(U@_apA!A`6Pr-bH8C6Qy@-UUWvNtr zHT+{-RiYOCnd@qHde!t;1<%ekb4EHZky3E)%v8aLW7wF8o&LZ)WnE@F<>n|y&1abm z5sAI}!{$yb9|qU?WtzjnE%doBIw8;sJ)pnyf-f*w!$vb?%N0LA59QHZeoLgNmlOvP z!Vl>qCSIw=Ru7B+Tm;2ElZ!h%r*9We`PzL&+StN9T^5l+VzW{GA}Kk_IyR=k5pU(a zA+6tt?Xos(;k0G6?|Lj=A2dQasQLbaZtf)T2nOHe`JAX?*>F+6L(W?%AdY%L{VEP; z3y+~$^!#7Dc9<1cu-H=v@G2ce;XqO=t5 zHc%utRCCHQFNfVPQ#b4TZRKp6PdW)9H(5@zL(&po6V%%XBmIJx2am2HU%lM4 z9=zFS+BPol%onumm7YnJ1=yXi2Gvi_8h<0_*UAfqyS@}c2F?}!Dap04`y*SLA38qU z%@mkh!`IV3`#u1Hc6*m{Ca-e@;TEkpaog@k4Jpl6pBTcTswUL%m!r-{s&-fBwno4< zU*15~^8fB)pY~tf`Gm4N;D#<-VtuQF1dN`BWjE07%Z3EyQ6OY{F>~q{!3tMBu z=R|dRlxe+ZH=9?|P9dXl)QGdPv*&NaXQya@gPo8c)xmrrR(Lbf1roYAEEU zDGfdFfmRvV7(VyF^q42nJPk%i9XFnGb@`T3Q8{j=4{=qg6ByYsG=H`tZLtdw_ zSi8v{@-ve@R^fMZ`FYA=b4WonElra81@~(AiI_ljrwYGWNKyqINWD9v#i#LlhSG83H7(Q!bJWqfrXba7+i*mzF@rHlqGHe@ z-S%NEMh$pWmv)&JS7@;!@kS7b7=z`-*YKu!s(y<7xruw*SX);R;FJQnHZ620br?Tq z>XJjP*7pZrgtOiGSo^pTj}0I&E_Ww0_$YV?4KzO|PENSqyWcvH<5Ao}x6eQN-7I|CB#+!cUl zL}9^_%WwvSuhHpT7KvTJ(uqOqoQ3)gJGLT7it&SO=Y6@7J= z{@W0B?~MjYlEB7;(DL;^^BpG;!(=0;cJq4Y15KfOLQ7WU?q;I#)ryNy@CQ1awBl)n zI>uX}GuNZFuw{J1~etV$-NC~KRor|`C>&I#2N9B*qbXj7yUGLmrMRL_$3B&aJhXvf5 z+ooj8OkSmCP>u+&LGLqvLyzFvFPb)8A{uja@!e6cf@c5zMjV8Z?NkKw@YO-FKb9}q z+UfU7yAjKCH`@e3@63cLSIl%Ub`_~V{5Qv(m}#GXWk2kUuhlP_>SNYk=ZYgN)Ay zExeIp_C-~mZ*|7S2%FO7o($~R$G%I8|=4GzExDKIk z^3wRdYOP{Y&*NlZX4@zRi|Ryl7gu|kKQ%r?o*zlo@1Ilg%;m5lvihjk=)pBgTS1hSl`6#H-`|j|Tp5+70apMcn;dk}* zdad)6=yw0~#JyeaE5AmK#*9|8Yo#?%&PKQVH2duCUu9A=Nz z@cm;Sbyj^D^QaI<<<&}QEHIQBk8QNS&$Ms#_pINrvK*EOu#u8oY_)qCW$Eaaelg0- zAph!)OlW1&>)8jXyqCGX7e>;*rgQKZIpM_UJELSgP9y%TB4b0;K{0x@2C;lrW3haC z`20%)RO*ld+IWEF2!|>eqU*JHkG)-%vvS|&FISAp=|N>&_(5OpcJdy5$h>Y5&kWkk z(NX)}UoHNv2AAC0uEtSAqP}bxV2n#@O7TtkeT%uTWN0tYio7y>|2H-0(B?XDwn1h+ zR(ww%n)tqAmp)3iX-Bd+;F@AI`Z+=4c+5R9CH8+zalNH?t1ChS;A-SRjeXs^VmM&K8~3Jq)Q!q2y_p&`OBs2 zvchTKW7Z}OPtPkDswh!Gk1$#aQf?l`y}yv z(fmDY%a@=7D&rKkDR?4Mpnjgy$3->;Y#@YWT8)*$F6g2khi9ry%B>}FB1-Hd24`|NMdt=b9a*WxKL3h4nnNQ8g9iyNaC zeoqgKvasX}`qn1!EN$Q+r3B5IsT-r2UvAs0I&Kwr^W%WbK?Xff$^E~x{*Q`qh9m_7 z<6gK3D{5wav$VaNnSEd;ZKpWc z=i!Q^Qr{kc&0nk!E0<1tS-^ee(UDwtzMTK19+vqHsuhzsbjx}Sug{A;NPmH%QjVZ> zzke`i7%wt>R;(7+9jMj*I5lLjhs=&=Mv8nsi&>g6kc)KE(O#LjVrcD~F`u}J=-;AQ z#-w#Z5$-PzlEU1+#gC&gIu~0>Enfl;7pfwKz?SpANNtpQY-8?c(;?IO_%pu8qhvww-Uuzyc8tv!5eZU_AoOW8r#?@%d7^KHtMR&PczJ@ z&;~Vc8e)gQ;r9$@ASf$S7>i8Xd;2iv>LF?yY0T}gm~9rnd)~~{amudrIHqW#+)Ywm zLG~`F%gf|Pl~z^|(Y^;Um7bkCKS-Te=Qp0OO{wj&3}dWc45Yya)Xz1?uxH^KsfBl* zCG|oeJGvzt>aT0o4}#OGHGOhpUXncNkEYgnrRGt0)MMSH*d_zjGNMk*V*AW&>`%nPUXkn07|1tO+$VNwofuFP-n%7i=5rDvNBIh zzi?uj(+P%i?dgr@)NpCV^EtV?<~j3$B4|25-F=k^V_Z*}f=(ysrANKZ?PJJx@a_q~ z>hA<=ZtV9Te46+7s(_R3y%r&jpJn9-LuqMIP)`;v^;}t6uSLI|{trCTet9IaO>wQ? z>@XoR?0>jvrk7e9?W>87%RD0P|06)eg)4}5=kFXQc9^cT5)QN0F}4ji$9r%;G&`Q% zk`ey?k3ho*DqmXpNLqf&FF!Xo5wa?lOH8kHKF4ZF7%$<7yddqQ^B2U9jP-gVilJ46^i=y26=ZO>WA|7x4$12NSw0 zbmn|P=n4<|h^XencXD>8>ZFj%r*hT9oza)%caVYl+ySx_7ll5l43 zEYPr!+or1=P04lD9u7wIouBV?rzZuxFB_0QI8#R(K8bSe&~Rbqsc8N+N?t-yEbiuh_-`yJIE~I!AFuY75-{yp+ZwC#yJHH&RwyvWT1ichL7mJvp#jU!CK3N`8TEDjjPjX~Y}%*t zlOW`th~7l*W_Eg3fh2VM=lF-UfS0g6^~;&eQ5#pHI99G15y> zbQe^e%2Da9Gz|*RBd_p9HyZ>Md;r^2x8M^S)mzRy);u?xB@U)1$+{O~zR}@_U=r3Q z14ggCjeWG@3w|#?Dg%ZGl6N%q{t_%yt}begQ?H~epnMDaqd=@~(R8JTe$0G`ITzO& z40VEY4GY*r3r0 zwXf)54Zp25&XBAnT^hOeg7a0D{W(76Z>=sfT(Vpyiq`Ag=H>Zj_=?o_vN@5(NZspC z>&<#+++4?)9yTvb$#p#J2l}ut!BV&7sbT&_Hnq$MWutBzd}itKmX>pC(V$syQB6u= zlJJc2cG{QbF{RldqI&z((QwH}&b7==@P_vQ?l)nMw5!|XFwyNzr zsdJ{Eq1Dz->j)a0CB>qCLISCD&=tcCXE^)w@?ieouA$xuzZuvS6z{`7mzezR*)(n|1Hn}b#aO(y;shE&e89n!=mi(pJZ|wm;toB(;~eA^Zktc?-7FS_az~U%FVX@0iUDG3mGl zB4gd#kNl92vH33{qsg=ps?3gwmOqq2iK<%3f~p{2B}jye4D`3c+uju(RJg{^0p;uW zEP*2?al3s?IGqtTUXlr^mLyu68LUwkGR|SI=JdreUThbIxCdCvHzwj_Eh*d#CJN~C z-{|}tJ1E8I?Pcs;WknkrCK}}Xigx<<3)$&9#JtMMqhuyQ&v6G_;rUd;MF`Bn5E0r^@Pi3GH}>*re;-M4PXx&H`ao%XK~YUMTj z77hOh8dQ(_zKv(1fl_CctQ|PUFvHAUZXspo?j5e?=4PL+cdWUyAF58%c0W_FRGQCl z6lCRUJg=PaigFAe-T*IrK>GVW?jj#Ny6o7i2aalfu-2)v2d3ZPKsN;|TeIMEF^Y`# zRl!A7aORGnfrd3Ed7dZVnwUgVPExZL?33X+$#^^WG0m;!>n!RpqDTuMJv0uxaDR{5 z25jbEUVAmX5Rm9txOP-%(+m?aC4(VH)CMp1hO8uwrVg}aM8K<-9Ttp1V}hGCmP9u? zd;@or2@@q#U!tT?>+Xkj;yuQ0wz^oo@%GGpWIc@65bm5=K_pPkN2boi-`o3uutCh3 zd~&LEz)n5CKNOLCSnc8DF*x&Xa?QY4Mzk6%NVQdKk;P#t&4_tO5JU0X-gLQHuFq>9 z5rB;E_TO5uGutU;XI6^pL9-+$7MC`N^A8AtO;<8xoRXWf+CcuC!852vkqpnR&m&v< z&v4zeLqF!vsZD~mTyiKt4O~fq3*Ya47?l1UaMplAyb_%E?G(kbgfWx2MAz8g;p2bc zxz!(@|8Q=$Gih~yp2)kvwB2GU^c=5iqb^+YGZx{k9h$vjPFJj)8;?pb0V|_g@eBEgdJa^ zv*?rnoh4)TTA#+caf$FaLy9xwKdPh5n#(HY`G+VK%xlS5^$*ie%!Y`h0|#o}gy@H_ z8S+%Bw_9aM*>-Q8f{>eM38}#k?#w;Vt352qc0X`|8D&Vx&kKLp++2*HB_22Wng`+? ziZkPz2t;@DID#$;Y0VDcEyiPa-3V3rc5e9GE+#ApHLx@x#p|jp;Yl7(Hq!LzjHjjw z%F4gY_~Ft@U3Qm=cWot@`3U)+al&0eb`RgtB;CfRRS8XiLxnr$>1ZItqr}z<*udud@(0vya8hf$u zZbn?e4~)-}d8K{Zf}{X5`kkR_C_<;E2>`EVm-OnN3~K69sqkl~G8yM8S#w_OY8l-4 zY_Bb|^d|P1#NNrO#COE-HBvdoNccA@gL?<9d~l48;js%q<+o+8X!F_KgV8I7GVJc!HyT1pS54L>OJlGzgEd_ zEiV7hg>+L<1*0s5e+1?+HxzG45k{w}bPOq@W{-Y3d$DtjV)TY5kZ*dm{Q7=27r$JjO$bk7acB&XS}kjSGuXQvdK5-G$vSFRcQ?f7 zQNeEl5WKZCkgeL_rJaWUC{48jI6dYm0p}aaP_a?5|9sl@iKA@X$M;gz1>k{y`1k%7 z+nj#nFebgt^o>TxOLhS{Y}UICg$0_nhyceAYkjDkW{XvS+frb8!m-q89MLbIEO3J!viNQ90eaqan-C!hx<%!;mnH<>BXU z)<=Kp=e`*#ac=>oEn%`hA}->se!~qr&(L6(+ms7R}u)7%frwp%&uStKL>hGmZwtbRh4BLS3^n$av zvJkvaybe1RFlsDv@IX9;mvJtLLSW{N8EX{4(as{5dDGn3EEQW@6_W8b_0}L*@tBJ# z&(YSpoT5Lxv(;KI#|vK*uxl^bVcy_lDgS=AWu+?XV|egjZY<>f&j*8^M&R`TTrQ-p3h24i2gw^?=#=?{h0 zn1CbdxivM;vs04~D2o>||K?L2AGk2J&|S?9MAbu8y{n||z!VQIIRjC1da!;eE5Ua9 z{4DXLg^)|1w#t#(B6IK7+2m!WmMzs9gyp(39L zGby#dz|RB!IO)19+bqg^uxhKQCJD0NndYq_Pl=xaJtziecb>i?{c%#&$Sv~St+4`= z1AAK_5+&7o#apoMPa668>l{?e5TQutbFy!p`8!!{7ie8;MdvQ_ zeX``HIgR$>!0Wk?9_=cR%qD2&_!t=pWai1c$YVOiq-9$*9RFF7 z73A?-&H7IYZR)v6p&D8NA0#cR{Ga*%4~Z;L@@JkrvgzW7vN~5*) zHZ;4xn(Y$-O*z=Tzzj&`$hyjVTzOZDTA;Hg@9#YF1o5@(w~RwXHIurEkp5xAjC?Ku zN2lA`edtHK&nooVs#R(B$Wx7aPeVPrVo9zxcl`^<7t6L^BAICNpeQ0i>MW&8AK`5Z zbjv2*r>b))!1bHHD+E&?pCBXAicqjQ0MvNgWgo!vyC@a?$7_WiNrVyJH0FtnvxDg_z!Hroy&>DPS0ZMfIZ`_3`npLT$#;N*U75jJ(N%o7o$Co@>xIs>+~#Tx|I z{?+)L{L>n8}CH#hbepl>jpD4+`L*5P6e5)9H$vCfmt{JyJ3%j=!$PBn6uPM6Vxn0;D)%s9+!aX<}W0pQi6)j^Gokp>u3AA zb=zMAwxL`a$KqYzMY%_8#?kHHoJjifUVGMctWy6aEirXVPD5nbm~E&Sefmcr>TWsx zZHrgNy5+aF0`QmvW@eFoiCq!1%d(SH{HA|A^oXC^sBn+ujyIbmldNX!S!zf>?YLA-VCj2^!HC87_Yo5;btt;83pleT`&?FJBbbS!8xR%@*Rh@0 zKM7*#1RM(##0XL#!4(w!A^O*Ip`!;!-*G_USTDxkc@Iwry9-;^%pbLB-#)Zv=!#hV zScKj#mx}x*ij$a79`ICrBgD^-aJ>l$S zCJgVyt&*61Srzzypa(NBBRFlN%9W82{Tnuw>-Q1jCsS9_to@NtYCVXI>Gr0MbF zf{N3rM=XA;C{yFqAKokq-2-A7;=@-8%ZYN+<~wYIh%Xx&55he!DNe-7i?gC~g7eNe z1c2Nf(0Z5phm}jKJ{%GGCki^}+Rm1opT@XXSZNk+7>1#5qCl%{u%GRaw!t0#J<;Lx zV#<5sOBh8w;g;fVi%Pd8ZYg0r>oqAGpSAOz!pArF>oMLV9iFk1j}vd$5xHSx$;K0rGQZV@!L?oO zv)C-%TQ)8tyzf^-ZJD61!^3Fy^aRa&GzkVDj^2ZhjeL}yC5L1#wyyLvp~Ls|HiOYl zRHY|mAV522PG`ONsap_g&S!zC1+3R+!>_O)o;E!x^%&$2xiz5F3f@j&(dgDGO&9Bg(6yPn~rVT1m8u-W-!#JZ8+RaZxHx~27JD&P*V zv2yg$(h+U(#p!TnjjnXtr?S*tPagxn0>#?ELBtf7jD*_;m6co|&s5z)t)>uXd$$1X z(F0;NooOC|JNfh2HI@RFJMfINz8}~3mtAESz=Z?dAlU{>?XJ)ntn-v0omwi4M4qAt z+GifCcJu5+kW#M}J|*DqZ^X`8ZQz~%(!k&Faq3$wJBY;T4p;)^L4k=5tIY~*&+%Vs z+T%@3CSF?7)zguLbTTxet-kUXknb?FH51nGwD{d(q&N(Ckv6zlF^Z?O!$}o$MaHBf zY~Qe^ZC_#)QBmYs+clC@f6LxnUe&hd2ZYCEiAYer@%!>Uh1N?jOY~tyaifyYoQ7_C z31;3XFGA4r< zfUQ+^)o@R>L$O+~GhwSEcE-mH5-*PxBN@O?eK&~ZMqV8otF+PFvMyaVPx|^PW2T#b zjabdTh#8Or#eHhC&#HA>;GM)A)v;kHdGZSU62wV==}iu(zO^(8BF|OlW?9jnPq^@5 z`oh!V1Yug+H4*P-HBN(>RnRCHgfz>w+s5~DRT*J5()xcMat1K!oN;M?P4jANR{`UO zDUl%Wk_aN}s4^iGy&fBiW!F&6FnWblfnc-3k8}uDK!@!+$^Jq=`CfQ*dQmOUu%X&DIM3JjBDEmzG~s#DJk&ApfLR^kK+?s{c@PWWO11pEHEn&w zMZ)Bv=DC5?ldMsxI=L-rr`z6N5&Tt3_%HLa6W%c|YZNc^+KzdYvP8Vr8WiOaQ{3QPK=Eua?8)kOwg=a7`ZcG= zB+Th`y&h{iX&A4BrM=+vS$R^6i$K*q;Mko#Lt!E%X7<%|h61UxXnnXD#Ck+W8 zo&86UXqvI*>*RdM+Y`}YgmIE-D0H?B$k5zGQd{}EK1{GFaWnZ3@9%BneSsXw)-T&8DN9OXBKv8l@} z$ym+%ULsiakAUs9-72O-%_6?1Lzq*t8O4d*P>ka_ zsk3Ap;QEq%O}|Mq59}^~I-|P*ey%yuY|Q<;;r@kxo>A|%$Scj~Rdfmp$0vLhY0GIT zxp^668aPCQ>Bj2({mtt=jZBQ5TSziE=$E#2xOK2DmflZQO!(kj?ft9<>8}358B0&( ztl<{kZrTj;yq~VawKZX zgkyB=GgL9=4MM{piO%#{nLsx?rM_;WBa%aPX$@x|1zqNPwpAf38GcFg#qX?Ir2!>A z-B@$y>e?PPMMoihiQ2$vmAfssZUJ7KpII8X^0X$d=l)=KR)=e_TRO;@#WK+_VAb$q zkhi4DytS#+9Uv|9AmI7-91$i@Gl^QWDO0~$YEFKcV9Q*hs!li=T{`b=7@!wUEL4mi z%4eBgxcL{;kP&bq0K@&s?>gzW_Ju|+_wGuiHimij+IVNOXMb>- zjw9r-UcnbBQZUkc3~Dl4Br>_A9r-zU13hiH^H9_7h>OEh4IDrKaKCi1aB`kO@;%hn zvYkMCwY_^C+~^pqwoGKW8pHphKlY^U)|tKKUeN~;AEGC7h-@}-#P zeCMYSXD`tmq*n^*Orf)mEMfR{J5W_x#h+44OhTdwc1sM^4E|*=t34W$2fyawlH`)! zfVs0N1@!MCsor+%{rsA!`5?{?NpFsKYm9`m?^IgzDUSLvrjKTPV}iR7m1(`M2lgVD zd2nm~o?5L$Wvw?~-wimoW_@5#M@2);D$Ef`2{8e<6k|#nBbB`!k)&!uPuiL4stb|6 z+l4)IMa!P-E3!k{nrZsDDA=qH5sML9eZG6ieYIXMa~zB{R7= z)1U+N(R4H9N5;sJfD6Fu4xL-O4pCp`FH)w6JX2cU33yjAuf~hLXJ=Z-)X$$2U26AO z{-g-bG6XC`7R>c84H2pHRUY(qW#!`$W9oZFg5{FwC7_A31B52jA|-o`B-MKl^Tye4 zbD}Y}**#Wsf}e1Q3rM&bk=FH=*^oX`bo+nf>_3B=e8cu#6h%ZqkgimvNk>6?Q|Y|~ z2#|nCuc3DYMFi<3H0d=YbOchUN^jCbOG59xgJAjXylcO+|Fb`=nKkQkX7V9(&;2~P z&g(pmGDc;rXy4oSK=`+a}!sehLD?g9u( z%lB1Ji%YyosU8fMLzQZJm*Q|SHJhU7M+8+TI_#+Tn_Ca17_{W0|4BBro(qI>ZNQ6K zG#+s~L{&|LH`Nz=D&F;)y^G&u?`Jaci4MMVTS z4(+SOTyB!y7kiZaULG5GvZ#q5Q$u1t+F3C~aqvSIM-{pKJ?r!|b50k~RtIP@;S8E5 zn*<~IKN)sm8pGcfi?+%^8&XQrQrcm5{+vibU2Qxn@hgju|Iiur60>d_Pg^Ex-ix0^ z{|w(P*gBq_L^ld@~PMkdSFKI4(EU?8%sRdqp@N=obNW`Pyj3>$`pt`v_+7vhkN^^6n*I6G&JQp6b>b8X@V2Qsirr*=3 zOl5D0$_wcJaLWY-q~*ibu!0F|LJpy#O(?8fI`m#u)L$g-T4?g}0wT4ap{UrI$Hu=8 zSBfyrJVf7ppRHrw_hOs!@%fBjo%qFv?HQO&$n(qh@&3)-+$98i6gdeSPe{@(Quad_ zTXwOrO<(%@4oqHOI-$5J>8I!5nrWJ^4HhwkQuE5+fNV5gc~tD()Ku>blhyQdI6!)y z*Ez_)7U-q_KA}~H`w{P=kQyeWGtmFaE%(0pBBG8AuGmTpc)K70!8%;eiXaX8bg>B) zjeE?Dk)4UeG-8Wt|G|ERC9Yk){<}M-FTpxMu-lyQZH>4EuH7Q;RQdc-JXw#Aak-sx zLWLkMjd#&L%E={2i!}DJ0ZU0c`>#ELH1lnG%a~rbeVl&1<=6Ot-)#RYJT&V82FNC+XTjBMNJ901&m+bULA<5J~9@CBm^g~y=9Lj=@W|g!CS~Yz)WagU0edk zA}m6mQ5_O~^#xd2*t~^NOI}n_4M{F85)i%jQ*LOYqR4nH3|V6&YlgA8*2V4{6g=?5 zZyq8=qT-N~uQ)9Z0hM<&8m2x}rhl2e7LsW`Q0%a@=6ixHb9`MmA6AgCt&#fsCYt?a z_xgM54^gRY>Z}%8SA4K9da?eu5F7n}Bh~&KX7m4*Y7=AfTh`i1KeUYI$)D0=IB?fy zZ*_{nj&5Hb`G3w%Q2%^h;G%c-F@0_M5BDc8dDjxH$%eF4oX`XyBN0X)5s{WhJwznb z?ZKYoI#wJ$XldyBw?voFI#X2$@DDXl+x5sU>kEC+R0ik#c7CkQ22c;Z9mC)n%l!7? zeeKMq;`@Stm|IWnp+OY*W?U^0oO5vHY@m|a-fXTLN&QpDO@dqwvMFJDdTaVBJ!0pUFms_L`chQ!`KiC# z26Q#@Uq}%6GMid?@0)9%G`z^aCgrhD#Fof%6>B$8af@5cEMgy(W_kduZD0=ONgxXf zYL8ug##Z^tmL`~Jg@8FoIfl}J?7;~me?@sPOh7pfkbe9BRyj@H^v`2!hws;H+-2bi z;V;jBBrMI_2d`X0OlyXlj1~8?gH+?JHnWG{9fJB91zwY+{$MM3YFfqZ2k>&^ev&VJ zqulzoapFdKxWppE3nh-tP>R->!7`a;6n^}q`F5;Vet_+bBcDx7S?LuAZN;!>ViMev#+t8%JQa?DHn0*G-AxqQw8E&;PH(MU%r+ zZ0(Ms;lc_6kr6#JnFB!iWO#S(5_uWL4fUD;@*N?@tFLB5k_T~osxFecdtb6l?Rux~ zj{%7Tfq#tV*8F?mIE$tS2x!=E^dF@5CgW*hsBK(8Mz)>XaG2m_RX&ld`I-iPZ_ z8aoFQc;62iYG);^IVtF$wZJoA#`aDyH*%iWW9!SPgtjaACyn6Bjkq0gY_q{@5KEZL zy@&1&nsrW+pb*uTlfTtxf;Fxk3=K8 zCDZtL+7tC+p_UNn7qr#q6@cjweC4lvN1*q!RFsJQGA}nzPRom#thPQ~e0z?1fy?Wp zEGn>`Ozr0McdXm12#R-iwnH;IR!?JbUi{KRPZ)SD6&ZIxsY!wze+`5p(^`N3qNUl; zY3SE#wahHPY+OcicJhW7dL z?B)DP^Er*ySxzcyXVBM1a7|>h28BHmUq7TcR1T`XuEg~JoTtyq;K;6*_69@dYW4Jj zt z(S^bo1^liQ`^~|9Au3%!O8cJ551EYMPMF@GSA!n>BfeNPWu9s$kxolU9_)^jGTJg) zEnRfwye9+#D>EPwD*~E>O0LLWrr{ocOV0pRyzN7pkNokpk`f9n#gHzjNiu4YGo8+7&ii3p%;V^fo@d=V5s@#@8XJPx^NsX<)0swrr(7I= zmro9`MxxNNd4WVnX65;?)#ze_Q+HTNU#QRY+@g-;-;>L;XT0 zo$l?yE>k7i?OJ+*LPnvQd%0A0a7O<`&ezug$^n)hyGv3lhdQNRVIvtyMNJL% z+9(;s^`8wWyZhAvA6FaB?@v`cMXtc+!A#$OBm&~zt=PiEpVg}fL%@VAS7!eC-wqsBtj4r~`; zHexoqnm-KpB1+Pi`7kCt!BeZk%`9&Fh?9AXr7d>9is4#HZ)ZK>hX^S<+GKsApZI`gNONAiw(tj8YbaEy|$dREi0H9YTAXw_6s zdwfH>zuG6o!Ct%TVIg6;h=t@whX}ae!h8{|-{vKV)VY<`W+G`Sac#;165`>0?4SdH zpf;~sr8GIY*|peAYFCWi6@H>cX*ub7;dN*D*mteki2@^CBAqq!hu-q-wy=z{zU(h6 zw`hr*?~@Ix)?x9FFCc3}L{4%QwHjV|s!h6Wb1Pz&S@> z)@#jPZYeNU$mW9b*^6kQEZ4mP=XV|WS`Jr7weo6&&&}UIyuZsnUBmJ6jHg_|k>RXT z=APz=B}6(me@3X``3xR1y7=R2{v{2I@6U_`MxoiadCl9gsSELL0W0Oh1%i3r3O7vh z6kifj%|gvfi06ozZqqOA`iq$OA&H$mJL<7g{*d7XF!j42cvLp@)AV3Jwa$={O6(ae zIBr7PDQ8Q#PPt7oVUzwx-DG5xBeLcOxWYSKyY?RmOvUN+6vk7=-E^y0`;YnsQk`P&r**v{0Q4)N^W;v0X+{!!t$20ASt9LB6vMxvTnEu0(e3^uUFf=Y zDlvtjZfh2j1r>}=DCJyp2%T9!P`7Vs#L7^ghC~^F?X3x|Cv+~ zVGn4BrStumI{ee_kTEJJ-noBr`ThN)e*!6HF6&eCt1?HJL?gA+<9T~$JycGitY+;J z;ezC395<(l<`sfLOEok{ltBjqR}Ehp2~9H<-^g?DNW!WfiBm3abLQ1htz^#K)D|tl zG;av7)z{qk%pQJUI~?&#Tk*AtE8F)=ll9}oIi~o#aE#>LzUjUNG5Z7FQw|5xraEs; z_6xb`{9#hjQngl=zrNb{Le{$?Hsw_`<B8`gi_DOS zUr1|+n}dxm28ERopb+4mwwM?2IobY?#OEl%n)%}8u&o_k*3^6oe#l7Zd07Bu>rPgP)h_g=l<@Ti7i z$=aJj^;fDBnI~XToo&oxdN0FOX=3 z5GxuzpR!y4MoZncekpyJK592ju=^B~D&^d} zKTKOvh6VxS3jn4>%7Eq}f6p2OYHkpp=0!mXX}Yo-GTOaeI+qnAaW^aK^uYRl)F4>L35w*Gw)_4~ z-!2FO5ItpxHi@2hhY(A%pCt+=LqjW#4G8{Rtl(5naV~szztY-}kn_X%hCu#;p_Ibv z8fgYWw)U5gyV=;{O4+9i6RJHFFN_66-9Gby_pL>!ZnU6SZpN)&QLm%;~L?}kq z1E2Uwlci1>bNB?=>b%k52$yUhmq_@VdiPooxASjwbq4qH#`%w+^3))5cTcic@)%DXN~~YDs&k^pd{J(Eb2hv7fV5}?j4iH z*0iq2`*#sWzG&$tvy2Zljwp3N=G{n7Yw0mt2!9h`amNR7{XArE$bz#?=$kA6U_Ikc zDN!v)O=9YI#3t}qHmGo$SE7xdkmlOl-I+?ZUX&08lXu}Bl?J-$6%Xgu%AoHA=~7kD znMgkkpq}O^2Fr!zrj25LKjM3*oFr97Y~2va`R{p&AI!49i~W|^48$Db$KStJ?~Z%L zRv|Me2^a86K=fy-jkYF1qxW}w9;gRO(UdTio?tcT!gq0=7S^&wKl?2(O-gq*0auIa z;Wsnd66>%#U$(&MAq%(89|EZb0TacrMGrMv53f8la%~{*QvgWAg*<#>*8rlvoE1}{ z_x(P%eH7fE$A#{YNfO7^!|LPlqCf!`N?%{Sq4AMqwrUxhX(1Es8bbaM-8aozL-LE&?>j;x*lhDtBZHX_w9>=RIdtg6X~R$r|)?SNJ8`+C-<$a3eYwS zPf)5!QX4wTLjYXp3mL=(MRwb1^xu)uPk$@#3>#}uN%B_9)XZ%$G%X+GG^8K7av7e9 zw+FNfx{V-jmEK)?;^#a7JWoz2ooSqL>o)scLc{8RxgkXzz>CV#SZXu*WLn~6Bl^<6 z5o3@URc@>@?p4=@%`Z$tT02$*w7>C4b+@BPdyyewp%KFPT~VbY$uRCjkXPG)dWO^OGR5NP!f6jz%>Bs*6OFvxdkUG@)agf5&@GgjHtZjdc}{pv zB(E&wWqIT6Lrto}pwCo+GoMJ?YZ1qS3a+IoA)RHMt^FTvkD9LVl$qCl>$HfmdM&fX zjgTt@`C8O4l_{`eEW2>%zQq-8vR;3wf-`Hsg%(|WN#@Vq9|>K-^Wu{wDU;;6vBWJU zq0z>WH37tds?D&;JHs9qew-k?o5f>Z4OG*H5H116fEH)j2+Z+|1{7)4b&3OmMv|CI4$@*%$=Vmkin3d%cg(W(P=DA73L2ur7G&a zEFWb~mMz*5V0Sg6EA2z_Nw27aAH`-2W-%Kdit2}Tv0IF_|0?E2Jn*`WdfZ`U?%crZ zTf-qTpy?NO(IF~lYjZceFq~^FeWODoerxXS2Z-*xgjKxJFsLS{SfuYAIZtc?Q~Fex zE8pND>(zAn8sn;ogE}CbDRgRemSeQtF1%6wf!f=j>J#bZc7L_ze^sn{Zz5RZkDp&j z*8-l4N=sfqWOyY4eCHTsZea!8EPixYmP=rmdMo>>I$SBeHNB~bFa7%$h$Eh9_8y{P zKqiV)CC=o9o1BYIZ}1IkZXMM-D6pWJ!8djCCu)pL-CyW81{+mQv0p!ns7?JgQf13Z z4L_~@V=FgVpcFREYKuPqjnQ5dD{YdUr=xGpb@!HUxrx$VxUj51+hV}{A919#YaJ<@ ztZY(^GeJ36v^0`A35yoLH81`oXT9GVR3=k`Y+)cKwucp(EpS71w_LrESxVoy zj5)_nc&LFbytP(`6RQRfot*Nc(2){%I=lnr}rWRBU?5 za7M_eWnpqj_UDUn(Sa0cU%ezg#r6!LSD?3ft#aqgNzybVQ$`C{J<)y3_lO*2Xl^J_ z>k$K*(H&aTUSguJ-Po2xus!lkpaA6@4n7vJz~2c@)XDwZ;kbv&{dn?&ZC|0^x-ek> zMiAHwmnbi5;l)5bfR{OBJ7N6fG*c?Ec#fKj3hj(RIx+N$;{|FzU$Ho&;u*dqzJ1tL zv01M)7XVsb!?4NUs`Ray8UWT!0jLw99+(zaN4r>bfik?beKi5YW%gP*8OuKuKB zvdlCLpM$yVo>EBp2#NsNo6Ene>-%@j{-KqfmZ+!jOzC?dZSIffvPqG03_lb0QQO+> z>g6oxpHnRY*Qd4>^d8~I1N=M<14`n*lh>8oX8oes3~VXwk%g@w&2blRQX>zR`PEkufAvHFYUmL#8;9oeJ&3qsflgBb zE3kAP_0Lqn_vURCOogS&ahhEuGugkCQ4h; zoQSVu?9ueZ4N!F{R{e`7D_~=!-t^wt)xDQbqH!6~IWG3R{H##1?kHf{f(3n?nJVoM zAobd=B$8uIj!DnlFsZ+py7S@DxtD3rclh1=If=3vpa z%(A{C!yz_hcDlZ3H%ueU0%>fBe2Yj6K)Lc$ZoSAX)g=4$na4)p&f&6PGoPMyNxwK7 zvJLezDLC(c9%1HFWeq!xA(@MerCku=j$2UyF+0DKppzYcr=^3@mNjx*WUv%E`keF)TRj)qu#fadlOm%qTZfMDxp6QKc(!5}wPPg8YDRPu%a_)Pm z=}f0ln5gf<&lILoWv3c2hZ$6E(HVkF8nU=UoeBiDF%Y9J9(-j<1L(9i1{2Z# zeknhrhi=nJ^N@Mba`SRC#S<_0ndW0TsIcRi zD+K2Bs23&gx42*zq6KaE{eWQ|L$+W{P1X}jfATl}p<`(aaS>CQu(9cHT-q*r4N+%f zkLVk2)2?wSx%PUs>0}`sEay`81pPNe6efoP)C0W^dx2#eTF0XDkXp?xy7C#rJy(cI z91xmzHRH%Tob)DW! zmvYFeQbyYLz$n6^75%9KugRr0<1dz9{2OKvcC z?-nbvPt}y^z2}p-E$9)1a}fT(JIQ46+`c=N&ivoO_(Q~M<2^*LD3!iGz~s4RiOM_P2lpXJsn-%je^%@WW7BI8MXe zHr1X`Hov+=u^@iH7&Da_h57BG{FqTBG!e$~y1WxqaFF|)Q*B5r0JWMEJ>A)X7i&A7b!foTo?%d>}#*FlHU z(`!W$fDldJoXdfM!Mnan{PJzzj7A8DRGj>}{V0GB4HCi=LS=FOU_*O@V}Tot~DLq#%I6#S7E)Lce@;4y!{^( zr&o~P)40`D)4-ARchR_!&7W;89ym#lY=gQMR-zP`z2=Mn0e9~|-miP)83!`b!CqY6 zU92udAFJ#>+$C5~Y8_cw)ddB>uOa~fs^4R|>t68M!6H@5s-7{rLH$H>pVFqX8!$kIXPjg%8l(IMkd82On32f z=Z<;Okyi)AZMlm#i{lbzq!x&KYWjDze*8r~uaf9l;H+eCFbK<9fZX%VbCMUNU}g)v zoFNWg!VH6S;%@T3Kb%Uo!KP0w2%NP_wF`aDLekDY<5FwWO1|x+8;6e)(VU;(qQ}QI zyqWRlY;^JmGPEj9ur)GiLCfII71}ru$w|Fwin#n>Wug!zc`!DA!UgSiw?xuC+$vxE zP^zrrMOoGm-7O=-EgJIV1{j@m(#?hYjIf%=WNTu&L6&QneqU%AQ#8Yt44~;_K%0At zkId1CsQ~cjf!gLmpwD#CbMYk~aZ5}a44m_j*P{t81wB2ME_3%7?9&%B@nDU5@iLFX zf1&}VoDTbc7PexeYOe+7hE~sXhoV=zg*gA;NH&Rwx)DQ- zqY&}+U%wWDTymZH;OcSl`eIfWQYMStR=6$XzVQ57RAeN_&Ck?9aHk{BlD!Xar zrXbtHur-o%)4!$nFQv*}OY}jQsf(xiXNDig9`2uLqpJetH=O!J57tg|!=+~AzI9c0 zM~RE&B?MrpENqjjp2UiFb1A4oaCVhq)ItJbf`GWjk76ke(!qPSN_DXwX)bD&Lw0W> z6fBvBp*mH;KVICj(ELI2?m@_IMQWbCQ$xpe14OktbXLr<{l_-`EyL!Yur^;XHREyL zpDlJ%u68SQK!|7L$>@A9+%A9R=uL;O%;BigP4#4^6@qvmjkEzDi5+^zgThhJUx|Mc zZ-wx$@`#&)e@j08Jq+19Qzx2t^KMH21ibpST{pQ^`~UCR9W0?sudDdqls}1tO{+)= z&>~ryw(T7E`&VRPhcUV@{G$0hRBSfg6{maG0@zo)w%no z|0Be!VuN2b|1NX~lg_o0dQ(1q0U0SzhQuHsZ~jtz==Hp+8u|mnqR+cQE_J@ghmZQSD8}4%bEUym*nKyZcg!HXm z&3)33jF)5${xe?59tmq>RO_VyKMn+U*)Jf#uT!-1s(TxKrs;KkYBfJarjuWFvTq{m z-afHv>qUZTnTCpyBjBvjsH9mY>9nmngP;Pg;8s*4*BiMsLVNSDNmYDE_C^_5f9R53 zeQ0_hBI#fn=7S9hHPzHX|7wKPao)P;LlKz(XZ_M%KssV_@MHg&yQ&rjYCIE;mCpLf zbPuN!G(7q*F62uv-2d?w@o(_Sc$$7pmbYi#l$nfKS>b+;nbaw9niDsb;pwJ3pzJZh z7OWDh#Hi=30eFgn8|*hh!qIwst>!rVBDc| z-)y{nYNJ7kVOvh=W$M%4WIX=Fglyv zTKo4_48@m!qB|_|DO5G&#!o2>mcT|L58G+<)0?^@^L!8AE3H!tN}6!B8KjB&oR+#U z$)NdC7C!A}dkU|JJ$VBm1pi6ft3DHUp3+u6Jn5N#4e@gvof0XiK_>S96f(khduN+~ zS+!$^(i+xw7tc!DE~r;P^|e>>9;yPPO=%lXe+`=Du}@d#^7cO7B=A~VQ)_$i^ZBsq zuS?0($yL!mAkrteHC#LuR=U8-J|>}2U=cB{-MfrZ7jB83A1Aue!1i^gdcIG|UW`rl z^SF=uBJ`3m9#sN9Ce!X(De`o+_t?8XggLcU^H^nN{(QmasmiHSsJQcp>><+>``XCU z%Qp438?H8;*~<26Zj-qc!eh;DTP3@ShzqH}kY3;>NVHzTCB<4<|6d@hi&^+M!R-Jw zE4U70!bY3BLu5JE8a_W<+hjOg{bK+Z_RQ zNePqR^u(@aL8?App9xkTk(2wRbRa&vVG?K%=U52 z%F(IJ^2ChoVc4_n5tXAfZmIZgZF%hsvhjbRqtZ9MD(a|_jF+X->@HqUSo2sp+1Cpg z5^SE+2Xy-+(Up2}>y3DW1#?|3I0Q|P4$|gZgMRN?)NoXVYG(WL=a@+4L@;)aHW!fD z#c3>oD6;SVKuWrSeVx`QbILZT3i>)%!@nUUg9+dn$Jh4|b~}gP_Et-P{27os zzf58{RyF8gpaT)QUKYCVtgWbN&~13IVmjs--M|p&IN;1=o_X)!?S&ArRiWZR9B*Ip zD9)K^q!j#l+PqeLIVTkb_m3p{8z_c{Gy zhUbL|;J{1@RxW;U3fN7e%XpeFV`3*fEIgH78nZ%wKfm;l=l*3yz?X-Qik|0r&i&6{ zL|BQld?;)X;(D*~t8xG?#`LGs)jI(*ib(!Q_>c*ZfUp0>VJw#Xq0fvU$b4m$!3*GN zfxk5x1SQ=-JcoAZecn1i!G_Y|kQsa{nCDH>;^yPc0T3?8@V z1wk_#MCGUVc=--k2Mjfu$@(_6M{L4eL+>!>d`#?!CGT;R-4*4FFnc>t)!fba;TF_1=vJ1vG=t0d^?x^f`cfp%;xFplOd?9Zr z6mcB<2&|hP;qy%@$|>|X6Kqy!zs`~w3xhm|HIvs(kBH ztwKw4g&dbdme-%Yclt18WW1%_tY=bDn`J4FJ|;e(5WtG>LNEO{Rof5Z_hNMnl)(X) z58IHe{Vt?v?pVe0qR`MPTej=*EF?AfCp|Wb{a;*bJx1y))mwio@R>H?b}?=|<%`um zxW~&NkY0n2Ox0*{fKt664xxNqVPz?={4gf{9|+XJZFgDqvAPjCva?fDIk#Jr@|Bgg zxT&N}s^HIwV%h9)&CGnIQ>}m3=0S_Tl!c8RM1zR%G*0F*328bQ{E_kc0fDZ_6}@sCN^Ak4HvLH^X3`yo|)^ ziyG&mlDmp+@V0mWpR1IW^4-gKDS&43@81;5n?wV(%v|!hjI>`9H!bG6SPTFFc>&bP zgGc_s4IodHqMDo#KdLuTaB@JQTdH3rf_shMTTsz{T5=}eW_9X71WwL?Ax;Hm=T14| zjzeA^ihpYpK{BN7)<)U4TjqRYEvWa-4v1*}dd5N-4rpanczs|B4spY#mk*63>vP$Y zEj@oO$e5bpUa=o^`sfrw-G~dKutKU&!SeMa3#qen56sUP% z#ENsI0m;l+PD@TLKIwZxEq$4jpr7ix3S(D9h+I8zqLQn%nS7-N?zoA)!2U;~=2k{^ z3t`_@yrSuKC_w78gzx%xqjIc7V`4amfoHBJPcTbYMXqp19)iGyAD>lNu|a9O0q)tKz_7m$5NyI!VW+$|{x`Ec`8r zIVs9Up99Ut9hd0X=DKZ$4q=@sWg4GQWZ_ZUcMtP_?-0Bz40fMHk+ocn$f#d#p}$2E zic>8tLdz2s1mj;eR(slt+v$17r~4gJ={w7j9P$NSVDp>XpCf>KQ|_blAbW_SAZ1xo z@~Keu1&OQFvy>J^UbS=B3lOWPj{RAMcIdaBD_{n08(Z_0hDEGq|4FF*wazQ{uu2I8 z+K_3oIn%9L<`L!q+|>Jz5Fh#k8pWTtW-p=HaS;g|G2bUWKF#Ap2IXU ze=qLeIrTKB`BNa3`f}s9Lk~6=F}tFTt1>CDC7Rq~tdm`33gnZJuL0%dskLM^69BrL zq&a^hOx<1VLO}^VDfZ9TDzwNyI0l)>w}c;NV%Llwl&AF)?GwbJ9ZMW*zrUu%(*;qp zK9NiNHb;f(rxGQ30Z3w7@8*`(18$TMRAA;y#&Y$5mX+HXE=JOTGkX z$ud-t+C1;brEnc&2J84G$jUCFEf4qEm0?KgTD=DHW;gKu0;@h*pKt7#j`^?q+~hWj z8nu-+2d@NzLwE-66o_TAzb2A#>%(t)z-bG-Nri)`bl_ZD3<&xkiHCC|-GRV^0^y2K z=T{p*H9cId3cxZKtuvX~>`~We`E|J$L?=_D8dP-adk_L~6W9gFNw#n9%{znNf42cw z)LyunA+jJw6rHRJUOvBlwKX|e$Y}q;kwNL+-ho8sY}7!~MpmbYU!dARwBO&PYr%O& zc}Y6)@M*OwZ!dLPfYHvqheMZHu!mO zwMEed-HMxxOn$iOFrt(0**!Izjh?AK_z^~z2b5xaqd%}`G)5w`3A^B(kIC%Wp=2F2 zMaf;RcO<>!U^6iQ8FzA3`yx%pROV>mrGy8qQx!kA3cE|h@BR6tb$HrtYIeMsU)Uge zV=qzt?z?+XQ%R^cW9n|NjjYz8+mm1ktSt&&*hXU{O23RTqi{RH2A#QaE5&7e@y+&0 z!9J>~KNQ6T0Dv>Yx&IoxuV(x#nnpUiL~^o95Z5A>IFYSZh%BSbWp3L}jm+s&yqWl# z0FS)50Cl;%HRIzwGmHHy+5QV2yXWJRt(Fo~FBC_>556YwzVU%QC6~R@Fxs2b|3mRv zEz##wBI>>DN6&f6IqQaM6P+0EFFGE@dI)dr&O0I7w+(1>b{<2_9eEYU_@R1n$F1=% z=pm_-9OjTF_iNRw?pQte9)5y%tsRhKVtq!kOqYH(xLzIY=N6SSBCr?0DET0}j>;9+v%& zukJsRmg>t9rN5sBs<6Rgo0jWv$Hi!q`^x$o@{zMx!IA#T!LI^-L63rOT-l^0rZVBn z3NW(K)LTF5;pBIZ!zn`LU1{aQ=fz%w{Dt`#^hu3xC4IM8xWX#LOmQpJe(gS}Qxfy$ zv7ce@35cto!qpFo=me{-;p71lO?Pk$LBjjR?lh|=p)nN@-Onyy$`C2J|40g&8~)D! z@Xk2V;ciw8yZBaCkXq(9jO+Btb5-dzs!3Hw%T(Lwr=A%9c749!-@DwjVEd+pth&ZE za|URz^MGY632>pa_b>66Z9D64;=YPEg|QpnMO(Q;T%StoWq2!UNvc9!TP(>#S>&}#_l9)vI}u$^H$qG?F+VlRcK70)IxnmF`DAt;67LoH zz$0Rxj(`nxss%V{`NIj&6>n_E97szkWEhZMRdSP~r!GC664`5LNC=5p08IExUeYPE zv?!HNb)u7Hl8vfk^Bq%4r>cu<=J20aw8lKBchjaWiq%pl;?#VO&BY zesq_x4sGV9(QAq4DR;exI*=g9XJnE#iXyfg&h5{HbSw?UW$61kTx%+yCFdBt^upk# zBR&tl4rV@s2Qgmmd9)Ek9}?0hdC5&ok<4%2Q5PpYy=7S2uqQMhjBKjyElb8*MCVj1 zkUlCt|Z6NBT&36NhF2Z2tI5CPGjHd$M5 zssEPkr!w=go}47_SMHj=OS$Smic=U2(kydCerL4nFo#(m(LI~51y+0;Pl+~n`3KR2 z$bV8cO_;W|$~m|t?+a=8s_uZausc8FPtQimsFzURc<1Ux*B_*HuH{us*5WSG&*~blRd){ zVJsORJ;gcm?)B@ad%Tt`>ihqZd|khB)sxlIxDaMB^7%^vMQQarnK~tkyTAMjU9$Tl zN#w28H7q#geC@G_JqRvF9z0WsjdG3(( z4ABg3Vm+(XjP(j%Z)AC;?JV%7>DrYqiF(EKFc$_B;4+2kip&9)zo&#=ngVhP*aW1b zMS#O?yf1GRGYfd7JE#oF%A<2?+Z|dYGK9Oloeda~N3k5KmVRqYb@6f|!%5CdQn?Rc zMB7+PMw7>YN2nmgVt-(uO0Bm4mHw$iVB6iuGto+>ZAZCJW0JkS4+0JuN5`jXJ7Nxs z)us`g{C%hTD`tv;V#B73_;Sc8)1+`JbTWPN+tDtoKKnwj%(=0V6qf$CTBMe&8D1c% z3B|U#2!S9x=Dm6Oj!JdBGY%S7@n)`!k;Wpsd%Gav!SvV~_TJ`1-ibYK$ph)IR|4|V zRN~Q@1{?1b%Euafglcz0jYhN7wL!IKD?VZVn4zHWlMG+QnitmJ^QMG~nO=;c1~lp1 zM^Lq716BNd<32BeHF}hKF#&rb9XIN5oYsQqwIz7@9R`!dK`r2Aoa%Ywro!%#gtn@IXJ!W&J-A-8ni6D{snXt-iIU#?O(Pn!k5< zg%I#T&iFJ-M!vOIkM{>``dXzX)ocvn>YL~ohsd`yZdN42V{@ z80aHaL!FPn@0yA?iQ1|b{JHGpg|M9Cl}|+Zk(mYzbDT6}p73Q%=JrfTV|Rcn=kOmb zNIF4c1KC?3+D{hsxBrR1F5%DJe!dyZgL8zsTjkdk-=+Jcy#~Yt-jdihsmC;~Xb%3+ zww$*{%s%Ev;pGMg0qSEf%Qkp?eMAGc^^iU)@VVMYwOE9w;h`8GZ&k=zQxiWr zs<}0M=7qQA-`{?}di-KMpK_4fr-Vtm&0965^bPO|wGxH~R#qimD4pr4eNJ0 z*ZXEoc`SQA8qX=E{nE2aC4&kosPFWkF<>&)kI0nND|*AOU6`7V_&enk5)u);{pt`k zke%+-_v6j=x9Fiy6%EjF;pgr)+%hB=o(KjD2-F-ORl(_7$So%5@V1S{Eyoq_yggNx zCK<5*ybRKxOB=ihHuP31?WnOVeB)B{#vjv@o;AgUy@A5ub`EZ=q_QhL)Bb;-huNeY zf2?>=K;38P&o`N7oGDd?yB41|=7978Pm=GE(*Xhzh?6s8K%R>j+|@$O1+O>QlyR=X z_Ie4ml=QH&iTY&}-v^sIy|z6;yv;1DBEpEVWnp)jwtg>hKwaz$Ui;t8URvgTJx;k5 zirqgk4NESxB?^~ACf@mZRExeKQoxwS?f1@)?j%*{-VQ5{yuu#l_$=1ELTt8DiHLmF zbQ9tK7(gI7R1B?bS=TYHGE;9BDaP`kUSGs*Xbs}z@LI--_j5v&4YV7F5;|Zi9|Xp> zt6A!|zs@wdnKvAyEd{%4gk2PG2FSK&j!Lgb_Eioy4GnG&+k$*Q+F=+A-*o=294vVp z!ku;fynauO$%2*FrSHOqo1M8+ytC9u@eOQ3F<8BMm+` z%Q~SvoW3b2?A~kh-}ysx^M6pp&i|)@S(lVtE1+%qzcoO?|40U|gXR9Iq~cWSN(eVO z%F_)LZI%x%oT5W_Bfyt#(_;n;sXk4EV(Uc=QSMQAFHl= z!+tIl8v?^nbFXgi=v>v0g)|7#)B@bJnSdV1Zx|T%$Wg7RqMk?REt|hv-|54`>Mab# zRKsbae_T}8cX)TeRiF%^grQzOJ?J`lL4N{%50aTnxH-2j9zOoNPzo|W&}mIH3HhE- z1&7HCg!vQWi1~$BxNW)W62)6QM3c`c`(0Mtec*o8QxDQF9yc|X1)u(|>deel##4DP ziVsaU85aDUMb}O(skIBePbNpPB$SaramsOhCp`yrz81RYqY^vfT?g^{uc%e=u7`O) zDI*Oxq&b~&!IVatpnDvgLk4lan3Ed(NW2~&z|}=rVQ%KPd?&2W2J0}(w4$thU!V1L z|J#YmjWOfJ+KJD>Q1IsUV%VEZ!&w<3H+Xth$l&2CW&^od@n2%#xs<%E zCF91ZypKMCGc}LPb>Ld_^rDtu$_<#XIM&t%Kzi0Elcq9M@sIp&p6^vd`8UBBNs_kGTF&i%*zXU=(_{FQvh>-l;-o_e+#tWGuv zOT~IhIF-a|O8B&58^#2_rwSg|uIJ#ZyRYQL3^4xxX74pTc)SlZ4GbDf7?6*X8tPm{ zca=3D3d29>tv`o7F`QN!R(fd<_`5JNu60sgRWgkJMTI(Dj5(&mIzTw$x-h`F-BK86 zZ1Fs-7kg;M89t1dxr%R18q4xi#kfkkySzvmim~haIVZs$<~FzgBfgA7$mv5nF$KJ8 zM-rpy>$-P#ezI4dB+7Z}CEA_j?paX20Tl@qNxn;vQ<^CeW%}RgYS}_ssL!`+WWFei zRNdq02eTc`^^f}0KSCKv-x!Kds0H~E*@cr-m2CwX2xMvaYivAK=zkMLdCjIb#GUbI zfB!6yZf>s6(B)2)(x$x0OT`!R05(BMBS1}UU11t_AeNgOCIQYDd#n3b1A7wlNMC_W z)arYaZK|`+@-IND5HzTiPcBmxKtc+F3AM5ixX9F-Yh3I`Kr2LF)NRx=Qqp4N)U551 z@#d#~S4f8iNRZFU0}B<)`nQ3F%Fu7y&^pQK@1dWGB^EUx3YcpuXXD=mRY!XAsbR`> z=;@glj3KL|w~7QEE3pLKu*yM8sxC1lP7L+tkW?)4+^bkl6jJB0OPD3b@c_ zS+uVt>TM2X0~LKv?KpK6`Sbu9SGIUI1>v3a^;S16U}f4}@sm`}(7f=uFXoQ@EPhgI zugCWPhp<<_I0IyeJSf#B@Cwa*W~Mk=gM#mb5Z=8J%_0yLLln(9hW= z#nPC;OPfM2pMVB8z(VSnm>#R*ox9PL^xt+mV?Ru)!`D_yD634tWAsGq+(YKxzfoKl zDNtG&Ev_x@!sjVw=5I7(r)o8vsG3r~Np1ykf*N} z)_*llHxfV*L-cKCWkZKRk-3%mV7M0nQ9F!N@*7X$;5ZyvWDLKd>4OGdrD%?8-@h`Hm+@nKI4q$U%BMy zZk1sG#cc{0gJ;q@T4oGZHt=Nd459?G-!p*^rHU?}_ZY^^>r-q|#H}1F`xnGmX8fGr zIF_z2sDBa-vs4P@;FovGvb2O2`XrO#DsZ5VRXcWKMiBAAX1~U~zZj{Oh%WZJ!*Z(n z*dL;!&VXaldfowRWiaUHp1Z}v+o_6{b>mphmZ9udH?Pkr`-G1QWBaCbi~r8kt~Rm@ zJSCoy3^(Z1gE7lE0YBqH4}T=ndm9ya%1Yhp(rUJoB=Z#$?;s-x;vz&XC9~IY64{~H z0p#PZCR*{|Gue%f%`55&a|w7PTGAWDovGJVuf}w59|fp%-qE+wXlZBxHq4R8+Wmx5jnzgG4U~+q|beEV21btmkeZqI(!E951&YpO!ZNMESxH6E zeXrZ!hmv}Cds>qGK^iS28U}>(WSG|4kippZDZP9)4~lsuq%mLJyH2zdyKkpNwg86W z`7N$Fu3_Gv^J*iZq0M=dpouT(A#Gigyh#%R)uueXkJsJ+a_%;FiH(S_#?#^v~%`j2eAz~5qdNyF)~KD_3pmjr3*FiS2k9n5eVP+^kO z^-cEGw{c#9EdY7L;hcS-jig zBGpw2mW{?rDY3)U-sK*j{*r>%E+MrM=KeroPKxq?q;uSw#v_ti2n{#M?3+$I>?V-8?ic`ElN!eb#nbnb0(qQ$lK%33*aM^#OEcPL?js_92n=eDS=g%p$CKtDhGsyY=3WOBkZs8m zEh0ScM@~@rJ0;9g(Lg(q%2YgBc(t|bOO)AL)thbfuIDNFN(V18EZn13c6eP8apXgC z8A$Ti$>%|$#+?bf1G2!=RA&I|&zV2)QMiAe(BAm>*$mxallq4@b#-q0eMHnJ+Okyf z?@-Zg1rFE%=ZlBM3yw-_?MsC@F}o4Z#P2ElYLk?0s+p$5KEYlb$YRzt`^33^Q`f;S zin2SV>U&EGZ5{b^2PQQ^k^Ni^%Zo1PbwHzDCba$$pmvQ_zLk0E)xGV@E5K7ek@vGs zAJVG7?^((ex(!je4YCIwN39#d?0zYz!~nKZ+~6FauHX4i#fcsqJnx8q*CLCAb*M+A zTePBOd6hd~o7DD;aVmL`CpUYEz~PNeZG13bNcb|a!Qi7t)B$l-!@EYvDw7jA$F(BA z7S-Bl-5Okt*_@l*T6>oN^3}}q`-6)?|Ec00q@5~st)XU_a%+UVmvWmo*6{4t*(bq` zm5mU52Kg%i&1_39ai&3#VVuj>b9+zt_Du9G&Z02xbm7zr9Qcn zi~K#4!U>AzE>wb_32aHPIkB2I*K|&cm%6es_cE#d!;|pQo$S4zQZQc~W}pN2I928u zBqh{dr)-4z;)y4Lc}}c%8atssDmw@7>8zkp*)|`1AMWjI80%kz)wK`(_C4_Zp2@$S z$oX|Z=1ZLNg0`@Tcc;}tvR!j$n_Zu7S|EA$X_)Jf=80M++rCpwtwun_32P7UI@$$b z_+;P>Si0|PCGjT>bpX}7fd;{k<7hWs*yDnxEx5U%$$17c{Q0gz+Itbb@+{KG>}GX297w*+RU zGw@et|(Y3T-D@5*i?(qZfC)0NPgR)Sh86B2xf zT{)b38L_z$7Wzcr;}^?*<-vgN>WF*Ft5rpAXkpw?vJJS?n={*^zumzo9$;z_QcM5w z16kjcj&9xgF8kY==cT=I$9p8?xz~ES0iQILeHe}~i7pD&Rpm@BJ|=gn)tG>(A36w5 zOUVX>jS>|HNTHw?NvnFp9K<)nge#Y@xiSeV?|vj>P+4|>fvpzOp{tP6Hmk2_H;5n( zG9`8&$M(2#>$!??d;N8=aaQaacSa;()=*Vk2S&?y5Aic;^M@Dkf%Qp>_<=}G&YbSc zoV3;yy?A0e+GMd6`A8n^)#Mn;ZBcH!I^NB_xVGcid%^Wwy5Z~O9hibi-Z87Tg~RQj zz28pI-Z^H;MBe3}7-cKn>Ja^aFv}e8G4XHE4$G%}r2H^#xaEcFp8?Ia^a_vFcDFnh z=dh+-Zgc+b+6exOX>s8JnlQe-@^i$rAVq&rUA=oWm8Sz8t`>@iVw_Ht{ZK%CsL04= z8r%G4cu@1q#rpTtsOjKB*Cas}3Lnxjk`H#-L4-UG>`{>`QKdU$3|SdIfy|ZqT9mmp z=jCE?9_kTHxRNvcs7PR6dPBtu6-ueAklEI!n5J$;AXG+ZBrex4u477PX7Cr7zimro zN0rqM0P`hLj8*VYq?vIjpCj=B`HVu%Cg$j+1 zkQs@y2wy8@lgM3sbi@87|C{}3==2qZDx~b8g0IX8_o%N``Ha&6Y#fVLIX3Z-FC}i& z_*~)2yC)}wR;ZT zO$SPTqlMz$Tj6i_y{E|x;G=9X=6v_#dkU@Qrd^%hKfH7g+1=i|oR+?ahKgHQ6SVfa zZG>*Yte8Dq;?p{65U$>lX5DVe@c9n5x$#q5-o?aBmssa+e*;%KYtTvaMbmiGA(kJp z%7*Kc`##V4%TO#oCjn~RqcW0=)gF%jKqmI2R!?oo! zLYTJrC)T?Sjcq$Kugh*3b$`~(8MQcmed&@)|D2baqK4{}@)T*}<`xjV=r^MZ(zz6( z>oxCYg#^fs7DMLDb5jQCZ=NKLD7>0>S-lq7sAf4kE&Dx)Z+lSWhxA40{vgwlyz0UT(+2^dBxB>GZM+7z z7S<3c`r#=heb~PI=jF~~X$kz<;O^`;8vW+^&BaP8fmEWVISM4N%o}eG@>~h4f~ZMv z7TII7VUi`TztLp9@==Mj>VbLgii+-wq$ZJj0N|O9pmHKj=lr+Bn!Iia7$3t2gRnw4 z!aKGX72smqVeex*Y_(9^)0v|lJWQBfJds_yU(e&+swv{hG=2X$`fP3(ZP3?%O)ST35X^ms15A z)di;YlE|ZU|81RB391R_HvP$lk367;W%uU{s5vkG)q?;^koGrdNp_NGeHeN|KB`9Q z_XONBm&Zx z#1xVbXf0%z9rjg77J`4go8-6^Lhu*3*cO8zR|?!vqW^4@jNe;*rF;Dmocr|Kak)6L zvx>hZ0L08H>p{MM&&+3ll;Yu5lj??z5kF57D>m#WfcEDb$>+y>G8%C>_^Z;pvE945 zNphKy+Y?&g=71lf{LP^4sC=|RnoM}XwY4kGNB{WkyXGyE4lU|QVjAk&{1T+cVTvu{ z?@JDJj!Gv|DCzsSQeiSQn#~$F6vJ*NO7=8#S?){Z=$3VqJXfCb^?oI_z#yV>aCXKe z1b>jOMIA>kP2^wBvc>|Za`p}Mo=1chH6A**B>iya3)0+;_8MX=W*C1_x9p)bc&W;x zI_Lh0Ft?(vq_PbLw+NT6rr}&oeHcGUy`P@4`;^wNz9();i2T(MEL1ETpS}EqyKWLN z;j1XZ9F_9^`LR$YsceNPbK}e;RP1{lKt+2GHx)*fTh{9v_l$TpG2#N4&NV+5%mlgE zZhQNUST6H{3wYg?W@IV|sKk{Nn8S;pt+&z5I;=riN9Y|7F5 zdZF=+Y#hzIFCUwixT_tLpuye&pz#UY-utHg3r(s$_J6o0xb3@I&p?avx>$fLG_M&2 zCRIT37ZmL#SD7|}NFtp<(pI5a!_q_F?7BC43LRGq zhQ#zoi^QtXlZ8Cqg{y^x7h5${H>)5jaV|Nzz9rp_*r%ehycOV7RY{V27k&GplEorS zk_~mH2EGxpV-LL|6>F>ViC`fV_caNPcNO_y9IRZsFZw(_KFAz(+pl2*lwN-3GQ}sd zRk4%3Sn}~(Edw>otrOCvhV=j8A9S^Qy-D-EEb$`_1~zl0-Fei++KKwfNvuUsNZEq- zPXEE$G?|8kpUdfBDKHsK+5BC`lPrSXmMTM^fGCAHcCc+p5*if(0^ zpf%AiU&C(~_D?0tyX)socPJWDGAu8uRJmES7fhgrkyPThFB)ghDbe;B;m8&U>uX-s zV=vCUdqqh0B^vkz+1ycCKvF?*rRNL~6yK|oF#FoQ$H;q~m^g#B!``V7TOR9WR;=lh z4P4M+-d)%(BSk>e=G;l4xV=8ro4s&2wWiwBi@4?_hXen7dk5u>X1(_sjmR1Knvlh7Anz<;eQ_M7`+ApWg-` z&9hM0u!p()hrx*PoX)mVDKSeGZh zOZNnx+u8f$%M=-v?PIfQ4kWM6R#Y6-BS@p&BZP%DJ#-znH#0rv`Wf90lJZs4J|sK? z2t1f>-2SP8cae22mu1t6g)3l9Q{JqiLiF|+ktrE5g&UWbcE+Vj0G;1h)ClEhrG`xo zn+d6c|L}=!x~tkyY8hLxJivq9acj{%IfA+>b}J$siMJ(9X5#E3X*W9KG~uM@+Hdmi z<=fs@-)#cK1DpafV1>UrGX_`%UokiC5~mWu3IUh;RmqPYziJGqHc{myC3S`)K5%3* zz!k5T;i0m{Md+7e9BA5?ve$pod~l;5;YoDlFb&yI0>w5`w8+7L-rv`zBDHCkF}tK; zF44pLH^`qK&_w4Aq+i;wV=S{Ua%|yT0uNJ*>Y%FZ z=FeaFN2CXnsmN_`zhm9{ps*hIE9+b**|MoRnG?OiZhy0Hhaapw*49`=BtG6QkMz(s zbDmW<;|6dv_~f`+=y@1z=^b!?S}>m*y96Av(x$x>;OweNWIk!wQSCrHeT+`7*>jR> zp?z4I#Y)q}(D^FPmR|`}*jSkP82;#l`A-F`v3Np=A*nca!JF1pMkOP#Iva`*!p3A4 zE<4BG{Bh?hYXbnQ^;mWV5sfM-Q~m$&KDjj(zBnjc-|_<@Stp@tZj7jKERT07+>=!1 zO7o~Wap6gl9K`1EXRiRYmEXxm>m0rt2}eNL6-s~&OWsjp5yqe$=^`ITbdP|4gz1P$ zVp24Jso48LegihQ)+2&V0WQYiuIgQrd34f1se3&=*>cyABQ@uZ2638>TIY~8CIr_! zp!w|QZ72y6aFAxyfqik}RL-oHZvPM~z!HEl@(PIK>u*r-Ce%J0oA1@^`cYRN;HwLb zJI8~n*Y`tTGqF3!RwwE->i*S_L1bE2w7fVTws-nl(2+NZLsX zdQhu}nsg93`}Kl9V!=T^7-}nVZYX!D;T7@ommw#LIxEM`=+5u+wjQ+bUSC^j?5~>E zOO8Noa!p4e7wVd-yAM|N>u!5J*18$01pz)u4#}0oMmC5v zjGu5pW(Dc4(m%X0u;K>=eo0o67Zw&#ZN;0)^MPoX);7g6UR|4*<7~g->j)u~QRpWH3yO zG>qD^omt-JKP>KK&BCKoTe7-j))^Oai6^kP0Al=$&3bD`#&xBp;tDN5Ws+Zn&Mw&L zBXE`(U#HQ z0)BN86IFV=7#D7&L>Vuvu)OF^*!{OeU)jsrw;Vrf(ZfA*enNVJWp~Ew+Nj<_ZX)`AeY0<4Ih&#zCHa>UNSK+- zi}lG8GxINE;FawgFQg5ZN%PC`%@JKKDeCbjnMKB(zK>|EIsB2T%o+L77W!d%wxn{f zyv8DVf&jie)8%}b3iX@fZ(r;KnCDU~Ih1+S!=sYX$@hbP6NDmN+vBbizRUUFS|hn6 zORPbS%HXMHsg=V7OjhU8?6Z{T)*tUg3V_4L#_7?#@`BRcc3IHSgVws|2^h~9L>#_M z{b&~2gw?cSJf~Vg@YOp<&*}3oLwzvQH|+ESN)I_D|0sA*8blq4y-%B}CWd~?rZ5d= z09vBQ>!oHn?#+rn+G5tn`Gj;MVtE;#6!HAW>il2q=KuQxg~Na4w5S$?^qmjcteAH) z{QEXV)$4KNe)krgbY5pPU8~-S`j1_T5kS0*dVo3$zQx$4(SeepZunVjc>?KIP?Q`o z-D=kQltNPB+u<>W>?H?rTov%XP*%xmYb3FzI1`-uGtiMNei(AgYPMwTh<6^&0raq$ zHFDToFzd87d-hH36JhaL-D*5pf>3Pt@(fL*YNWWnO6q+<#i;!dblSWamDVG&t_FO! zVL~wC7snEZuL0Cgcw=IMsN=L*7e<>RS%B8nt4|QhwEh^#XIR#3O*SY+CKgn9Qz|wE zfbVDfz2U{Yu-hQ<$d%Z4*+V4SZ0BakjERXX8r)m|p^K(|{+X&E(`6~C-Dh{K0=^{I z+?2zgd#gylA|L(L)NG60%#3!T;Q}}Qn zlpzxZ1eyh%+h^QrLP0G($u2%4r4{TNn78P>@YOnFUXaIiQeA5%O2K`Ei6l9TYWRF$ z#mQ9@YmG+93gq>sv-U3WbN(taKG8{vO>8bA>w8rM(4H*|6}?m`^x{p zZ{K{pl<~eQF#j|hq;m&+_SegB$8*LaU3Ba3*3`4L>VJ5BiZ)}!pwoTyN)O}^{$Qdt znOpR+cc}AWcn#1;n};}EU$EsrntqF#sKBcDGtL{HqA31qZ$b=hMuufHkNABLSSkO5 ziBymZV4%AGv+|==B(7BZ$v?cPr6ce&AIjP**ezW5NG2mLgNlybF8fXIa6ge#aQ-7S zhJQD6J&adPD9xTwVz1=$QVcoAB}uBm%2!Gn@}6hu7&;H90znBNk*;z^kdo0{{cARP zLsi#q!sm5oD%g9*%XxE4x+4ye!DLO)M3&tM>BL9Lk~_+j7OvHXUhw!xtl)R=%MjL* zCClS8!RCqFb$(LZz_{5+usQv%NFj%c`4FGya;D-dqtFD#Y+gA-lu6_-DYf)nxpz0{ zhn`QkRvYrK{4vpzPD7Bq?EX00%Hx9r7sn@-H5j5#y)v8Mbx~Tpw_3uDo~HsQ#i#BM ztXlCoe?HZGm?m2B@fGMm+31kNz~!h&RKB9!hgahS=Fv22w$*VK$3A6X`WaZ}I6`I0 z5sge0=cgR^NJiJBnjOk1KhxaJU=q&x)dD}B`57NcVtTd8B(djy`XR=e<->8QDBFUK z8(}FD{`&}^I!MNHb{xN;cX$df?bnC&gwS*XG<%oUZTR+R3MvYc9LB{x;B!auChL6u z@m^Zrzsxdeojh3(+|v)A#rZ{RT7+#%Pu0#?^pLwF_vVST%clpESD#cZ&sv?WPn z<3Il(SX5=!H_J$QaghiJUH~g%f&~r*Mw6>wNq2)uk@(z9R6x~T8rrP)1LuNIoQLbo z_>ps2_cnu<%b}d+R^kk^@s74%OEUtzb!NdAvr3odXXJT{jv%gF^Hirav?-0)TRJa^ z`)dVF&rf8|L(J}G{9!ELF_I-($-MW``k#LK8Z2;65FipKTq;3vY%hbUsI;P)Q9mvb zf7!QCw9{xa^16Dlx+BuJbL`{1?pYEGW8EcM^O(%tVM?0rGuLOz_}itbQXk1<j!>ji8gHDe#^syangAX5SX^7(Afx~pBIp79`uI={=5 zdZ}-Fytsg?k5@Bsc$SxB?=WYsoPIu}QE4&fUfhVJUL<))2Q4H^!D!*EzxPb&@%<&r zF992C%TC`Nbot5coict37;(zC)jaSd`~fBkv?D>Jyb$Lu;pO{9r`SGFVZ>k5F6(54 zC2Ukt#O(0@Aq%b>bUqAPJO9WVIHdM{!0f`=^T8Ga&A>wD!&*iRhqrCVnq9ldf-J47 z_Ff2$A>DMH0DruFqerR$DV@npf-K3$>j+>!FUOWY_eaOk;5?TmtVWmC&)3vfzdqa@ z-|T`ZI2$B)sI3>Cw>#iF=oy2@p1Xq}s|E{-rF2)-Cj$LyG0CyA$eUE~>dvLj?+}m2 zm{x76f#8L>`6;wge(g^Ei+mBa-h7?Q{%w^<-7>MbWb)VbZ~lqFZfg!3g!2jW$Q0qP#sl<6zIC@2W{T=r0xLZI-%hi<4*TcvaQg(l z`pERKjHt<$gJS6|nqD=Vnxh})!NDjDX?87CZuBhq* zG|X6~cn|?(AU@Dxgtr~uFQw+{ZyfQknIlbP?MiN}g*i%1t%V>bwqRRcQzlMeKzQY2 z+d0O#IA|!OYK<&z{7bt`ae-#6{$^LY$E8b4h#MW~swCoyCce4s9LTUhHADXtxH%)G zG-=NR6p)-P$_dG79qY3~)Bs#Ras)AeKX9TBlqROXq5K{F3Ha?+bG_5Y8yAv=;9Xxw zqg|bE=Dl)wa8Y@^ewp`0i2%=QA*uqzyB_%GJ-_{q+xp*TB4K0i%zocY_mOX?)=R># z1|1i6MIaUHZHmWJipfFqLVDNcgo~zAukQyOjIOQdzDL>ei)7zT%<^_Qtsl<0LB%ag z88njY_e+1D3AQQ!fD%RS4k_$z9sX1UZ_E{%yx2dAfBG&IxmRjItc)(0)C;r{gPy3s zo1HS=DP)+Hji=_eB}8mW_%~5~1x~7*D!(gRi7YAkXy*|yzj8Nh<&${8eT>lQa!1`5 zlqpWD$~(%d=i(Hu3qKuX3jg7eH;;$DtNtS3qQ$jmB74he(-Akn83F%m5SGVYmz}p& z9zPOk6^`^ygLtM0aV;x7ypH|y!zY`oO|`8UOAkhgoQF4Q3{?kMBq+FtpoSLOo0v&M`L#8ePq6#tY_^|hIew-0&qw7SH#8Cd05|w{~SDr<8jZt zXLKuTU-_cDlgJpqMTR6Y%S5tQw$<)#Gt{iSF8S3EktdQ3jVF4%@AFuK6NnpPmpSx~ z5xMV6JIe8Xl;^=~}%ZX)@mM_S54p6#IsN0sjH0E`08Z(62-~64X zNg(P`0S?4#BQiwO1`Fca&N}&f+(ur?6zZNU&ojluxv28sOtY5Xq#fegp~T!^G}hTX*JLH)_!kNA;;eV z?rKiaj+F_LVbg1X57b(&ernRNvZFwt#(P_3t)zSeAH~Dpnn06WX22^yl+1twXP+_3 zG(6|`4Ef?((%#n>sPAB-z z5uEq>)W_7z_3@}*(QDK5?6Mljhf0pY)RuZLm!<#i;4qW{DgSQa_@uW? z6uq)hx%WxF4&PPzP4DwfEn@!R$voKNmHGpVcsP=SE~YLwOI>v&6|FhUu5X?Z**@c^ zlWkT6$*yV&U+`s8sje6z-KDeV7{9C1PwFxLqAOdWti7%2>w8TQ;P`&dY4jpcUjyP1 z%|#FqDR$#L<%vScf9du&ot1voi!?M7MF{#ntw^kRSx;ICTb&_ZHobWjA~efKHE9y> z)OpiUMD(b7J?}SkC{GqgthF{{VQt}ZV)RH{vZsdJKecLA1~jD7tJZz>qKM@wf!i9GU&zvG zbMI0Oxj;LDPYy~vj<>cyFli(o7OgE=6+#L=L<@{CNL85zczLJ5x~v0I>W=8{H`G6B zp&2R`xyxv(r=5A26)edAyM*E?@k%>Vlv8YR3b()Z()_)XX_Mk4)#0Xsan@S%uILDvW$Z$j~}f&emfk0MzBmmGy9o^w^Ph)9X@d(_40bClPx;d?zIL0 ziZ;!Ej>obXUINaA=aQ{wmm9=o@^vbg0{fkVi@g@}GJef@GbZR$fs0*YC~JtQ(V6k) zeAYU~M*>*uo?0v^n#X^0{%h5xbnS-J_oVJ~AClN6RmJdu@8S>RQVz)mR93b005Yn~ zLB+Xj14(4d`;vmnM$5%1x*vCcS#w>QOBt8&6rtluKnXRN@j@PVQ24|0F}D6J$#2@T zkW(CMicS#LFZiK9VKT5^xNl|>=IexdHA|v{+lA?DyKd=8EApxjkr})3L?ds9VDxGv zf)Am5zA|50sU*l{yi0dMBuFmH5_<0r^Y$Sr^8pcq{C9jL`xQN!O)-<=mpOP@aw zCBXfwwrXhu9*ez67{%cK4kvTX5|}(~c#$5y;3hdDwwKw}eHV3-w1a56V4SKwyD>$q zl{z`^=LoHUNodSe#er?%KJybEQ<0SKv6Fs)Fd}Ukq6g(WRn@lD{26AwA{njgF{MtM z(J!)dX^xX`cu6{3)s6V3p3+h^>VO}2X}ZBKA16rRm;wRkrl*WwCWNBHdSn5t+z;< zG1bOk+4nY4*vWo#5^Eh#)}4(9-r%P3knH8vOx#nei3OBQp^B@L)dL-(<<*!BJG3Zg zvn#G6Bg*QT#KGvpqe^F3olFE(`N@1)>f}?1Kxvcb&u<5)Z#(LKI+@O;bs`e3OcZ8{ z%%X03AJ#?tGGo@RYJ-*M){H~-hVuu{MI}YQO0(NLlG;or`cFKzFlk4BRrij6vv8?j zpVD7o$8Y$F7!*v)8?5&#;Yo63DXZ%0tG;F=nem~^f^h>E^-tJomjkYoRZ>*w6Rhh+ zb;&ybZzMs&TIAi_+6J=K&h5+qPJP@l_HC%B8T6|mNDy&oNdHvKkl|z)foLIKZhuL< z`ar7Mpkx&kek3mNYHa4@KYMX|uRst1B6^byi^qjjNtgQjL1qZ!__6tP&Kh4&53E|( z(URE{I~q{VAIT;;)3v|pCRRT-^C$T`c{f*cpBA-GP5>5*o+-NQI$a-L#|(s+#*$0P zKTemc9;(ag8u!bQF|QaePa$Jx>AcOzH+4aA-lQ)hZ8ZucYbxUESAHcHVwLFe53mo+eO7lB0Iw{8{QLlB!kE% zPG*oFm{nHrbf#fp;cfmdTFs~ED1i_lb%&4MO|yOJ?{m?E_14x_(hfQH(0H5u%bA&^&-3Z=dU#Pc2>^K2rJq*=imY&8r8Iv{3P_(AQ&;7GX*Wm~hU+D&gLgFGVy`yoX@pr{Kvh7nz8p zp)YIuYafXh(ow*I&8F(*j4<>bT3TPpkMkoG*JT+v^_0tnAsE`l_&C2%DQ3xvxOqDztJ`vab+3&#Gxh>H)OR)S8UEnM zGHm9r`|Fz0DM@38l(6TVcNo++Ljavclv)`Y+H!S36RW zO_#ER>M^bC?KvC5L=z&PzhH-5)V=!7=I93Tozc^rtj3^i7be!8)<*IjAR&or zQofEs3|?zwiA}+d5E&k}MU?4LNr3aJ8cJMj2asurC@wf0!d5m!`ii+~p!fC!Xg6{T z;t4{;UFb^b#pcqT-b!%w`bcAuSV94g!aTSu`*?XkS$-~st(6dZJ*^Z3i&nzyi`(=G z=5x#byc-K6jBg3@N1GIxxmNQ(607=~synJmK$+BfP3VMPi{*1Dp`E*qVykQxJCGuS zU6mELu^c;pe9w@j!qK)Bg(uU+RZ&><5FMT#Q<(HzbfG`(dg#ybRQ-eb(2nRh#)ov= z;~tJ&WH`g(>t6z}Z_EyAp@ml+3&xB|w*JpG8``ir#M{G1+~shjcMvVsCK;KxnBiJY ztqtKG7v_QKU`C-$F6tP~5-RFypK8n@{lYMy0y^R#u>|JV6bWI6)Ll&87%JJkUgeNY zXn3Gvh~MHZ?{L5=3oic3VoM>fd=>%j3bwWcj!=lcUD#}KYgfzuqf2{JWA$QU~VWQ{!6lpcUjpck5oRuF_GA*v} zzSLWY(i41``3v~riof)dY8EO$^+fdtfguRCYbntd1`ANkNd$g)&84_&RoI-=m+tBj zM3AIkSgIiiEh?=iCnO>;RM{ZXuRg5&p*6Rx742v%M1{Mjp^4(;l9tj-5A1iKS?EJ_ zUwk(pC~vQ$!bCR^TH)a}spieyQ5d~30@%u-f<3s;Q79oa*H%7_41M4wdbqif70&FI zPh}dHpLyV_jdwiH%PM7AN~~J9%cwtIzEUJ%>=r;1kA+~uhx}@{k!H<`Xt~GWCM{|B z!SiGnm7fcZ2HAb84@>!+gN2|JA=ai-Oy}Z&+ME8$wtJ1G6&_N?@DR60zW4ioyCZ&m zX#w8w?RvnTz;r7nGYvsO^usfGKI^;e)uN%%g~ebD)XhA_XCj7&oA_o(k4eYhc1GH^ zJeev`b;#UV9N@DE61-Gq){7hTM+Wlx_MUtwioJ_MX26M3|6cs+U-<5x0?|3zqZuYU zOu8i-;xA=5KD>nZ|Kfue<1N*slFOL#_PlF*?K_n+dkOI(zPqETL!)Wk+XIqfju8S@ zKd*EgSJ>E^6FI+EJkzNJ|X^%IBXd-CUeLV^hg+-F8OWhMXy&1c#%`xT1q^Sqn(DlLB%kdm$Q+<81xS3%iYon+;g6tV1=FW_&-g`MkCTjT7gdZ3^E zV6lR^tzCg97m%4!d_%v@S(j__V$5 z-}x(k*LQsPFzD?qB>GzAw#y)(&hdYoyEEO&b5DwRga40QbcYn;Ld0m2PyzOjZ?7Bp)`%dWY;EA6V!>`hNdaieW!blVAee3o1Q2|I5gC%CH&^N! zAoe%r;6M(z2<@WzOT9xwqKXE8aR2}+@YAiat}Y865=iJ&>SUsui~Z4e6v7Sj%B7by5apX-i~c}W>%Y4ob*{9>rAHGb4yky-Z|zk(u4s! z!v2$z;}+OAoixLI0OrqIZfAy5^50S!pgJ6!E^8T}bZr5<@L^mUujs5%YsNq0joJ8- zkY=x=C6Y1_Roz{J-^;wl7VhTgS+QZ4@BiNnFNR!bgkLx*joDm#}bwCRH z%C`{%)1SMFR8A+VJhJ{QN;M?Nh`xf3`vMD8AHBZU#K)(mZIM99I-dPp%=b&@M0*_|ECBwV*&#(chi+ahgv}r&_Yi!N^Rtd?tZ%R?6a6W0}S}CvOJe z!WFqHt3=FGayYf2w8-w0h{u})33(wf@`vn*aRh+7BciOv^XTmFKRnqZU|%K{Xx=6o zUsgh0BoLK?wRHE7U}gNAI?Q4t{NS9%+MTGU_f>s!F7$Dk$ut#hJ%YoM6~t}hzA$H= z%qF*}Hkfdlr732r7g{Q@@+C7WCAE0t&G?w0?mBB*+L{n;zTwbO6|uOQ6-#_34*V$4wb5adIQm7H6t z-8@_|pCGqF`&jmaM+qeyl${C1z4G2T5-UF3g) zVMFvMNvGcvH;R6h&UYR&J0%xU_?Jz4l|u|?cb>cPC%8;$EgyH&ZvHz~`Fcwh@ehwV z{!WUIE6S(f?z6?8uebH4H5ksN%Ibs1l5L5siac91HCaEhwtn&U;m+n7-m$%gTCNY? zInDjU>$W)LErO7ZJ7b$Y49cLi#)hg&_pDyo&fvRlK-l9R@DWBNnW-Z7HOvk+AII&& z;b)c3$Q}t`HHNR*Dd)-|=pSB<=Y_r@t`zq)f9Z~W=_aZogJx{&s0ckwhNwH}Q(J6( z>bdEE2Nl>zKHvy*TiPyk-7Sp2DgK8SW^(oZA70w0e|TIo|M0vG*gqNAd|D9y%SWSI z&}!Br=Jg%dvAf5Aoc2gDZa{7+;kqhiBF9+fvq3OJx>&ZQyGRu6eAxX|m*E)c<^4B* z@Q=z`30l`xq4iIig~Li6RCN;`%-Df%?=0WQh3`KT4I1tt!|>vuJG-8P8f^LA5dl^z z#U$g=LVj;^El!M+kn=&-jwlxL66M4H<`h%^GZC8HBeWjpURlSYRa{43dnS}9cDfTr z0~w96ooRt(=Pf!h4R$WXy!&DM^4->{ilkoFo*~W0f#1^~h;VM)#;mlG`3aq_ov+c| zsI0c2_@MY_Bk?>}mj=4(shg_$k%E93Evkx&Whvwfz!_>d6?(=JQ_z=TjCa*%265gP z(kH0|+rMXLUveq!wwgWQWlRaIPJd6IqQn^%l7gY#D|ILYxbUp-sNm?J`{f67&=gv1 za#`w#iUclGeY%!&qAwl1&|3@%=(tZAmRHgz&S?XbU6yQkBK3viP%9@^Pe81ts7)@_ zBkCGl0NZVKE5O!?GXPxso@uYuO5YNzGECn!mHIsCfy}xRJ94ndey{46S)mv=`!?MPt=cGmjK2++?u06&+)ZFo-MU3u!&4{}dKj*R%b;pKS)p zJ)jkr+A_I3Upc4!>b+_sNIx&~i#TBNxMi8`$SD45G z%5>}n5As|``b&+akd@cdDBfGaIT(uj5huh=`ppGITR^GiuZMWbDjp z5=Pti-covst0n}s?b|_Zx#SXijB~N75mgc!ZqQsBxZl*U_vth88Zb!CuWg-HUX5rU z(ol&r9ZP#iOwKNuD9)#LW1HAkBlkRZ=R;L=PNH|7U%#3%`NFFWQc|7ATS~7ss*r-& zOdljfgm$*?6g`*v3{*9Q;l#9x^S|F`Bor^Bu?FVS-<%XrTk zVr`OavOE%gZ;LAlI_Y5C9Xu)caOU`t0?|ijRO~H7ubHQ3CX;hR9DBG9TC6n61xQIv zR)3?ZS&k=x^Oh}E%cHuV58eE_W^_mQF#@+tT3(vtV`pI!yUp?fEWLmDE>?5H%4E{Q zHodccf9H4aJ@=k-&$)lY-aF_0em$Sh$AiB7Ru5k1yYlmslNAv9A+%-n zI;O-*UH4!`V#it2_g+oi`8Q&Yul5M8@!X953B9(e%RYL*7hA{JQS`C3ePS#N0}|C= zA0dqt^(ZFssslXxp8b!b7@8ig_;G)K<^Y+2uzm&8DRdu}sX&P7wd7dxH2jNfjkxET z9jhew&8lx=kV*1UnACHFpRCU`n;QkQ37I8#P=>l}wROauRn-V?GkS0{U98e-_BI-? zFHF_ez&daOKOIvUNGN1`$xzS6YdxN6tq4nL{GVb3T6X)qWHzDHnO>id2pXRF0$m2Z@jhJ^gOLPMTRlOorm8kY*OJ@OP!U9pxr3y1*A zil$2QtXi-cJe7KxIsDz?E_4M}2%mY8jX!3*v^kzy+8QNG!G9Hy^DQ%tGASglx~h|u z(+)!+#?D%Xvv(TiV!lUAiyHoZxy9y(3ruGXm247=`S%4rDq67z-j^E?>cuTC@Y+9) zE&6nAdWYxt?q2 z5IF`J6nm-gs`~Qs^6R(JgI|n_E%QLvWHaZ;ZYV&K68g)^`4acT$Nl3d0|7H!00>lF z9ISs3^H2l#Mxb zaZ;B+(Sohro^Brg)m6+*ZYe_Uj>@z0XBmJjAYl>ilMst}pFxhC zTGfWOMi>6-3P+b@BxpEXUmI-Dh1LJz&QfV_YqXwo^jz7IZ2tiI$Yc8b#O^U^V)J38 z$Lu@mbvd+@XEM7&P|`El%jqqv}6-Ud} z3)z_dkU8yf)Bh?lY8tD?E~>HhdV%_9qFv*{7lVJqxHw)?yVpvwnpV?&7}ps{{OS0! zEH8UZbma3-Cw0Ho@>~Tqunb@)fhA4BQl)5bcjznR}tSj zo3BFsD=-Eo)v^ael*OX^O<&=Bq`xag&3u2RYx!{Zh){WstJwE1>ftL_xbRL)H$!Gw zZ)Mzn_|1G|eqX*d^e^UnbluIuDU=_^(&JO}eEA9~^~|Lr&edDIM9p@|?SQV=8LYH4 zN;m;nn!jv-m+%M9PW3s4{N7qui{f9gz-)boLx40CB-i5bGtIny!sD9#inMtL;9BpM)B6#Rt>duJ3PDe?NUl&u@tPJ!S~}YBDp*^t&B_{ zj3W(VLhuv4={a!TALQFf7YDZ2_eFq-XXX*JF(=XpwFT`=o%wS}SMbBXJmahX1~#U} z0{_F?Ccjg9b;(yCV75g2fDYFSa!+JyWwb!?6<->hiXh;{=yZ9#2ck7gq#C*7GR?$Jhhp0&&rJQRs&H#`t_S~5$A`iBCnJ*vS=2H6Qg6}8yUSz>pYJkt`?zsR#vGKt*B zLqB2PuItgcbzJ8D6eUH3naaIUg?Yan_bRph+u$`dH&-$AhO&&~d@9SHp29<>IARy# zd4b1T@@c?t1tZzizh6R9H*T?edOr>h^x3oxIaDVE0wuoFmvxp1V@usk>SwFEQ=R~n z;&v4>S60j|y-Usg${5TKfQuZPcJf5mE=I%T|MU_~!uFQg|HW$dV7e0}n|=w~jk|k1 zJc7t$ma@b89yKPz9B&H=Ot`q_JdTB+o zsc!DP`F_x&AS2Vqq+!B^U8iA6L(5R??OKvvX)GvqZYm-8HJ9emThhsn&{=d z8Fea~ikQ~tTIC74F&{06ILD44KZ5`=J&{(%u&tr!f?qb7?knhL#@YwvzUG{2pj3~{ z5N<)+1C=^K+#`6LEuAMt7@$&(dXq~v;wC)&D49yzR`TbFSGM*d9WWOqV{+GVcbMYs(;sB;zm9(MYRnr(>w2D~l(wxy8oxb+ zyw(I1U?u@IgOS=g%gx`OPJ_TDNLWG8xSI!!Lj+B9R%?N~=$kfwIg834Pl zWzV;9kvd0j`Fq)fVk#>M&nGhF0^SPYk_h}GxmN?_DLHF`Hw%?NHMg~#iF8W2b6_xv z!At?4C$_4K?LI6QjkV)9-&Az{;l=Atm|-U{ORtXaYN&jd4vQ`Jrzv7~n{=^OoQHYR z^Ijp?V~%=!CvhUH(qpV12jy_!lwRUqg@%1O>Rh5;nrnR5&QvJmIN<;TK@44wsJLgV z)ObyaehC4o%RI!sEy^y+c^v-Cg)kB&x8zN37{KfthnK@PBUohQP zSJK>YsC^}f;6GIdeC^FL`P8@*`wag>9nt1Nc7|KRxH`<7wc9BXn|SM(MHQ2nb)W2R zncsG;tcYFM5ZMn+Wgya>WjpE-Gb(=bC*`;OksW4O_1_r4T`T8|qbSqiv8GAst^&|7 zi^_UNTE2X$v{vXcD$!jr75;kVgK-2;pN7q8G~Nx~IksGqKlZ2jd?>9KuVe)S-s-dZ z<)0DdO$C~&%~}u%PrD{3E~hYBtKyCq31V5vHf)*cJ%#HW1$_Qm=`KlnNfvj+z{8z{ zyj5Ov@$asNqawBcY^RzEc6Rv1^7jY5y3`m>{ksl@4?T*uon=Q+r#5l)S|aiR{yfsg zuM&d@?L20(t!?})t(Wu8TfE-vwn+;J4F;6f0<2{C=Xw)@o;^h}l@%{_PtLbwgw6?t zNV88$AW(#l;90$4JoEnQ=xt8J1F4vN$WV05K9pO9ARz#!-829ne2Gg!G>#;3216Oq zH?b}gK)Cxo*-fkx*VjUk_cGiBn1ib29PelXZlI;luaHvCN47^Yal&lSs!i4TCKKzH zcR@CO9WA%@0h7|j`NnIeho+IG&K*FrcdyfIKC@58Zjkp94MW{s%_+!%{^_Z$5HDqf zN3;K-FN1AFOp!c(si|J`=THZql1kE9JB0imVhz>(&6%kCZQgQit8n0=qzfr~Uxjk! zT-?I~DXRQ$DzVD4bczQ1>Q+yJ*Y2yFz_$jxe1q(J= z?{@BZCQH$XrRDzac49;Q?N)vct%S1L`~C_SdG*-W zl-;F?IWg4@hz%;{wkp>nNw>JAQQ6DX4L&Bm0gOUoz_#rhtiupdVgmFW! z0H|z$*@Y%jn*)GpcgDVcoiD_=++Zp!RaFQtE?r9xrFwCQ#aVp)|MZ?-`{2^ zYkksul}oCvSgl8cBo8&g=k`!BL!;W@I5XnV{hxtB<@-54T3Mx-JJrqt z3-1i4XqLTRGT~_&}_eW&osUT^kIyY7VAs5shITT}sq{%}!5R)>QZ60WO$53yh5;TmHFIw7rBOO@G-sD{ z?Og11snI2xRN>|xnUKe*Ma}T=VRNYa&W-0SbeVsG`^owUJN;+bEm~^P0kb~CpS}w- z43mvvxy2(PG53GVHh5d-7FtVaxOGQupwFevd9z`#D6bLC-G)POgHRy7@yNe{+FAV) z|D0W-0p56l|8+xacUHYSs?4lu{4LlSgu`3MksAm3B7LYBNwpOxHQ`zjb zXr?}qXj|gVtZRnYfI`-68n-&?9VSfSLW`FKACTtXfQ^+2jjcFwQASEDKh(ES-aihaP>gX)VwjMK(j)LajxWBkQwoixiRwf#y$)YRU~+>%nNK8EQhXV$~s;QO!>~FTC%>GVBzV zd6j3v9AJLIph~n+tWryCeT=KYZI9KqG`7I8gv8sU@3SZg6jjm*y?NfZpHEI3F*z%J z?n~d-d)uL2;Lj}6HkID%%Y}@~yTFjrAKowHW&1TEP+wXNqD6Bjg)5^5yVy~b9 z>dJ+%fZ7=($_7>eNC=^wg163slA(v*kt#;pS7FejTSTBP6COi`+gmE%)_pB5KAt;b z^!@foCifqz`=0IVYFVqPFyRIC%?|4oslnb_!G07lS?+%iwHi)AZfG-)g0R2r2O5_sw9W^)?>^Kud&w62aC}VZJL|%rM z-!eV@d6aqaqUBEzvJ>2Zcu?0FAhk5Em-(FRWP3?!a?oypb3wc!OGL54?g8fSHt{8e z*n@29GfGH3o+@stph%ujG!@+CNp5;TMyUqksc5M`5&z0u3;?KsJ?q;}G>CcUQ8yU> zR5HyrzpA2|B3l7eIK2^7dPCNSuFp3Dym0+{3@>TfDk;L2A{AQ>kbM%e^8H&Za5>;F z(CD?spwBiny8eY5f@$25(MrTCU2{DMm%_BNM+>zJ>I-|r?k7uKk{Kr(N)8a}yoRa_ zv)n9___a;-oQvztB_86DW%(i(zu5=CE*l-muU{}0L zvud;t$bF)CK62u>>W%<#O`woMAfNEDtE70zy$se9Fo~Gc=QZfH%d|SMvAN|GA~3gS-omFzB~Cc)<^Ov+`G1cprHjwJWYuJQ zP!jGyl`y1B(v@N|#B@kaDSuC0zMmJg)pH6{JAt=2JR*-Vfd6gIDa={7U~{Xj1Sk4M)lk(i?eq6&z>%Kf}`un9XIhL6ohJ+z!}o@V(1{|7w?I{$`+~LaZxL zXZMc@5i;DXEH1HCY#d1rN+2H0mqb7x0o3-m2ehFCyEDPw(PWls@xItZC~SSn7r4cK zArZ0v=^@oxxM>R{yKI{Zkv0>Z3DFap^)MU#$Vi!^0O&H3kRQMwU69bb;|F>25|B4$=ENAaFD<`xM zs0fe#-y_DassGQJqoF3%fy({EpC{zEj?Xwgo>`9md;K4tt3i%#kQ3P_7gM5(Nz+Bw z3{aDxYrU!-MVh2$LjsXnkHl@PMiCu)?8ea?*B;S0x`*e~H76Ffhg#a?^A@7W z+HkkaF{{mte|G=j-DP~dpVW`NH%R;s@5}OWIgQ`&$i3Unl@B5cGm<{}PE9+;@lhE*QJ~EZnqtPb3pAPc_P&v|hDqF)jWg8C#+6g)931_IG}_SAJWcqc7Vhe8$;E?Y9Y|MyRNzQ=*{B^b{W&eR+i`uaTinH7M zpTXIQsWye*HSu6n5PFzfyy^2)V3G5*WJ!HDMt0QINGvM^ckkjl#BW0wydEwbFGB5t z4}1xU)ALeTnGf4flP}_S7g7hL&^=0J6Xbw7dgqKkr&@D4%(UIEoXy`$+*-NaKe2ko+3xf`{R48jVnVC3QiJ>dzO)Cpfy+l%H=lnmKW{puKKl^*fzH?bMwf|?=DzHE%pc5j zMsqH2GEGxSw3RII{0xk$`w#Cr)<^80U$Je;iR_3C{I;K|Y0(jOyO7e=r^jLr$c6Bm zQ=f&t3ee`2o7lfOj!OkdO z)ilVDHB8E`w$ylhMnt*#cs)(po4L$*4~KOnOMR&w$!=x&V1Z5PhPSMZuN82>{Aw6@ zlF1zK@WhSeKpQLfsU;ZQB{(A^(d+y`&D@nHN4^J<=3*6JoVQan;X3t!uT2^2C??7K z<$`q6O_remt4-& zt6byLtZU6YLpO07jYmT_ce&CY6rhDE^*L}s`$Mvu)ap+Eej2h&q^)6c@(?jMYopi_ z!b^WwTsD=6s8P$cIple^D_}kuv)+lsk5U1ZO^&a>UKJ!m{}>itt0Tv^CNr#vpo`W=Qxy|a*R}!;38r1Qp|nfFk)*_uB3VOck;Uag+DFjnl*1{*$CodSh7-q$g1r`UZm-oeb$K}7hqX`-( z1cy`II9j{^Mstbm0mpd$=|$2|NU0X1o^D4qMco^Ot(IDw6qJn?F!-bh-mPu;8`&k! z$Ph~sR#X+3FR%&xxVrUdyBtt{kTf+I$vDDEP^*{nk(t%&$w|MltD{G^*c!I3WK&(Y zU-y8cg~90v>HZe2d9hJPVLSf_k;&eukYfwSz7UO9Ej9YW-m8h(=y@$#!dBo*P4s^1 z$IT#v`^-Bt-)duT`xhBLf8^$74KpGpx17QY`duJOA>$bR?WvE}eocE=lpV*Pwb;w$ z7i-k*>W@j}_SM55=m=V-zIKn-Hj9E4tfzYMVjUl#079pK1-7ouDXsa_vRf1zR55&a zM3MhPcdJv_dF{^_fI^_kIo z&%i#*DcQcZl~N;jL_Vx*Ff(BHBlx*k$koLpqlrc+@narF^_N0D#Zu2WyU<9(_J%xC zFhKC3yvuC}%Bt~w8IhjBf>TEq>3azcpnTohholW0fa%d--f6Eg9*Cx?4w zjB`6OXI`EG{TJfw4&JQv_@%Er9?Y6?>VwJs-1eheiZ&bE%pra<-^Fa25Ck=%1bL@^7*_Na)^%Xk!XK=pS=@Z>SDNIZ& zHNKk&QAnB^h&@YB2)QMAqGL;HOdD9fT)m%(2+7q5KVhUGdso0hK0c4pQq+I;FEOdv*9=0Ynh zg!~UrU2Ax5lFaiiRclyf7d?VfQgC(z` z_*t7OZ=y9a$C$n_6k->HI2o^%bPAH}{ zD*c`Q5s^W8%=h-aB8M@<`g>_d;6#Fc1zURIp?G^K2{gCWlnhd@rW?4H#J8U#wyMWH znFxil)5UB4YADI1SugAjP`oO@Id&KHbIVWx_ovoo_J;!--bd<53!jI%r!#fb>2Yl^ zVXG{0I?sj8u<<`TxLX1T#;fEG_6(@<5rk9?EOc^`Wt#Q!9SeX>;Wgr|M8LZxKdah86)Jy#aw)L=}utYT2&1#nKY-uPlA94 z*!9|P#FvqDUrjyj=b6KHWRruoqofXy*W|i9|7%F6Q{Bz;lJz%u3{jqtOnabNLXl@5 zqSAUSuy_sNY(`%Y@36l_zN6um;7b9#gG@~I2-}R$Wbh~}Wm6>%r_yA)Bh?mbB$^`2 zgm)Panlnz1bT`h9_)xtIk3!+ob-%{%19D0#wH6LZ=)osuAp9qG`2F&PKsl7RmtQ97 z7-!p@J8Wr6$A3o>`4&fFJ~h#L)6pPI=%H=`uwUnfO4VRr_HWzCVnaNAM)7!*mxVJ9 zx_?lh&p{MSI;y@*#x9;P!szd!$JE^3UfO+uhmJVs4>3I%1m!LuVb%%;_Uw~AEHApw zS0;fy+$^zEZxp#`)G{kjp(i(qtBC_iH!|pZ!_6Rj(esg{7I2EwY0`{fmTsFy8)BxW zvBaVyKB12vWMHF%N;)1)=&k&SSvD@J#4g-tsZOvlYtlqNB;rf#rK7HaL!%jcOYVHY zW2?)7$lip?m%c|{_UBc_$qF&MvG2huqGY?*wugKY3aFe@WF-q^UC^%1b9AUU}W zXI7%j!x#^;_Sy}YwobQ)dy+xFf1Elwys&RiP(|AWUb+!Nqy>w1=cQ%4Xi~BEZQi9$ zJ1ZE}FXHF}U6hn+<=IkGxTzf=?WSbXuoH^WhDFcylfvf1J>ENsB>rNzVWlSLRRO=O z9!mNpQ1I+vjY}%t6fCd%ufjOf?p}(O((5mOY;=sO7oL`Sw$tOQ>;85mSy~n7K-=S^ zM4HGCT^(CZ4=4l^)~h4srnRcS#j5J-=t^tlh%50v|E)I1!-;2I;hWZj-b|!>P27>A z9y|>q65nR(*$TWaWe%`V>aq&!C%gv^sGwBYcANW>KYI#}(aOb>(K{0Hp|kPWE^)vU zBVVe9=-Fi2E$wMF>T;>vyoSptc({MFqH{Xb4Lfb7jkl(qJ8j(E4qVyh=5Drc<}$0u zD}ouW^{E5JV7_HzK0v3-p7SmT`6L)q_S#petzIrztHA|+%rZi^GJ4erFQm5*)lo{j>LY5oW*rk`uSL| zwC4`bHN8^25c;gC7$9h6@w*%emh5!;JC6E`vQYDdp84}cz!sZ$9(~sIU@tr*9Pv(% zQQi#YM=PckW>s4}V z3GXk~DvAon=S{wb37xNUHXXOB+fBJuOdD|kB)MdGRd?kHHUj@tRy&MFJc$Kh*88PIpe7_2Q(p%Wkv^jbX;XUK-yZEdo5>e;1%DU z>{ua3HLq2grx?s!mqScA=m^E9pggqnH_6IZabZRWIenR_qE-oLok^@)iS z_cz72_puIN7Q5d&h8bsu9PwCv%F$ic8SM!OrE2R$F;~EN%r^l6A+TJE2&%~iUE zc=7uJFx4)w3#l+O_?<^)70%LF+v1~+cbQ&ZG3JzH_-RRJj7dZ9xsLYVu-%@_>Lv|1 zdP?8=F!tq5+%0KR&G7JrK-Oxi)||9*xrDV+rQWV$z28L2%8>me`(It4370&9TQ07& zT6Uf#?H%*mLr4EiB{=5!ns!+c8{1BWI95&dn(xED9M?mxuWf$cKfSt2!t5p6zk z;!+FO(5ch}51u1yO#^S;M;XE#1oUU{UDWln@$o$L@#jSTjv4S!urAodAF#9}=QzYK z>JF4!IfASpD9sXQT$N4M0|IK`iDtzDz1ST4qB(dMus=oxR>hupJZ8($)C8lW{dw)v zAZi@+=65i7L)+G=?7&-|Z)&?5dlQRpo3ce_xo>Ty(qa+6@lNH~9N1R$6$h76TNljJ zye3BFJ;wEmA(iWF7r0TMyBj>kh$-B-q-4(jCTWHE#S0iPZwej^FHeh{!@*nZN+Sv4a-Q$**t&?=MA+rn(JIXXF9{N_he|$s+~#3wqH%_2?YQ1$(eP2hHH<-Jg6I!*RHDj$Y5YO zAcqh>nAM|q&$1K8vD7Ogpe|J`deZyANVi|PRjnw0G3mB&a^di_g3L8xH?#K>r_1?M z!`f%k)W`cqi&pI|00dRy4mDq?C%2@s1!87k_Ln}VmjfDMu-@)@l{O#l_o+MqpF+pf zs4~<{O;54tG_97|XgOnmt!;9cc)_eehrm_utC4`|gj}@y=ab|XpE7))MY8YY6>nMnmo#Eiy_w8a zR_tR}mzW~V`RzZgdBjV}y{kKV-kKSNQ?y};W2VuI;u;gyW=HRG3Upj<3^CkUAOl}& zUpO2dw@f4a6m{n{-7t=L@0AtG{b!JwC^;#osR^6uE7{6Nnqs)mSazL))F*fl9!X_7(VbB}4XbhWF-+x#8~5upti~)2ZDD zI;4fj`20MA9JHpBw4M4{`R#XrJv0nJ2 zAW2;Fpu77fRhxRSU!p2+u zd4ytvt8#&PU0k1{uDzxW=vFi9KXNT5+chbC7TZ^PF}cw&9yA{Cx=0wVOoDL)epicz zI)hkCH3s5~0~AbPXQ_X5XH3T}0-khe9qdwzmLBaUq7LJDgg_tMyQ{*-vieTE$Xi^> zB^TnNFf~0x=^WWV{Yl@_b%hjE8ORZ+k5UFSA9g^ z*cs4ghKARycvZzk2Di-lA02&m^ZP`agnOwDTGM2Hv(u^PA~ggQWV^XY*Y-zDolT9P z_mYm-y;6=+S$V&OJ8rRK%ffn9U>rV>DDLN7eZU20@p)KoEHYo5W9APCRo{rU`Yg%K z=>|^#XY@u4o1P1=L9&l{6n(Ts26eK53CWC^iGj^Hs5e$m_e(z^4f|(g8=}Gtu@G8n zr%veFFC5R9YY^z$w$5jLM7I3p(bMykheij*1T8H0oB+#@vd1+w|&%x&ll z^L(Jcz$?ukSf#3k;)L-)Vkf@+#4Y3%7v&=p%&kOz%^K_cy0Z?80vGWs^6j#&XzJ>% zpaau5wklPI&JsQTqs3o`GQox;27IaZ7ZWv)2vZq)thh`m(Qi}6pAm0u(ve6aRaVAy6RdZ zQu(4QwpokGKqP+%cI-U6f=12NJdyDOApH3(y&Nxnj~C+#w5(B(x15o9@EI)QUr~lF zc=|z=xq^-*=n7L9%E<$Cb9o}%%j_f!n1ZQ2aaDbE@m=*F_4KFHf&z$_U)CIT_I{&Y zO1OSF#cz52kp1tO424W>!hK9GO}FvnL3P}=WaXKYU+jr4L{Rh>q%BO0+?C7JKAS)P zIrq!{hetfRFI)==%stn{0M;&W#y~GuBA*CdeX4oIm>}PJPh_o2C9v{FCCP;6_)NWK zadNFrFmql#9H<`m;Eb2Crx^6xmtPQG|EKmNW&Ee~gPY+_Vjg%3RZjjt3l6uNv@3w` zv6N$uisO%@q%yH>IOcNv5>H+*nc)Nvx@amYCO_{3$5Xf0vFeEPfhWCTv@W79O&CG} z_*sedrVJ@U*%zEp?3`Sj>N+jk4KUN#I&G=SR{W*=Sm+mWe2Q}nB z%++aCM@=wM+v^*r@7i33+1d?|71{QmxBm0Ps3vveljw+w5AQrZ{W02^^((0(0l%W8 zuY!)GCmffwuadu zFpvLqBbg{~vMZGG!Swi?zP@_gMvz_&A%**|r<0ohoB#UG0jp(Z0_IDh<`C36+ozH+ z_2tcS1hr>nuGq0rWjZr@=(2kR7l5<1gj#cy^S8O3O$%vs<`zJks-T+MKs3$zbBO^L z?}TT>n{J=_F&{JQKd5T#OAeQ_;Y99CRmxA!CNU=ibQK`0UfK-t>)m;q4vo(>12?I| zA|atA9@MsEo%+9T$Dsin3P_J_^LevqabS{`8Yc!>@K)2!@xlNlVfYZmg=Hf5Htqt} zGj{lF^fSFt!4K5ceZ(ByR+eSp?5sbR_(0-IXTq1iu~jRx+A_8|^o6n4bHu;!Wt4X# zYTAMf6F{hUcJCe*E%7(#L23atNkRD{3f==@l&wB^jJ=`Cl7*EJtzar)+I8{6c*dDi zoL|Jn^}VSYUd*v@E2W`Iu&AaQl(*uVpEc2zMUgX;1bB%dyYogaXj?eRVJ{_lr;IUu zS2QdNjnVK!wI^idAt50w++qua6kgc_4V;nWrDLgU>wf^9YH@RniFJUgm+DDC@y<-P z4Tj(r=s{)8xmJEClSU1z3L|Yrz~>Js*gE0nWnh9^q4R0zC){y+I9&I=88nsOE`R*{ zORYhKgX1@=-I1FlcY|LBhl*oNTAeB6V0@xJ_p~Q->l+(BCzE;)H}pa3`v0f+wYB!| z;K!nzIo@tRhqIKQnzCpe=GIY*ub-!L&T{m>#r5KrpVT+z7t>p@KL6o~U#MK~e>)j7 z5qd8+?<`c5C1*5zPYFTg!>(k zN{^WS(>y{8e>#Tr{`vdOy0RmiwBerl&A!1kFl_DTq1$v$fAeKni=|C1y*7cd6AO{Y zOZ@XP*C`9D5y)1`-1R<+of8XY)%hl$b7FSC`)bfLtBz#o>Q%n`yBuSN-+nCoyqw*m zx=1&)XycEE2Pm7qt$vmt@4Qk~ulJbbzuuIB_F6J{VH+JEz9ygv#_OL`6`gA7#~e#G zTuluHsC|uP*RKDPP*>^OS+-44DJB^kP|;I#o-FmPcI~p4y6hB(#Sx_ZVb=TE} zExXGyc-Yj&`d;6&lY&}KhD(_))tYROva{i zax3vmZsGS+GkU_^&DH?@sjt~leNj4Q8egsMz!K<`qjzDK$7a%tnwj56iZLBOg$PFG zcea*P4_zhieae5Mw^FgEy3E=Z8)lr19n`~;s+`sQ#n(~6wKMFCutxTt)%m%l^BwvO z6_@LQL+WNuU3rY@0(!@L6D{6&8@)4lbj7*G+2CB^7Nv#8nUmVb1t8|vd~W9-8^qL_ zmH};au<|o5o8E{qMktr7tYOKt2KshroqzRJQyY=$LQ1c--Y?=&F!m;B=)`T-6V~Hj z2!XIze#z=7eu)*)xZQ~^Iq!H{Y!E0hcBBPgT833NMXHO#e*TnOu!~qJc4$Men0l^r z1Po7*E$0l7CpRyITx9>F_EHY5e${f3enD~c4j*7L{!Qe2HL#;d?B66ov3k;o>yqQtlDl-vyPu9avxde92 z)%rf#Q|m3&jiHpY ziDfn#Txtyn{rh&y>3};@ZQjZ2Ecp8wwtP>+myV8YHX&oUtMFivUR!KPZ;GA~%>43( zhhV)FmSJ7qH;R~OY2E#brzEb7W_W=`8kbZ=|6*PASJ3|9{+~!^sjD^ zR9PzSZr+Kc7xUw>!@}fBXbj@4^*?@-0dl7%QN(8ot$ z-dG7mw`JQi-h;r%;i$DD8zRq-aWNU8j6XKHUSmhvcs|a=VWdWEiKMilkYCc~&Z=`r z>Afj>4_se#Rl>%z{F$&@hv}9pK4=_w5YqTw?lt`3U|i61=G_Do{Wf5WUM2E+P?ksq zJq>6?Nd=H)RJ->>aL-S6a%}|@JgQO0TzM#C1ITC|_nX*#xVX8Z5rfChibPh_1L~tu z`6{xG4gC>qQ<<-r#a6(>gh0Q-hSx80C*JWNR{kq`^=CA`TtA0u9UV2kSX2uWDefT# z>(n(QvGAB;U6KlI;WLko$tn0VOg}c^*2D?)n(ev+w9-nriUV>F`5C?QWY43Oxy+{4 zvu9*@$Z4OcYCc&=HDj4f6gCMsAV@ltR0|3a_OCBHKHo_Z1MsnTt9AUYz+zDI>W7u-)ii{! zIhL)ylNTGe%Y{|_ak=x-7fojB>_B=Ao)7M$8`%N1jCr+8^sKlr6-yx-PigwjekkbJ zld8C&vJR0nrO)oU{0B55#b&*tDjBgWoXsGvw=GIW&+BtVMiCObRc?rki9Vm8+qRE( zSJj$;;cc0NWM7s;FR;jw9s2{Q*_W*KTn!(yH?6{KGETzN9d5viD?iv)%Bzea&v3hZ*h*i}fur zeG1o6f$Hh{pp-aT)i@^30i=5%ScfUr^N%Pd?4W-Bi8t9cW$+%-qW**1*Ml0F+-Cj& zL_N)s$PUkAfoHlTYvgmT27R6p64o=5XLwPc#!k9#&QL%p-SDy}djc70M` z>0vnW3#$JK%3i*%?aY!`ekJSt~CL2WJbOg8i!{XG0@^+Ka0j-08Q1H_?U(!OX z+_W9sBp;h@4gQ`s)4-TfuO3K@xl7F-P}j3vP&bGME_fM$J;SlOwE90zblsu|e3e?Q zHr886*C%x~)vl|I2&q^5@vz}D#%9l&#+!BWajwKZtlLL!r8x%2e5stT$~1p;mea$z zK1vT$@rh?TQ6;#~Mm2RsKa*Ef8u&8-tfP;di+uc5fF-!SSH-lM(kw#*QTtZ=oc$$c zCzdOsUL*K@^g_r%PyL`IL}`@JeD8=z*s$5;2)@^@ z&fM>>tWXS3GF-1gT|?v&JLb0L%Ol5!>qle1Qt7 ztMZK8?}kc(6Vss0cG*l-O&08R#77RHC5Ee{nDDgDuB=WHEi&-RMl4OC85~-t?lxVr zDLgETJH&~BpUCKebe6Nn#S9=9g8g@oJ7O?PlN99P%i1x3KdgOfe-);D%VKKsYY<$e ziW@LDwF>FEY(pCC-jbNr<~@m27P&8Jg=Y0kDuyt|^82x?r5k)bVA9;D*ruhZPzYmA z_?V2XF#dhz)0T5Dg&2u-4MG;plEQd{k4~mxJFkE5ALy_yl#ktg+#bel-)|B^eN2va zD`E#o#KjS`QlElrO!D*RPzqm(HLL5K(7(jYdkKGOTl*R4ruvmP5BKY_I~98lLS9*s zzH(7l(V=$8@)|eI!bbLZgkZA5p8wh|z(|UC$18nq%;$fGv8>h#^mC-`Qjs@j@|+A# zAF{334g+;hs&G}PB);a&>lV7(hd%CsL!Jq&=hxjEytR0`nNo2st3EOAHh6##F6 z?P+^l-UjDplI#~>FtFDnnXIe14Pg`l*pYn4(yywUu(GeaL?O|4rXSFAVT(-_b+Dwv zX=Z;s=qnVPlY5E(=?H(wkzY#-Nst6I^5NAITOS2X8zLHN^bsW+C6`miaC35d*D&g%Ug6srO`tpD0h0K(?D4hW>w!PxR&9fSdxh5Rct-6>KglP} z2Y%L-4}Z+I60iA%M7gwW=Rjt7Zcj}64TD^$^n*3IwLiKiu*6@^%VzM*fFX7wKXS?h z6e~!i3m`-LW62K1>2ElCDR*ac+-hKaxQo=gg0T1|)E{DSo*QUMY5!YGGK{|OpyTqc z@CJyWICE!M0ou4>E3noixHNFZ{JxZy>u)1p{^sUNg-S9HeddPR|H0OKN3;FMf8SbF zqoqY@?OmIqHr3jpR&23%>=j$JX6;Sw8bJ^{iLIq5iXt%*d#_NtrTV?|yRPfp=RViD z|IZ&epS<%qpU><0dOjYa8a6}!^XIQS$}XMuJZ-!lteFw|*_3@&|NcC_9ooicTQaS8 zM?VlxT2p0%Lyo;azkh-}&zp(EdUy-+#)(-=Ivol3Ytz?BI28)F)0clf@%hVZiiNUu zX2E)z|9rHh-#HauI;M{>gaqycxzP=)A{^#6#=Z>hf+RkCOVOj~%0%8LQLW(u$3DuZw7qJJgGe>K%VtYIUccS6$2 zO`Myy{t4DSIxn+HaRQ9s%+w1Ej7rSuhJE2RhE=#?s@H{_0v{&KeX5V>*prxLM@^$d zGuRa0j$Fk*LbPA+8|u0y#r9u@3$uLBZU49@;mF4QQtrb6@B6bk4cNVsjiP`}tY-T> z!m*1!iLub2nJRowK-gj+87jK!We9b6H>t1SZ_H~wm1k;lkl2CcuFv6DC8wmL?fz>k zAZ@skS0lQgn;T7Sq3O)+_{*vCNJ|5+ctMk8HZ!F?IN*$sY#bO>gIY*A#ys^v2||{- z9m$SjI#SOC?>B?m;LFA*^nZLRRbG#-KsKMm;h50ndJxl};KU zBKz?*QYXFi^vx>h&HgiwWN6bPbEa`OJ)lfdmjoTTQ*C)qc!#84C)LMPw}*~#oNcct zNX?HGbX+NKQgnkObu5i^aBs;b2oFo{&}YZE7awRzBDgB%zUk;>$T~Ub71BR4=wy6KD zOE{`27Atf4(%BV;etd}!L#iE``m>jZSKa4%NXW&k{23F` zeFU%4hjzwDMO8mh2GV}ZTChSQ1{7vX`4ICSg#4?xeVIdUQQj})O20YP*M@{O%d9l_ z{7>q5I=tH2z#`2hs2p0o>|$mwR&6&bXvj}dP_6h&c;Hvi1r=S*()1F0&15SI*+8Xg zBc<_fl465lqzBIe;b7F)+c=Q~ioZ|~zu@o=3MTI!-$*_q}t$ zCQp|nzn_L_XOtNGnLrd?%-Jo3l4p&Yr#-=Yh)N;ouKeo_9Ua%27I0vQtf-!0B+$)R z*-#EL%Vk-Z{9!u#G?H&`BT8Pz*!^t>(;v?DC-N*iw&7LF?!t(&38#eHhl>se&){ni~| z_Ca%EG+V)GGms+HS-z!8l2Oj@ppHykUPo*B!isS8=y66D&(eX7x+CH%Z0^z#d6F(s5_aBeN6MANA+65 z{KBN^sg5E0IbSCJM@bq>^;Bt??&=$+q2Oz`8Lt^r9i=j_xlbwO0ei4rpSj0)yNW+* zslEG~r|FwP+M-BgZte}A-RVpKp>Qhd3$e1xh3 z9W+a4mU#p4BUyf!8QB8tK#_i)jru)1ST$cac+Jxx-?2jk>=jVIV0%3d$s|2Z|O-8A~G!bT)8^?Iiq}s z`}~uspy9V8BenR#OXI%d8d*wtmT4SbX8P+_?F4?w_!Rf9#M9N{&B7DMmTJvb#XBa7 z0QVwh>%Agg`WLt%;|2XAm`Ro^NH(}FxQXxV! zm)dXa{Yov4L^)7*EzM0fgJU+6-(k12gwOED{IFk@>Si(==qGc+ybLH&|Az8xnzVAM z!1MFCrbF}e;@>~&Wzq~*u4eI#+7cmML-C$H%Hf^shE03ILJyA@gO6(eZS>c7id8c>$AEgxFj(0yN9bezWGSx7g1M#t>nP-or)voa0HpTrB87H~X(R*ta(WJ2f3#{>qr~3gcOjUTj{q`9= zHjeq5`h(!6eQl8=z|cnf@nJ1|OwVQwHUh-^g+OZ>OO*qE`uP`^b)wEklNDua8Y%`2 zeGORKI(td)-VFk-%cFPfM+FWe z#|tFJu)$fjvQLsBs{XTG(GN*;D5s~EnO|cG-0+g1)ssWX{Df9cK{2OSBL~~FKt$*zpVghmgKzOa}t3?i>$?(FE-y=w27)Z;hRu-8eSlsYEp zMfzXR5DhPx1-N+zQ2{+XrOD?WMjCq$8EX#6h2F;cX0a0MTY$jd0<(>>9UQ%Q z-~Dx&F>Db&d!~gxG?gM6u}~)31<+|Nrsl zCfC9LC;kLB7G*%$PVUTMXrVISNQ=QucMQxHh9-g>H$UvuCh%&FvOcoAFbUS*T=Wup z6Sv@dHTyW`GRx{l$1Q1m#nK9H$|A2+mWTB&ga~_l1$c*A4zo1Kw^~-e;UM2~7Qg#i zi~hG(7?D;pK)@vA;$pcZdpj5C{g+;5F%319>9k^$r#TuebT>ULLH1SX*ndQ%YYx|; zdh@&|HRFKDS&K-0dGEk$lNZd&h04Ay^&7rSX~rQf2Q04Y_{9O-T6DxEV=cpb zzk5$n@vfnlPWP7`W2u)pUz1~MIPOrw)MipYf2R3RyufIs)AS22N(gaONMu)^<)b^-zW^%HwwX>GJR zj|Kc1g0k!5pc?e)o++!u)4aB5P9+i6uMXEgaRy< z+c;GT{yetl&tp|mSfBZ^$-~7p_*;YbxE6jny_IC}qfO@2+Q>yp7M)Cze3wcNMctYG zB#UcL%jo6*NFssRce9~)@c$>ZUGMon#nf(nitocaI{(t^AsJ~Cg|{599hmuTWOyY5 z3JR$TjT@G@bcZ_NzmrGvc&;q9CgmhrW=HO@Q*|>Jw5jh_0UPpInA}jhy|Ut`=dMx* z(Y~~;Y$d*6UlQsxcguoS~O~Y@*t4Tsq)Zfou&Pt_F_^d=c z5YzhpelW{lY5O+eQ0Jwf_HHy&?DSic0q~v}zDosc>7VmHJGH(j1Ea;1VFtCBtF(Ka z7Vz#O)W{isB}ytz{(4fqIm@I>~dy4zq0;1t5#s(&$d|Z)n77P5Ot%Mf;goj#5MS~ zy!-zp?cTXr{J&CE8|eQXv}URD=(-*sbU&IgJt~NXjkDYaeYkeJoiX-vwl)bQ(v>N- ziWtGa>ijJO>?)zn^^5K^^ZrAyN9iL-&-wo&qU&3(JLY4Xd-NYsp2kTn{pWwNO6#Z_ z%;29iM}?22Nv-2O*-qxVbS2Dwic0;JwjiFx+-4Rcsrj+A_CSzFVAfwjcMXZE$^{II{nYU)49$JQahx3LN>C-Q3J6>WN6t_OPR5pMxb zLsq)$I2gbFIpZS2yx}P^;vXxb!fc`x8cpqRjA%F^9Z7vb`>ES*L1aNthQiZc@b&yP$U# z2q$D!{71x%ZN(S*GS87~mM$RcozHJW#cx0c3}>mG9UA%IEOH6NQgLw(@3QG*e;S)( zcCXYMdwVNVnz%O&nL0m!Ed#BqppoT zocapgGmV=65XK!oEX=3>jv^zkL2nl zE?yX)o<(*rk-Ux>W&d0yO_flR^H%84Sf034L2Bp#`-zg-HNV_DMkI$jfeDY5l}p~# zj%$eOdWcEsBN;Y_JU9f-3=((A*05;XVx%vAs}naJn{3B1xJ@@1vaqNW5M*oyHW7cs z|9AVJYI>1Ct<&h9tC_C+nwkHivWsE!m`c1QN|Cdb<#};+mY%Vsj_ohxB6xbH2INIK z_~3t9MO~h4qa83ecc&UWWp!nz4^E}~^2+#V0p_UHLLUD4b-X`{I__=E{jKhInAiDtdLB?eotqGXW+tIVH7)#OGNu%H5)=T z9U31Ikd$KKBFk^}(hCHkJDN}b#-A!Pohm0$QLNmGY4JDpN`oHaTOqS1xfY!=ck|m+ z7Bl8I243+UEYRJri)~bAA8v&grrbR9+eoP@FKL<$hemiC83OZ~i7C}=0Od_nw9{3i+j=v)kcy_9?K!qR@N zW@3k@`hoK!j5kahZ+Zt~QD|*Jh?T|zKx$*!7Un5R*5L2=fBS6s3V+`oy|et}I@qPr zxh~ILFtla83tDKR(xDafFBQCd%nsq;u;wSOAvOvs2~~|}i@ht>tV-;13X32s#!w}NNd^ZnYc!| zcgYulVW_u7;*ys{<&senBVE*wsY#BDNAF|mWED4JJuNQ+GtlWX_cxq#oWK+Uo`DDw z>8#nRG`2StMZX&{lo{KjgZ-EDD1mV&BBPF z-M~kdSTwsOn~A0KsuWdN-T)n~lCCxWbAwgS&y`Ws@kga?rdm;=BT45O^7fXo);Y+# zC0EzAiyi)ezXiY`evKozeALc;W56#B3!w(fjs;OUuGeWje*{-c*02=|HX^2uO3WZn zc&#LYl!J|Y<|H?CGRRoZ3X+4oT(_HOh9()BZ!^R+cmkAIth9{PtA@etgbtv3p9C`e z)7h|ipHwhoVe#UW#U8!sLEwVa-hDqh*F63N3VkhA$YUri=EG1cVCJel%a9Id8MU8H z^Bx=6pRg7sgQ*kikV(4DhMVon907%4T*pC|#@QI)2p8$tSC>d*1hh;|y&;z3sz&|k z`2FdeK>!<8$k!&w*bpjK0eLv=q}nsqItBr0xowv~5VW`JL1{XM%OP_spvm@XW{bpJ z1R>X))d`{dv|F>Ka7!4IIsck{z>KGcacYPc#1zIDbOTWJuaX0gmK7zd-I}|ki?d|4 zV(-4XTJf>{m#ATp=Hw0Lo4dafxR1fy=yXc`5yzS9{h@h9Ly+AjRr8nsQe|o=jh8|% zTvhVrE+66q_2`WJit*+G$pXt1h}nG?p7xA_UF5bB$1_NTQ((Bxkj)X6KtIo1&Jv%aRX%(Xzywn6EYP0 z#Z{+9>rO3nNmF7p6Cvz&l8s=_)bY#CtwalwnhoxKojBEO-!ACvuBP7{Gm3o(bHBQr zZD|n@H6t|B)4CmI{4TGeIyr*eqk{FcHXu}#A1t_B9`U_m1ZuLZ1cRlQD`hA zhz?woqn#bnHb2L+Cj$f5kH zAK7L)r>IhwVy+`1%WW5)t04&)R^J;m$M9J~9*itxp2d6gzhTSNYR6L#x71Q~RBpqY zrI-)2E5iBvdAmvh)o;V?gOG54|D84gWqUiajm!O;*%?6V9jVo* z^QEd440phw9#s?d^8oix!1HY=O83k6G`@3}TdhUCPqW!*V@Y*O=HhB6t6a+r9^^By zZufeWe#}FdmI6^jyl=18Pi**JdzNa_*lMuadf+$TlpoxeX|XGvz0XHPvm#_f=QKyX zZpC;W>lf9MO85oyQvS zPFpxt&;X7=SQX7IB|iV``lq=dBM5C^mE<0GH`n+-BK^iFEZIiM@@P!QVx?7sdm(p9 z`Ca{(Kf<5A$TjzgjJj`aBw-J~wTwaP9L1Vhs)_V%+|XHF&k>eAAl^tVvx>9$LE;c$Sz0Wtd`7^oI$`qhRCSXGcaWSm(Cu)&woTS zi`3(SilYpKY)M03+TUbeK zt7HXSn)48yzBP_vg^-&lv8VgJkh|IMh0sN9SFA0GjM7K$duj`bPDc}3CUpXY3Yo9A zATZ&7E$k*po;^uT)Lzs!M;)4H*v_CN*T-wkom^tvTb*JwcYuu%EYdApGP#NeFP!CUg4!g|J4=9YezlkKg`g@l{9S z9JtlsAELnPo%4 zsS0^!I%AtC(BN;PW-;+0YOIHnL z3p#(Ea{aP32~i-pnf%n@W~?LvmvC-cRc`jY{<-kNcR{7Q{_jL0btLb~%;IZTp5ue; zan$VH*GTnXZb;VReVssTX4~#o2TI4>uNiDEby?f{I);{IM8Xrl9Ue z)^n@DKZZt70L00Q?nqXWBfL+TW&>cwS)oWs;p8{$9-GoPE zhR@=@n6|xemkay*qZPNM9_0@*;3TN%UsQ-&M{LF6JRag{+bMA1B?;Nra{0L^^5x34 z?co!;@afz~{+H&A%?U!f1*gWWKR>kR6psdxuBWnGzz+Bt{a;M^y&IDM_c;fOjjw!p zG2&UiCmQS^Spa4y-u8PVL69Gij`1^*8$fM+O=~$ozg;6D9>8OHs>yv( ze6dAZ�kAw|@&zG?&KjT4Ch`HtQwDBk~8niqlR?DgBVf;VjunK+hmXKu8~em#5S-dU_cbmJ zK~#@y7<1BsW+bx1xg6gS$KPm&UJ@v}#;lcjf}3j4rvrZqrnnTP91OTnNJ1v490sV1 z`&{_bm`M-B%`XOr&0GA8Hj6e{d7jZB)+S?9ZU z4vbcPjo_6PxrgU2Sxh_&YDSk1?1zTsgZppa#phT%!h5~k#j2_i=-}m_D`rpXp5#y> zH@}!#q=$S9ZJ#awu0U4LfntlGUH}+2@Ntl{I2M@(WNwg>gLZflaUL&(&qRHo!?~+( z>{+QsL2eACtaJpuR~NObU0zD{_3nl7>jKcMZ0H zVokM~xHJA}x(N#^wy&8eiZPD6BNh>1v?d=|S{d1Mi3Sdyt;?*M>gc$TEjbRAL>1MZ z+&q?BzeGwtkV;ZOna8`js)66uSA!kKOpay>Vsu-2p3A058XUbX;Vby=b7QH|(5%z4 zRLHaJ0ELFwC^EeH*)Y=;@kTzo9{L)p0#mtP|8fl>qKe?{=p`T2nWnv@6`2&7?Ta}l z4V(_Ur2CD&ZPgh_srGsbqWs5=+C&CcMCshjb#Vvy!S_b?#1|nJ|j^;Zr6JjesjW7sIQ> zrE$N;<6~Gs^xE9YiZcCz*Sq7e1ovd|!|Ke@uomBf!nrHwE%zWUP%)4L_xKYvl1f_Y zg^|MkNL^rp1K?$0YIEY|n7t>SemdckZ=2le0zfP7EIBK&5->Ff!D}MFM+bwoxJ78! z+-31^zBW7nilZk4_k)fW15OIBsXc9u>f5SvxHZ9+p!D~vuR5<`NPuw6SSu`EZYgGa zB674o@n=4b<`QSei`RrpdE$JSKrxmA1j9y>I0p(?rhuY3)MiwqyVF{_TRc48+D2s z?4bgk-}1#wQS>VHth2L*mTl-d$SI@{fN!7-KjxSJxK0T$gL_#c6nm$q0Y)$~?;w5N zkPGM5-|w`)YX?1>;{D8gbM{*3%JU&(=Pupe{PsZ|Q|$8x$WLziV?`U`8zr2+#ebHA zf((IPVqOW=B>X&1Hr7id!`Yy$DNFt|7JDJj=#*R>6W6mdT7zE~CgmSk4CXsK+xJiN z(}};oiY2x#YKC7X{qjnG?lQ!;6M_2Sijko^XSSA1^3&06l{vh1ao0O< zR2Z1{C+pC5xI?KIFYLcQk|Q0J3rr0=N&K;voh3=WhD8+@jgv$N-ek?N@4{|3gV(PZSFNmBT>O>?JlK&eG;}25A zS-!hoNeRpU)#q?O2UXaBY{NCDpI1#cPEF`7nM2@TtU}a!bSbdPP`GdgI=eU{?R8xr z?;MNF?fOnlNfRe_tA#j?ZZIJCRQ>Hj!M@M%ImMMJEJ$lc36CIefFIhT3?MQeJG;CFktZ5%Iw^<@+Hw*)i zWl?jlF5o-Q_TMS>A|rnkp~X{2KgwszR31rU_{R9Q{8AahEyEs!XQO_M`w_KI2z-k< zE6rjiX94}jIPUEoy|jutG~Z?MEhe!Z`3k?)Yo-5Mg-?*j#W@N^xX_g@B-STGK>%** zS!u?h0tmPe?Y5uh$MrpNN17*|!PFsvABHC(?PsnstDw%VwSS`F7gV}n@zAE>PHDxE;}?|Q#sXpFf3a@t=_-cb8jYIEnf z0G4XZ_jSp5MdAR(O&GHBIVBdv_dCHS`#N=hUaWo^YlvTd7=+28hrUz>$Y1ff95;%9 zape};%{VW&C*pL}Uyc{06R|juc_|%MDyLS?q0F?7rD5$PZ#%$-GKbePVNa zfuWjbfzi()5m>~+^?;nHcwSy|k{*{_U$)7QNaZ(M;F>eh(O@gvOOar^$L~vj{Q(cW zV$-kJrAC-ZGjs;pU2*@-<{wkU&iQMu=9bL+vwiMnOe}Qkz!@i8{t@e1^!RYFX5qwM zylRxqbbGXzUBxq@{5UHen}bYqGNkTR%nP zap@jpCzq#iVyWrPWAb+@ca-HPq{xrO~woao<(WB?Rwg>Av=Yssdq}ctb z8l^hlNB~>ATcZsJn-MU(_Ww{sjd}~wL6pMp)Ua4aLqc|<8Nv+(n?Cl9MpGS)euR%X z)oz4zaIUz2+=L0?ZF`%}VbmT-%~QwK&Kplruom$h4T0s? z_!?`yQf0!hfK&VXYYq%S-Nx-$W(hcMZ15u>6~MZJI-?7k+-sg$X)jE4j5VKesQgqj z2N_MsbsAu1atp$1<(Tob{-wU5pzF+u{Y7(mUG354;sJ~~+~S>7!Kbd2V*`sE(n9`F zi3e2qTzr zppf*QwTXMx^4}#-{v*2iH!SkAiyI=M_spzMTagf1B){YM<5DMUL~{<9Ls}UqcP+7W zS!Q3S=YCQ*V2L5384Ka&zSA{{b>BRwxOx!L!ZiLe8#Hf7$kF|OeSoZ>L^8NxbMC#k!x!9m8v8{r*+iOkm1xij>F)W) zHPiUM|8DZ*4NqlJp_ZTe>>kN~nYR>UC>jq%UEJ-=@C!1Hu>zVT0DaQ>WD4XMg8qS5 zODB#S#0qXd79on@DkZV}6#~-Ukv?AOo3iF|JE&K~S$gLiHt{G5Qwj5^4R)n8_55bv zIF`J7hnI*ZjA)$S!A%H%0QQ~pehjDop8Tb7X2B_2h`Xb;MLEHMn@y<@;@VuZDd|Yq z-F~sj0aJINlF9LGLOjXwNPqoxf%#1h0S$zHyNKW(E7Fb$norHvZD&e~CE614$UaOi zdn)n*0G$3>$@TXX2XTc1hb98`%>Y0B7Fqrx_C9*=nSgMfMnq%z@LqAG2$#np=sBZJut~E{si}F`~7l{?Z|`HpXoPz!F}i!7Pl||O<4<0(>3jzMgY7F9%={3KDkeFD2zN%$KIyDLRnY&@DYfOV3H3-Lc#l4m2|~jFRVG^sIQ6 z`AgT?i-L7w>G`X>BLOeVweAf0k!|e38TBQD<(~f4a@Y-eH~;R*e?$pyBdCsXkNP|C zf}n-T&IG}lM*iOAPCtiSQmNca3S{cx@nlskHu{?I*~FD&BdFT`qg}&)f{{Ijxn3|J zuLHkI(d$?IU%5~Wa3r@9^{>A!i+10NVv*&UipHmtcbz7b*Kq3{&$ZCSafJZxlaV9V zP=vpUCtI(+{c!{{1F73fm5a2?M3y4Kqd-}n@;u91y#yTRa%<$HGPky6XQk>apRMQ2 z{K(AWTsq-95-f+?AD1;kmMpGhL-h>J3tRV?<~p|DkK*U<9dcY;=4sIGELrYv1r!IB zz)%c=>8G8TuUt)3Ws8D(3>#Lt@u2E|RF6PV8+3j**!}inv>#F~E-mxM`uB9atJ#vl zl)YB5Fm>4r)UwUJ&n|QG0%MgT*jyst-_p?(^C^=erm3UxgJdSkX+E$8Mx6T6*sn^g zquKOTBJA?>9iKpW@4!5zscgM*ELrywoh9yk|1WTSXjg7H`G^01M9YosM%*x!Y1DPAu}j%lnrL|+-Q^0YOMwNMNtl>L zbkb;u1P{;X*9PA?(;n8Yk@qfEUX)_h=C9-$$aeJCe#5zsf!&yzzj_vO!OG1m0#QI^ccFuqa z7?cAaBY`Ez`4K1a=t-se5OzmtnZ6t`)wc2i77J5hQW!DF$E^3~boirVjci3@;vE>1 z;8Tr;=&w{V12s(U5rXsD05)>1eL5p4(FcDr#v=KqwI7LSrsftIj>>a1Wwq%!3-}Dg zRulPcGUfZPf3)3tSW>=rpkz8{LJC2XecauT8s7%Sjm_DYfM+nrj+D#(-Dnzo;Dg%P zF2y~AWiI#f#i7ycFyd(6bM*2`K2QYR9{3$F16_0$Fq%UjRM-C^Cj9107R#aO?F%Ba z)ylkILW+`OKeT1aXG=5YBEg>yEoLGxq8$|2NbdefQK7#^RAmIbF)ka}W~Z(*g({n< z(LNeMSj~ALntLs)F@Kl12y0RXhng!UfbI&gx!9KZV9M@^%^i_N{F>0K6J~%OW!8yX zLO43po6L(zsGJ0zHohv+L4IsHWExUqv3;JsyvIUck%*&NIue&$#Dj%Zs5UhxRWm~+ zHo7PlE4C$V+RYng^Q{b=Ew^ca(@Br3q(Ez#S!H_G4gIY*>$*dI{X_jZ0n{yhtXtFq!tXsTr!@rlrumFm4WWs4e4YclbnX| zAibLl&ReFbr<$YKv%vkWH&q8ZqCJ(r!@Fz1hUV42ra(dkyvDxzc zDlnPV5{xnUYp~6cD!TjonRgSkspC4jB>GNPEO#|kcjL$Rp*Pocz>Cm#Cp<*rpPD`~ zQJ0}MUzV;{%<3^VD~_s7Z0<2A zw~$MT#kONs!T^ca-=@Pt$B03{{)c*~W7YbQop<+sls@*pR>J;Wxag^OEe*|9FPF6I zfsATWSQ)zH$42n^>^bbk`9l(rr@P8^2HIn^;QsFG6VE<4wxM!sZ%a>sdGfx8rfihiUddC0+#iTY;n9czTmL!=yC1}675brayZz>PssA&m zoNY(^BbNAobv#W{$|u)oI{cP*b7!e+vNaynmS9*%bBp0e_ayF4XAiZh%Q({}Jka{r zntz(@E($@LLy--to=1PyMBJFAI)xjgRD(wpxWzcX%kOLn4OA-BjHn*#{h;SM@am`?~6bzfOGMO*h z2o6%*^raj3mTQS)5?BZjY>fy@a0uM1W!>%K?xit8{1Jw7 zh49u;oVkO?-9+%y^1Blvk3x}#1zQx~aS1v@Li27L@r*J`|4Q77g9r*&UnGi3as1OS z+{u{ySKj*KWwAOItPYgw`3i+RO+}lO$=}?}9fgM0SjvGWWA!V$=^4DAERNZP`(Dm- zqi`fp!<^WcHL=BF&uid6qI*^Y5S_>@IQYS0dzb5dSk&55liS98KM7A;m~+20WE6-7pE0iNmgie*8W)QUX7udu*rn}wuG?vo;&f36R$sz zMdRZFo|s@ygw$EL&x)H$o9w&TH@y*biVfcBlC;Y`3AuP8HN&@>o3E)%LrO#h_gO~< zlFJ-6!bHYo>xx>!+b=&2j4)Hvr%1ENjl^mfu!G7fp)tq>Gwx)%tl0)^@4%Gl1H_0} zy`kM=q5K$J)TK~w2&nr@pu*FrF6GC79aL`5Oj$o8vs{;`uISG=xK6&{-Sn=6HF3Yh zeimsa7dh+9mH?`1h|G111>@w58fdrZ168ZcGB$h$|X>Sn5U1tpQq~ zIq{Q=;S#XlVH9MzQE4kF*0f68UcBB4#S8lAvLy&i|M@p9-Z9$~-5pBFW`8$Ue-ScVsDknx5KUQozJ3+ z>5k=-20G$8uQRXC_Z=dft4*US)EqD#e!J$cTzzd91!K8aElMz>K;g-AsY$+q4%4U5 zE{NgcX=3icE>&9?@(I~s=&yfy9h=tYoel~O;&>5%jR}w}A&so-OObgv+f{RJNF&kin3nP4h0#iWnX{gWcm7BaM?Bpsxf1Ip;&%U=`qq&`!Fv3lDY6DcP?S=?tOEY zlTrx$VOU-(&m4_{q-1v)1=G6tJL*G=&Mp#H9Kxg=LBqbZSw?YMV9_yz+0kmD^w;vl z2BzlGjaI21adFZ8a)$GxbL9(~YtFl!_)fCiOx64+40}k?0=mOxLY>I{kDcDY+{xTC zX?j^i%ZDvNSC$;xkh%&qE??k9+c9;ZI#^IunkfS&`nhV{n;phRq1{K4{pWg8A8uaI z`A1N{FWAd}$~s_GYPwsM3r6Jgh0cD9>fUd+6fhF&BvCEAVgK@daT~z=v;-J;SlVt( z%>GNVSgtfB&!9cnK=vpW2C{zE(qB+<+p1F0uE@U_!P!zyU)cWeO{ZqepVe=($V@$3 zi^z|)zn29@?7&hWUrZLGa(1kL;I`#ILV;)4!tln|A`{!>EIhJV)CnG|%9|F5rWfDT z`E{i^#$teYb=p_ z;qSAIH+QXMG;J&C5GXU1MimpPMNUIxx}D9tH?%=rol8Y6<5e_rsyqJ#3G1I1K6IBX zeR?)LD0=c@uTT3?Oim7^o^A?6`d?WyOqTQX@z%-*E z<=i55OT|l3I|G*+fTD09ccw&>%uGedrpZk^(8&LoC&&pQ>MB?sagmhs;n3UA1u9W0 z)*ow*o$`k>8te%9aW08SOTE{4xvCAa`df!;skwH$2c}tSUpy5~r=8T~x>5NB zV%b*;bhp?D%J=V<_m3Fl*{b88=TX3hW)}GnnM&VZp6?DHhjF6-j@lpOmX9Uno_wr5 z3~4FFnHlz0eLhG5kbhwC&ysubA5nG5d<3gEQ!xA_eCkSqz!cf5_MBi97xwzy`_9wm z%beAzWn;#qqs^mjS8yTIb-Y2d2iUF(lhb?J{*s_#)%YP`YU)OREZrvQMN(*VkL;&~ z?x9p^Oc*){Gj8F=lBBHY*|fCYa2%>^33RF#KXvdWBl^YLi0t@rs}$De&`%E&;{Ihl zH2o-F1>!%0POZGMVUB#hQ!Omx*%0!NvL>|PijH~Z?H;AiHk4bg^l!V#Sy0%M!N`3n z2^Ni{pUlJmNFgTrZbM0TPAx>J*-Y=>T_ zP(K@D(j(qc&@1Ww(rIZ6d(ORsB@R}f;=f?Qj8a&;jLGRa{A!?PUm!#s))*?ap)J-G zYI(g^O;`=*&;#@0*9oe>O=f6-X_qNfP7avP#!$~f#o90bpmsYPUF^@&LrrdTsAKD? z7jR}6jIX8R9<&@P;v;&-bsi^_ZbWw{P2i{0z&g}o95Bz?v z$nbMBXvxX;$nbw+8$dP4ngY1=UA?oV^l-b4-uK$lec`F(@xMn;DoFAFG+*bm?^}4v zYnzj6X_zYr_}g?RIna+YFxaBJ+>5g{2GV=>CqLO5$7^&|I_c!VTr4aRD6nRP*ULQ+ zkBi@Hz`>-k(UC=j=~&j)1NCy06>H{tv9L_9-CY*CyCm63-M5-Vo3ABNDS(bc0OJo{rVKZp#}M3zZ0q4g<~^Tfo*IY7s`- zjyF$UW`@jh4~>NpY%+W>F5g8#Alg?45v5L{!#yciranA^iPx_F_6`^>_!6=ODg>PI zMpo5~`0D&`k@i=>S2{n8-{Ezmsb1$$52wg9(!$x$k=Krk{EkZ8n(>}o=+QS{cGaB) z`RL|s7QH3Q^8$ktzlj1^M{uw@+S9$pfnDnS+oWTpgJRQlwS8btuNyrkX!>KS zL#!6$CM;3wuD-Zdp?vdp?cJ|hqx6|MR2(YzjN_Io+&A|(d?*&)ZSKCG^Oh=Q=zq0l z2&AsJYqjvt*@^MA5)K62POJ4#);$BAvH-6oM_O4N6z(sd7kn&)LP)y8E>V{1;w&lg z8F((qhe1dC@UDp29dKAN($Y!A#$Dve&V-sx;IM=+@tpkwQtbu)i3L30;^MB8&OmmE3P0Ye0!PeH3+)KO}F?O@76 zmhkichpo4aYO8J6wP`8PLXlFWXp56TptzN`xD|IV#U((H;LxJQDFnA7!GeV3YgDREo>1*CUZy{Wa_G*&W@)f+Egq=uFqvNfPN~<7A?3BQ6ck~LG$o+69YU6@B zL_-@~Pv>yOLuS?YZUz1R@}zP|BZb7+nm*}D?86>|I{qT0PCB6fry?(ZjRxLFdpBSsvi^{Mpui~#I+b?_~#OmiEaMIv^gI8G)Sd`x0 z0c;tZ;Hppre}PGb8iK@__%q&U0v`Mb@Y@e$wd@+aH~*VH;Cc5cDQwwB01xE}E9lpI zy9XyDIUwBitu-wZj+jK#{FtQ&g)Nyr6LBi-{_N`azDyIfXjMhEzbKo_)@TQMX->EW z=drqLzwpGwX_->r3+62y32#fUswdIB-*o?1wFwM-Tisy7DnfPcj9#AHn_MxEgNe0V zf<%Rg*?Xa1gEy*E&<*b83F`s&qb_!HrAbwwuMk%B;?j8t7NCR-vUYF)iP1zSI{`82p)%iKw8eJ`n+Lr8B%+(DfJ>e9$I%mKJCB)G z-2!6y&b8z*ZQ?&Tj_*czCNtHE!+61rL$$!aV};iAVWL|rQ{>Vd%VH?D&~rj}3o%yM zWmF#`nRQmjMM-{9y+F+3OwLDoX0mI+C{tsFA|&KmH&XM$<64V($##B_Zc3;v0OxO_ z$i{>@No=zsmu}vUzRgs-1IsS3e}|sysIql^KP0Driq*S)X=cvBcsK3GE11%Ivw$~& zcPK}dup+bXQOiR_Blr~~c}QuE2L*LGgzF;HLeTk%BjVmv5*Z@xNcrJd*yToLkx+N3 zJB;qDQ*ihYjoRBXoO01(Hu)q6o>3NXeapNK*DzZU8-NY&dmVL(osWoFFfkDlT?m(n zNmy4rFDj_Fe59wxsu;6sV!LcD^93v?S_!)oplT&5X0L#$W8xZu6G!8uI$ylXB7`^K zxJ>B(A#(R=)~jq8mjo%*Drqv@%Ii_ZTOC)rx_H#=INO6G zan?BTdP)kvB;@tzP~HdTrn#wUCaRmA>J3uO?Z=>c8~eQ<9fLoj+clT3r?vj_eN5wv z_gmYgT>mY%MBQ`sm$AiVReq0jZ^?X8;fNM7U7!)9xBfTro$It&oA}#mI6&vWDsMR0 zh@a*6?d+tiwQ%Zall zMIfjBp5+bTzH7DEQ*QT?ScaYkg=yAR`t^4Hk-y)P{tD!_I9QKD%1!N6#Y&H&q`xR9 znoD$+-O^Q)0MVbwAe(z{$uW07wdM=vsM(-5P70(L5A<)Ft2bmwI2XR#h(orhHhB%@Pe~s)F`cFA{`iiiR_GOt?$Xgjo5zQ-@CY} zrj^@blV1x#@NUi1^+7fzv5k;ktgCQ}a>G0vsy56ZG+*EryH&P5^>aBVXLRx2+;JL{ z>+DRH`b1)*yeTj`-mPALjuz|JEWRN7#(XH|qHfh!FQYXm!%a%5jtv{uxxc2%ciH?cE6Gy$9 zJk(SVG?D^aWO?QX$Toy2y5}L5KPi!q$8j5uO0AY$c+b9BR>S+PjTpZW^XG|Xqs8(E zwkf1usNt*496U3*q)pYRx1~xs7I^+h*l?#akiByqi=G$s)7}%K>U8$Km4rOR_3!iu zJ^1oDi-d(i7r|FuOQUsQlC6*lU-3+0-eM}%Hc&HpA}2yWOAVd81M7h65v|#3bh=|Y%&QOlyTh0a*1mPj*r0gw zEO^rHaKyvZL?}qaN>93pAugnb57w`cd>P#b8Fe;>hOgnpyQe8yl8!@if8ErzTYHStL zG?_}sJ~{s1y_>C})qQ>=H_Kt!VwvMYgnQ*fMXhG_7_Vzg4eBn{u{BAt##64APsW|5 zNr0j{Oi?S_uFBef7lOm)Poe>7*(jN*LnDBfLh}t!8(OqsW6yxEN55;`m=lxo9G;2W zL#;Qz5N<*Ye~r0?fagLN3PvXOJmR|5g||Qk?0ZG)n|rvTXCJ4EeNYa1vp-rMCYF?h zT1;xI_?|z=U~3;{gV~LBkxuBYd67t_Q&6oJlR)@cPoP`k-l5%aYcyT*TG6LDL!E{;sWezas;08m=l|E0`rmrgCj9V>uiq>x7sqj^_MCUW3h-Jh0l*j% zAQ)%cTB7R?t- z`Ay&TL$?I)1DCx@`)T8@b11ddGTxA41}v0a{%0Kd^SYQTrz33BW0=kQmbkmMYr+UJ zupdt@E7yjsVH&Bja3=qp2vvq@Z;>f-K=Q$OfrW)GNVg9=+RB0M0uf_>=JMPu`NiSuvt>3_ z`fb30kW;3)bSF*h>%)2t6TGml<^8w>zig@&rrdppz`a32#<89tz(&L`FF-vr zGLXOwP$JW+1^AG*gPw?3CNjN}b?*D3!wO|CumNOb{P9F@q9q`2PiGG%=ko9D4 zhSEPc&)(IRUD{2yXDI*YIEJ}XS*z9#d!0~Hm%PhUzp40psjOr6-~Cig8t+Lj&a3U> zBH3Nn(3B9oo$r*7l9GKzNGNN6r6jWv`S$WiE`Rra4K-zij{Sq9tGDqjLL_VRBlNVC z&Pr+Wh*PzJ`%NxgmNFVQV9Qjx?dK=m{EQ%|zuDZeVNu)83?$XSzxrGS`~#}us6;q! z1GJz@yP7LW4P*t#*du&N+jv`TSL53%fHO59ZtSGsv&*NLq_HIiMM8yXu!-7Rz7i$j zd&(>6!UY$uSU`6Y`7c+cBlj;Cvju+sWeT|tCK6VZU0dv%>po{N!MS<+@V+5BhR=Gr zriv!zUXIc=uEG0CZy5SCs`$e`o0X}C1#J6LR!_6vQ5ODY3z%-5P`-Py z$i837V2L(8s!y4(6+Tu{k1>@-sHhT?JnW) zQ_-8gl6tUW1nAdr@kZIQ-FVFBfGee&F+|m9hpwON65c1HfL}JT{o1z4sgc5?45gbu z)m%+Q)VS-GCNGCQp=6|Y5bc*1dr27!bukE9JZ*arHrvKuvp|&<71qhh@hH(1hlaiK zF7OgE@htkRlGlBh@!FQdoC+P>rxo{_b~C|_AZs6(dA%+xUrABJoq#D&C=aA$YCYF7{iofL(>i^)}S?&n`gJTm27q1mwSoS0qUdV#j zT61ExWCsrmzuRYS_1$rw9#%zuzS;3)qo3R7;uyq5VisrVoD;yV7x%HF%?VnUvotD_ zrB*@LzKDWs3+pt^mbNE;CL4px?O82~yR#hHDF;z{iyuNd)uD_%@v5>WDAim&6LGbY z2zJ5j?9idlDQQazPkBJo?}eY-He+9;CE(1u^f7*bU+M!aQOh;eYr;J;N77?m@_>*T> z==IubGf(4F%(oe(J%)>;0?IWHiGE&TCC8j73_3Ac3DvX#JY!Y3o=IMz$ch^AB}ug) zlwQd6HyBfh!SG=v&)Z8}f$WH+{DQCI&U^c{N`Kw1plKZIa1ID=ePv5ynaA|)q^DYj z3NK=)!H86_T|DzG;2uFO^2^{#NW@GE>iyc9#IAJ&C z;qLgt;Ktfg0Tcd~sJ^BRvdi3T9Fao{_x7{}91g_c6~;}SD&!OvFEQKgsLyfpvh5dy zWahf7cOKdoU>$S&WtZX#+)N@c z9@2)>zJYkluRab+V^!Dz$kVqFo%61ism6HTT+Nl?V!Tda5S_ol301z^ERzYd=~U=; zW6#m$#G9;@wH9kLLv6j3zQsCAvNN5W9@r^+KDV9}WZPEFFW;!ZB~XdpkL=4#!MqmF zerAhOq6AJPIpK{Li$e1zzVl_0b-vE zrBIHMGfqV~U56!=3HI1#>(iu6O)QVwt0y`x8mV>rvF^s|zRc>A3+mU4(ChYIK5y$dboZi-a&l_5 z!J)`Q;bhXotT(U4BTU`z!@8^&cK!^lF=?^3<=4||4sx871{_hp)SC559c1{!Y@O7sq`mv0;+*w~U(yn)7t~mAyZ<8!tK1 zr+CM*86{Jk{2~Jkl#PeZTpjIV(8lcQSk<2o=LM=Td_mRpGXZ# zO5vnkqd0L^NuhV6uga}LJ)0^!=gm-XPC2L4oiiVaxzX0sKYD~Iku$HgU2|7^C_lhf zM!b>}imq?&S;{xOJQWdkC5N_-<&0uw%e4!m1-d=H6lUR;EoL9tj13On9h!Kbpt-t%jRV)@M_B;)iISUa z*pUozJitLRxcs$XPS*7E%xHL3d|%pb*Q;f^bwBcTua}CYtrixIcFR;NS#?srG9+um z@P*ggWOsL&k6fkw#7Dii%Bvi#69}nQ}k%;RO6Q53*|Y@PgANat%A&~%-RmtU5PIY?90H(-Sthj z^Vsc+f3IBw5UpgxH_1spe$W<-+(Q2Soc0)LOtPh$Z^wqNBXOE3Yb&#HxMVG}&!C$G zf!6Z{xa-3fGZ&s^=O>WII}_=O4uT%^_z9wS znH6Fy1@6Q)w2J=Jb23Izp8cr$BYems?z^ZG`u%p&>!q&BeFb{S%fS@z@KgMs54aPI z=ULvx)4EPBRZi6hk)Zy#g)E=mM@-n3DaYr^#Hr|EJ_aRj=3~3bCUvcm(JIYWXZtGu zgwfG*3KmHk7X4WK#yjWSNPoV3p@Mi<#VxTS z%)R~zZuGeS`#_+#&$%Ym@bvn~wU|TWaJYM}kL$lqgi1(EZsfc4Mr8olant_t+l)Bk z$Fd%5Uv|7p2o${N$>5U5!0->w+Xa#5+mw>Uf^A#0`*Do5~jXmq3@ zAxAxP746DYX_bER${e~#t^zgd@W&V0%GrZw ze|H=fN-C+BlT`PjYb?_;(9Ybym14tk9f$7sp-fZtgNHJzO{TRfJxq9E?xV>s%e=nm zeg;n18zE}` zD7j4;hT(m3qMl<=))INiw`TCXy{)OFx37slGtmpv$f0jk$?BIs`D2soh0a%wg2wHR zB~`;r)b37GfRn*-bp3}WH2?H`YZzPk_5({O=}#R=0VkUcZ?25|Xb)K${u%Lv+>mX} z4o2U8ujY22mR7NY(DF6(UFQ2GSllWX3_0cZY3GmzS+8XHF}bkngogJ8`v>f4ad{@S z={u!aA_1HiBczvFa@1P(Y+P6-S#&1<#EVKLl)BK3t#D*F(ptWoc4_AL0_nT04?xaj zW?b{bUcR7`@W`CLV=Lplo+o;YZNQ=!SAp|oJT<>e7we73|7L>~a_uy@t5dIESptqD*bf~`*!p3o2 z6{$6Y%)=>lU;^Nl=TO#DoQ(j4I(_PfYT&7tMBlPQCj4c+7wvdBojWA691!f7&5A~{ zN==(X?nL1$6V{w=bI&)0UCumN7VnE*Ec(ZXpVcF#yUNBS!mQ{SCk!V;&N;?^G}W5v zHQZ&^<*S6QIa}EpXzUq5t}$*iIAiX{6DWoS=IW(h(HKjMwJ{DxOFahWOtrjruql}P zbq^~|qE)T(dufr?IJ&XD3$X^S#5en@s*^UA2=@+J!dW;O1Zcyfk0Tl!#2)gC?vId^ zR{eGrx}tuYd2d;X{gC{Ufpkz)GyXNz)%zv#WY<2NvWsER0*iJ3$EKnPY9l(k%cz-X zWx-p4@+`PxT(|)(Lc>Vv16RSI)2V&;*eh%~%aGDy>e^ASu(jslHT-*dq6t)o-#|lI zUP*)QYZ;Bk8)5?soAJSm`H)2t0$WBsI13F4n5AQ{u%M?ZV5Y zQ^dX5Fg`cCyeEn;o+$TuIhO!3-f7Ckl-hl^k$ZVbmcpp8+~Ql-q$c92XE*h$%R`LN z@)RS!V5qA5L{7(iBu+D_k%bnWSeyJXnYia!h`#C@)sQ5jU+}lA={-hkTvY^qWsI@Y z`F-eSRc+0x*WXuuD1rg74x+*3M*GfDW7Phg7*R;UPbm1i(cGi2yQJvU8UF8i{a#?V z=}fRx)z8QK@JC4*L3d;Fl{orER@lhEP2i}urUiha$ZaZjn5f-i;;JzPr+#i{D>-&* z-31#6;klwsD%4Y)Q*K{p9R$>792C=72&4=ufatn4O<1j)Lo|;hyw$9bg!dwxE~5sQ z8OROwj;W|!GJiTcLhjG&GKkaUZ#^5XqwjZaoJOW6Qdz;WQIw#+qoGg#Y(LDa6fkL6 zhMO>o86TB)?+T4}if!agsFv84G93lDls=qx3xjCg2rpdGGkPE>8F+q*6Ad{#iKJ<) zQzdfgS4jq-m`Luug2O~;{qj)G+Rom)8s}U>Q$2BkR0cl6p_o{Y<-YeLj>bId9+$+S zTkmc^mAMy5x;75xGRZbGO~0vIDahERLKI5avWgf_cReKE+LHz!7V)+wGcocH9Q=$b z%nS&(jOe_sd$A!w{mAL~fKW{i5?bk&bi)9rjf%QS@TWwIJnM)P5+O3!GXSxLAB{CL zyt_Di@6&E=SWYf@xSaNfiSKE+XoMxFcy3cZjDkW#ifFA?p2V?0Gq-V6Od+I*Sjt9$ zJ<@br{q@2zxgrF5=?>&{3=Z}t-VW2l%T-YyL_2&T4$wS`fh8?rcWwqNge_H2^JLgU zEdHX|_z-WuesO@VI=wzf^wShVd{13)dg5bpfe-~0_ptN%3%!#I4)QZa?0Dn&vL@e0 zMeOX?QVJ$@#S`?Xl$Aays-McFfj1&g*;gPWc-~Nnv252r6HD(LH<69^Ht`EptZ3wO zRl43!YRk` zMYM5n(5gxy708UvAMyz%2{^RU-BxlCBXr(=d&u|(y(VQgG;Be{7RZSw6Hi+*UTs#P zAVOzSvU{`dbdbv*zL)Y9*0))<^(sq@-|+)WD*#XV{!~?thMS6-%Hcf2#xlabG^I;{ zf(#4(78Tm)aPXmn?#BQ^*4g|Q{v?0OC#5}m&*6>V1a=H}N8ZYBBEVhbT{bcSiSqIb zBv`v$vo34bW|F+|*RGrok!Anj@SnV0(^Z^BAO&Ia4R6V`t2N+Y4gv&$r@4<(gOts^ zUELTulDHF98%DMC3qz5IgiOzx6s_D8pWfeQTNqo7Im#x6g?)!{x@xICzKBZI0l(w2 zl5KoP+)MAI>H>JVt-fZimey3Bg!gnw10R~#K)?KJ!_=#?7*v%fP=JCK$rscc{bq0E zGY#O>=AUH5dEQQE$eg5X1~!LHD-mB})L*PxW4g|MlBSQkkAyMpMDQ4_xr>lL_|8MQJ= z`=w+r5yQkI?%n#Vrzs11Bv$gqOHc85AgZ;Ymchcbg1h)iU$xdmAZMpd6I$B`eaXC# zV;Kv=8-#3pF8r3mIxXffAXXVmpkrdP30(g(uewAYjQ|Np0tPhZSVN*p8r>%Ame^HO zOdE2;@D7n7cU-?6bFwnrP?+6P3XJ^)*D#&oPYv2=&qTYw$2U=}-XxETmfWUGESkP2 ze((8i+2AVi+q|umHXS8olMz;mBAXKfTiE{$X^8D>3Hquk%68=UjxY^KpU^~4rS8PCVa8hcf{On;Mti{M`Wd;4FRP`+-qoPGu zLX0!V*Rc&fkS6}D<>s%yd1b$rsM$GQhZm`<{N>Z=8&WVosVFR2|8=d|>99PQ9kp0; zs}kbSTy6dpDT+cin<~-2k76A zKiloqqW#?*RG7PHvc|?{cPHLNo5AaxU*u$Rcm}2}6)w#~4#w0jMoM>mW~siM`L3nZ zp}|p2yHDHN+vaJ}q=c@VS)T+&*{cGuSb5O|WvIXZmwHr2e?Tnb+xW%~sEl?K*SDtx zxzN`tgWC^v&_Ct~r5}q*$ZkvA?ZX+Z1(L&9vxf*6!aoP|KYnXzYPOE>TZrqH(5Pt8 zq+1VZ11D zW@*v+lRneF*7^r;<|>*0$&&?|R{)>oSv|+C5u-6wx}%A#5_JkL2^(>F9vji%`&V%X zaSIyDNx3!PWcG3XogocNS#5hyx%XrytQ@#i-a;jhv0Bhyl}NKGY#sFZY^dNzafI<0 ziq6(r1;8d-_zV_CU&0aeY}RC7t6s8IS4c!8El|qLXYS_LOyV1@*z8UT(T=?}`I4IC zS|AKABMpQ9+l2fdxa_|#{<{{bY=E3z)Ev^*o6)QeiFoN_fzDh~yJT#6=*sWxdnQB&!WO4dfPKbP2rQD&yxa>*bA;COI4Lvbw?LL~KS$_g zp%CR40Cde+Gp3zc$Bn_>5(yAvY`Qm%>~D^8d3SKLj8bV$QaSp{BeV<9j$=czRw#=q zzaQji(7VhZbZZ-b2Kp9_z_oYeuqG~qXu-U2fB{}y@=11mQ~PpVNAiqK6=E{)bVGAj z-bP(DPQg>WF<|4>WK>hdGbfdd94&Z7=a@io6nZKLE_tdjy6IU+ZD2^q=i~iKpP4GtN7dNG89d5j?G8zq zleDOr39K*su7_&a%Yrx6;;6*%syVq* z>eB|^%`gGuO-RTu?8AxZ-@?S8yfUDlszH%D%sJQ|*%9M*IjYP_e9c0Ksa!J_{v$+- z*GfHVRL|U{u@H|(SNP$a^mBZ90OdAH;JPnacr57l^Xu=f zxTS&C(JlMd5SvePw)0S*`c#lLKRiCB82|KFiGbCQ>uce1k!`2i6tDM9G4(k;E$_aT zsX0z+6w|BNIW@o;V0kQT3e|+s@_c60QhSX)wW=!~&IL7zjp$jbeu}~Pg z)5)5Q>Y_#oJea?SYK$Sl?=R4A&v_~!`@02&h`Wy$waASim$L-RJtn)?v^y*yz3if> z)Vs0FiNyF<$%8c_$JldroyWS zza8Z(dF%-O6F(l4Uv%b@#~ z#r1TXQ~6B3FPhn^Z(Y{HrEMLpv1kd@Jl8#<9F0?BX$XJ$;ZPQNkaAwnNLmoQbRVWL z*1U);x8=UmU1N6W>lq$otPO(V8{)=rm08qGp4;8D8;ZG!HZa>1ybsjOo-RnAkY&C9N_>U7PzNolR3ZkzdP} znGIjKVK*ud6+T{K3fnYnD+QowF?qvWGFEaa!oxoe9X7B86V_d>ruO>kLH+ET&A(>~ zm9<$pHnCa;(OjMvI-8>$UeRQuapBHn{eM1);6~!MKp4MirBq1bk2|_(?p*-5x%?%} z6A!vEx~Im6k^#E)r4$t}G`!@AB|m1yj;L46`y!e~%0hF5d#SA{g zf?=X4tRU?_*JmUD&JWFym;oThxp>e%kd-wzHXblOuUSkl_cu(Jm;W_0YXIXA+tQrO zH7ce*bZ5vi9_H$q({s0n;xQ?mqEF@9YPpZFcjn$q((4=|s+SsdBogF0BU!)e7h=VV zDu3Irg6INX?e5=IxaoEgFIli&u}w73Dh~Dw7?J=DLIBCFwn!7+RA|M_3{h3%jxvD} z%r{Yd5b9|>WwVJ;N8%n=DrZQLzfHG zI5VU>MNoKc$!alPuu4n;*H9Clw~X(xb@rzBTTV2`iYde5^-DPikes%Td=?fVX)8*D z=*{>^pk$anZIsCWGUH?sWt7EexaTv0`GOJOXKVjZUkI|)aeR!|FAyD2Vd=jAl6b6# zWUQ&g^yr%CaBUwxCKV`C%x3!Dla=H<{wh#p#lqF-=S!07=Fak#{o8S+XfRpGsh02W%i4|d zoqdP=>cYknb2UaE4rW5qj{k@u5l=yWrI!3GN*S3j4ZexW$D}Nb+3StHd#2?DE}y!) zfuF>Qwzm0Em(M$VN=h_Sg~*3Fw|_h)u?z$rL}uWhH?29Re}omZ-Y9RUVL$RQV~BBe zBX7eB9GxxXRd2=ulHuiV4HM2mfe5^Ny`~Q(uy>1I-_tt}e&p64nj6Ydk>XqowUBV? zb6R?j^V+S(&DV45>Omh%)U<^_!+A9Xa!flX1qcf`tM~pP#B}=zfv(9K%@JwH6mU@p zVem=3|H^5ytw2ft`vC@+jAflw&^MV9_c_WN=s_Z8D^_5&>&}(sS;<=pLg~@$jVXC4 z983g}#tnapnn^Q?1^**SfZ(q`NtUDMuEv@|4cZAgPZl> zpS{5%-2lupYH>c-O4j$)@V1(cXITf^K2;+2Q_j@ITPBz`7|MIlU><8sV|WkM+gj98 zZ&-fD7FI^|l*E}EN}-iPCi>v#g75yTX{}Ac7cH{Uo-P}T3T>qPeRr33yRCw$9I5)6 z4#GEF`_dHaCVtZ_d1ehcM)=_ zs*;5Vz!?E%X8&F*qx?c}aG)#lbe=&5-LwadPX`7i9*z^6*=+J|fOB9Ttm!y~bo?p=H-+EY`emU_ z;eC?!G7(m{QyrKEPtV$4--XoDB7_uT8Wi%|x7pqT-a-_Q!ht>LVG%hELM*_7UQ8uO z!`v;Jp_$OHWWas37HeZo-->3ij({Og1mGuJDp9v1Wm6d$Hw}Z<~cs94^@R&8y;&{KTlGnGf3E z)j!|Lb#a}}sQvYMWOX}EJ4-L3g9NGjhIhoUd!#Lizd=T1UUEg`qmx{xv?EWFljlfY z9##yW68QXEJX86vEb^3jFk385V?lfJr2!Z{)vGM=84*~KHX1%M6O3$0&I==OPNaL6 zrFfIRuPjRPdpe`H$Z|JZ^j9Hz%mF;Jx{y(qcfeoBdSu+7-$jq-b|2fe<&si#1Yv!> z%bIwmxFFfMU+JKtWj2hhI%D(5`}VwF#6xRNiLA!9is9J3U{Q_0y_)(;YZb?Ij4&eL zu-2>L-hZ)sWZgBuQ**WNNqfqXPKpsSHgFu4$Y?ckz5k92Nm5pfRg?d#_A zG3wL^2>tG&xv-?gw`lAe2i%MS32|uJfY?LrvjsRzN8a=Em!MOoPjXoLycoW|4A7>B zxU>}KXhM2RDpd5&lvY$dvcXoFL9lUbD6nQ(FUTTi!_`sogi$L>bW4G~vH5gU)*A(i zFmQ#alwdUhsx$zTruGv@T~UlLvoz(^v5TUt{ZV-Tl^O#Qt2?DI2f3RQwiwW|9`SVQ z^H7DDTvOEhN_*UW!I_yBRid6@pyl)?09!qwigeb~;=$xaiCl?*iyTTV+Z3jy} zXEW(qbE)fy7pHW<)yHEHT>z8|iJMNAqIp)~w@zB9qci9--tbP?XgK~&Evn=yW0+*? z?F!Z+YHo>S@{7-ME$`!x+Qh(V-aajnyc8dq%@Qf_Xcj|x*$NT%guyd3b~wuT{W1cT}qfF&@G#r7+HIgK5q${Z5me>BCs5meOtMEUv49Y zk^hkLegFy#iR>>3v)hCl1MxVjV`#r=E-ESwHe0vxG`0!<6cEy^1V0OotiaZIxMB@1 ze*Xb6)gMnDRxXUZdTp#Px{#&+6n92+y)g>AXUO&{`T1uoMjOA--xz#?90m@?#6J*la(3T*!jZbkrkPo zu`|)N&>$TjL9{D|YkT?dgU(BepmgIul$mPnmNOZ#nbzZ~{ZVJXJ%>Nn{!k|8hXd$E z3D3*v>h_60Y!0)-LxVrF@Zo<1IiX_*pXfhj6iRn6fmtCCf7<@Ug2ba?rWfYzKO4s| z}Om_&%e7R67HZy*k^i2ah;^K9sUn;~FlcSeq z=BkeJ^W6n}O*+&lu5eIM)NsjkO>})@b^6vfV@7$(?I>oR*(9qze{x(N%)^oeb{x}l z6Lopxz@q#}N&v+*s3ccYj!gFFm|ZXU_ld21BO1~xx*nC1uLEbqN;^lT#5O1|@9(lj zgq~iqtxSo09Ab2-2#rm8on;?j`M?^phH%4$d&|3ccz;3L-*-y&1l&lPcHf zX2OT$7S4Hg0dnffp6c8=lkTQ?{RdZZio)4(&3;YESU35*7>#a)_SIxhT|T`B0NajP zJNJGpP0M=RT6gHVnwt9B^P(GdQrxhm9B*+QiLK9e0DXzR zTv-two{P6oOr^aVn*;``wkqnnp_|Gr?z?`c{c-x~A=f=!zBnJlqTi%rUmY_&A}~k( zA^4kC1kS3dc}(5*3PC=Cz`GOYc0VqQuZgGpu=LHL!%jvv9Q6nvc8;#kj?2AE>&|gp zP?M91-7I{H2p>0dbRb^YbreG{S@r%*Ovz&67!##!vul;&)pb-9ZeGrrl6;k_&)HAh zus|jZt0>%fBfjsJ^*z%^-tzJ2WM|$G8C8S0mr{XO$al8wi0g6hz41Uk9qsNgb5HI+ z)Cf<(#e~MC3jQ3A>lSp2-~`(mY2Z_eZl1!}+!%!c)p5YpY16|>V1 zsHd>^V#BZ~Mk%HHz0M*&{L!E`5COLg-@LwMTz2;I&8VYZmQ!E$QJ#0Bee>2Yr`d_O zL`N<8R4WWlF-OZi&K0=DBf-RF+xXrrOb^Du*rg`wd}UkKa*0i@%VL2a;`9y+e07`r zDPg0b_jDYODjt!V8YA6wT{i#0F>t_fE$o$REx63q`|XRTYw?rkWP!Uza>eRaRTTQe ze|1*2aRqeX2C(_G>nQI$xddCc@EZiU^b%fVY?o~*3l0B+W2?$qF?3_TvlRi|>+{N9 zFPg^h=Pv1LDdTU-$XXFtMrTh){t@xEuJm~(u3`?Lw3xWoHyzkU<$@Lx|2RA#bFMXZ z3bKNO3+DCY%y^A57OoPj6)@ty+_iV~i3LwDtT{NW7#3?G^ClCj6~JGZWHz6^p>SeD zp)T#0+Cnb&eu@#qDFh!S#PnAjv(~6tJz_TVM}qMq8^X#cudLxeKlgU#KwjF#cU4Q~ zSG7!<)#nS%!=Hv3=a9ZsmOtGqJ5qC;QkP)(ap0vl9O7;40AQKqGPY{45#bnO1z)Ad zIeZl!)O=|@YlWTwV<@us803q5l#(WW^jb^k;q~2sb0#L1E1he0qqu)?JhrIjAFhdx z$Akg4o_r;t2xUbuDR{3fYEn!5jl)nYX&s@(s@dM`!K{sQ2$g>JpsQ9LQ+B=5O?a#R zGC=#AySM)QQMTVOH#VC05FL*Gx2}mrwlSASO$n!uh3+nE6B@pu$#|+y;1#=`YieiW z(UymLSUQu=>9VDf10nc5I8ZqR@fH(Rq9SG}sd@^$1i2I`I4NO~ed_j}(ecG5}o3 z)Z^n=!nAYT)s=!coxv>>Cj&lW_ipLr-vHym$!GtEU_mkXhd&VNimN&ArTpr(s zh9r=aW@G~B$qDkCyx40j-wn348c!@hl!71XARlV4mmhJ2pOEG^Caf96RG@a;Vy>0? zf83D6ZUML>e8yz(fqJD~GE8_@r2tEW{DP}*`H&qC1?gk#g>igjd4O8J{{t( z_YN(cM_Ij{q?niaY?WuHcah~nFGcY^V>PUU9_Ld32hJ+axlVS`NOQM_aGzRV4E~dYF(A>7{3I+- z4xs$t|E;85<&G}>Vf_b(_SEeE*3I-J1qvs8AgPtwBQN64!@e*2?~jL1>ePlparltF z*H1GFoufvrEYFMn!BNy#Ydh0*LGMD3BAeg2_&Bq*Ijm|RAJ&E9fpDB`T{*CY*5$Uw z+Vz1jFC8(b($smcOb801{AC58qd2G%V&GbpMkm_QZ^L!0Yp7kwpv{@X1M+6li0L&E ztE-LihUaH(XDxO?-P^_1G%6;HuxF#{3jh~uV;s@VhsGu0DlIyzMItg?S<+*cTnnF; zd`_$=KTP1qQIpu?QOjzVY3Ta`oB*Zss|^!_Uhw3+P}dZ)Gh%ry;!M||KSI*E*j4ll zj$&RS&hHm3&*M2b_q)ideP)Tfbm%5(4h=CmosPOVdM0g6j_0hI$;7Zkohb?aC4^@hV$cjB^tH3Xsg=+;kx^Bk%@ zo__MGs8F2#$$l_qBC0+i%B9js@a2} zQQ&nQgDltEs7nwiN8Tp~yYS*dS>V1bP(v7#C3ZC*`2y`TWrXKIOUG-9GC2ZdU%IPE zrcSkg>eqWtqk#sI72qG244t@r$qY;+3P+1{DS1CN5-(a2I*KOCfHLEBfZqsF-c!YM z)?7-nrg2{^Wbu+0+74d?QcJWnBTBY|jV$tNM{L6B?4Z~LLP4n6qi)$L>4R}=(+7?= zt^vS!+KEug@TW%eBW6leEb0MrBT52<#O}SB;84zs!W~1BRGaqfs=FuQ^|S(9KGziF z;frcI1sEL4iH#NA?-hd)z5&f!<8DdzJ|-Jv@s1G{0o@U%QjPxg_6A3b$c*mu6hc-r zSQh6M{M(84WkW^@XJRd!Tt`hga*(~T;X3QYL+j{fv)+_PkHJ`SIBdvi7(Bh@ebjth zQ-i@_6EpS5LyW#7?0hU1kwpf7G<`Gu<6qUbV3J4UIu39NI$=X zoWy&YHd8S^hBt0mMSXEebxW9fBSF5g3YRl1CQ(J=8f6fe{<28XaGL$;j zU9{@HlQ1=O=37E3)d3pD5*Qn0yS2>v4b_w4_Cn24Km8xB&N{BC#_#(mB7%a7Gz_Gq z90*8<(p{rlT4HpEh;&O!cWs2k28@y}>5UjYy1T{e*>&Id^Shtd^WXm5&g-0=bH3lt z=l%9YPYL>5y-<&NIp&wRpVL3?Aewask;`@%Ow%J^k(EhO9AsWkHscg4cgr8p_N~p0 z-&G@x0D)xH^&5trjpFl>+|_uA^%#=F;!rzSlL@J1r#f%KM9%iBg^=utLkfUc+d{myvM^3K` zi25V_MvQEB?^&UA@(3ty+SX)F=tw=3lHQ=G>6TM#c4ec?u?1}==UsCiDUK|dmXvXTy*o3abfZJs8>ez zP7p({%;*4{s=1YtKenQ;e(5QyH4O{*QH2 zrnC6vFm+m~ejD=`{JBg+oc9LMy_7B9X{vY!mDA1Wkk0<$1_o)I#*;jBBim$htzsj; zNmq{1c>S|%*<+PW{XqPy7wWhDpdlAk-E4J^@L!TVvv%I+hBWw4n6{IqCv7TQYwOFJ zW_tO)?eQEZ*^8dl9fgzRXY^k~rrS}To;n6fplNLw8C!Oenj*io+n3a~O33PiJ+agv z=O<@zuh7MSwQ&a;1n=ga(wBih4oDYX9U=pLz_!G@Vnb6WndAT$J$q2EMP9j6;UiL3 zmwDeUVj6SY=xM*Nm2d!`d&iQ_gYa7+E3(1wSng%y0IhaC3gmq+>1=`YXirf>H{@^1 zlspp_w_N3{1w>UT`OecRQm%P4v8+2xm4LUWGKn1uX>57RS_NWJ7hjV*+dF48w?I3} z7`UfEhKgRoz2}cJrL&59Or9J^hSJbP^!057l=Ya4RjS@oUbLv@gqv!4d_k-y<;wBj z+S_DAu?@eA0kZ0YZ3w;pVf#0N+em=DxK2&k{x`Sm=CP!5f$}o$;gv$a@5}aMI{JVO zxqZcNjQv^G^3kYvOs+oxErA}OM%B7`X|_cmg=4+EYMGk@0c@Jc*oM!}seFzSO_Jbk z(e@jWMxnUee7^{JHY02d+eyU#3Ir^Ho7Z-%fpEc#E>`$Vt;xTGXIQ&xUCj5vKXNBwn-QSzv|{wv}4p zpb1KGD|wZq4i0!+X*`;UV0P!VnzLeC!^W(CVx36Q$A4oxUJEq2PPkg5!qI*t-BE@K z4}x3uOm^HuXlT}$t?TARPP{DUoXph+1C7|Eg6#GmR(bnZny@;Vrcd$0i<87yJU*`= zYVR|jgGd5B>)l+d)!&ZpToV_J$n&%s+b{*mNYa0Iyvz#?B>Tu8hDCQY_9LhM!HLrY zP0^Sxsd)6Z@sr$;K6_kEgjY_ykwEFU$7cI+DN@;4b;>QZKj*J2y+i^P(~J(*57$3A(r{?*8AkN+H@hH-X2(%kiprkwBBJ+?`Re6`wfGXV#xUF>yv{s(4w|$v5Nn#l$Q}*#{$ex9NUa^bJ=aO-<~!HMvy3 za0eQm_%lb5ku6zxT(<3PyeqJ*>f%Aid7!7X?+jg_69SWli2@<3nRA6J>TCWRIgnjEUMN7vc z!ZX0iu2wAU*c!JOPQ^L;{o>eT)E8|Eg)w;qEABYP+ppJPf0|u{LY32Ck+EbiX3lB6 zOZ=GC>jobn4Q;Q4flM(XsvVYUheu6u@7jPwJ=ZpF4-d!6ViZNE0IWwkxYnDK6^v=TNJwdXT5! zovwVX;H0&v^A7K~NbR}OWdSsWl-lzyZs`kLROKGo#*mN3$o!a&{F);17|q-iP^J`q z@CW}BL=yV|UX_`rF{iCseTJoYYzV^O=m>MEIwBP!9ScNx8E8iV;%aPTm^zVjmU1L1 z5v(hkUjb!?528J^U3FFI&27#+ymSVwv-sNpZ`_wPTt40oiH0f4PE3o-vOKc`YD{j3 zco_wIIQ~3M_{796fk>UXtU$}FNKcHfD!edcxH+3lATf@0c@#hz*HwI>bu7D)8J}z_ z>=ON@6bbU=D)3rAl~%=}i35lxbt^8U6ni?|?a`_#`h}0Zi`w)0U_|oKvrhC~@Y^~3jFW9I}uF}t0b$+eeUrVwH=Hg1_ zv`E%D0#yHFjpx)QZ)*Z_ylgz8NCWS)F*$R%qUgzriFP#)A2z~(8_?oSi%uHy0itss z@ERQ(2Ak4PHr}A|w-~KW1&&aAD+i)!w}__ShUbBz)5<^r7Jg?lhK$84yPT-DI;8KO z)}v29uZi<-$uj3DR6mG?F1`(Cyj?d+kG)Fa^oBIL85iZq9uK@-afYH%hkVIjVhLnV z|ITXVVENz`;=Q&(neg{cv2`ZSTqb|sX6P#0%S0)<(xL~ifwYH%LXb_U*M$obt1t9% z=C$So*MYnq3bC+@y~A^NGfbMFaQ_1!HO=<+40r(+g3}TI#GqYV{%eNC(NAH#Z1kDO zm8a%q-=Pg=+#RnMYMv}t=ZV<4%V;ugE$V?D}*rp9LVo5s-9 zYA3p!o8MDcKHml@t2HiCM#8wH9a~B666jbI#ask&IEqT)j_^4{*m-ePW#8AZ0KpW& zN^fT8GYoWLFZ;%)sof)|aD|yIcVPw)H{5mHp#@(r!WaY8eeF!@iH92AL{z+BF$cM!^{ zr#x`(h%91#|`31!_5S#1t$Rlz*b61*uK|COk z-y^0}%Y1$3T;H~q_o(~@F1t&3&dUdbzr^kNM-IueN(x5pcbt-mjPb?3JvgHy8+sLA ze0aTfB{D?(LU|@{_J-W*htpXiGoq^>tFEdj^;AhG9+@Up&UB3{ORVeotm8-|O|Vw~NA zu06ApjvK&Y=~OQ8GY4=GnceGYMo6L@1mUh=V`sC{)q}n|-&K+Ify83F#9`mgrm9-E`I*J5m>+~#&5QlXMh-^3rotJ1CxubRBdi@242 z-xiG)MQ6#iHY+@!xF)~b%=cu!8E42GE5rTojrQHuwO)h2nAN+Hp4#YG~!$ z(7LLs8b|>f-k%oUo*VpQo|P`f(Qh*M4BnP>q9w{n%DVad5C&wW)EqL9V&f zR>WY>x*IF^(?V2<$XVweRb+XcSwy4F3qT8rq~uqxZTr-5JIG`oS0)CXx2u$m4m@8p zdy#oSRMcm1o=uops}pbFQG6uEv@s1VJabKS3oFkeVMdfavnvNenJ?a2SropRJ-qxw z^&+MC<@dEmBFQ^y_U|t(J#C^S9PyiHd6!~4V$rpRnyy_oM(Qc%k6y(iTDPZG0`Va* z^`vccfq|Zjsv}OoDkJp3p39X+&`A5~H}_Pb26v}BMJT+)W&}SOAYQ7+;8nJsYGSOE zWUMg1@U>3xc-0tYwJ2L{Fr&n7oV>F6+Wa*!T@dN)J^6;(GJ1FLKsq zbmwN`-u#2Z`dz=(z{)Ueu$$>093_3Go)f7dGF0|qHI_#99cu@cVUci0REn(HIcV0~ zfj>5aqmh~kv9D4C*JlWJdIwoT8u;m^m()!+z>N=U+|i|vU;&h852AN28|h7C7aB(M z8=RZaR&^nrzj7Mrd6N@0<%3>4&n-U4jJ~tg5EyPKUkSiz{kJZyyXz2P!^NE{<0q*( zr*;AOMO?3!tE~_fP`|Tz#fWmSPGzlips~Lj0e_ya_rq@C4g}*w?UVKBez_dpEDI1u}*WuIR!%?yr_ zWlzg75B}diO?mM>O5$$>k{zu6F&>2$34+5Bbd*Q9J{LNH)7j>jse`Jr#~3UXRz82m znr%4T&DC`swqvx|wSo2VIjQs^@adJ*l^w|*OnZg@eOS9}y%Pg3Y*yOb>NH9s)hITf zKEY0G9r_~F`GQj2MMEjM6~?}d4&06C28-f4L@lYhYZxukxcieOv#M1q&8rq}5n^-& z8J=!Vi|y!g@Tw5}oGw<5JY&IbvGA$Rj;pL)M-;j{OkdX-Bw%+|Cw5;%p6AafCGmtC zHuFAaZj%mJdi8MhwyTA-hV%=%V2yxma$oQ9!~ndy=>s5559lbx61sn2bwEo@i>FNi zo;n?pgB@rB100C#ga};kNkF(7K7RivG(QC#gblnGGajT zmg1u6@>FBE0e2y*qE(Osl_c^WMVYVhZ>f^_b?i%%0N&0q9uW?H8aH7!5j|(IS+rmB}nTI{@{IeIXL5wxEn?vq{VQ&r$E>d(q1!%l{RhU6bYT=;Ys8DMF;g2&+3<+&1T7{q6Td|S0{8C zsVK6}KC-~XImJId+aF|P&@s3E5NgDQ4;+0&J{0akQZ)-W0b;{?MWvNuKGrUWN8)$90W)g8!*37?*kL5R#iZc?JVtb^gpXj|-(R@m>2G`tX4eM(rU$Hy{>8>KFV{iu?Y2-2$@8lH0F$D{S`-euy{;5Mu7TF3j$S*=7-~P(CMQu+t?UH zkJVqzW!6i!q6<#b=|RfKoy;=jg|j#*&!2sU>~bV>=iff!xI8Q$u$v*#4oBt-eHYU8 zFj@zG-JYYOB4G6`WKv9ja#jB3;_c#vYZIeR$+eWNl&jceirfRW2TV}|w)Q2#Ab__Pl&Z@hT!A?STYH@g%~=6{UeJNvp>eO#UB-+JBlu z4-P(LHQy@!@Uv;NKQ~4&&^a&F}s&6X%o+d&Kz_jSl~fX%IP0cxbGI%gAy=7yM_x>{I7n&SeHSF z!m5&iVciCjBz0&A8W>73!hQtTQtUKrsl-0ZD@?Cfhn55JY)M!%Xm z-u4^fD$EhgDXGUQ_Nf=eY1%2jLFyMR&5;ANhfkNZv(ov0CpRd671OTFt?F^D@P^64 z=Ko}H{5Z`p%_c{tGRQACf6vMG0^@c>m}p{oT^d-R-uVA+E32sW&J@istd}~b|M{pH zs?V;Zt4fHK)D>RfpR@miL)NiWq4d999SGCgcgYN31!-VY15s+zizL_Bl{^!_1#Esg z>CClG#Z47qSXA*5Z_P(z|Mdx3V8Wr>(E|!qTYK{SIsrfkhC|(M^EM(du)SX#!7;O# zM*)}jsvj#GFEs*^IK`L4i}?Vx>BR__@P6MtaTMZWBgTUjJdIO2z7>6rdM0wbPfsS^ zRmP83xauCB?tv*%spo5J8b__cMRns@OfD1*$-~kXl)g7)la_x9w56MYR^Y`&>~K7??*Oeb#tFgTV!N0Q~|bd^p<{y^d~_{IRg7Po#+_bF`WF5jNl zP(}Y`A5*u(fV}VZkhm49`l{%NqA}g;=i-{Z!Y`kSXfIPNlrKZHPZSb;_FO(tMZpd% zk{l+0QPO_y!~Y3+=6g$G*H0Eg=P*8?p~1`s=yBq|C^b!Fc)3>BXzH?G(7q)r z1P(;M%s)Tpb3yi4Hc?Ss63BO zQNZ7=%&rIE+@4El6cl^WDy=6b=ykfgr$`qu@O^G->MLvIZ@%IB6q!{!0^j7-7D3JH z=&fwWX<+9M&PQ`j#85le!zEEA&9w+QY}yCHu;ji|%GV!gAwnn|Q|e?ms_*dVVoPq9 zlUjHu?V_3VNTq4%b|~UA$Xgp+HzRQ5Cc{SL9uM_pUS-2Jly-69W$zvDbw_@~J~fL1 zQdq&KyY+wu&(x3>GP}I02GxnOs(K)C7bgv|2@prXMBECF1@URSXZD&c=WB$FOlUV5 z7tH3jSWS(&#|sI(SqR4*Ie;&jI;x?xf;w2kb50)RZ|s7|dsFVOIeunRo~Zp&AQdFL zN$a(ew%vqj?5WyQ=hr3D;|_jK+!4uqAmd6(4sYezYQdPKoG5k;ras#BtY8Bv%c6$F znOTv&>5R!q*t3XSqlz%c$Xqw8EBVMFe|-A{jmp~u5Dy2*Tk^+x#C=_LX=CXuW^6lq zto4wj)V5jz&?2ML(aI(L>i)HpE3yU4qVKGK=gy)ms@igo*6?PRuPb)C9qp_z-cDEM8)ta0 zE+??@DEtn&jQ483uPEgYbkj}_8+-L{nvY*tMTTNwT%>YHv0d$2UO5J)yMWOUAm!ld-J%~v8)1-AK@2|(Ge6H1 z9Wg{V3h8$E+818SUvao!Fu#zpIfK{vec;9$|1d!n#4?mD;$Hvezl>f^LiI_7ek1Nd zrJ*;-EOZ9YsAUcus0o7wO?bO z^cVNasVzT;1~;g!@MKS4>QVBw-wg7xD%n%kyq`R979RahW5kp#bv9doD6UR`l>&+F zhW*B&Av?W8hQvzp1NoO;xroFl-@L(#NcjZ*DgU(Zg`K*S3<5sbbe?T}xwc4+mr>dI zMRv(qpB0%SHlCW?H1mFg6>KP|ned#&0juw+v{kMZWB2pBAa|j;NgsQ6m4)w$bv412 zTm3^^=;2CE$`Ar@$?FZP<9@K0mdYk(;&RwFC$Nprv5|jNVh7{>-+z)sH&F1JL5;e+ zt?K8B(Rvcbx<5$HfPMrjAqI)(+bXgYifuSW5*K(6M$(a{%#FQEzW=6vL|x#rXgqlY zXcq~3Gk=KUQcj%y>Gbza)s%)4p(w&i=G=B_Ipt6y(d=buL+{3KyEcu;BLD`lAvSo1 zj~^W<1aaZdWtQI>wWrmMsS(K!eeIdqw&-i}D)1&XEpc$EFwRZdJ5o*R+vEBAsR*hk zv`X4J{fHt@qB&@28pa5{33TZfqzay!hIy5rys30{5JoG}i1@1-r(yJkf1KrB&yRYR zu#3X+iEpR8wRU1ohhP#vqi$ULaImc54> z(#pbw^#>w8+(7@D=aJ^|RI#C-YPVG|%~_$Xx_mdj=$x-M6#FWpS3c706`~R z594cP@)+(=C$dno27DIos*M10Ep!PzqSeH+?Q{)7;cH>^eSmiU+4}%FLz-_mtFgLE z4Xu)!Ydao~K9Pd3*-9CvDt3!P>G80dHOw|4vd4SL^A}bp5od z{-Mb9N=)WU^Y81*R#s;`MV$SLANN3N81+|%!)*p!zJgf~sRRDOLCur>&J%XLE}Mw% z5#%OKtN{C{)XWrg$LYT;U0HaQMV=qUU>j-|G_rX1?7D2+US4&(LTxEGvBIK_aKcGm z;RmMg;jfj@+`1wJ{K6u&o>L8LCAB7)5P7?s!KT<|3^IVduT2^bAiV*!NkP3149XAG zfdhIO?DN8gw0c5eTk)XE5Ho(3zjDP$;)PrVaI3rHyF8Y(@b!G+RgFbYrZW4j;kRXK z1zo^yE$pKmJ2KPc`)>Trbcs7L@#*sSFS`SysXv#}}y?s330Ni1+q2{iOf<| zgo;#n>OjXL8q5DW&R-sG4G4X%cDBHwBudts#=Eeq@N?<>U|H&OiDWw1vn6aSAnGHS zrBrfZR%0>AJ2O9)j_d=C$?iaMZc^{gF`Xu8ttEn5cz0G;j?;aSE4zg)N4_Kryy2Q* zYxb^S$w-3z+@jdK|98y_a{)JJVTA#u^$u@l92HFH4Dv+kRcnRJ3L&ga!Jy{pKR72^ zIz1_?tHc-)rln6h80fgYS9hrsHT8t3q1h*)h4NqZU~5G*p|4SMm)t(HGd{c$R+uJy z&g@~ee;8Tywzn)aeO4F?A`MF(qW&J5!-m2y>vHJc4PKgnYH1qoVaZ&E!68f&cX@I9TR zCXbKmU(%?VQF`_hn!1_~YUfDTwi5Cc5YHDpk`1t7pAoQ3nGg*zZ z>r`JmS2NwPmgDU#e7ZmcRflUl{FGn_px5n=tw2LU5}T&~&>So+eUGePB>Iyyihh2h zw!19Y!vE_ZoIL?jmhL{;Hsi_GJ24{0Y%-lFenD>;Mpc9N;8=Tr?r-_jkuK{uGW~v8 z)-=dKF>(I57M4OExC zMeX$!5U8WCO+i`CV*VR`lb8`5L7yb3qouJ}-aF8T^Q?Uy#Y%sVpTTpLUN zhFK!kH#J));DwwVkIM92(=@s>_f!vYYMdA}=`CF!Fctj`dmef<^6YG)5S4}w3i_MI zM7vYv`Z*-z0G^Ga3>WN~u;oQiZ3GL>Y%tRWQ`-<{s>d++WC?|I*h%+D-oL!cT&pOZ z3Fl6(BL|~DoD+0_0hKZ65eKF3ewZ>N~DUr8jBk91~hTb^Dw7E7*3?O?J(0F zeuBu{KR6t997c<3IS4R7w#0BNg=exVW|Q=}ojk6QnrpY-F)z|O<*&6>X7e_dm73P- zVEd7{A>`q5Ug%i}p~gMwUo^@=dA3H4p^XRfqM7T{8?w&A$y2{?;cHp}rL#_6G}c*a zvS07g<~;%%nHzE%gl?~@Wa24T+iQg@U}byff=`54Kd4vy9*&uxpVwV-)5*CxsJceP zpS>*XvZ2TjGyML2R~@G5|8WansOIZTNw0*uz-AL`0MnM@EH?tgGWNV~$*Ly3 zyWVCy9w*|^GH-#cb|yd;$3RuX64-6Lz^hKcMDx!k^{^8C#;L03BFm7lw%N0&0h$q0 zm;{zg_=X(#R?L7;5izQ6dOw2Ayru`0{p9R(8W5))g#R=Z_pBLn&V0GKE!h5goU*jMePu%Y%a zVm#KzWfpsZiqx;e&)3HyKkSqVq$mG_lXkMrZ#JO!kaNT{MY~y^ZLpfk$zrRB>r?|Z zrvAa+)rvJ2CZe64GY71>rmo)uQb(jd94>R79+X-b?TlcU035)4dRC31gaMvP6Nyxb z(9C2u;+O}7**8pH!wAJl|t3~Pl!FjrNA?j@8q?~NO zW6@@ySMLQIipnK)rDMlFyYtiZ2Uo<{K#&4oUp?MFc3wBRn!Zo9(vC!WaYhSpqd~mc z651{HzrDro=oG87N}KhLc{Bq=81@Akx&3mNZ9kJ#xV;!X%vU?x9Z_|Q^OT)QiHTmz z?u*tU2u8jbpQW#>q0O0XfN=3$m?T}Frh=pE7W`u*3#TA7>@1eOVh%Zr4dJEm640g* zz%wkTI+->iOV@h~`^`0xu`mhx=1D^*ksuO04lReLU?r*W<6>IsG-uGFZ?26|)u@Aa z6!5_);E#NDAusPo(S9Aymgs=&R}SBfiskZJ9Vks0C(FgUV#~CL!@w!8!StlZ*unx| zl~Tu2=@@L0VMuEfcxX2xoGzG4sf%$C$klAL14#Pxk;1i{Ki}ID>m-~;L&}M|alHxG z3$Fw+P=jE_A{%S?nzJse!Jeuy{FQ+hm<)1vxJnMTQNMIjbcuO1r9@PC?jjr}jkl93T=5@?- zNed|xy8j4B7&lup<$B$-UOh|`oB`ai2NVcN1;yzK(Oz6oZkRY?Rz0hrMv zK@Yf6WKqYm(QA616|B|LN4u>>XP49Tk!VA_bHw|TEhB83K~^D7)iO8L$6uk_6J>jY zYO~=zo3Gcvqt3|b}7=pm6%TkoGh zaNen1Tn}VnXXs5L&|i42qglG0qfUVJT7^(>2rZ9wX#`Rp47nShmr;A&IPO64?b8Wf zyp{N4`*2NSjf3VC9UWw?%&Mj8sd!xMlP<$-X`^iEKwq3q9d4*ZuJJ#_+L}-jA}?3< z zTs6}tKSNr=5&ZvG5&pCOc(_yINc4N^aPe?z0t{<7dY=w|8sqCTIz%FrI1g$ZdA-r0 z$`jNuWIXTRgwsA~-D9;vlJl2cG?)hJE!kn!wnI7XJ5Q_zYu^ulEwpPWGxGBtVf;al zGr*#LmnJ1fHvDk2IFD)`h7{;_u@s_xiI#%bNqIASJ}{MsN}Sfz&P@3Bj+Xz7F&p#QW)}f zT0<7w+htRjN~RY+iwx)E*jZ#?W{&uv%ltp9n8hdR}VYB?c957Pvb}!119CW;6tr>m1jQ|zZys@xhWHHg;uZ+jt)hzBFrCO0%vLzzPYdEd8EldE<;ELaJiKDX7 z4R|aHF6txZPw&8M70u)%>3gOJ%H*YiNpNNnaEiSQG9quP=_Npt5M52Ty?QTsKKlQ$r8l38?+; zoQWT*A=jbX`OPjVOYE2=1i1KlFt_o&xUmiskWP-tX1@ni~qs;L`Q?XJbOMuCiQ?p|)SKQrI9JCy2xIcW7%Ft|qhq36_5`K+DV1LH1yy{OK3Q zT=3?g65+6q?nhehHLAlAYuS<+l32xfa(MGPyik(8PhIyh=WL6Gkn;~+X}q}67ul9n z1gn`nS@FNab2i;a`nIppYna5mjxghAd192ix#rIhQx!h!imt-H2-dP=4{J;b1KQ_5 zyTb5aloVKzhSc}a#zx=Jtv78)rwBafB~MJucvXyEKOHYc`WQderwJZU6aL^yDWfr84thlBDeJ z^KGraoI!1%)c=^0wB?K(c?-MTWGV#KRX_sDl4A^T=wDT<|Bh+PUCAk<(%YsO}hjT3la@%rDXop zL=^Zg9tJeg5oFbzKR;-jmxu?&RL4c>rN)XJ?qXOyOsmHLkn=3 z|30iHKLDDef@j|HEi@p;(Y{&Xml~qk^gpLXrv!;u98IbLZL@;`=689illZ+-;VZd^ zZ)$w9(9Nt~lZ~APW1Uk#_Hu~5_4=W9;+h~$GXQeC1gSyn)ku`(x~d7)rvl}fGfy^4 zz;bowL%%1i6~lnM1qxoGJHpCwY$ei@oCS6VnjWY#be-AxpFkzz(;WqSY0E0viC@Gh z9f|HTRP2Au@kF#~w*-IE= zf8~@+)=T2GNcHrHaK)axroD|YM%k-?{T+HhJa7kq@D#nd9&Us!C;W-WSsYM{oq zq4x_qy@;6S-FYV$zBF0eQ1U7io(p5Uc%U~ckb_@I^+oG*YKf_{e>HU3-0H$MTHWgw z%lf>)43;Gm4mxw`tC4PWF@`L%n?ON6#675^WPhJ8>xY>}&9a_%4Mxlqz)atf#%vmr zYW=Qkz-MPnSw|FPLnsGXoft>EP;AX& zkQY8|mxT75GB>&nf+d;uI>9wmz;1VN*B8+WjQRqHxbeYis@PcaJ9Z6B>a&+-SpHnJzC7!j8{WFTO0jf(Pq0w(_Xf7& z1cmCXqMF9}bG#vyp0~gEc%|{fJ=c{(cpPe4 zx02fJ;?VxBXR6?1`@ln`zanWOfBta9`A--o^?m&_?3Nu}wUbU7k)d&br2y!)UT|9> zukOVTQH9%oJm}f84+PpV$xW#E?1IP-d8hJ1W53(YI-j?QLY(?7o&22^s?~p_{a8*9 zqM7!~lDAJ-)DxlruvQ%GG?BQBZ%}BgXJcTZrwtoYE)0yRz zQhtVMqgu}puZ}_ zgh%2!75?T`)2j!RP>hJX(T?j_k^!?(leJAROfG$m~n-5j2N4X159M?76ST%tBjpSyrOmDi9sy`gdkn=RMK+iN3<_@ zs#{7f$Y#_&YX(e{Xu39vV~nb_#sO|p5#5iDl<=KRw9ebYpL#X&dzC7xX7a)fEWeeB zBhg=K`uLxgd*WdWfHSUGPn3(aN|76-ql=Wh2HB9SVU4qNCG+|8VF~=eXM%cirMW)4 z$?h(>T`TnLq<$`*o$MuF^4*>0vpxL;zp(ziMj_f?M_Bd;q?{l2a8+au$P1pLk33bA z!h+84Amu`kkekF2<==6hC19b_MX4-9Z_-L^n8DznHghV?=7OjdF$ znIa@;Ifgy17-f1sB?236)(?SK#e;4#5!!pBY}gLZpr*KIeCd&)yrw=es&97N>PRKL z?WBZwy!fyj3kZ7W&}!&_PxU5OPHmhiL6ZyxZp>)S-FfWft%tC_wC8ii`iNd+en3M- zJH@4-Xp1T032TvgjXDad636+~n!i=6zt%9!NBsJTMO)MAqlH6m3M;Xy61EF(;$NI8 z={}d|W}bg?Ebd?}74P;BfWGe8O*{hgJYCxBX*#2D7#(x*5m}g@oc}~B#~363Tw;F_ z_cDGCe&9#F(j3o-kjf@_nh@~OH_p=O|H3m6^X_q_<=XWv;Nv}2RBDr}f+~4{LtoSX zgG0#*OQma4fkMxb;!@bNY72bpzeraJ&dO30C7+$l!s|Dtf9?=JWKU>#K3$yK9Me1m z8}msOvX1vm70@^tW2MkcQ!!S37f8Hw>HjMU9Y%MP0U(t>Le zk$&14?WvXZDdXJlslc8RHV%W=%&bx3ANBMTE1j~lcRt^`iKVKvDSY&DN%7rWdmZpnd;u>unDT4>Gj0_BIKNz_w~AC$!TzgPERl5Yud-<|%1oOp>q_7~#tx#JZi)1oG%SV1O+2TpeGTGfhL|VH zYB<{JKf5j6@h)j7fp!U9mvcMkjS!iR9>AOrj_&r$&O=`Pmq+x#(fhjy7bsbW#-c}B zv`YUbhs<5H5~Ez|QE<2ryZ=>O^+V{P+T;vRMq0B#-b)$UTin8jH%$so$8HqoZtMWbP#ZXt{^iW*Bm<=8!9qF%1o#rGE|MG0b1 zjB(bjS3i@M{b#t(%EQxEZ?aBo|HWV*$EKP(A|^ccQlw?QuS(N<^DTOv?zpeVDobeKa+4QCNx1G_!- zK~vx6D2%|59$Y{3bsjL}4HXcqFp_p%G^CE+kLkT|?o*Hv08vKIuB1&h`CX%`;&cKg zGKt?lc2eg6_qY$o{)5xbkS4^!%vojZI;ROi`pF#&INxeZ`NVp(h+K-Zz?}Fahg1*k zXKE+<*uiS;EJ_)Ry0yi?Mwx@1KviOkgeS=#vf4J6n{Ic>w?4JarPsSR3@6N%2q>xl zi0eaCDYP04?hJCWoCD&rmLN)HP>UJ7$Hp`2kKj4iXUV|^H6oYyF32-3*D_&1dTQN* z`I84HXV#c&ZQEKFTvIqYO^&muY0jOk-+Y2{+-po!sh6=B##i{t(BN2*ax$x-m*US^mzE&NMZ31B+)_ z;kCJ6E#@-jKXZOt=XAHbI6BvUMcoaO&HmFg6QwG27uHWgky4(Vvm+X(&z}@H^0`GK z$}w~h6u{)xPv>bWrJnHt-h3mvM{TFJd3#SZ(xgqHUg4r!-)N%!xO{Tl=96WyB!FyL zn921*FLdraZX9d?6UlEc3hhR~>ckdyu|+>^cQd&sO9Y1pRo4--mmDY&3DnFsuW6uUjM)_o?_tNoy#-1vs^yj2ItYWtAp}IZj>rr~Iht|E!QP zQgf#hyTA7HnQmE&?jcb@oN`&!%zNCn%Zc9~4$7j=qI+o1-c*mMO21SqugJXRKmrkT z`-R|{L0x(EWV+3G&qtT;QU5CH^;mC9B6B$2zd7Vdz+VwbPmZ68TbnW=uk>5#o=R6q zLRdPepIQc?WcC#@Vr2Mc)|_3Xe+%k}u#B$ZKBV26!%D9?MVYVSA6L^Brl2Wnx8Z_< z4v_$VDEKPCI!|s@n@>y8NzJ)ts=(89x)AZIAH>Hq?ws~HS?-;&l{>NTWpd9=@2ls^ zql!G-B*4++=8P(E(Y*QznOTj3Qp(TJhb3&G#a_uxx^O*8o|YNbk8(d+41D>TMq(#f zxi&GD5$2pZ4^pm&L>rHUssp^=7JnlZB+cp3>DDnWo;%cdtJ|#Bd#s{ehxuWkHZ^pT zE!r|FPDbJ>(@eyYRmrADin}2;tDK8=aiI2xy6-w`fe+3Z9L6iY#c>N8NS_`&?N8d? znjgz{rVq5Wl{YD^V4Q4fe9tb);0hpQ!ro0D>8butpZGT{i}m$)%yV+tzi!3Q#vcy7 zxW9FDC@ErK6~_mNjW-+-T23#-jXCCqAjvd(JwNMQsV4}^6#zybt%Gh}X@f7u6pg(l zngt=d+K5juB1cwpkz{ql>a_?Fn(d?KXGQcrm62ijTpNrmOMzi$z5j>1_Y7;QYu9#B z6a)mMOO@V1M0ypF-a{vVD9r>C2oO3bN|O$uNUx!XPUt9Ix<0G zvDVtZ_Wu5S|CmRZ_Z(x)I>vop=hZxNy0>*U*+wo^7QAA?QM+CWQF}|-Xqqx1JJ9Rg zsy+B~5aXJe#c-N`7=JAlOgs@Le4KK#Dff#eG#T0zU?X+6xpkzeyB9}<(<9@BadG;Q zS~NsYy_|X4kJf;NywGCLbE(MPAi~R9xULd%9>S(;(>dnZd}c%oKMXBr9%2zWpBB3; zS(0!yvP*`k+M3O;1zNP|g|-C2h}T{R=k20)-|>vvKp%?RegO*aH?0B+ zMQi6u^}G{9ZIAX)m!-bdb}g<$2;r8IM)kM9r8{-L8qXlk7YsQR-t(u@2K*o=e615C zb8Bg#Yb9-Fw6meSGTIL>Ilf@8M&^bjv9Za<<(J-b@+r-)Bw4x?N5!b`-%@b^+HO?5 zodX0S)w^JfWd5Meif?L&CK<}`A}p<%{F%r8)~LvfvU4c*jd0)vH79=k?N8U~JWH6S zWoNR0=TKk6q=$-E6KySK6ZPF-;v+w!zt-TVS=r0Y=F`0|^b?pAD+jyfrbNUgX0a;U z$vqgJJ{XA?4|FM9yQ9F{UsnCRv~uKLk;=#zh{Im|*`4t?=|y>Xj5rsyvyn4`9k1%G zawhC=YiA-BL}V(J*nB-pvNPiT>N^3_k$OLas%5a99;k^<$k#61x&65`C08yVu@a;Q z&)q9AV8NtOEV+)Kv_&n*B=g4xc_)Y?1Kkxz%27x9A?O5LK(GaC=wLyQAxJxqKD!lM z82^RGOPnB?l(GM<`cQPQcdxlt%T4l5j!0Qk=dy>z#H-`N-=yH7KCIxt3>}dWfh+nJ zzR}U3Y0otulLcw>M33?EfTyixB_62B{V@--2cmEO z)`h%-ICW@=!W{V_g@$orWP6QQ_-^!OWjtq~2rUakTAHSSpg zg`JG?eUD#xv-EvlMfHj&G}Yys=}jDT*214h5eW7l+pd4%-M2SMMYIZqsq?4!(nifD zhFNW8zA@q|rh4xKBPc3DDq)4Qvj&U=EEkSBk}^(n*ZG}4pReWueK+Yw)qVSt6sW*_ zig5i`6_S_*Co3j2?#{HA>Ikgg&)rb3F!8R4K4en$425WH$+@8Vc9to_Iwcrj215}^plXhu6;)$Qt3Gutm%A~#QKkiQPzV3Z@nBE$mu=`Rjlyv zd}zWgoy`dU;Rosmyd#}TL11)WTf?i8Vd<>}3GvDia!S1csMLI+&-1(CzNUBwOC7bH zS`=hfU05bqe0)fS{liQM#f-Inj$@2~Z^f7|v}3}HQ9as{_busdt=`V)MvDNswy@BY z#1Nz@D^b*|rd3YnY9chjxYSJONEv4on$S3YqeWcv{#h1cZmTK^-3w+F)+F+&Wxm%E zFRG^O<#Pl6UMrl;zRtVt^_-MjrAjVD>cv+)ibTotKZCPz1hU=&{-~>O@Lo0zz{K*r ztlB{bb)LPjD!+e*);ch8HSMX?HJ_g-9wXDU?t5D;4f((Vxj||{_%dmvRyOm*bXV&EUmw+)0R28nlT;NtLIDB=oLro==Y0&l(`GlNMf(<^Qc^s=( zwBW{>#Ub}Zx*t{piU6zAYp+!uRO-)t%c!G~nsrdfH-4kwzQs-B6Zci`;4cGfA7pmt zysr3-bcF`)^mBOJs@&u8(&&FhT(8Xz0>sP}Xu9*y!Z2bxv<-w02EC*XS2{Y17Dxvm zY@%`|=mtJMm@qYUbj%uzAW1M1snHuwt2}t|M4Q+Ws64$x^`$C~YFW`&b0Za0W&8! zd^MCZz9qeaVeF>_=OMFedz!u|ToVmcRL27&Kh`q4fT!_XC&wE=mc+UB3lb(owWXL=;kIkk2M8fni`2e&rQ#LTwo3gmuw22EDDkHmLG~rh7pMRy$a*@qM?bkjI7=@G7Pr5@!N5xB1O$n z+#c$O>4WJ_6Za`ic$i@}m@x4SNmf;sJ-LI^du&=SDpOx{>xfBMM1P-!{L2(M5s^M0iSG)8pW#@ z3noe?2J;6)gW?Lw7%kGBeYwxI`ixn{q~-Eut^*2@ne`_MQ}f*W5R8zOi{@Y^kTmI= zspKh0X|P7e)aSAh?1i$>^*xJ%gPWJNn>q%kzqZOVEf^ci0?*$m}Ot zdZE0K*vI`;Sm%#X(E6-EvNBBpjH%<)<;=!_ZlZjt1*r8~{7#mt)IVVGozj~=&JKn;`)CJ= zB3TOARNTfOFP5H8JWY(()`+19u3=yV#t-_p>VtMsu1gUC!~1iaatp3{Rt#V!uMnWe zlZ42Rt!GI=lyQ)^y~+I4!R$=ob}dpuNSq62b%GW)QY%_wu68uWDs3bTE?y)*#q3UX z2*5iqU)!d!t91{2jS(VQ2M!(WK?Epe-rQvx8$ef*7_OvOZjXrclzXH{8OW7*(b9aV zxz^OamI$}lm%Hx_wl)#|v-T_Wf=e#{>|U^t3_u%7cweodrDT-M2k`4Iva$;Wqev@K@YFb`5XKy73X9 zKjPENr_dj1OlyhDlAV$jEB-gd>k4|@nk zniDoJgZvKGT*L-n&lSW(e4kSg0PV|{$vfs}!1`>+MOWuwvR&W*CbYIqpcAzc!sfh& z0)!1JH~DRM9*>LPV{6UGj&4dgw_*_~7mri>DP%#{Ei{}}{91HbCuNhDFO=$t(erT3 z+r&t0G@Z`vd~l1kvsR@uStRb)BL&Ja2XVbJ$*$gxB_^d>6-Ej+o{|9gF9#OR86qXg zMtzoIpTXoYu8&DTN90zTVg0G4AHQ$YUGk3VkbpOZzc_jj9nD+J%bUnOEDl6y>j9pG z2g&FsO?zdS!8GT=0bqzg&@o&u?WNh+UwBD^i)U3b8M1K&G))JSCZ_Gt+Aa$Vitfoq zUL+i5_hsFh+w3ga4{@Y0%Q&OA`-bKAEZP|zZyY~ko(X@abU(FQEJVvn(DYoF)s-O! zsf^zKB%fpL;S%hDbMdr;_YL=peTrA2nCHSHH736-F}3TwfaEdY<7Y%sC}?Z^!L7u7 zV)B^5fFkc$b5JL9WI$T&7rq$V*ots?CJjYpey*JyMF$4R)3{Hj2=SFIe`{Y@Znh02 zRVmbP@O|BVOqI2wUTvbHz1=8II?#2x%3RCuJ6mVAUK_Vs@RfLWDjnI{^k&S_zN=d4 z_)Ra9so(r7T>i((5w#!o#1tNdbRXnyC z%cTzV5Z$?MffvRHS+&n$Ml&QCMe_OFG;9YAEJfqk0;80`C$>`_#4f3+tX%b{iHuD= z9-6ydeQVpOz_(L<>>Su1XnC?B##48KiC6N_QWY4_LCLwKe0gFCVE&5j=}hrc$w$jB z&uEfPVFf$`ApG~I7wM5}eV<1J$~5`aCHCjIU9jZ_KTM*mr!F!Sj7Vf5<)es}7J}?c zp^9alR`73`ky^n=w7(}7eRFqOGJY%@`6B>PqGUSEIjujUMZ#FGxj}~Ro3I*=+6U9r8tS#Uu=Q2M_N49 z7)ezd?uS{&{aWI6>ml~98Eea&xaGJK?)9J>AQnaCb{ES_RZmO23 z`P`ky2~-3g(bg+&&nk7p!({Ks_UBF)&?_0cM>nfM7wJuAOHqmrj?1T$m3p*7BJZtI zTc&>Co-bu+OB9w4sP}VxWpvU5L!4wG_sfNh^JDwnI`9aG{}HuMx^)V)C5YSD2A8Wb ziiM+kVy@4ir9uw=J`S;nQ};-ri<9%6pZ2IbMUUd=eYQ_5T--EDI>fi%0--7M%JKCF zeCC}jA8laPiJAauiMfFWf)$xLR>iw^+|)P=4AbV<6rAm*)qOSVkdvg3`^BU0 zZNQw|ZwU1_Q&cHrnQA_Q?xaclhQLh*7kuL$q2q`BT{=g?=J4H=!w#*DDeWsfTT6*~lJ-mHI`w;9*g#!_{HH(^q zRv-srmIP~{3Yf?;6tFl$s@!6ZsyLlMj@Z*7jtx=?hz#Y`nq3}H9(Ze9{CrrtElSG> zA=>3eVFv?%@}*h#9est>LOR@6xSpH?U|&X)j%bE(7Ut?z-*=Hb)EkpK+B@1sF*7rE z(vqbTd>{?b=TpnM=p@nsoOWlM1!fYJ5`y{8o)*)k+V|)$yz)jVPRZE$VxP0fip@r$ z)F@PnnMVD*eV6x|$ckFPGE^K{v|L8cvT23M8nB<@V@--z!r8j3>VR5}o*$5DH&I5} zQ+u4GVhQh&oik4Qn|mSDZm`;Mi! zyw7-gdS9$pVCiD4+RGIAb6kqAe%+W%pn}Qr=m?gc7$mG;%E`;e8A^szPn)T&?^x}| z3~Zg>OVXO0F48Ft6ud#Smn5PZzy~jqhn(uwbEYN;OtRwAH zE}K;15nrkooN&k|W@M{|qa5A5b5%x?;yKEDo4ez6$_QAJtS_xb#5Q>UXzh=}*U(PI1If1v>*X z0U_+@r8pct5K(n#@It}v>x}v6(+69;nhxoN?-<(L?Z#pN&2d`3<+{zuy6N#%G@AN} znQn_PV~-}MvZ_?E+vIgW=3+)Vft#KIjumzpZwbVfw0M>as-vjh?NT8GvSjhB#mv}M%K;c8G_4y@+#v?0yDy*pL5VL4WV_N5xB};?3jm}hwU$b0X z+qc8qDu*v%E-d8F4g^L87o| zlJ*3Xu)wH3@gz^6xfygal@11KmnO0gQrnDvctJ?jI%D}lhsGy!;k)ZY_3j&}Gr=!RuG_Xpm5mP* zBmtVU%=yo)0`I@7twVj@Kkr0wtuAaM9{vG5X&Vk4QF()2NQd|GaZO(^LpTqJEWHm< z9E>j(lJ3dIbnCbI*T zzol>PG9owx3_dqjJ^1zvFx=OEtjlVjxV#j7Q~{PoTFs)w2@WHr>j(5kvwzKCjCT+C zdJ$MXXY;A1Sbf*WL|$9PoY}H=NHxOo&^xWwbt;Qq*vb#l#Kqop`@fsHgBneO+?c^K-1 z-SXbgKIHM`8%vC$Pelz~?9Wd@$J6Tb{LYH^g`0#UkLR`%8BEcw`hLolq`jHlheE5v zDId^UvEBj^=hIF8T4M%eeQ6-KV<7PscI;=?vLE=tg@GxWtMB2VYbK&wTPr2jYea!& zS~J_$8Fu`PIA7&2!Bs3^xKszBOy$IS7n()s-swS{r9(chgz7AwRu=_d^2Wt8pwK%#gcktaXJ_~ z7!ICFw)-x?FuJ^3so+?j8uLo$S42v~1yVViO!sBep;R#qZg%Y3z9mO3=_l~L?dNU! z*EUeG(7_XaUF1Buq6BKOzyylVoP5%-pkDOwGI2L?=D>0sCOw&6MjgsRQ5+k2nDH($ z5YAB~)_c)i$v)CznGGq4a5uD(_@0}@^jZbB6q$eTV-LPrJ6%AFxo37>OHcXPfMY>v=i6!{!v zSvOUUjkrBFIIUMe22&47tnoiJ{_ISidt;R82ljX&y3R;Y-zG$?T!1Bcz-b%WRQ+sc z#LiU95}H$k;%z8F(Y(c`oKy6t()yRE>;k4Istb7$S<45gII=m%dr=J^WRi2}uObC2 zZlsK9<*dvGcWam~Mg2RciADwIs?_+#B8M+7JpR0HsYn;lo?qGZnY4x!+tm(@Avc-G zIQP)7HZ%EhuZRrx#i<()Jrq%#i(54F*OHL9IjeIc1J9A|hGyFm-g1}y544(`6d(IK zh%?lXfqg^XvH=2##LDic{sJrx#eqH=lZd;3RIWK@*-suz8Rs#p4G{NF+Idvzt?qi6 zCrQ|WF3ZT<;$J9)fz76qf@!&NM57p|{pSO`il9%O<&xarHKL45g zr~N#9M{W8a9#XP(U#eIl<6p&t84cdO7{Qq#UPWFnMX2T0VQv(|TQmBVCI1UaRY_;= z&i$i6oc@mUePRw)=Lrbwy2TN{1267N^eyao8N??r^3xI$Uz<_19jg z0SbTrv!Vabb^o0rsZ{ZwjmdupyXW^#(&G&A>+R%d^_J97B*eN@s+7&PCCoHCR#bzd zU;ka8@4pXmr(}IruO0~Ow>{Q+p`j3PC%x}qi5k>jc>hG6L03o`cZBjwePzvI{(EF7 zrA#NiiFEt5aS15O2YOh_{G4YTaF_q(mMSo}Dk4yM-LqzwKKd2Q*C(R)Jo1ElKs&IMMW^&2_`;1^8h;*Sq6c9NA>W8Ok4wgR8Ls5jC=9 z0q1{r`QP2aowlCsXS1YY^&z2<=tADYi|8Zq(0=OOA-EI21wT(4ChmVZSpNSXcXK`H z)d<(Sh@4qNn|~fDrE~Y_4eUD7bVFz=C_?cQVt1lTKI6z?`8erMNxRZ-;&;8Smt}ze z!h@AX3|v)8rDQ|+*&3*m$tIr1VVxK3qLWf=IC)r$39Bj#g;GjqeDeiq(HMe>X+27; zIt}kR@$EwB^oMs(N{?E>?cO(mq>h%b%kM)%rTw&SNv!q=GiTKsC!XQ`uPtrmt>H`T z5~w12{Wtyh1zOAU7h05Rl&whaew?J0c-5Tng&E{r?R_k6Ij}2x>->HCkB{8;-Wno* zaLQ8FLIJrmr@U`QC&`GchRW}{9G@A(`w{-NtYurpTxuAv@wRrnjUMuf+2c?n$yMap z#5rLLH^U^kHgvVYe>OTlQFhF2cSLW$(~3EeZkcn-#wgv0v z1ukiBqhfxT(*g5)FViAt!IIkCVPlJ$h3FeH>d65<1Dt)}&{K{0u*V=hnFg0UB6@!L z5AFQrgoIwcOyEgcDh3m@D;->VMHxSVRjf9Yqe5sxioCq%dn1W@ElF@`%lqgMsF4AV zfI&ajWa6lDIFz`a_EcyqlS>=s#HCiVb;V%wfHPX^b3tjp$9i@ThIGOvbK09`o~1>z!P@%d#;HT3?gzDGuD4sI|AwIFxw>vb zT>p2}Qq`r9s+URE<^}e+b`JDdg?C4p3Ugm|vq-qb=KQU54uUE(G9b-xOltfuJdr=* z(}% z1D3I&5ZPqbU+@dbTQJMESC&r`zx}1xJck1xV)6rb4Qy zx$jA1)*@MAEId9t_s+^zMeu>XI7iMX{W~o^vy14BpCOmUgSv*>dYg)qal!tx?@YDf zGl+I2*9fO>6-j?LdMcrzEu+{Y$vBOooQayY$0+SG(9?K~okqqahLx#mb&Z$EYdtaUhO9S4+Rm`?Z1Ap1< z-l=~hcq)oms1B%7H7fd=0r^~#`XEjJ^ed{UfuOUhvyYX)8@aDXQ7C zrL82zr++C2zt8Pl;_p;U)65*{r_vEwc&psMezCdID=$Ug8VBw427M7ey44jOd;=^X zji~+}V2abd;1Whm8J-_B1svCZJ9t{H^N+6XKl;J=Y)#GdE)}*ax?oz@-?Rw5fT$_0 zh>_VM%ff%yzGZM?vQySvRX`)#6mu~2V!~zNku|P-`;sd9OucOEKYi9iFMLz_fJ_*wf7Wo(GN(L27Xkj_9{6iu3hleg`G({ zeu6sc<=DEg3+|Qb+pUzU_?Tscz5!(IM}7IR{kX4D`~Eu<@Js9FD3dbjP`l98-UzaD zmPWec6n}7|DS}F4G4f)Tj~?_LptL%nR|g&LF~e6!oYMT8i+I6%7fgG8prO(pqb6RC@t!K``9qNqhpS)H~kS;T^} zhfAws(WXEMFkTFE1m)X7PCpjBZ|tap%WU9LR@Jjaf-9JDY>boX|ToNUdUUX4&>7*?7ji z-CD08<2TyzH!p2UpBXc|+^Cc5YIZp==zWcvkC%Hc#Hnn192$L=X!zOD4O4gDcJ{X%e#&>@tXWv=y!@49r`q%@2r7 zDo8Bo$;Mvs_wn)DC!vG(Io3Z$hi!E+kPCv+Wv|ITk=v zn{VGdPb>g6#R(UV{lc83UnKJ|6qe&?CK0gnH*5+(>~<<}MbtDl__Of*`vJWi|2}Sh zIoDFyUI2f{IOr1*TmCsw7H!g$^Q=N*MD#rNWmE^Mv6Ui%u;&vS7q|A8`}ptiqU0&J%`^D}IW8O5nfk_AU*IVa2Gqb1) zHo&}O`MbWE6A2Jw&&=9Rmgl!TKAcdRZ)WwW)XDqzxB5sPOCOHt?l7CUV;GP_6AM;$ z%@kM%Rd!iBcY{o~uN9xBCe~TSgF-3KzCR`acvwYln9dFY#L)S0Pg^won^S%(`S^EXR!!5N{e5IejJ^gB| zmbVrLjuH3cZ?kFyjAarg7O`)zp>Tl@S^c)lvOBd6C}%*Xp(W1WNWhl|jf8VeNji@k zdwLiYaH>a)^+4q0c|MgyR_hO6ztPWMPOSsSs1#5Ag;%i1^-^hd!YpNKOKCpP#P;%) zZCiYn8S=4F@gMZWODDMv7Ojq*!eBtU-@sEtum;*kW$B42bwMW)Bc@=U0C$@xI0e%W z4aU~xpE-nv;h=m1)oH^>-MLk(#K(pSF|$0de&(s623$$Z9JETl1ak5@{xr0=Wru7K zqo;|}N0rm9C_!lnLXAZwMh)hQ8ltP&2>ferKKbS*Ka3`D<(aAwYj#UtauX z8jUfXTUPu5@py`pm%z@<^v<73kOj^KSta-Z#C*ObmkA5q??Jx@LabNL15n!Nc!-18V(U4Jux~CHLL&oQ!!TS40LNvV1D&T6Spdr)7*pKF|atz0gv( z5HF^_7A>41)cu~K`vl(prPRKGe+g2D=2Gb1A(L3kxzA=DvIc(v2AUSOE+E0`*q7@% zZf#Z()ypsBC|@j!@T-_tfFR7!?}nIr$Y6!N!fat)pO~C+8jUDXW1j0PfOYKh6@<5N zhExhYKY0~Eb`!&yJJY}Vh83yGe~+Sv;)B43ehgr}9@)Cz!ZpqjRjw~_p{T3aSIfPG z6b@Be@=9>_$)T0+l+}>$?D-3i!rHrGP#V%f|53{l+YIL9cl=2~>7Z@6%&7OvPmi#u z;i;KZmG`xo*(Klk1|QR-Y$A;Vd>UBWb_P0Weg6@HUSG3CP?qoKpR1(Wte<-7pe3tc zG3#y|5#pc4CDvnfrMPGV7?Y?aDZa*vziHTw(FMH~nff*u^R+KM{?^{>wH1G>5)cOH zR-?{zPs5z0Hj{dn%m%|JSx5s}zxi1=ccq{y+Ew4m5vI58l4v#bl_dL2rViOpUy2RW zNDsH*zOJxHd7`G5cMAECr z=T})|s+sZBi_BD!z<2{Hmyeh7!<$jc!>Li#HWP;{vIxO!9f|Wwsjq^zI6_^zoAerGn1i2P z?d(JBokj&-c?oRgw@zhJjgmN2{@g5T>#181qHvPu!_=o@Ux1c1``jg;=$c=vUCP6! zh-!hZLS4{NtgSj|o#k!Fwfb8(E{l>%J<2G{Du(g14-46Ro6l$1#%S=z6Tz`-m2xKw zzN&gcs$iaqtj)F^q8&S$bJ=E`(fkc1LTqY>Q7fQ(`x%avWk{oh@ja0;R#4)^Z-blq z)NLtwN)rRer!qGh5{&STK7Dlj{WjF&ald@s-)>pi3JKOw?fEUkpdLzthEgGPCh8>@ zM}1aTMLH%T)dx*5=p|%`U%`a8P!PX zPK%AyP=MLUeA{J=*sIDlvkL~tRh?6&>1byp96md)WoW8p<~TFyd7^7`{lZ)G8MOLm z+7JRPlK)9hSVp=>Vz=OJD9@*z%D~);pB?UTpho%TSpV!m>zIOmLxo_=maemQl}c!1 zDDwm-Z?4maJ6-A}#Ha0I=al9Zzj2j!6AcWLkj>-EKScL`YSvL&O`2)LuiKp7JkKDD zlXdPKOD{)2PEoZabdLe-GLqDCv-TP@&vewnq-+MQw@KuMN}iFSW^E>(L>RbyHs_xlod|lCLSsI^GVg7 zu^HHT4<}JvR@djfv)`6Ot3}Qxbb8ic=Etc6>vbIQ_~SL}Nqyecsn>~Kzg7IvE=9}( zXA(Jkre1Rjp5^Q<;yb&Lm~W^s=Ea53g>_kZOcz@|-Z3Co%IlyKFbDR~{C|1rEjg0eqtZ=dz%Il7T-V+#r6zMe}lxn^^0s z-a2qMID&4637WSbF>MA}R7icqnZpPkqDstUW9kvE>AwPwNz3uB`-!&!R?oBP-#=1Y zGYu`zcbAz9pi|g1*-`=Z?~CWd89mH07P6Tp>*~hDq{9h=jGA(!${kCBU;FhI?~5Fi zqDvzJj==7kSfg-ROg7D@AwDrMMK>sOekRec4CfUV%I5T%VbI%YUkX=V7trdIP35;w>+vdfSGtXXO@BZ{R({#rFW_!ev{Q9CdFQE| z)YBRhrl}y{T3q;SpZu|(Gf4~C#VQHfiEZgIzO2{fhf&$@Zh5$(axu`BxsP0TIjAQy z6#aM6U?W<4@Xy~q^P*oEUTWS?gDt%7Z2FIsBXg36T*l5Psm0~E$4_J~;&9H5;(zj> zB2jMc;)_p#lC&zVt#{k}(|%7&uKlNaU=sMO^s&XGPi@0x{{kZvz@Uahdp<6I8r> zfIwNM4V82gE)lOeXiA+3$eSi3^*qpjiZGvOL0hHz0o@?xkWzezg6P-&^yY{qGnAm8 zBnK92xG)+q9**?=Lo;U0!*Kb^$WIvg?n&M&%Cld&l^GBOP519Y$vb-f6O~oscCZ(~V;CDwslgR2A`}taIg(n#UI-xn1i%d!C& z3$6ma<5qsrTvXC^INW(ELP?2tA784YG-8BS@i8COjm~YtHxpfX9Lj{iIp{rgmfmv3B)^pR$#u;RuAe zC-Fu9J7V*Ef&USV@4)>`xC)s3AnyW5){I^Sgs>i2Sjfuqa1hGw&RM?s_3QdkdSk^G zCN{ld2rHMsiAm;i;W2jfSw{+Q#8J%U#+pC=QE`atm#2(Bo`jkw=T})1Qw^V)S*H|J zhEcJ#S0K$7)_i6=SGt+cAKGf`7FuDZ*-KZyTCh6U+P{}NpsrDC{AX`Dj=*c5#jn5k zw9E^-XPoDFiH$1a_2f)@HIVIk0HPoUO(tSB^##M{?DtAl>}CEf}PbuFJg zuCdj~UaJsgwi(jIvU0{t#@^hG8Uedzc3r%~W=Om<3J8kwV3Ke^*GOw zeR(YRHF^hMi_atTzA76BJq|_Ht1j>o?73*@d<51Wigx7c`Q}n>&Sbsy0WCowV$$&5 z8_-J)bv6pvq^tXb*=&#;eeJKfZ??@qV)U}D(K&&E%H1lO`{6uC2!WnRZF_q>nVP8g zxmij*3y#+-frTigF7J6JYs63LHjW1AFHz?r8jjrb{7T(poa z@VrHb^4SW#c;uEj(aZD~UOHw;I&|+Zyas8M?&)Wnv0IZ}bx?4bYa(0ovv0$_5qmRH z59t;y8S#?PKR6)ja?i2O0x1F<o(gj3obo~+&viyj4y5x zs2UQ#P|hzkjnKNyZ~0aC%AW5(e=8{i1PlC+|I@lx^`HA>|E&BXI5SHeiR7j1_|L!n z|04uHe^T!M72yaqkbLH1fBX6Tm)Q)G|Ea+{<(y;S@12t9j-RbUwbwG9&Zt=lG|*m# zvy`>)5$xj@&f{5JwPVGlE<)n*mm8KlR^Q6D9wM--;pF3G)Fy%=R}Z-EYhuk_F9qKQ zn(#QcF9;r#s zk#KQycKLo-qzDHXS;8OTVb{k=7e50Ca>1=ALqsaKjflxhlUah zY$s<(Rit&gwSW$JjqFCn4n-8xNP+SZBRXc)yF)I1oduDJlLd~}<0|A@H>FoGmt^wr zaKjxh#}~RE_4`a(dOGJj6|UJ{F;7nWHxZW`s*Z_X6JTHzHnYCdEZV~|=q%jp78_}d z6BNLwf9H=#oIlffNuN?NW@=5=T#1h$gqu$#OR|y;^wFTe=qw4l`Lr>cymy zl70WV^j;S1+b&xai7ZC=pXn0|(hmDacGYE?0xV6ppGR~j z`iJ+b!_9)(lwK^y-h;9?c}sp?JN&-CIL#9DTlcm%{m?GEq%kAlbKKTDg+z(g&fSum zMy#)Dd)J2yhw+E6gfeBgdgV3=r_87^=U=pcX8B#&Y3q9RdfOaBGm>jN)&bRhm~whhf`SUJmt@zOm%e(zk7IKDy>v>?Ib)Z=0mjhzvk#z0V1)wi=?bEB z^8jy#FX+va-*5wXxI1B*tn6_-Pk!QGNU2xol$K{uTI0O`rij@T)Xq2m{*@I z1`;0bQrbO_w^Bl0(G_iBSk?fhX3IK4U3VjrWNtH<9Z?dodKJg) zb*0vU-rP1y?h6h+1UZT7q-{AoBYn+(O0xW$z&wRuf(1u|V-;d0WlN3a5S|jUoi5BF z%a1xeW^D6g*UQ_m{FbyjOj1Zpn7NA!dXVrQ4*6*aKMewP!@V@QG}h;!vk8sX4(lyL zQMiv;6%rW3PIAujZTj2F#v;`PtO#8znvpc*xHq>OekCW zvY#;Nq(ir3>PbuW;vsZlosAdt?6rslhl=X}NU>MFVulYXiH?nj5)VxcvcyaWqs2z^ zt&<@SpTBL1Nl}vXvSO_D-$@OckcrZ5jq0tL1}dm*GeQKtkan;Vts3+54rFz14cW3y z1ZpT;&||p5%ccCvP}$(5)#fj}Z|>2*20NZ~TK8M)g4!iH48~w`J8wE;v)!#`6;(U@d!u$|oJ7Ybv$EE-E?3xqaf^$m8)I&}RF z0H%P#q=crzXbKxxM{6HHe?lL?-D@m2Jqzbq!{7?(nw-d1sm!5gv~YE*s9>Jss7Q}$ z1a4YGbzi(6%#Ppg0w{v^n9t=sa%j6E6qaM3K0zq$ytv(c?!97_LfocP zy9jqobr-ntH6*z+?AeMey~E7bd9F|G4IMG1D`ZNU$eA?bok;VScGj5_%TYqtFQC!+O9Oo)}<{c$U55t*2{>|sfoz^fwT0sRnap=VO zlP*ROKG#dkFL?{M6vmH!A9~LVLfD=bFNGTrnJ?v7d)@<{9mGZKIdfj*?v&yR#9`)t z?2}1NXXZjoA&A-dneRgX_VZ?9(#S{B!EPGZSSCWg%W8eqa2pX(n{#Q_IG z3_ee$gq;g56jlNYBz|bK7+#yiMaqO*TWEthO;Sp3cFvJvXK(!?>ij>SC@0Xq@k*u0 zS5SPEFVj*+Zj3jUKt9gXV>wT(;`1BT)g4GP(*l~uxv zXCr^^?}Av_7wRAT$q?S-BA$Q2oIEN_I~Q7MnLoe$E$qabT47e{cDoQ$yBPG*P&}j> zJTtzeV~mU(E${8sMJ>KzB-(P&G$Y7h3gmfDGh3frm#EvK`cYNrR?MtHrxaS_5{*hy zCti5ASa|1a@U-(txA~m7;FdkOmBIMrSDFaT2nxT%?!)^M;Q5NArfwZYZEp+j=BoNt z=~e$O3nxa^-4fJ3hA7AqD*rHc0AL=e-AwK1PtO&7J=jeiyOK3{#_&D^(gb|6St(+# z`pT|^k8s#;*9vOD0b_ns$)I;ydO)XfoRfl9Xw5~eG1ILV~TE>*3u>~lQE zpr88^Qx~geLx>{z#Kmks&$x0N2(Gq?xU=k>1e^G)H1o$B4$X%ukvUvpMmPb$?*`u~ zNeK12B1|F;YPsC&hMLtWfHdnLaasGbZqaC^6+-v=MQy{N2|nS=<(UhM*&I{mAxYWS zcHc3p6LU;A$vXxgr~$dkVXF^1St4(e_?)#KB_es3Ibl4t?SzzS-7d-A9d%?Jp-5ln zcTx{?T>%3-cqXGOgE5_r0aVOrDB6^gUd2p|(D%C}&HCZJOJlA9I~{YA;?D+0Nx5_| zxExTQx|VzqjB1);E;pPr{|R5jahbH6m@-}+(M+CI^ul=mc;KvIjuQp+{vY1nJDTnP z|NqvZT18u{wX3$EMeSKz5X26uHCjSq#-^=Vdym?C$4+9a+PjFoH?eB3^8V#BzQ1#R z*LALQuIr!c_(%SD$@BHhE3e4oK5oms3)vAgIa=`ujciPS%Q$bdAKp=}*LuHpJfp0?(R8!7EZLA!Mmu?V@#`aP zg{5-t(Gm|FnFQPG&rLu4hoB_hJ9hz}wfTbCNqT!09O5zIXqf3<$E*@u8GvGYcJZ8Y ziKZ7@F1?$DUwXRMztIXS*x`d)c?>^}Khy^vHk zG;q2>g)2r1R(fO+N_)#*`*uzrS(!eLc@SIh3SbUUyIpukhd~4!3Yo?*Upg}+vHe;d z=0Vy@s!H0X8qd2ZjZIaQ^6)|3MZgbotE$+1oR;y~VXsj<59n1>LA!N5$&@|v`I8&o zpAVjI5Fcq6aD8>dqkT&PJsdR7H(24+#;DQ}v+LZ;tHoP+$r#fa?E*SJO-Wrt34taw*vnf-8o z3uLXMcPi=??XUWRptU@-+q1~Nbq+2yTSJrdSGiR6k}br1Z{CO(hgJcA<>nsqGR`3g z<=$+Q+4i9s%vA}sg4mk{o_>5e*uN@hI1b8js(&0Et$z|iT$Vp|fBaFB&th|&j9#yh^k%PdT?4K8%_2e!x8GbjHM6dt?oz?vCo-!MI91pmcJ z>Wr{o4K>bAsxaa~AaB_F^($qyyR65EYn~LNgUH124upn0}%P zslK#2p-eKn9eg2mjSG7O9+Aqu1IMrdAK6vsBv4T_2D(aFRan|=QQuGqOWGbKf8;au z{B(m+=~4Q<7$4cNTC%xj)cRvq^K<%2q%vMEJ#bHONn1Ag0V>|4S^zmhl1pP_de?W! zB3{8$5_8v7ZXKDV`sjWlZDt;8DTNnD8=D_IwBHtq|Zpp&+ z=Xp?p8>Y4WniS?QlAT$?G2i5JniZveJb5HV4Sajb~Z@tRY0B z*74m}8(y_fJLmx0q@LBW`Sy$@s858-6QrRF!c}o-|4bRBX`Ox%SJRlUC^^=Wk!vRT z*u0KStp;~rc(X0cL0LF=j#`Aa@(&wz_;A-;900Y=1xbt-CQi&DkB_W&*9~mfT!4KR zgzaay6lw|o_U@xA(Nz3p)87#`P`km9AIH^>)fW-Y1XdY410`Y-sOJSO41s^FF*eq4 z%AA>=*To!t(Nzg)=Wxc*3CzyC_MLAu26v?S_(#eglGAnnFp;9V!>u;P-Yg&&n(krM z88@Li_@StFVkWk_S-D}wBo6T#6JR%ACcsl2@79m1fVKWh{%mG>$G4Fnk#9dSJ4!Gt zh#62hc80EacNOOb8oFp?_wTo$jp-n%k6m~t!46t8-c|tVk4EvX>|-~BT=p)LyGDq> zmiLisCA|f#m8<0CWZbI~Hll4uT7F)E&ar|m4bnO5=GOoN6Dlgld`SK1nAV0sQYxor zDA~>RVM_3y48Nu{Qy!;XFjbdcmG|n)v&4k<-?p?M$EXnJI7o>=QsALqdc?^+Ar}LV zeG~FRm-O3NUpo>!_Vo?kG+lB3iW|k(>lP^aE)|wQEr)#?FF6fT6@4*GyT_n{hju%} zu2X3DAte*lV(O$kqgiq>KWsgl2^Bm`8mYZ|o;XM6OELH$c?!zJcQLfbt`*ZJf?OLa6Lt>9s@Bss5n@%3eqMoYC=&pED*^s}+Qwy$1o+kU%+Img%{OlXrM z(A}<6i#t@Not`i5v6pw$wWZ(Ii@Qdldd6lX^r7MCquGe1Ex3TI-^BK>Q1AA$NdXos z-^p*x@f5KfzsCsRSjQw*r~)|T)%u+ipULWJ0ZD(qW1HreR+>&$#8LZSmPpHWV6Hh8*Jz0IaE(Mi`I$(1J6QB0z- z&d3aOnL*MlULw zSFC7d>cJso`#o@`o~u#SP{-aRA4AG4e?Oa1;NpCxL@7@YkCc}OcYrze8fJAMtKSFa zuz$7xA<*DwJ{!(9LY|tP_{`xUP@X2U83r`2ZVzepcmZxj<1!V2J|!oseLo>lK?q)o z>|K#&XIrPvlH6gN!qMw0%eRK!Y#i3>954UigiP1T3Ab#on@5*nvHf5x-VSjsH7nNr zUqLJUV3TA(Ta}Pff#FZg%tX-m)N0Gilf#n13PY%|gk%R@<2Jr|dcGYA7)r1-cu{;F zQHby_L+`4VG(~$5xr=2}+0&+~nQgU zs4y~IWcX$>z*Dol;XGkO*J;|`uAo+hkod!`-P z!OC&Ak;t`9)_WX>OYr!?rOb^ut@Gh#&w=Wo zw)E3JYL@aV{otl++QkRP3>OmIza|}6|0$Df@sJMz+UV(L1J(7J+wgm>T3KU~$(-}_j!XEPP$?AR5#G7!!9 zjv-ke`XtKKrxW1_Ufal{h&W4#%=3(aC{}@}IwONhrL-0e4>b{EPlbrmQixv81j}a3lF8=8OqOO{04#mnsac-2mI0h+L z=FKtZ*07iAoX@LIJz2{xTDPs-U2mcWK){&9F2yH3h3rlB!^LzcUabb0 z`}n@h>%z3g)FFlFfeH3(-f$HQiO0+B64%XC$NvyKO>t10JHKV3E?C6;fo`j%c?lPN zb50m?R<|ro1&v~|zJ1$#g@Hi|K1>Mrqt0B4HQL`#c5%_Mx);xh#z4wumtBMtYgai{ zS<51`-#0K|P%<{Rr-#FpK6qnnq2_S4sUBORWlIQG3`6BgY;7P37i+j5)ok5(NMi!} z6h2PQNS^!i*@tdmY&1qrt2w?{{w29%tl%|w{qMcz;%m+G3e^Zg-r`4YR)DTi`3^#h zASJio6foSig;Sn9jNI6-h(rzwoPurMvECyrMp?0vgLe1lA>=5!CkoA#YxM#Nd_3+0 z;&oe}oo$J^xZeHJ3IEnr>-nE`o@}`;U~dYqf77YYrN&rPSJ|sClFTv~MUL_8<UlzUw=4-e}*h78CZnpn>K3Ra8|7tuIa?nq;89C`Lj`nEWUTm(i zV>}_H0I{(F?dG=~4x zJ@h7{mi&-FU__rBweUk#g576+0^+#o!t%YvikgrH^(Mn5!^J4JK%>;pTpmB~KBE;Q zh6tbO=1|5y9Fkq_62oiHt=9U6$CKo;zO1L+E)Mu}fijxMlHQp-0PmtDO7d_-Vet8l7+8jhuh@x{&eVUYg*xVeSHt*5LUy-}dk#_1(Lz~Kk z{E5EUXeUWMjvx@fN&a>^^Qp|Mj~?Un#oeZ};HI^-l*3`cqrq#ruUj_Q2KG-GNuOiL zcc4Lohi=v+@~|gX`7b$=wZTDcbS%URB7f#}SmH?m9QwG&frC2%rMz7$_0TF6bnUff zUe^6+6c056ZU*Fbr#sL|+9Na3eui@M$K|J+Y+2CuGxIAO$4N1FxC?97gzuYN#q~#C`=AZD^PTBB2mf=~f z<@)b>w?#-1RtY6SLY=h(8|9vkjq-FF1X(y^D-K)VH7k_;;&`Kv560FqN&L#WkGsPpI^F8s0lZWOhkzGKQmViJu0&|&Qtm{*ESoAj*nvot8P+K zsvIi4k&x|lD*r$Bz@-#kH};2dR0M%>FxgS8n_lmbS9xg6P3E@)--Ewrn2XI|iSVl4 zVc;rYvoZE5b~8%cvD#W-Q;VS?74??=8Z=8RNzjWgC?BZ*_XVSq@p(PvlyBnw$l@Y*FKSiuuHh}`wZ+AlLryxd?t;Ag zP|Y*8ywXqgDf{~n|9D-^-K`agfq{lCYg22$emx_>6h zgi24{zWtwLM2NYt|MxY(DRqhsnEvOf|L2%F&O&gx^naha@PF1__y5OosR6GN0&-bT ziK15jeRkXXo?}OkuN8IXP7BwbpM@4IE97$l+Z4HgceFkIuPFZ^2wgnT&Hvw68@*?7 zhws4_EkCuVA+b69b*rK`1A6Mo#oc`kpP^162CAL771y46TeUGJdC+iyd3!CU=+wD1%uf5({+UE( ziZWOkETLxzzuG^QUK-uoPu1mI{q&Zz&a^q#Lh2Ylia`25+9#8s(3z;?w2Z*5GaolcCN`}lxvX)E;~o)-t@TWwMzlat>3KbVe8Y9rOhsAanS4c6rYoRl^M38k zUnA})wj>%44noYoX1>_+^f6}?{W^+4#T>u_*PX-&F))-!HT#E@BlD1H@y0IU!T_p% z?f1`MsH!1xBu-mY-iBT`7fWaa7uorgh><9)_5UTmLq}1e;6BV)D|DzO;hxgwh5h|< zdT#Ve4De|NIaMqf6}bhKhr2_ul*r>&r2;z3mFr8q9zqiGIBxW0p&0A|K+FW^TxdHB@Q zv>|Nx+0qP<#{0)>@t{557QQ{lL{u+FC#M8+r6?PL$4gh@xCmr9cVx2L#Ea&Rdvp{J z^kYs9I`lq-UK>wSmxSAUPumIm@cPi{KQ2MWCj27pxyUW_%ZkUyRUat7jyg?MfQhK4PY1jxG?OI+}FAvLqO{DzEK zULBp`q+aMI!cf%-l)3#;A?^j5LkMjP?|6N9D!ky~75$5ds_68Nj%S>Cb# zG#C{;(oYRY)_fB5tIv6J56NN%(Dnm{*Q0k^Sxl!Z1|ME+YVh-ab_>{B;%AfEWaMrZ z+A9Z{L+Kl{h46?a}^^F)*1iHIrM$m{3DsvX;)th{U{ z*6H0nmI&=gT!Z{N-}&#rMy>9!q7Qc>KP{!_=lK{Ag#+HyZ1I8wEgHoN3o8NuKW08; zbTo*1+E6cVniaEA$pCJMUs85|(lE47VbR%MbH+S2F5o}1syZkua}6v+fVQu7dD%YN zLvSj+u4*ALI2n>VRf#b-QLER9=#Gm0Xg`*aEY0L!j}9%gDS5rh6n zKvO;Iha2B#{BWFkQnZ@NHBFl5pu!-lKRycv{%})IwQ-uI`Bp_0M5!~F=P)FiF=kvm z0jxvQjO+KD2i&P)Nl zaLJCcuufNF0e4rW8V{R(5gD^vW@Q#gUQD zZ$fM5>SOVfxVKD5;B`%MAk4C(T2@o&!AqPbWPB#ykv$KE*jThjl9$sCrE=wr+y>r&pM6oP_6+c73_l?A^?lIMdv*B zngun`J+3cfDIZgHAA7_=hb7ooNsfzSH$CXd#lIDOo_LE0(b59j*UU07n|QJ)`w^KB zn|&zqQYvUQv?M}`81a8!iEhe%?~>kC$JyH$)?Gi{2Ou{(d*!)pkm)K8zX`A<6`1lq ztcO(g<4wbC_;&8#LwnWNTM>$6yLbz=B5M#EN2BGwK~94kYE$n-?A&eLRr@R6FYt~v zjX)&UHmhH2ZAUrPS&>E8^v6IF(F^|ZwyG%<>1IQXq04FtH$_9gzA<*}Z~}t<=_9f5 zWwpp#_wb;EvT%kZ28<2N*QeJ~sOwOF>UkwEQ zLw~*hXfY_Giy*E;VW>{q#AWk&D#;Vf?JFX!14%*;?DOu|ZXgfN4Re7ED362N+o>hV zn@@E>>a%XfLhGT#ZOdz~nuWcZIj3VXIeLVm)hTISou;AI9F&KwFCLBaT1V1jZFWqY z*Ok2%=?v1@^Aa3K4FdN}VDRZRK-k#pv{IpN7+;{Z zPAviPxCAZ_G5P&I(5|qvmIxg3c6~c{VG!SA2vj4UhDc2KH<@D)YdsUzF}0-W107#o z*P9-Q>&C4En#k_fi0%JTT*nh>wSpEY6Q4n_Ntd2U+O4jMcZL5Xrf9~F;S)ndAE+oY zX0G4cmH@e92GR4?PQ9h4)R~0Tjiq~UD!2uPMMwW`>Ek_HC98R-2_tnP&D_`^9Z~PnW8o8eIE< zMTaN63xqR^gw;V=%tLTlnMaDmMnLq`?OIPzAuIKwp6wAC9Mze}33aN0_R5V`M_khx8CY~%55 zdBBvFNw&?$k;UG?klsP)q=x985(n?VNdjdVFKfQMU(kw_ki--H`6+9De)V}4?|_rz zT!y|CDa0KeTKM2{bCKPRJ5AtzS&_VYTm(TP3Q^(oxRqz{$u@28t9jyh@{j1&D!qo7uG~qo< zv#N73f1%QEf4s>fTJCX`$n~4i#=k+*m|4p&MZ^klQN(i5)q#2W^53=EsQwUFF_`>a zN<1ZPV4HsQVv{-pwj`q==_8aPm8og=9xfDQmt3=8DDP0fVZ+T)3}EVNu(uru-)jZ1 z*2@Gl&UCh$n#>Z1S%X-Z<()?p);L<}HFVh3W2Bej_%R?;`&)f6TNp%yAX$aKsh^uA|EyjxFYMu_z2*M;jaBH^k3 zTK4A;mw5LaEeq^r_G6~xzqF9lY2YsZA-MxISLhfg3%!)dRYFWccBXVU=7EYBq%U;S75DfZY|hPSUHqjubH)!9`M5g0r|l` z)+@NbX?2Z_9`*7_#ALqHzAnwE)lHBt)Zh|rG$p;n_z)GdBCqq&K8emH+$MG0n-JCv zZkX|n5^A#dZ+%(PN%PA57dw*OH=A$&9HX0<^mKZq^bWbf^kmFHxnM%pr6`p`*VXke z^&-U{F;xO2vo5)e%cK|O{(iyHVBf2m%9-?LTq+_xJ3%cv$opsd=6Fhif5j>5Ckkr2 zAp563yA|uvjG%w_!A`zgTXBs@aIS+U*?s!esIDi9nR008f&A5t&pROUT{gmE<6q4x zEf>=Cvi8YvVU)_f3rrNu#d!ph)$jaduBzt8326?Re%dlR=U}X}=`VYH4CkrSX84)Z z1@ith?w={7TOgNYA8RrE{lq|NB{*8k_i7?8a-8ylzbL=Hw_L+}!tf~Xmvcr4^VAOA zS<1g|QPkXWop94dU1OvT%{KYd=^Ofk>l#xAHoLz5y!VJt`K~+?gc+v~q~LM z7{aX=7HXT4rKBWrU{mAr`37qXm-EBYT*8yG>gWmFB3Z_@I*H}Z^kWn7-)im~6df^z z{_$iDHgs_KcdQa5So&Ff#;Ba*4$?x3@n#-)1r*7>KeGn+5} zi$5%f2g0a$s^!CE)^YWD?>hc!^Z!y; zCA20)b~ls`jE))N3@GfBDAo4-HS7xG#lhG}&&4;46Gk$m);!p&I_^GaF%YAt&vbx_ zdyRh8t!N4@iZh{3v*HqLOx0GwaA?{{R7ZV11)Ga|Cu3YMb2rFH@+K1c6`{8Bl7CBe z%9SNqmD`b-|GEn!P)3(0zq3%?=vVwV>?Iy;Yw($+Q>$1V-3 z6d|Ddd39OJ1mUG$N^%f9V~!IBHfx)ii8~o6eAj21E!Et?ak;^cVvGFQj@Kw@TcW?_ zopAf}W|(YcGqH04mh(?8@^s8{n=X|2H<6v=d${~TM8a;z_Q z#iS$mp;x1|pp%}p&Cut*>$)U&LtZg}Q;S(XR0Z!GI-ucl7~Ftjh$^&YLAxs6dsb_Kz%V-Lp!PI4S_rGQ>N3fsu%7 zn_Kpnqw|(3L{ThnuoG*_q|vPbniVV?e&?ag<#hM&yG^-P+AcfC{8DP$0aYlAWdFjS z&0^!I@@!J?my9gZo)dWdA`GFlyQ48QQ--eqWcUcw4E13_n4F5M(T7+UJnR-gKl#w3k8-z73bNKiQ`#%H|EgX%14z8KnFZk7O zgQGT-_pxw^65!_(g)ag!uDP}=OxBsf*$8CU!oZGibPF)_$aTi&J0C9@2a5aNeO--T z^U~vvPKz%7uPD)#^ij;!Gb1rE@#%O*N2{z8ML!7xptm5a8uy5xV6<;vcmrGZ+%J;y zm)GU5SR(WBX%pf-mvpaI^`^On*BZ%3YtqX*kVwtkBDXuTwpFwWh4n>M8POYpmHk%G zZa4vZ6Bn^uAGKcsDgOP!GGh8XPVIW4k#_@YgpFREOaI77KQ>6!^L>T*yWH2KXk=MD z;a!J^gGLCv2ZT&bsOfQY16f{^5*As-jl(fIcJ3)Ss(LTW=Bo}7NPOeIPa*55?xw>q zKF=`fNo~eg`g_+V(Ol90nE_!#Ql{f{`9m9*H{@~A3!^pF-_Wigva9d<$)l4<16wc9 zY&NA52AJ+1^@{$Y5=|qKIQl#(-XS}lHS-1Ht~HrrB+jCtG4ATM5~?Ii-Sy{ebiVa; zQX#X*=qt3D4e$||d?hTxkDQEBliahD2kk@T-f;AVW}c8Rt~A+4AWUG*u0sbdI%7aj zmBtw#$dHpg{!P|V`NM!YrAL&!Pu-|%eQ@IRLpRH1V{VBA|GL$0WiR&2yYo)anGGVn zwO+M^eNkeKBZ<>K7HJ#6K4G`z`*!{--(m%v&ED>3j-*sGhBZnRKu`sU1-)to70srC zwfk?9XaU^ZU~lug*fjTgQA#3)ZiHR=yaW72;(CKQOHA}tZkez$e1iXcyxlBav@@rT zY|^$;p)rWw30De;Z0a#GBFzUtuunq#Dxmt*tmXRQ z&%qu+U1RRkFJncQQp#|2dexo1fe~+Np?skEUAeWJ%U@SvWW_*YH7}x66+70ulw`+o zvfAwLQsF6eZV&A{V!skk`&0)OzI+z#7Vh^Xd4j~vz-`@M6GG>@;~P2qTx>M|SBSJK zxXR*PASuhi0?>RY4_R4!3;~IXSqponI(WMy)-W^Ph(8O^tK17OEWpo z=sqPzK*jCQCu;sq49118(KDk<;85V3Wjs*mTN5^YVFZudThM#Em#bVdPd)t!7PLQW z&!q!4`>-df>-xwnKK<((5&JArMu}qij?u5+_^f~7?(s6ge^@~OFCS3V*n`Fg<0Cps zX^_wv^gje@-8cNQzQy0NYn$lAUvl@8DLuUx*)#>2ld>#2Se*`ChBS zgpdY8QkoSV^ok&Zz5HKN+~0kCDXJHqD-rs2=uVN3DiljSyV?#`O2kg$!vw7NlAc&g zAqb!D9)}X&kqvh}{UhXFa}?(mOekEs<|;&3`YlWn#>Ikr9f}XTuyJm4Da%}W!>3kG zO{}8+a=E|l5}8!BRo$Am2G;7$&GIRK`L+-Jsl5vuA{u~-`zCwI7Qe=MInC8SOFq;U z=<}^kXnPMmMOrr@P9&p%YEn`vMXuJur50{Tl}BWxMTx&1>Y0qbP%)u%_}o<8Y|>vr zx(M)87ZT>$Ggbs~r?6#X_&K|VkEZnULhd!uhy60Dy3Y9)^?LK9dqR7KCjZV0z-+{b z-?cHO9O@&*@-2EXueEVTkNrPdu3E!gA-5XYlMCf{V`D?nDuo;>RofhF&>}v7fpX|g z5U1qngy#1h(;j@j+|9o1DzMSFutvG+$W8jKm&pfJv~*Vg5&Z>@qhHFuq}Ve~Po?ES zw$jHpyw5O7Ux(JRTuWC0wJ8<(iU49&W_foON2*R6;d95m-?&q-vB$aKK%l@pzc$~q z_n}wrXdF<(Hl?P0qNafyc0LvUekr1jyvs!PD$r?>Jd&ex2+0hyHqyf+!Ey=42bb?#MDDO9JZCpJpm}XBL*w(d^`62dpdf zBZAQ0aw7+osB)*?r=kUp;iCKkYqgX=Wb|sVi>VRdo}9zYEMF1?&B=MnXCLnhRbYHp zd|t7_5`lnfd9WaS-nnk3@Rez~-S^l#hx)(zKwct_&pJGYP5 z#9pcO0URgEfFTu=#4LIN#Bkh;+HtLYQNQB)0baKI2O_ z1L-6(lSj5;;((FNcfV3R2)Xf`1ePApw*jpTUG??GM1Y2R$Dv~w|| zM@}^fDLP8abH;VNEP@OQmN065!q>jO=Uh#1zgH%MIt*c#B)J!H6P`Hs0E4zrTuyAB z6AWW_t13pfgaM zr*~%cZBMV}KU9n5o`lW90En)w^uFY(KjfaN4C#oF{wo9arP7WIp2n(MtyPEyd32B z&fZa;v<#Dd2&Ch-u+bzF-X##kJH${s>|Du;-)?Ay2+ShqbhqV;?$1=K42sIRDc~jq zsC+*$ia??yy+nv7UCs3gGOsJB&!Z@Mb}opHo`9jbunbG(2$9G*58g$1V$nP~n-X!U zbpfT3nQ_mxWa%zvJ<*BZPE(KKbWdy1={j&`6$pq}7r%RG-buBH9H`aIrb^HiX!8Yr zLR&X}HBuT=x=%8Lg8wi-SvhC7lzkDKS*9z(Np>~n94w{;4ybGA(pTdNn;*`t%^Fjo zv+sGoR{gD0VPYevsXM-3*jQY5JcGI2Mk4;|YTnG_x-<(kCR`esZW@1w)=Q$B{dV0wbKE?&PZl(v`zQ$*&r{xNE6Gq`+9=u?rrXl7gA z(a#W^4}`yUjmJ~onQK`siBJFwy;+6&Kv-3x3};p6 zs5B>TPe|%RJp^&z=sBsaMffh~2dd$y12SG8m7jAtwgy_pV`Xs0r(>Ap8ezJJil{kd z#{ja2vJwfA`$9vI#Iq*PRIU7a%Uspyzx8ln>`I!)I297Qpehv1j{fvB9Ug61r{F2* zztrHDGKtSCc0b=IguK;Hf8N*_yP={B&}(LRW%tt+_*9gQW0iCNZZ@xQ@nrv;tww$9 z?vKY0<3pdKC{}Q1@GNFckvuyGhs3Db`8NB`&UN+O6vs``URKM4wOZk0f`S0 z`5uYsH%Omi*BjDT)5S5kE!QsorEyDRl-AUhO+{m&sj`*Fq zi5ni+hHnPRY^ho;n{*CiwbM6h2-fPX`Kk0ix;gC>-&}Zprv=%7F_IxWCm+m%zi!8; zLr12Z#Lqm=i4vFhdI!CN$O6{}-4r9j)^7-lvM#k({&Gl!))8+u0sKtYkbfcvn?*<0 zlPq_p&cSpOo~~4D4%_PAXjKkoLF=B1p69%I_8kBqE>AH!w_(xqPJ&15(xGM{m2yQL ze!LTTHR+OspVA!t?d$Cyio&`QGCi6xgSGlyJ(z_a^T35z-ZBVf)1gS%*WK=HD>&MG zTNCzrBr%+m=ONa*h^)@*(U_mut%SVpi(*+cG6QQ@BBwd-R%?#Wyw+ED0u_$M^2Ra9 zWI{~(ls}CD@syLnJ~s`>`(VWvy>dd#7 z4lM!s3(sLrx8-ew8bGByU@Sv>R+jSD;O2;6@smkxwuz+dU>n;8B;T2 zIDV-6+$_6UhIH`cDxqk`@0!dxj>XgxeO$^XDtDAnl$PdShYi8H?!;(e_1T#q!JAsm z%RjvwBz5%0%mDC5$vRds;+`YUEOR%P33YDCm^%YJZg<_dTV_v8w{JA4ryaijRZ7zz zi$Y4jw6h2M)iI1ieoD%g2Y<%8nKJj*$BoW8X^dFaRV|)(Fvnj4+x8?%DHf?z=Tn2G zC38wk_=+2GpVtIeI&7bwr>Y_83@)h`-G z9My`sWPD7--qtE5mCQGrjHFs6_)CM_0iSy`^9_+eD8G~73o%uNxru!SiTtveXMwv7 zefM`F#TN|wz+V(aOV$ewp=!+2nE4D+MCIWo9ZO}t5ml~xLvM@O#sWKM(t1Oo-h5o0 zC!sa%z}tzLm|~pNB>5InE$_e!#h(0(70{PygD+iNuUSdCN`Fkex4KQLa~H28{;7GQ z7_)K5IAKc&NR=YU*OT1O!&KRYkn-x_nhglT-c6JNO_X#51dliU|Vd@!%Iv6E%* z|J;~0^*+im$<9D@$V*KbuaHWKsk-OYpNNQdUg#|4o2Mh33^-~yYKz775A0hjAIQ-< zyuCQCMdF=b1DGLnAOHTFJbC<`be2=b$2EMFkERcQr7GsesTotCb_5GqUr-l}JH%>| zCU(i5r0IE{yZBxf6)doE7EfyykE$*RxINJ`u?Vy&f5(_`Db^6HDd0ajl6urks5d-l z&d62o8??C`8ii9gMhR`qjeM;4{)-@?X3Z1#lk8jGLRQHDO%mhhU{4?u-o@1GFxHIG znbvP58@5}>I=&#vX)C1&*EOAer-KLE=&XPm%%DswiujhQrjJf@wj#Rrx+*Ply!;hQ z9v2(4EsG75p`IO8E9t(e%e~Ia+KH~Lk9^kXsT?ZBK^5Y;snxPmU=p+`fa~k0Fn+>n z`G;mfifSmobs{o}AtTWM-&dsnjz!hv=eAbfbjFS=*JcXG3EAP6(7Gqkc<=e{*sFO6 zj)aFuic?~z;B=Bms^tgZ=lGS9oyk|_5AM}M_eYN!@hZlXcS0J}y-)=eP z7eA(_^!MW)c^uCmJK#vYQgojD)gpJ!m@NEXke6}Y!-|AaE%5q_@HBU;9^l&9QN^YM z$9DI4XAg9nYWWC=*Sh-)G<30K(#=4SXafcwIXn4%2zDGMOf3sa9Iva!BpaQ z=i)=cxzq>hHQVi(6}d^@iY!IqWAb&lclp}h=`I_o^?bzG3qjb?IO96iMa5>9{1&Ia zk5CbGmM-2@={ueSqy5@Zlo}o$94V2s{>0RZ@5sZ8Ycp4K46I32>8b1VMPXT$v0=MK zZJ)mDhsHgGsv&QGBd*b^xGIiY2j)o}kF!r&fm(+qIPx{#eE z1nL&DorP$5sBNFCXm)9o0LJu_W;tKx6Mc@tv#0eXjKg?bGS*UFd8pSvPU6M{8;e;5 zE9h7KOKg#`(lhkAs0hxLT;+>=Y@^6A0hmili{szsOGAzqcT(U{KBVwxvs4scx<5&4#cxOLuKA#fr;IH}^65=$FpOC1%Zd=Rb|v zlqjxp`MGeF&IGpkv3ul*H%+XuJH;mzAo+(zBS+?t8KOVS;m4pihc+7%WB!hEZTCG% zO%NxO*|sinfy`ar49~vz>HxX8v}C#47}l_0gw@E@4dd4LNhL3~TFR;o!vz4XF}iRt z6^k77d`fjfo2fEt@cfM)UthI&w~DPyE%6nS&B2(Jfl+fC-$s-h3Z2W8))HtoJ{$aH{6z2sa7 z7g<&6Y^I5&fn%eFj4^?COhr%|%2FYhMtWN3H5;Xy+{!5pHd8DPQRd+PAi?S>qJuKAE=Wuv4)(4a zq~A>DtFk03lp^%Yg=()o`IzI|L+!Q`2RwR;YPwW3M|*bG>4jF^Om@%vp-p4@zT*X* zx~r1xBA}jmN?ukLr3dRKU8j1Hr7mCA?YGohrbDXZFHiLfS~y1kN-gAGyx91npnS}7 z4i(NT_@Cce_|Ncj$h*L?A{1eOIFH+GLZwusi(D+)W58i%nnh?uX#eyP#WS))Y4Ojj zIfhWVQC8MX<^d*p`V6rZTeA5u-8${|zE2XG$Dag;M5;zslJhD$}w;xn_vc`TF^(ym|9 z#nU(bJPt-0q#&KLDezR3!$Nd=e_5+Gi_HQDQzf4oZ(z@As3Og$5}_%Ev(=J6WM!o% z@l(#5V7H&_IU};uJ}`KIw8yjfLI7%yhw0H|y{Ec9oR5fJV=x)XJCh;%Vq$qUylOrQ zJ#vl#Vy#{GEI%;5pxV1pEy_27)bzH_GhyC>je~aci}URM{Cf1W+rUZ*{B^_M{N!67;1?T+X+{A+s!= zqxRuGPwe^v^9&{XjFWt|SkjMDgENN4&rkK&GZRSPJrMCfcx3($ft~hYj!E1Qp5)xg zUQ8oZTcI-+-ZU*423G6$&lUUl+p{D1EoxwjFKpGW9p*jc1ij%(Ch@;Vt#7p z$uF&zJR^7gD`-`#ZrI0_SFmVboI-&eROR(cT}=sg_DGgj!MmiU`O)rPnVM@R*-gV= zeAQ>3E6%BOuE_8N`;dAKXKH~StaKk)_7DD4T671SF{YIg2c8q+9Tcz%5M;oUb3Y0Q zL&6uq_(GMP@^1+J=yf@VRO+)v!nHiscXjk#GJ2>d>_w0;6&%BBj?|`E6D#5)Ce83M ze-t-e#M`xeKxXRL1x1)zdv`G*sNp&^v1**VNV-JHQk^; z#3j7_!tTt&tu$CMqB=egCVn*Zbkq}@3&u#S848aZx{StuxBQ1dEJcYvIeQTs*#enwVErcu{|KwW&hJ2oUVa&?Kni`sxZ`t z3bi!P@nzAvj*h3jH3y_W29)}4xV&8wVQH0QDIG*(1FBnoByUZ_%~3=wCDiv*$6GQs zNlMVWW&r;7+PTN^_hZF-Fh2jEc@bU6fzIIGH0r{+AD=#z`OrCLR}d>UbNEM+q0Y!k zVo^9EG2U8dCo_${zen#Bh{$HObY?0l=C~ULl#(V4h}dQLXQ}2-VZ55LYX($T@4;WJ z9G&9-E||evw{~-191f4&Byh#CnQ$}C(XFFJgI-~Ate0o$LgrZ6 zfZyKL#nHMQKYBvn{oE5S!Xfh#h-Wt=dJ<+Is}CV^dqq zBKD3g_8z4tzx%#l|8vfx^M7!joY(6-BbNtqh2*-v-_Pg$)?Ma#=qPCzbn^B$jL-X ztyW4e@ryrG$hLF<`WRy{`_o3~RfGYJwSGK{uALI(;iu?Gl<=(y?JJo)pTqqDF^g`}-)W+P4bC}YBU802@KOBGxz_$XnzIxqrk>-huG`g7K12ZJmHVzsE zSG(nhLFg~gyIGYV_XD7UE-0qWbbB4qDW>!JbQK$X(L8AB>O=s5UbXb161{wI#|MBn zrPzl;P@Ed`QxDaypK$ouwM1kd5H1*5oMywdE3GD&`QlO41$_OPxVUV)1$zez&hm9a z>rXgD|K<^&4vqDok3AY@pyVj$A$Ypvkmy|7AnM#XF$Wrf`+eFi(eF1OJJeI$uYH*? z2X+m4B9%hd@P5&3Pz^xvufOP;-!4`e?=CIYUs7u0+thCpX*IT$0z7>L)eD~z8;f0l}O5#t;k%!)g2Mr&aH z!Mw!>(0tD&8Y`h~yqJ~3xQ}1WxucZ=0>TZ`ewc1j7(}&094DUqTdRPmo_#7CTP&tmWbz?EJ>?0Z~d6SInchblV5)=iKxvhgqJ818)m;r zjwY0%-c^MGDo(~U+hDQCN zqL3uOLb9`~`h-Jrj7mv=j59c_KY!P5UtZ-}fC+5YI*|)ePJY{Iu(Ok&6m3tyn^Prx z7oH}9D;3Wcn<9*=Ij(Fp1kMSjrJrl_oq;E2X2Yw|hsTK+t!fx)N7S$T+n@yQQy6Io zCfizbQY8uem*6lm#Xaz}B>NA~lV%m9eTR0t!eDqI%dJ>b9 zN*YTxsut^N`6i*}e_Gat+tsn>XWt!EHm11(0)a-9@W0N+R(Rxvt z76JQ<7j|s^OxlO>=1hnf_MMrR>Vom}ZyhNe6!wofiP>O46LTN)j^s{%iWNdD;QKfV7QNYbaRDrz#d?Ej@fCN$tbG^0H*K8eY^u z=qiSI4K@kn3Kw{Z(yQ$wZ`CZ7D&86nOZ$47P%qn9 z6LCav1f&rA?S9G4cR#&e1*4*;tYeM_mX%#9N`rC0%3s7=`81fdC63>7qo7)3A#QGo zT|dKA=_%iO>t~N4=B-HJ5Pn(;u)2xl@~Yb7K3kY18a%%rmLw= z(E_>p$dRmul-QsYo;_5Y>4eIx;yGvb2ihBgGfkzU*@`HKyuz(^m+oXL9@QHSN%lBvjbHJ$A&+Tt5z>}`_jqT{KJ89H!}V0(+zA?C zpJYLRah1EDq`H#h`C>Xbx0S*Hbiq1!!yY$nv)H4q&QAU-;49B2)e#kDO|5P03}l|i z(Mn$WxckA`6Xy&vL@AeK0BC?I?>MTwk?RIxl=k+C%g@%%5ZXz?FBg}c7A!U} zUQ}#uL0~RhDCZYs03uB`YWU4DqoBV7oRV)JOS&)LOLt7NqLU63RHU`t%iH7cl|?m2 z`?Np^tJxgFQMq4crKvu2@b|iL0GL&`vaqO(StK=@<;rCO3y`Gb;W|0ROk#f9y&%~V zWAu2s$wS9#jIB26O92QN(nl^;Tr7>Lc4loi2Ol z#90hqx6gm_j`!SBZOvcZtQ63KFnCVd&AwsEkGlX9_S~tu(}?dCy>h6UyCSa-l0sWY z=S-@vYlaF_v0tdh)Ky-x?IBu47WCL9kAK?r#x6OoCm1?fFTF@f>acN$WaE#o#Mv0A zaje1fX7N}~yySL`wdA*mY>NQh>r#>q%|sh3v6>Lo`OG4YCm zgU}bT%WEo5+G^}$c-($!w%nTx^C{zr;)IDcVWh&(RZKO1EYUfp{iA;8ju+pVZ@)hzU4GSL0&OMV+5_3#;E9O6>eSYTl!On-X`_ zBDrU3ua0vsw`BVE(n_m_?C#rW+V!CbPa6YW2&bTAWvYkz6u5Ky^#Z=C-~^Po-FWyF zjsfw!Yn>Nx42T<8zK~QHNes^B*7_*u6!xwrux#--|J#WceQZu~CK#v%2zo?6Ky6XL ze`h~Oj0uwxRFxIW?G8D!daaA0u646WZC$4iuozI| zCFriIFnTZk>OcJ>BdaLR%Ag3zaqC3$Y<)=Adr*7K*n zD#{=9@JGSZ9w@LCG2l{8DEDF3^5AOhu?P-Xdxi5^r2=5X#?|yVpEp`H?~1h!cq6+h zXY5K6V{v@E|Ckm6zDlr`iaHx?*#vcQ1beb=-RBTZkfwRw-b)8K0J()_j-AC(C)Etr zYN%*G|UWp+8ok9QN;_bWrRYcy8lVAtGMSAZ#jv& zan$ex4H;Im9&u=Xnl`VKuu_ZCDC2*f-4dz8Is0}nyRJ7~;$nPr#Ml7gWg$LN;lzLc zz<<3koaQE|N8)B*wLiQg(7-&cDe%0nd(QCKZqC^KgGjFu6X{``MubU6nw$I_^>mJY zBJ&=j{W?o6@t3_12O)@zgwaS*1CMG{nFFv4D2HsSI4YkflY>D40N&~+F8=ygvbqPrBs4N38fCux)@c{jL-LTU@bZ;tyC?ib9V8go|y2ryOhq z^$73j%4{+*l9^)}FBy&jKzHuh0~Kpy!j{cSr<4B@D6GgSOv$cu)}9OxPZLea&GG~{ zYN0xjCRbvg(OuIUEyFp>DJjO^7(4q^6m_U_j(<#DKYc;mwa1wlvWx_!C*Dz)ksq8T zl$~VSDI#V@6pYIln_fm=Grvi z3c-&A)&ZJJ5}!{l1QoD8DrmBE;guc?SSHif%8`vBKkUzAH2vkEj#ZS$dq#+45aDkC zXtB@jNvC~LuP==)ZT=BSWE$X~i$`mQf=*cK$}Y3B!E2b0N`Zxb-Rt4gI%T=ELS+$= z<380XbzoWQpAU0gg_Z+M4sD>08G-KbZ#t6WesbTagnN4feROOq^MmR~=|z|V^FQBA zXHXjZ(B|L0c5)q$)3o`W>a3&!id9#MW z!wG|U^5v8f51a`mDa@@?=;2fJh+~s`oq|0;#EkxVRaKRUedW+k{;G&{Tt4{G`yL z!E|g4$+|d7_uw~vI@N^7j~4d-5_~JdK*4@iuOqomks4pR-#C7jOmV$kviuv8UTj$- z%F$(@&e|&V`wcxl@nde`p0T$OaWQ~5ymm06*e5hMe3`SGMmIk9-+cR5&>gCv^q;S2 zl+^)i-N$g?;kLKpBGzhtM*RtT+^%R0SVkbi z<8nf|@aFnX#o2v)={GrDH~(w?pV`Y!*q`Ro(|2(qzUE3-69e=vB}4brF6vIg+`0V* zhKe;0FIy&Wl>h(1{~IR(_Mb_1)4$IxsDsD5%kE`8j^l-5m#U0hCL+GJB}V7-h#fq! z|2pvhFziOk0gL_}^QQW*K((TlamG#1J8aF>m3qpLQ0*D*V1^Km%)-K2M1fhcVXJf3x3$3MNO4vfZ0&(Ed_qxt$}&h?S~rZjaHtZst9%^)$(eP zF+^oJnUlzWFmo-3(elt5xK_oCVT`WhaS2(*hLvbq6M$?zbbRM_yYs7;m163JEVeW8 zO|?y7Zq+8B!U?gQtIjc8{2*|p*L~HF=alr97|-2MGKJVW3L^t#tzrUpP%#=T_>Ji} z!sb;BmoW|DZ@1kdQ2HvN#w6{A!uRl?YJ)`k6p?YbSY@LpcMPc* z*Dl9C2NR4(k@7kQ;kt)#uXAnL9WU3#KJ_2=*#?t;B3#m1@B-d!ps1G#gjaMbtOMTn zwWKfLjW|w`*}Q$&5R@`HTZwm@#1&xz+Pmv-h`Y5tI$gd@aO#gb^MGO-iWW=xJ~|rv zz9_J%d0yW(#+&E;q^?9VBZ0kL9H+V-{DdcKEWD%adeC0sup+GLbH(!k2%`|AuXOJN zXkM3AH8TNmBl07uDwdm_s!~j=HZ^gTlh}?xTzvs@r9Gzi9Kp#cbjAx>Pl`i4wYRcZ z8yxwRa{$ghxY7bRPz_-eG`QJgofNPT9gnnhlVgM3n|IQqrK0%D>$4}z;bKbqQTP*H z`Xtv?gd!nT#=YXjH%hI9Xn1=7v^wfIbIFAY|{H@7p@hBH)TXds~;4nZWgp@7Ei z2_9C{?ijCe@OIB)s>6W`E-TG3mUS)Ua=lWj>gAI;sa;vz1Nl{dv<>151u5VIKjUpP z%N**EWZT1Y(?~=d_!YUS+t&+X)ksW|2YrCiBtxZWz7SolL03;gW=BH(FC1&8pwPFL zIS`(gWD8ZWCu-BS2Z7oM3G}h)GMe@^n~AtYg}91Ql_)!Lt23c?^RK{n(MC1~$QmAH zOg~E1m+fD7BXMUsFE$lL!mmY{IEv^@OD77E%QnzXp{*xDIg7c{cRhKSc`}82?-i-P z{V|&XKD8t6>ZxnH_45T@$9U({bLi={#Q5XKhre!y>jUt8um>;ih-sFJS+Ns%Hv(u#saQ{Q79=E-P=8~?x ze^@vBRbDy%PZAY=HKCVKa&C?;*qjJ6e5~IMl>X62Z>`Fq3_d3$38(|6;5_hc0@6%v z(Q|iA!SUA`T2W4MaVVmaZ_arJxp{^}_A(xvEL|XA=2i)$0(}04<(F}OV@)+lkZo5~ z>1BR>$VSxjoFpYyu9|>s;Znvba1Vd=YO}=jCWi&vt-JS>aK>(^(JrjUapWi>9-hLu^bP zdumzvC9aVVd+(uZsq?JB4Snk!=7+-xp8M{3RVN%|qSqSn_g=AG#FsGS4(2q}S~Luo zsE-Z}CLl?KL>t$?P%Z00>|GLb9z?p>XuWlDCzrcF!{qDSk<{W8l#fhO``9r`@5@jD zB{-URm|}|h`O1|7z#$P_d(pb2sZt`>vSRe{czo#RAw1U=kT#<~1LOOTj$v`VfeMo{ z?)(;+V1@PoCwI2#3*}_VCDO$T3!{X0m?g%vM!#oGm)ML=SdE~jKpnLB(tLqeRdiU| z^PY;|*f~DQ!`|`AaTP~6C_BGRv`bAze>Fkv{-zgp=qn$jN&$2e?l?as5$fPa zA6gIY8%w!2^aZOYHge@F24TvShRGZ(1M$W{kIK@Eh}Fzyl+H4$R8)0qPQjsn*5pM5 zEgM^qK^1TClg$d)z4IrNCC?<-_tz8m^_f<#$Lg#htw_FuomYA~cJ)1=UW*y;ptOp- z!KIC&`?Zz4O#);}pYwI=^>;dzl833E;3@!ACF1go_na|fgM$pB(rOGB^6UDse+dMD zTYAEwwTh_#XwC)iGdo3sEdni&&DKcXEN^WP%csR$^Sd6M9&G;-u3+)Pl`}CM8&YBlv6l)?I85Nsp? zze1OqY~dgYQiCj}ITC3s46q-}0?shEtm9JU6-+NYfu3jE5hjtBvJy=JFyl^MKGS4&#t4Y^az0cAb=A*76yT%HZ(J&ia{^Hg`MT z=BW?I){)k-5NKb~CH+{ySB=f9B<(Z7!)9y~;T>^6=@1@nQIh!73t6)tM_h}c)!aG2`&%Z+~MWaYN#A#e$G)xyj&$0wQ)^;mV~g_ zn`Vz|j2pKJS9BVBu$3I>*!MEeIHrC*w=Kr}yI)n9Qh(eu&@F#B>DNtst@Cr4k}kgC zgES|K)1Ie#;&O-#o0rxLq0iotpXs{%Q`yH)Obqmwe$J-9I0WM7kqTVuc#QE`%a6RWt~E3vjvz+|qbs$SJ4AJ%XE z3s&;vA}YPOUSvfJMZ=G88uw$>5I79^ix8suv4L!?K%rFNqWAQyTRqZOFq$S~T1EOg^j z6J&qUWh+GA(AxE}$gCaR4Zh=k9M%<;Rg{2FWZj>WAI{r|x~{F2Sa`DEooqWUj)Rwj5HX{%TQsCRXGb$`*FcJp}j@iz!R)8(JQ=5ZBvpvZ?|q2L#5lp_iQoh zw>y13wk_+CPn`z9-q0!-Hamlm?CG{W9IYu0X#6cMzx)RV% zKp#7KIxwh*^h z6pyLsF>PzL4|>=iu|F04hG^3NU1_}%Ga|0D=j1AOk?@y*%C~ngpK8lSE%}($9OAXj zQ&|I`QFy~Hy5y(`Zo9?4?!E|dx9M@y3O@1u;qRg`Yt_xfEWg+5M!8O6)x7n^_O5E{pis-;+TFAj*{U*2o^FO+VS;jbF1#Z z_%#pkg!sizsLwY;ddJtGuk`}`=i#d+9)ba}-v$*|jYoQo93KfSecx9s6H<@Oj@c*` zp0j4YDBPvkbJb0}Z{*=UJ(`gE9^Lwo(MJ$n=ayF0oUdKea9nfD1=}BsZcVaUs$49Z z`EIOpErF($m+QFd8aEM+<7d;-lyiIVhdPH4Wm(&z1QVk z{O8zSOP|{%l_KITsJ7f`Wt-Xe0z(F=q>OIaz08PW-jIJ5JbZ5#|LqK$&>iGo zO39^R*09hyGM0^LVdCFXOJSkC5d=NtG9J;A3b{@Cph)K8p?RDSFOxytd<-WuY|*zD zG4671KbM4AePyBCJtl0Wzh+TCd-^5{VkRAczviWa%a0r#jC|^?u6M7aGFnWYLT;^3=aD?#;FDDZ}YwFs4FzzK%O5;|zn=vK;tmG)sb96G`L*-l4F7mNQ ztbQCzjv0;uWwu~CJjh<0ab#;B6d^YpF#sHJQXk_{s1ici-4f&ks(xztYIY0PV8s8I zz>G+HF#yDzn2Qu+HENbA8F`(b#giiI15!Yz3|11^*2f)boJw7ow!%h?jWx;BZq zFJb#zDxvhrR3gX$SmnNDuY0q244)M*vx5$OOg_~)c;}2My{|yk)a@$Yt z{}kIT4?`Kxe_M2c$xq>cbkSp&P(XZld`%JwHywR zmS+Uv@yh!4^ubg4C9lAdw41svwuY~0+g6#Ygz=iyYJ7SZC;KM)-S&BE1)f#`Tz)Dw z(yY$<`J3|OvczuKVjx_xme=n)p(8Yi>8b%zu$xl8B`CtFU^Jwt$E}URmL!IXyMo+! zPRCfAI6l63EdOlfv0K=&oSdE6p~q@b1BI>-=ghi+6*q4i+i;pft8^ikV6iaCiA@bo zFPIxT{gq9)&z*s#&LrN}M-C8H`Qf)GWHETpbqZOFzW2GJPaS$Df6k?ITWqVzD9uIY zpW3-)R^QXst}8ztGfa>z%D)aB{`3~p9Ac|PX<%1n4XA~E+~FGBPOeWY&9!~!{A9cf z@zd?6pL#=wA;()0lqnp}5lBiTeQd)Zo{%8Et!D!UVLXwv zf8xwbj1UE4LF4xWs+&HIMrXqGRWqMg_f64oUmh~1pu_>c4KoZfHjt)OGIZ%9~t)Ldr!sJ8by zO!w_>B}UZLLVvwL4z$kNDAHxf6vPQLij9t09>xy^7U(=Cw!V?F!IY+SBs+~xeah_dz~6PBx;(0JfoX|9Gi`{bxj*%uGA1bbEGSc zpJFeI;;l*G5eLNP4?&(hr~VTHe>{T^6+dt6iuzLr?TqN1^BB9!QuQdF62twMfTd?P zNk5U8d&Uo>U~eUjK8Umba87iye{u@47HwO>uy}Ygkk0*B>B6jLp;=yS>2ck5=^E6a z^-dB~0~b7XK^b7B^%1}C%--uFm~?R~6sOS^EU^RRmdGe-pfyC;PbZRkM+}>NR3voT zmVZ|(4g;TlE4+j_3-_3N1~BssL>EsjY$4i`4eY3ArDE~>bNux-RqV^U*SXJ?MM%3n z2)P_{pTulZ{eJ1@ep!;}Gp#=zPr>c9Gnc}DWnMrb{PK+H*a=S6SbF#|Tk>^pN$@DO z?A+jfwV+Lu(~|0{xUY)DdJUo+^6*d5@i72P(E*|ykoh*hHu-*$U}s0dsw5G|rthD_ zMCA}IxZpGY!tZ}D<3}G z3=l4#rXkvi66!=pHyczGRr6BDO}o_5!BaP5(uHyb+P5ZBd`=pV!W=sb3h}3q{bb_R zmC0X%(ur#(KOEkbs&;B@c9_a-t2J|yir~u%6uu`e#4aQTTwK5v_YT)MHKcTS7JS>l_?Jst53fV%pu+-*RZ1bZc+iQ4x)dkQqC`{+dKJ3vR{j7jju#m8M8 zcX1LsaEbWbSobcCaD>52mKr`ay=^EHLR?K=W z@=p4ph^ zAVn=dc73cxtQ)Y^u!ZDhaC<5AV^mD0g7R}kL(

_wo`#RW{0J4LD@YHO6hcDYpH z{DCd6u-J`AyRbA@!*~ZNLiNjT{G3mHUIx49PuFa=KqrooU;g@7bjSm%;zA-<^7tRu2J%8UaskRVF$ud|h>@zM);@J*Q5srL$hY$=#$diceG1+uyAA{#m|8eh=hxPhMcd$xO$VL_E(@C3<6=-K_$ znu=d|D9iDaBa$d~JmrgY z;~Bgp39*`?67LsMYkiNCSPv}_ z=&Glv%KMJFxyTlzH&xX6OAuSiX{xs^-AI;)sF~v}aG8yQ;0?Kyl1gE!20HHQvo;PX z-US^?uLM|dW49s88OMLPoGiAK{t|?nRejlx-b>u4R|9kImZ&OnPyN7SRya)4&ppt8 z34*k-5QAkIa-esCV>FEz9C@d{?ikF^f~GK&hlThL0j9&Waa1gYuZ{$T^5Q*GztqHI zh0-~4NYGMV1XSdDfn0`lR{3Hb?lM0`o1FE@PdHG5bD6cGnFGz~0s$V!6R2(KhdqPo z%@0OYZvvbTh<23%9FMVxv9sZ%2A;mJ_ANq6;=Du!%oybv>sD7vYaTomK2F-)Grsy) z1PF;}sDw4L)RG*5CO>cNPcz9seQ;K=CkT}-totlM!`#&o<-05*evGn+f}!Xol}tJP z**p-Jxu0hyZ``PGciXS4OnjoUr`7SJHp68u;d*?sDmF~X)>yGn@aTClrHjwpw_umY zW8;q(MxODc5IxnC1p2UMyygPT&2sK&UZ5x#KThZR_J?}U zQT(3H@$)3voUvP!H2u>{8NF#2i@|Y?muk1xi|L1erCM9vy9#mw0gHWHvvB{j zr1f`PuxCYD8m%j0KGQf>Zt3Kht0y++Wc`Y^P>R7#~DoA=6fxFK_Pc!-@0k-g0M z8~zIo#H?I2eCqEuleUlXC_)N8I>BVH!X zLG=5GOHDsYINuGbanFR8COIKsxS)2%x`|>F2GWmc$QSS8(N!pnKflfD*mU!70{B-b{)sJdTc2MSNcBWzs#SsB_F z8^5OOK<&Uj@7OdUJs{ngh!6t(%kTb8iW0x9>g(8RTtr@=PEU9}y{{ry+N_F9!d)7m z?@K3KGYn#~)vwUzq10BUoi^1wJj@^wd4Yb5oW zWE+bD6=q;0Yopk<6s$hZbcm@EFL~F>4}Lo^8{=8=gP*F)sA@@Mhc|T|QLGsRZWaHj zX>)IohI4OzEt4K{K)Tfn=cn+fQBSK-_Yx~2ZH~R44H;?YdvkyGcR1Zc1=Vr!+)Gsz zi&6Vms$j1ATn?21?_(ZM|5jqQZU>LxJrzi0iCtI(tIfEZvbDCwNf#h<8U>?6(>~wsXOaDVXI%>05Ak}6YCe6{E89;BPMq>#vLAnhk)nL? zZ}RnpNajZXNev<+N`-W%#vk*rtUr2{&NisH4k($o?h(nDENUia4mn)%{X6JSB5;I4 zOrg_IMD-`L#1{2;R5623kUyjCzQMXz_{r`uNYGnFdVOCoe@WHPRin~BVtlpYz@-vR z!7EJnI)%v=pqUQ9mC);eN*}GHTGHek`cv=1D@$DZd4Aoo&Xv1?vPfLv{cVDkG_qh7 ztxNsk>5OvTQGaNCwyAj?A4xCL6QJE7Ypai)6EqAS$@yiS6hSq?GJx(*_CQhwcRIMZ z1)kL>ReHgbsn*6|;Z!rq!O$}EJOQ%1J3?8)g$B~DM@gCZWb6H6*Aiz$agmO6Lu1~! zf|J-eEO$K$SR59M?*Pe|oI#jG8?>q&Q>_82URuL*TSZL2wZ{Gs@m9K~dMg@v-bNx; zcHC&!RHIwox1jDT5$OnUNV<-74{L*6`y%Mf7$+&ryZv_E zJgE)6NZsW%$rd(q^MnZaeNq2O&I<=Tdb#j?x^4SMVThBmSJj7W_zE;;O$UzWsE1vI zZD7+m1)taQ#$Y#gK5!Hqh+0ND_dZav5x=^dmf@YL$(RZ5t8{y?UlKudiqo>_BK>%9 zDnFl@j)WZjDtN^1*;DXj4#_*4Qv2r3DZ6L|HesY!Rbjws z`fO<*Hj7$ylKU>zt11=pSoT34ci(PNzI#&e7{vyFrhGtL>~yqyglGNh+nr&PONos( zm?7(ISI-gQ{XJtIujwN;2F<1T-VJ?%bDEEkrcW(Ywq1`|i>@YANyXa`I!q#-)tq(~ zG$L`Wr+CGN2V7XYMf2ID2hFIc7ut6byIY|-5rEB>4C^WaD)JNw=cO3LZ8&f5E_9=s zEvzMA`xCvp4lffTVTIep4~CJ7zMO^%V6rMXtKP~pt#;v5l0r+C&|$2)v!0uI-e<8kb0@tEy6PXaYk8)T;SXHK-Vx z8V=ip(LQc31Rwulb$#L%mxfrad1%=+_M)ql|P-k1o2sg?`PZBF~OdxOtgU7s|b5|d05AchK zHWyuBM5fJvEmdKjA8pTa&e6<}!!u4{H$B4=5q{FFZ!xJ2USDeQX=!4r=95=KkItYM zPV!E+-VA1AmzmP{smi)7P3A>i{VbjT>@FBRSLUMkx)45cqIC=v4o8AQD1aF=(s!Pn zO$Z96zN}MP*f7#od~?;qDQ@Gm;8r)CSPe~Q<#P`nm(pHmB>6m2Uv16twQo0#eWi5n zUfA23C$HH%&q6-z{F?RAhE90;{{AzgoiOGX(O&im?@OOQi!85+Y^FmaJuJQ!<>l7Q zXm%+lOrI2hZmN<;FWxu2OUaSf1P^t}g}t+MbT~k`xB~Zu2E-E5n)G8CQ|&w+rE{!A zqQ=(L=QJ{iV{1%GsdM0XX#1#KxX83r<0ih2v`~C@H^!^1is{`6=%~MR5Wj}lbTZ6N zXG7H4;6(-}rG=x4yo%Q^S2GO?lP($%@f(1j*#1PK{bLa| zS^QRw=fyjif!Sds%rTQez0Kd(oZ;P&J|kAnsp~R#BIkBKzH0KFhXSuI5u~f#AFM7~ z-muAb^|Co5(swmo4=&_Y55I69ClsMjIqoyZxVaR}qbkQvzKlD85 zl@2v-bcEZJFTd{2<&79KW3O{y&-+NHCgZ{wR8bLdro;O}K?YEX%*k58#DtE|T8;@F z*+9)5rAF|Mt%+aUv-2(KT(?qHp#Fa5wEcOfOw87l0ZHiLI*G;`YlrlFi=r#8(Dmi@ zj}0#5&C!6-%lZ4Lm^ldvg&z$jntb5TjrM;nF!()K!5 znVRfw*E@}C^V)F@-{$a@<_@m>hpHU%`fg7O9?NBnyX54G2yWW|`Cf>SJo%$CZT5~;%xd5Mgcru`1U-!9l>=?SJVC{LEpRdvVR zrZABvE6ielya+lWXi7=>l0DUqs5{NnbwYE2!Kklc^3cZZ{BDL|w?(aS&uBFr&#nm* z&YR#smxR2i?$q8Jo9ooF1Hp?0c=|=I>$MPm20dz$MMi0odZB>NH|yAs+W#eh3ZBQE zb%OrQJm7w%?kAOwA3AQq3z|M$6Z>`ApCPjN(YMW(rIY$p#b=@eGZm%wiTM>{y_{Tq zw4)k3E!TmEGnV*g!vmCcm32?sIt#nH_V;URzD$2uCb4PV#!RW9xKicZs>3cJURP?~i1s?;;SeSTSgq==*qtkxTaBDU5tw`9 zJN%n4!|oMcMM4tzc`^|r!JQ!c21ld0P$xTz=MB88&K2p>AEYfjxUjEdrqp?eNM8sk zsJc--q7d6EIPs3@dt1xvAlM(@A=98K1-ZJ%M)0m~t$WPD{$yMmtfX=U-eGbf1;`Gg z`dP?GSJH?P4$HuL*KSvgsDHB0&uTitZoVDcTzC`@u=xQ~b;v!&d|6CN+1Srdn+C_% zis281CgK1^#Ly0SaJN3(^UF>V)AWJe@aekCJly-8h`HI4n;I1}YFa{JuZS0~GS&{G z9VtKE>vy*Nc?K*ANtZCKKA2)Ms#bARGAA+ZQ{ie`NtPj!Ymskx-kv zv%7noGW+?Q(*p*=wVX8yB~&xPbJ;lL%7}5);r(fHw%vBXG)$ZUVe5Sknv$iC88Fs zqzZq!0SnhWD#b*qo5#+`cMIK=C5$`QDWHTh{}R;7A?n(Z0DKN+9iIl}P{gM}L$M0T zI3A4+ANR{d#IsVhxmmH@?-i&N38awU=K`@kPT{MMY)&El#!xv=>zVk${HMAko06Vj zw9NSd=b3*Y^+W)d&r1F%U1{w?*h1wr`#c}30Jjn}2ikb~Cs-X&aKi-KfO6PkoL`u%Q1)lF6?%3<7GTHi;pnDJV}mW6KDw1zn4RX`J0y#~|+BEZ3FU*r?12eSOuw+XJ6f>&N6-?==*koI(5F|~JD zbO+|U>WGg(r|TI+z1-v6FHx+Ge3vgsFg;3RN{V-?AHvWd2>U|-PgQjI5^Qo4^deop z9_xsrM(d0g_a(!2=r{u%;re}VZLA(~(L4-rz-B`yNC~!>*6Yi=O*i?IBk6KQj}Q-} z9^>VC1l)7(%1}C6?J%})p68c->AfF%BFIi(NHpaSgMh_+3bgYk_fyw)m=~PeyI>B>g3*iTJmnQZj5EW;Q%+=^1-ic7=zYlZzj3LKa8OqjUCe*A0>z^O3YU z`7$dFB*(_FBWb+9T^W? zq?|_8K9z9}aZkG_x^9uqK?$NrTl_HV&zrT^R?(>wncVBSYh9RxMSjDtr9SU3cP^v~ zdB9zGa4rfpp4!Pn%l6x*(VFf;uMLl^lY{3f=}J-H5y1q2I32p$Vb^>5NkDHAk(PBE zg$1hY>mL$|60oQ-a(c6_`GVTa=QbDAJw=@qPQ?_eTX9({6A-bEqDVrA&3&W_N2K0ovyJ;pye*#J z)2=h?jFZwv8B_FOu;H4QHGc`NB`OD%Kf-Orn995T?o4v~Lu-^9jFx9U@#p6>m(6^Z z$Ep~^p!1SH$~oVIr*3_yzBgx8<<{ecy>md>-dwBt_H?t}cJF?An8!!^7m0~>^!`O& zuZ?P^b~ZLn&x=&=|H3vOB1I5Ty-XW}Z1ouXdO_yBTVjD8q0?MXwY@bBSSPagHm9o$ z6D^YbHg*ZY@ys0!W1A&1XON`UpYC&cT0FblAY9&_kjx#&uT^io6Ni^k$o{_VY;fML2a{uJ2n5)?UkS3qs4|4#Qq0)Zygm?+y9NCCs{|(v)5v; zi3Au~Yzz+kJn6Z-utuL^5(mnH_>5*KuWK=)7qV+g00JxoUk= zDF=SR$9$S8cA$0TxZFC9$DzXb_9}&}%@UkqCJ`0w&YCgIcc8K#V>PyV776WsHFHot zqn)!tDsb`2Y%+N^I5Bho=l;%$c_+k=FiP;T<;rH+99k=>Jil*cZXtSMa#fjA1ImFV zWPSuv7|tko`v{NQ2mMM!0^6KI$*%MT_uSI1bboj^;p<*|qlaANjK6moge4J3IJEO! zWQ7r<C4fMb**34X=`o5 zKeBGAQAElSQp>x)%l00XX99gTQ)_Erdv+=nkn9)I`kafQLH5DHJTK#Ahnke3n=tY{ zvVy~KJ{cD|E&FU*6Nf7@>gzQ2bG=iZ+}c#`-8!kdKN$JnD>j|X_1OgY>-wgCz-;6T zEl#sZq0!^f?O~HtHM&6l*moqKQk$#K=ObLDY#}SQfi13m*I382Etp!Od>Lz6r9)kg zkrjEnxx|AGxT9ZNwvvnh#uad8Jt6C(;o~q=RZaPPi+jhitE2i#8Yl_T(BBHO9zbuJ zNo7uMa;_Y75)~-Dt2;mBf5K5AUX}O%c@5qDHKANMWF zzYOm%^w9RJtt9mcDcPGY=}LJ$2zYp3BJe=Gg?KsDX=q)KO(+Y^$LUYH9qEA#@1z{2 zK-xEx z!cW7iPDQJn;7(8Kgg43*r?`{x*PFb56h(@bWSk{PITUd}wzQCj%uu+eeYfQ6a;(}k zYlJxzLOH`GXyMKP7ZzGffROR#jZ{zzW>NuvePBaQ1#?@n!=4HI6F9pB|8s2fF&4+99TOqc-;zSpsFxo#)PoF$AUysxix7BZXbH=8o% zK5RvC^)P2&=;>9H)#G}Q*_WRMFA!HOQbI4v1DnBB=%PJ~twA3sOO7#IWkAWHBCTi+ zslS&HbP3q}y;pZSww(1``C3Y0TMp$x8c4F`6j}l4gaW9d5l{c=#T^~oL%e$WQA5ke z)@p)QGVjRb@%))4C*Jbf(b@MyUX5BC64fGDZY(NO=(|4k2=&g$xOKX)OhRTe1wtPN z_M>jp!&+qIYlv?6ACqF-1<&c7Te%7wmo;1|*C#poNFs8IGA3WCa?f~fTix`xk`E{T zM&mlHgGyD0oS^Fz-XYtigWo2pEHMjD%v?)zYO~8OH*{i+VqBTZy+2-dJ<~^mdiZyh z0yQv%IFk_Q$lF&6)7WIp2@it1f9krX_~wRx-rNwy10Odo%h*=;bpN(nMz*s0rXz9JsqW{@k7vD@lQG@02VgZ9#I435<4Nb!XQ< zK>vP`{Qi7uD>65L8-Ii-#$}Dc<;gfQE>$7FT;NWj&I#^CI2Zh(H@JcCJ}+gLJ!!u= z?8B(?>BCWV6xFfGmsS7Rwp8UQvD2iDy{6TmS+8-zl@JycVUv`k<6X#@?a+R0mjVZ* z)v-_RQX{oVDtU@bZvEBq$2}20<*HreV?E!F&c}y(l!_UEyZ*0k1mv(l-!OQZDN#qZwpk4?NmWI!0PY;k>UGLYYC}k{Hb;Q4>h(iTmmk7nimF5x^ zyy7b%2IZ838@G$43Yn(22%};|%3$7$mU21CrXkRri1KoHU$P(%l*JX^+QcZy}%azoar{p!A|c*y7Wi2iwkXY3P;QDH7$8{DPTdx#QoiE!2U z0NIpAf77-@HJA{piSst~6~15q_Fgf>P5dAUQ=m&}X8+sL2n6sJ2R!nCTtq8Oj+z6DPN^vU98?#R-Y^ z^_EWE%dfxJKMU*`#8_NOu`i z%zT&}*sIgbSoXj~)YYFCP7tiFVR0Vt-6sDJG2 z_ZPFYxM^OO0*#JlwfWtCEb{M$5xkCWas<2)eaxkGgx*QD+vd#aCU0}O_eBzy7>~W# zZ#RYboHWdsq}hj!h*k^Zy)2jY*9gyY++OPp38DDzh%q&;2U2*g`g#Kbck$N~({ux} zevbZ%f4*7_D7KCk`@d{iTFYd!ItlnD)j=Welbok~F2B!_^jy9qnvSZB z2c%~9axY(JI^k~<{Ue?~f~U;*tCB10X6Ogm*ocs@tW#_*+mJI^i*427WMvr7sbjx4P z!R3v`G&H7cTr1zVm$VuuD~@G^KYghjDK^;teVLtBq2#pVIXtM9!CGB3S$8I4zEPq& z=kbc$VS@278#8>bF*1pNo9Nu2tDoXD)T7*ZJ;J?1`2SNw#4NTAtu-+TYLJQp zlA0mHf5Nf;pV(>DU)bq5Hxy0o+!S{G^?|WB-?Mrzuksd#m)qP6j2DKHTRl2B9d^CB zEK042g*4E1egL#t$ZFECc(zfg%Q`W$7Cd@Tirffy-&)eO9V9y)g`JYqZ`werZ&;w0 zD+YD08Z*b8jgGU8(kOTGxKdHv8E@CgvNU4E&0)DdFNaE{LEzedw5`xP_P=MtDw7N7 zo5-WPD=UtFdP#>j)SsjzvU)6U0&cxTTi!*yu?y30KOVj%P%`#!=FE>o8>%)?{V!>w z4}7Hl7qrpDQu5-mI_b4HQnk<0Y$PyYv?z1FpeP(c1`+p$*8~i+8MRfmhP8s|3u)Kg z_il#jhFG;GFASntIVp4N&|`#g)FuUK7`Sx}-^sY<3wq^Mot}43*}WpI!4bu79Go&V zuIzA8u|M`{%J15RcOIpabB4OadXDtr@cR_3+Zc9UH?91@qADs^>x;HIZ`h<)XC1Yr{$PV~99T~T>&TMT1Gjbb@Ml_{-Kv_q<; zXj(ZNQVz;hxb*~=6V_9kUQ0rk6f@7%;sO2i*N{lW7JJQUzTrE8#&lI_IZI}!;nwxj zItPW>Q>on-y}Dw>=uGWczI~TBT{sA3);FR$KM&rHLovXXIlO!8n~K0 z++q+ik~S-+In*1m(=nxp)kqIe=KSdj!F*r{kTyjey%}RHucpm8XK~;xFE58X_!;1T z;?vnEK-uPP!g=nBV!{jFCUAYn5u&v7^lll~i90(?kZrwOyy)e0<3eURuFpglr%R8y z7Lm&#a>9GHwjEFLi!$WmUz>E9_PW`MphITu+u|A1DlORcevIn>;5==nStG7N52$6i zqe_IGXvlh36*U8X6?)+3H*NiBe1n3yGdB?u!UE2jgz(;Ltd&g)96NuWff^~H+_WUH z=jG6tB@?*{t~Kw-l_*kQ&%uUoGJBqZ@WM4hD+4F}cV*Av&V`*9_ok&h$H~}_Kac3m z|9GNr9U^Eu%ZA<{TDm*H+rq3Y*7OJy7PN8su(gHbHGY94j}b*^>O#|U@aO9SD$#=T zsVVeR?6z$#<;j{MnZIT3vkYXi8+B!0w`@8@5^jupt%lROz3ddIXIU3y$zf36EHw;- z=SH?9KD0S#ETo}Mnvm9V+cAQTR%ze zgN3{$b{M`gcs{WKw<&~tA3YckkFIslM_asVKhn*lHIABdFVq)3#8Jpk7<5`2HWe2ox=0L1-yA1*qfH^THI*m!h zvIH0Td#Q8OIq%eTnTyKH?ZBE>dV@$5uDmU~vkgiGmlT5@H*oDzP34yCzFD&H;!%(R zWe$JTBSw5*GJM3b7{m9uHvF!`qf$>Jd5l+AXH0!CofLng1jw1m7ICcqE5dF;_sBp6 z0F75;AroH4o>iQ~x!PI`bM^e(SycRfE?1BL z3P%~}g)9Rek|7bvSx$6J)>Vt+D0%cF)6M);XsfG{1OiS%@Hes%kzS!RmJ}$QHT@+& zRdB2n1*HuEDmUcl{u0}XZLdrL5-@+Vn$u^Kidxlq3N`NxG%>A*u2Aw;B6nl$7=J-N zLWqg8DlN?K)|bGx6Vz`7=scqLJRR?R{A_fMUM8cZJ(GF`bS_t)Cn$T-HV|AFHOa`U{j`#ZFJ$it( z7S%HI>EpOdNapLAQnqJKbksZ&t9U=~B8vdpaF(iXZv36kms!B4>kb#6@m^W@9)}$I zr`}M~S5|q!o9E8T&_OPe7WE3Fp+O5?ddpN7TivI-^}mBRZgm$P(!%-YEgM}YoAV)T zq=v{sUjd&bM2JF5yFWU#jZPYX5@m#d1i89;mXw%W#=`e8iU18iDY|Li42>uL=Fq{l z_O%2e@{|n51!cjj-Mf*Rb;)%epZTQh%$LQ)3+_RtnOQJ@r!z)cp*Ag z+AQJya26z!;Fta4i0t}J!78eCoter)vD0b{xXIz z*^M(cL%X6t`?`i}qP2QI#QCFLR!w1YXFsC;5bZiK^7aDe4J^T~XIn*U(3vuq#~(7^ z%z!q_A`s(YmtFc=wrRF;jJ;nFL3bJ5`bTSJ;&JF}Ab3Q}B9J_=&bj&)7(I$U!tpdc4J zQWpF&r=p#?WXMt}$1L+^#(7X|LIZg45{3~+Q%(SL9`K&qBK~4ujFnJbA>c5{lK#yS zNt+&*Wsf}6eWfkqBqt}O`%Ixu1{~BQw2dDUJpVIf{OtWnngzk0+xISedQWjpbxpld z_s8n2wK?$nB~_ZuEs1{1&wmHW92QBT_upK4MmxmWA(cYDwp8PxWcM)dXCF7!bV*mH zzN?0p6ELPUTL??$rTuQ0UNFK&dC@|1&jZ3cGY*p~e><(`jrJcu-NSV!JzDCa*k+~- zU@klqjSbgXB$4-+hAcN<6=0@>026#wqe#faWmjMM(v8nnYDHo^U?(3FZC2giw2-C4 zwo5|SW5?qkaom263%DaX1@;b|sD}>~tH-?dF7WO~fR8-g8n|iEs+-INCP12~`?L=qoTyp6bU-==QRxx6YOZ zak)aN_cXARs?~Fd`Ey0{*zNc6!q=krA|Ljqu@{v88{Cme# ztJ2ZLN9}nOrei69Wh0%Ba5kU#q9J6jw2%(b;=w5Vr=eh_;h~bJ^2ePtaLiO4vcR zD9;)dom#xe_E)@}LlkIuUoUGGs!3$g>CQGkX=^TJU%0subL`iv%;36E<5pCg`=v_T zMxb$sf8zP@$A<(=r(whd!-`5pMLB77F^We@+GUC?5N$0Pb%sYvYl*FHps^n}`*Mn! z+I_(xq76wS^)Q!$-J!6H@OQ4rmDi?lqgCSuyp<8=^rYx&gS|5*FXlm5EY5soPmcpY z$O5E0@i(0py?ss6(Q>#<)VY$uv=7v0Vp%+G9b=q8{TMSPADFF@^!tQe= zivVq@Yb_qvH>5ilikAb{*$CN&g-<#wbcIv+ft>U&vRK zX(B79r?utYJKBhhG@Ht#>LvRbrOKIobHsQMG{T=BgFOMLZvC`pjR(dR-#z$)@z(b% z{#*2K>18j5H^ozBD!x_@+Gg4c`SSico)7c)ivFUD??qpxj<)mf;Nx|Ts_@DaFZX4z zYn*=vHzaaa2jCB7){NPbCBo6Va~alcDV2kIESwIpW;#*)~<$z750MC?ijDj z_>F>0ZQ?h@cL8I9aI7R)FPq-yDfNAmv%+{EOf~DtA>teEYl`%wo>qkW^4`O2=~$H1lFvo4gEwy4qt3kb;`3X zV%Li8?GvQ&l&yJW0gXU)Wr7)kLI-BQ%)f@I+V#qQF|pz&T0I}I;He~(?n{*tWk7y6 z3(APq+njV;<6;l4H~6$=h+ow9X6TnuPJ{2Noh<61q$^-vYv{Ckt*UEGS|Vk$jq&6g zqT_fsdfFa<4q?M%;d(#k6_ciIwoF-x6^Q}2lP#q%i1-8h2gXEQPJk)GZbHIM!gp&^ z8&(?Hh7|QG14TVAOKOQI#0>oBEYH|3{d{nx_Ni`Wn7?U1SW9u2O@;LVHYx;?)D9JFoHAXGpE zon?%Q{p2O%0Q0(yG5?QMO+%1hjl(l*VoVF7BZb~;jb8ye)V0RfI4DsrYlDs z8GX^pgZSYSe{TDL;>3S zD5KL?{p@n=oes(KVDhIC#`9~M6&OyKj>Fo9H^P=&L(?J2sjFogAfGjSPM!NFUJun97%`Uct+catn(ils4uoQAS{&86_9Ubd*0Uv4 zQ2$Uc*^f}j|936wEjrr*gET_N$(j~R zWsGB=tv2L!wEk`=6=yIZ&eV;>FIrs#6_o0;r~SdWa8d~!bwn>1HajFh%8G_m_P()O z;K|?LPhF-B%0^MnFwZJ|AOE~zTCZ)Az&tPrkksg%e6UF*6L#+b!QAZK6PGP6@ zm%P71uE7>vwu*)AtRKVUzBbinmLWZwx~q>Uvwz2oz9R;~=mfsW%{(U8BHb<`HHfK6 z_x!B?W6kr8l2E75`-@+il%(9rR>fDTPM=EYt~fFc%6}AYk!kpF8^T4&_B!9e@!YhI zGcH;)W+kQ_5on_6&gFoxXPcloIT3sXZgMIfK}%Xy8ii?wOvwkd7a^&>Q|=p-Q9?*a z^cJ<|fCgLMaDnZ_@>?A+uiKE{jn#%XEyokh+7nTH1D@NE+5b5u^zTHizbK*VqH41* zR$FjzX&%jIPFv1r5yoWMVO_1CWA}w-?>|q?O~TZP_gr<5nBkXM3mMPl(9A12yJV)n zj3{M%Bu2`|JePC}cnUAd-=sC)R@7hZxMN=Kg6{%+E(rf%Fq8u+MK*W!RFyJ!k||$$ z<7Ukp%4Gr87h&Mc4@$6#5C)|q_Hk$YwR~dpDu_JR0K7-GB4_!71dFX;!=xkWOT2@Z zB{)cHr^w|<`%ZB-<>=739^^58{@Cg2amLBZB}|JLURD!|{q77=4oEW~uVfNkHAs`a z6g1N|uTO0~#YyC0$CH(1F=g=W>^}1FBEEbUyx$WOeE7qDBSjfZ4y6aH2#eu!V+m;P zvIEtUD9>BDEX^V}jfUntd?*bPDKDo5H%jw3D2!&q8t8rfw_0 z@QE*9kJ`BJ`}kQ#UTVY0Bf_WV;SVBb#4JXdGwKpA?{BhkpxAoV^qk_i2s3mw*~WIV zjZ@@3q=fj)oP0#a2wzAjs;0J|DZgUjmPw?tQlegxQw9YhsJTq^zMBv*efqNXNMlA$ z)4bok{M$VyfQ#ATl>q?M1lD)3NMTtLKROocoYRl{1!q3|GHH>7`D02pj5q{1EE^mi z(rJX60|sZ|(hzu)L7ubBQ0zfW6YRX&Vu{{!ZQp#%l&Q%e+r$e;^gr%SUPJTzpL4U7 z*xG2q&j0`4AoQE=)4Am1SeBGbqVqq4a1o3y+sF$RC zh}@CqY7ac-LO%Fchpsid&1YPxR|=poy|klUM<%cR%}3zx<@KBslvj?NgCU%kVi@yP zVaENM)mFLmkJ~nYpX}4cVAC}<>NjrHLG>JP*4PlaA=y&`u^nyOC0fK@8>Enwm4!U#gFLHk>DVbUni2M&*Zj3>n zOjcuLVV{}=gQzmq-!DK5v~<|%577(W zRn3dZvu@F$;2td>LF$7Zhx%~{`>ckW5Y*$*mb^m?+CzlXh}N26!al7fWCjkK#Cg#^ z4XQ$1-)48qCM`<#d;0}>TlG(H&?jm!*O||&>{zFmYV=mG|8}B8K_ZBCQZm`z?|=MNdtW&4O<*GwK`QpFg#VqkVZ17yk#tFDIADsAKYyZoz1Cgs44G&vcA}2?2J*ZLR+C-e8ERHYTXH(~s_vF4 zpZm~qs$f@1-;}}hQlZB>@Y&&0@n!dZ4dQ|l%byp?s9Y$vxJUaQf+Eqq%Dl15X&@h_qk)&~TjO9|Xo8<72_2#gElJbl~ZVWNAj!E{r)NK|1 zBN^o~_oNE%2NQ}(?l6NlY54I4mx@D4RjH#$50l>$tL2QfZLMm{zqdClX%e-9rorB z>V=rchFc5}s~j!DoX7j~Kf<1sXgXVtvAh6seO`R2bTF*qLZwnZR3J5 z>XNMb^fm1#ZpVHL@khI}rD$OF!KBlBx#1q)C z`G^%Yt~n=f?Aa>X&jwB!g2^{Z$uYpNDaJpama|>?UccUV7}Gd6X#%`QW5Vd1?ls#H z_c%mk|DG2}H6pbrMVZt$+K(g^cGSCZOqC6!6d6mV7NGFFBN!{{7g}WJ8jr^78^y-O_i#chQKZa3Ed{p5k7lP+t}?JfWGV z4IfCVH=xJ22r{14p?p@!TM)qL+HSL#z57fH{lY)ZooqUwlY6{W2RuV3NdKOBgFbt? z(AZDx)c^E+T)H~7S?g>TQa;h~6qY?QLl?WXpyf66`(53o&8_LHa2qaPz3*`*3lCDg zh)MSXTTa&MPvg8n(V=r8?*o5JU5&-FiPI>`Lzzgo=~C|L?9-qX_Y4AhrM3zMVj=W# zb+nRU{Fkvrb_ryPM_C>o%=@luY2ZESDcl*tmAsV<8B6*Wa?|$k84o>)aCMMI<%oc> zOxLi0gW8m7p4f-b+)nQGgddO5)=$gnMZFxB3am-*QV@5*BH6IjroNX*DH>a)I->ij zYVtw=1&2A6cv?}mxa+K!UooL)!m55J@G@6(f!o!onr9&7QQc|vGLs*8C z*?X7DF6(7#rTJGBAb3dqsuidIXlgTV>JGFr_b_Oy-gaB|#;P)fFlT_B&@&6%&jl4L{MsAkR^MLx{=sRZ`qD`T?kLy)xA~vdamj;4@dw?N5InN z5O4kWGOG2c4>;d+L&uEk8JJEqkec{?yK_C+8m!vSKg029!A|FKPyGQS`ZVIt*LIukc1}A{+lDg-? zTI5!*;WqRSMp38rGntE z_f1q6=%FAMO$$6x)DtVpSdHH~dK`4Z4)(RXP@b=;35vI~TwcV3s_OJ|YpL8PGtLvw zl2u?qZt10(P8I#p(9oAy)RnDgk_rrVFPECX9Z%Sombm-PpH3uOs^%ECCd>jUR#8Ig z?otkt-W3etN0XZgQvh;MPPO^>;PI}G>`dk^WFd#M)vaz>t)8*1!DWZ)eA!k?enq`^ zGp9@j)KOT4qq|kRs_OU~I2W+zwF)9r5l zRXiOy+kz7Mv9kmZdnApexmk!Fwz=`3sz;>Sqc;;H!I=6r1{Z^Myflq2v6 znsZSg+ok-n?A=&cPPS!&;?QWyr!~b3I;eZv+^_%x5oO@mV^)__d0GV%aDTHgC5_J&7)Cv>tx^HoLGxG}=2h z`%MvZ-tTqkMkyuY_nJ)%C&?mj3*OLxV})xJxUy+F0!OA^(!XDPqV~kg;bRb^=yMgW zwgSNvi`JOG60pn?8W{*J*U@rk^)XZ8PXsb6uQa6B9^PxFJpEzCSi;@UM#9i$n3=94 z(#~)1SWUu{t)tRkli7m8okur%G!m=7sZttiNYj6@vu%S2?(J34S(&{LR2KK?SnuQC zP5YlK2aZGXT^2d=smjRRnfNq zgMLbP-{`h?-MB9KF2T`TvE+lHQ`>8wP^f>r!uP6Q+~FIOUs#mXEg4RwE0VoD7&m6m zIx;bwmu2z?g(wX zoTfbsS;Nx;+bGHpQ!?a(9ulP&4wnKYW#yTyV2FX`F6P%Dn2gXrB6gef3uwXUesE&0 zS#c@9HGxO-pf~U9EiYHU&SM>J5yvWYxOe66O&6XeOizgiJW1_J*P8dj=~pS8&HDATkE>Lt zG&q9RO_k6dKKHe=PGK-%Pm$w=)>*3VBC0I+XMRpg47rM7=R*-DK4GT;clZ*N+c7Zy zBF21UP9MIW#~{1fs*4+)2PpMeFbX!L~S*(l_M}6v5gj7l)tc=Xja)GjsN!41=Au+qQMREfadWHgv0OxYUsf z^003`?BU~NjV^*38UC+n9)-2B3&h3oeEk}JgQpglIlAhjIP=%>M{796dB)9+w;0DJ z=VxfKd#gv^YtRv^RiEPl;pqMs7d{~)?T5{B^H|VHP1X@LAD$XKglEgxK2@}ut)qa) z){O06C#TJ^|GLz~kNBpFE~ypGI>qE7waQ4G@%l9bqh%DB{>C?(GqJ%)z>}&L%7T$6 zoF_n?;bjt5a;2sqW6rL8a_f7uItl02SpGSz4XrKBw~&F^#Z$2OC@*N69ffCbRLwak zH`}Q8s)#)O%7j;!Hyz#p zM-Tv0Cmio9Gn4w!T=Wz@_Qj-nG4|N%NnRbHy6bA%eBV4It1mI)n>Ex0``c1rr_4gg zdbY~e{y+QM;c3LIWu}MW_XDut?!M_`-$FfR{Yvb17bofWYf{VM?4qjBT|rW zCLcJVcTP7{2J3~gG}%rY9Ohvn=ws`KIqQ-3WQEQTyy63+>_k^xnCgR$?j9Jf|7^28 zvj6Kes?RG?G(2dsx0T9j)t%^-aGI9n=Y)P|j2{j}BQn4mUsuw~Sty7tp3@i2e3qyY zYs5El`zsnzb55_@QdoG>>+j83C9TTVFKOF}zYitMk9(PUVczCbW3hR*R}4~UQlynb z7(Y|BxW(ATR*Qz3MBvUx`!Z+y@NtK4y${IQ?*r$}eLdBpl+4sE z!4o%822t?25Aq*64x%j*rjxL%t8=~Ev;xa!VEG4kc!S_A01t$ef@f>+*n6cFiVZE2 zV<{!`X%b~pAJgs?&=#=b(2?&}(E{vrWuI`Th-&&aetED}oE+wiKCNbc3q_du_6rQa zdt!ZaU%S;Cy}GkdeO;+*Zn$ePqpUcROMNf4J;nvM1{PZGn7RUyw#3Qp6!aeRi}v62+VjvS-Pw*he|@q0Ia7e^`C9* zzhIxAk)3t?Elbr7%1KLHdmIAZ>qHzHbX{(Sry6bj`@3xEL|2@;_S|6G7)f`_Q+!v* zmvsZ7wl-0q(32+H>hplC&Xymwl*a}WZt=*(Qn$EQ-v=mK5 zrn!rzAWBz;HG&10mQcBOJU?ornsJJ(aKoHQ{0Z$KrI!n>HRfYUo&^6iW!WC?k-u_e z)6O@YG(FZn8+VA;gJZ{#j6vbo2I_Pywr+MPdG1mtT2abQQS6y3lS=w~kZ$p>BIF6# zq!6<=2ODb<^XmonI`8Y3PaIrGAh~od9Q?!w3$-*&Y+zf6%qWbwfu?KpwTS29F2jCV z>Xm`*fpLv3C(FqI8s!A*vN`XBJ13{K9O-|9S!f^U6q^G`rt0Q6hMSRbpF68R7#4M9Q=m6Vwh-NkvrKqqH3|yW$)$eqB}0w@#L*i* zG3fyahqg>>+G~Ef^611r8><`r!ZWSd6C=w<9mxke@_AZ1oNqRMBZAf_$op3)+_=f5 z%%Fya&+xOG?%U~-bjHi=_4N0UxW6!YJi+2yvO$_zO#f%i%}x$UHR!Hh=ZW9g6wAo- zuWD$_N*Lu$3TTqObau_c;%R0lk|-1P`~@VQ^@eDX@?t^76Al9vxe<&Xa_{BVOC-Me z;2jlin*QhT$yqo@DoJL!g!7g(g9Ta`Z zEKma(R1Z?4)1|!iR*j=l?R!6$C{!u5=-QDhZ@Dl)Bv}~-C(6*6SGzeN#hA77=s=FA zZiNVscF4>Qi+2m{W2pY(s^*gsQ4})z)$07|Ox&EI{bwaqBu&l-Mb5)zK+RxHwTn@{N|R8zqy%NwG;a z3lsj=xbvs3?&bs=`$rLXS~|(`ZV>K?DEsR-F{BbvenFp~T*?be)~N6Q!5DALi{^;i zRV|_ds~L&EcUIDZjIX>>_kK6_f+g+y*l`KT_u_Do=RHR232z}8_8Ji%Ebz2cymBQUZAHLZV}%V)jsI=BX%yd&@0H37aKiP7AEDYRVTM z7w{x}i_UJpL_qt??slqjHpmVSmcgo)%lME9{FgQ(d(e>brt)UJsp_&v==5-*mt(-3fcx)1|99><%2D?izRa6H zWXaqXQC!?gQ=-;io~4tm>tuy{2RFuiSClnN|HyzTA`L0Iq0usdi59OIW@>BHbCtb& zi&)NE^j+3ouN|dnxlN{_zCd^3V>{iG$aTWei{o2Q8a~`PW*fEDL``D3tGO-xwn4N$ zt+p;8Ao_~Hmv#tm&8XJ!^?~Sf&wd*Sw4Ss=vxeFyVhaJ?QQ#jXM$Y__LGY>o*VJ~( zr{tMvJMn1Fk*SeaF{pD8Tjl3Bf#`bNUF765N>!f7gf^5|tINk2zp85dJ7V1RM8u4v zQ{~kR2guGKKh?^f{>~o9B4G#;U1Q>#9q)98i+RewT|RJ=14JFLAh!T%KO*V&>3^gM zekmIHdlq{rg(lcky<_odgv9?>e4v`N+NJ8v$kZKea|J-Z&4uM(=2qRv_-o8>6(5AR8jzm}l8n2I3qiOR1e3wU$(T_3YIO!EeiCO*Zx0N0Em? zJ0OLU-dOuZNL+!+|2x~H|L*;PeEDax2j0T}-)_^?sq{geHMbhxZI#LETcUv4{PZ6T z9G~mFq6a7cQ<}hEOVmRqLmV7Al||kmqDFPv)cQd2LtUkL%hwc6@->{q81Vwjx){!} zkt^ElN+-!x`=?B1#KX<8O+NyPMdR|~s%H&Mzk6|VG$0cQQ{!B_+~&r`d~~@_WM>n7 z-nephOEOmc9#J3T8Jdt2and}FjS2*aPZJeQmbXg1YfFfHg_Qi|N@U!;^O#z$%}T@M z&fwa&9#EA6P+#93eUaS9`&mzIt`QG_9Nlb&VU$!kA$ zWd_QKO= zKS%2yD|2m&%Z22t2DV~rllD8La-vA+X0wha?fr6`^bbp)TOw*_j-6) z;s8PQ}6~Vc=xJF^GG^*u$zzOM}bjlSmKm`qh1Pi634Lmkk#cLtdA( zD7!R_2+GsT7tKm~mjtU^$K8rWld~yS{c`rO);wKZ)g}5Iwk~|a=n1)kV%sC1Q4*+b z0UN7+`A^-^I;+-mklyRSgt0IiYlhMXY24TwmUsR>wCq}!6syQ6?9%CMZ3*Q3n?aa- zU)>{!bT?Qg*~uHS$`ND5%aquAQ>x+-Z8loxeZTDax{NiO5ys*JzX_`n^N9DNu_TOF z8C#7EJvKkxm$mE!R`z}vb$geF0;;@x# zVf~I{KjEfT>XMOf<~$=7Eo$twOuGjI0%YuJ2#FS@c@&4~eC@R;)hjBska?AIVs+&2d*SvLbewkd$BfKJrCHM=}yR8cwKRj4K9@ zM+Y802=H?>4@fB>sjBu+yakeiNL;=wBT{PL!tHueAKb3z-1csY7G8}slX(FDPW3d` z8}xs%_ZCobF3qAS1c#sj5;VaAgC!8$CD;&bu;3xM3@!l@Ah-t^Ah-_h41-G`cp$jU z;Oo?$WOx-=Da)F^Lxc151oQ0sl#gKe+-m z`Tm?Bf00A~*OTI8sx0g9P370*A81W>pZ}1nAhNrbq4^nb19SeNoEjR(bav6ngx+D> zY{h?Re%kqM7ot<%zwmPGGA~TJyyOFaOd|S}goq+J4j(J)!&ZUas0Lb_Zh`6#N7;_eqM4G<(HKwZGVT;EaLt-P&jljHqJl z%dTED!(h~!Y*Nv7B!jIeL;+IvvYm0v|C*sl5<>j&>$#w6*xGc$-7?BXU3qRn=Du`e zUavez_MOiBGQWW|6%L58#v|`uS;c09IyaAHHysOCprr^;$bkQ1SJ>|~l5sH`EPQyu zpnrji0W*27@2Zu$Vo%rB2-Oq z(d@V3X0=c-KUryi4D!DwC~kCLTSV!mtd|cXn_Mg}rw931!X-~efi_s`)8{6JdHK4%9^IBV;_FS#<(3(g48QqtD_G+Ep z`-Gv6VXVx7^}%$~Z~nve>F)5W+TIfX7^s6=k`Hl721{nwPB!1R!S97aC9A57B0>?J z&EE>eV|sfMB^auSjlUOuukSzpzx%~q?_m$eYKV7oR#VIIZ(V?we}*;Z0ThLqPduqH zPThp<4Y`I?!&Zz;*RXPIgfU5xCK>K)Gt)#G6*ajc$$H#DP*eu)Ss+MJJOw*@l-odK zq6?HI3p`Adjlh#|BH|!K&rahnCk%S8LQ4#gWj^2Cg6Y4*<@uc3bf0Bpb#N-)$mPxx zs4N_qg3r{4R34*oWW$mqpqZ?lt~JmBoEdAlb2#%y4;X$=`W|0_o-`6N@0n(24-3XT z819>I#}Fc~a+SnD+`Q;5$;7QY=`fo#3h2$}%gk-ux{T)<=bE}495++)XDJ>dAf&*d zcJeQyjqVlg5u%ksR> z(G@;3s?4A^&WY&0GRWcLou=UK)fC$pt4d#UT*6kPLl(bS_rVBs!Mr?N;NMEuq zm`4A8454e(sS;)Ql;fk&x5YRXMQ2>+x^jA6g%4uh$<1O|E)pfr^OQSI+u!r-RC3hn zPaqOTQdJemqeu;o#fVdr*r2j811nhNRZud|)X$?YuOc`mp@a=g@I$zPW)6q$3-%8Z zB~7-aP=rKYgAH8gy~ju4QDj6?JV{cGfOZ!v6ONs*c-?b=IHh18)1+!7%^K-WX)B`J z(ncf%Qz-s@oQ1UzOT#zyxL_7$z`ah_I>>5mMeUruzSkXx)1oT#dPbH8RR>^xB4KL~ zTb|jy*>5pd8_!Nv^~AMEHWrbtDyujjijsX{2(nOQ{_3?vRJhK zVchwqmS=T((_|YbZMTb55mtO3G(X=J{#uCt`i0c2Ze%*)3^X9M=vj^hK@?fG69{!S zm9CrlJDwHEZZLiN_8o)C`6-}=2Ty$HqBG=nbZ%}wi-NcmEfK$b^Al|G15L_)`Dc6$ zvX4llLEg2oVsn5O(OcDR()1;6&lexO)3$_ilHM_p1;Gty9^}*N)Pd*x#8D!%9EeyN zB!{xU{m=H8znhN!_ZhtYwaHEWoOKBJ{dK}dHaF@P)PLl6oJ=)2e(J|7=~2o$AmP1X z{p*Cr<}t%yuVPFn?k;296_)LJf;$uNmS~4gWs0C0x_+DayyBRD%vXxi4VCfVAA(w0o)T$@I}EO-fR)x$)c+M3f8OGtXKt(UvHrZtb~N4V z$Js8NE%wV2`)`;SDH?5VQp|?^Ov^>)P|nHEsy8L8MkohB5NvB#7UX8F0A_NYl7%!_ zY+!&gn5aqtL-UDyTV=ys&fLQtz`;OJ&6*XwQ)}QISw^g2x+5yrhMHnF>-{PH5|s=7 z9k~m$WZXVYdqs9C8)gz2c>yT~1tB_jDE*mdI5wY9J=N`MN;MTz!cwbLA2io+rdK*> z^bsR{`}9rb9`B6MU)mYlJ#;v%h_)3jP6hy;g`%s{;T?J;L?^`LMNQbJt*_V%t|&LO z9r3cHd}c5wS-lh$TfUePD!Pgp-7QeCJ{h?aHsxL1I87DS{)~7oo|MPy{xI4-8rgeu zf*4T|ffyJV{=ZclXM%ZeVTnZZiYiniyIjmoN=j*;;>a67PEv***O6|w z2+(JY_YgP$SSM za6|Qt2_oDcRqStVjqZk=hB-!jQ^9gcgP%X9&+nHGx~8Glbui_59TWXy=C!p|nrbP% z!YwW;=yb^!U9V|^`P9i*Dya+It;f8L4v@R#g>;<(pr9zaSeFW#(m5hEn9;^%K6f1V zU0iBUbvJk-rFlr#-aw3`AaN{dD4rB>0$ZSdLhVX+e7MoKZeY|x-|`Ef`CY(Fx}MAJ z{QLko9qxk%`)u|AX8Tx?TK#%1$HIemv*jSS$gnYZlCgt$aau(DMY`KQ*GZe^s*o1; zen$H3m;TYy9t}ERsmRwxfH&hH8arMFW9HCFnCd!%G~NL2?qX)1PNaCKtkJRf!kKlj zUwwZqqr|7}!!+lV&kf~?k7SSt07G>T= zb^X%`on?#%bRQm|VLh)%U6fz6ha)K}73cSr{1-kw?4IR_3ROJoO`5U3OKVJ|rJafP z;|$~%G3AhOhQlx1qN1_oDTDw<>`r#dr8PzQg@O6FR(hYv~>+lhJNHyZsw%SN$F zPzl_{D*w0ztY+4#FwMha7$N)_n*oi(MkPHj#w9(oqNJxFcRbRH4lGwlBec5LO068X zCIHX0ks|<`;RD}(!1UOt{<6oy4c$o&bU6Nvs@QM*b05*%}eCM>)lk+ z{<2Ogtv~V5U=+!VY&2M{W3u9~;RcwRO1vi4wo}tk(dSN|n9PMHAl|t3?@KgoP1B)6 z%iFe;Q!7@g3!yCGNP*+^jym$fly7IHsO=U;I7J1@UU6EaOLLy)7teLt;Is+9)R0Mk z(}*Psm|;if_!esXB(r3;Ch!lCt`y}$oJYdkHFG9m?#zKva>HxUTwvUpB_ z#K+vEsWs7GyFdkdourdm^+>P~gg-2K6r)do3!+rXE%%R#sXpO|zR< zpTCD+=lXA1g+qx^sh64^V#YrMROxb~m)oQDINWSNHT~|6K>ax>ncC=dr!p-zsE^PTjo{G*()*ipn$Mt#Fg2(o6dkWKcBFHe}C4!)s?q*WcY=Dc;Cv=%-6#?V={BT ze?7*j?y0LL@SSs`-W50E_5--SA`p^aF0Ho|Y=%+lN1ZRcp=NV3F1|UfyCrgK)U$E> zGnUCZjr%s)*Q=r&qQ?!}>^kb?(n|9ko0un4!@KJ4P7R*{10wr=z+3vp--rY$jA+yiah9Ig-+rO*w&`=?Xu8$n?G72g4VY2Ge}{= zRe^6m?KP-xOiAHb6J8GQU}Tw}W*G^dqWtaX?%R#mrajwjDVR-kba`eD2m&0pCEAA= z0V(qfS$#)XooPzPslkoZuFv%JqGnrd@j|5D$o9Mx@X9AzQgpUzWb`j^+~@m^#git! zAhM?JVm)7}r$;0!YsAiUuqi~4`%FgXEtG_3Q~eq?C>!=K>TCni|KMei|8E%aqPhwB zv*Q2B7?<$(j)-e;KYg9ub8AeuO&UQ_r`yjsIuK~D zVlr2-j}QqaSzek`nfv^^V`h&~Tk`PtU*oaSr9|g?ZDW3*2@IhtEh20`JC2jCBW56b z^<#w^bwTKFcS|}}jmQSCje~D+iw45lT}W;Y@T!-?^I+`CBGILUE^m>(V#Yr+6Nx9Nj|lwgMre;&p0eT zVqw{*TZPR}o8$^BOj#+~92c;=x+KHG&Z(4N9VWe=y+>!+E-145PNF{3SyiZxg3C^r zoz;u^bOe>ZO3viO3}jf-Yx+?h1X^}qTEKt%;P$s=&GxUZMz4p7cWtgXG-Z)1D({3r z?qkQZz>!;I%L$G!^JO9zD<`9EDm)SuM)nYtj=WX(YSj;NfD{B zlro9w4)nHpCX%JKl3h29O}^E2o#6$I8^Vf4*RKz8WHg~(I1&N^?}!DBTnRQ$JgmO^ zebkovpJ-%7|GrDtKl&dY$O4t2Qe+A3jQ(A~7pK&y5BmSx{|%C|wv)s*PpQX0?D03- z;B;|1?EYr`X+l|lGLRQUCg0^to7B|lxXP9mgETmD%Zh4rtKs-S=X6@p)?@M z*!-dwfm7_v+s%5+{$CG>{OYJxG@q~?XbsFSES18uGu$+#^8hDp1~{UkKvt^V^7nVo z;k%g4vbZm**`Il-QsgIM)hJ2fh>4kHEo=I2lxHno7hY=4P1auiKtma%Y)g`EX$Oe- zRyG;R=MvYEzj|}5m|PVgpcR}L9eo+1w3=G2e}d4poj5!gLK>z#L}K55<@gv{$HG|- z05FMpU@P@4w!F-1{e&bYiaX8PS929u=)Z4sGXc|o%wb}j-YUsIGc|1{NlJ1TZ0NVq zN_UUVo=F&;*_^~Rg0ZH0Y`8#|^)ru@8C$W}fi(!}IcZweSLqw@>rY4= z`k!!hK7Y9xzb>}cAR5j~T8ME~5hc3)K6qWxq8Gt3Gt7ortWrc86%_Me!rbpO70S^= z5~f}GAKmPy?*d0NJYpaMC<&_~4$+T5-DIIC%j0oyas924+(g2qh@(_3K-^|B!^-Y^ zgm@rD3?y3|wvyt-okyc2OhuJfQNF1h!@Q59x?t4(&VHRJwNnwXYwayEYwV#D zpAZ-rsA2qz!rD(Qi9M$)vEI+q&yRhQ#7QeuxbYnQRy(x6Nv{@J7hUEsWcse$VYI?y zd#LtK`dfQ#Xsm${Cxi>L)c3^H%8!Vq)bl^E_eq8O`Jt$*Om2g^p@bOjPI+Jc6a=PY z%{vNXF3o4P@G&?$SdKDlYtwVjKy*<~e0-P-5a=IPGs}=>Zh&{0b6{Es(F!{5mVa^m zMaKi%+3O!@2){Z`4Jr(s`F(`IdVyJSX0X$9g9s3wGp#@XMaBmM0RDXYt=3YVKHqM7 zq!TK-AbZ4zNNV$jW|w>Rn|Qzge^FskQK{9EeDNUz$@WXN)56A5t22~q|II}8)2Go& z?1qD~M+{Q&X@>I4d|g8n0=w3TUZE$e_wQ|Dx@(0`pFxI$_N2~P9D;v=cgAmj^8;;P5|P9f?y3$ic5J3moq5SH6B+ucnk47}$K}&~ zwoE5?4$)G{W{#=}`QwmWL6*m_X*+#UbH}7^;Q(%~B3EM<7qm4H&h+9OToj-%-}@PbIkaA&aqP=vFeOIoy_|fc zKWUy+-$naH7&aCcoty4V^PM{P72^uuw>Eb6af~d|_`{G}Q_@0e4!2Z&1a%U99<>yZ zb!NH25+HCLp!5MP7P{%!nF^D`OOU%jy6;|HF-xMrnE(wMZ48sJ&R+M9;~LMeB6_=)7oH zm>hq;ZUhB_;cM*E@_?>AOZYI>vhUT1@pK)t|9r%iro^F#o?U)*m z#Ka_dT|q^0A=zcVKAXL^kGh-HRnflYrqROX%EY*Ie!o!dCy^_Xi}_AsEOQ%nsLD3R z$$H6_2&)){8;iTK&qp51G60#2%$Vp-s17*PTr+7m9T~@SLW0Rzq>r+e4zhar?ZDe50Jw(^4@$7|IUWAxjK~#p4o^l(90enHo)C-S3>1 z%nveEq_SQ9_OM^{OS)H$`cRWW-6$MLOn6k9~YqKhX>|$CmLn6fJ7&Rr* z8{TR8b$n!fF6GUZ3t@LY1r$~k%qY+g8i$Z5sqgmU-+K2Gq}JEE@y$c=l7>#0O|wOs z{M?+w+6m&IU{O)ISot=&@(eo@m5vmKx!Jv8&lM>D>H8Y}x+?E;ZW*kV!^=w5sQmIU z=dO={dsN6*R_ruzd2GCF;pv~0(EN#eiI4uw13itne7ZAS}%|3^M?nm;gQPC0yx{kr5Dqu`u)I~Bw?cv{?CCxJ?bT2G$M zkdYIc0K?Nji~71~l1_USfjaQG8Mxkdya^5cf#xuMfFD(K68?;(2TIhK@{(TOeUp0;j*Z|FA) zCA-6|YlchP{Wmw2L)XMBY+7C^*QED9y_+vW#Q&w#ugo1fde_LWIMK8QSYIocqPg{S zI#RkL9gG>95RX1AioRD|?$viww6-xwyk^X*5|(@(hKqkTv~6X9+QZ*RFE~C` zZY+42@#NHq1+(0C2Hctl^YHXs@y{C=^^PY}Q|VaV+Q*X*dz zppO9`Ljew}qE@fI)uVrcoyNBB1OYupN%N>5D8+GW1u2+jR5Am4aTQsMtRBL+PgCHc z=p0$a(>% zWPq^>@oHG+S``q7O|wy5JM5$h0a43x+r6 z&$MQf9ng?pX0wE|^flYmDQ37UYx@xI0Di!ocywX*2WaROIX;tcq)H_uEhe-{HtcBH z$2sg+DhTjYVChZYHWLGjkHMHAmb+#B7wT4Z-2u!}fm~araZZn7fEEDPF#}RzEs-CT zvD-7EPV;@U!hhv1hYh@yu2$ssuot9-g7CnN)lP!3{x+wXw|5Q4GpKBYme@-3ez;PT zF=Msed+OLswUrufr*yka|F6&cWRmi8xH-HPK_gD8M4ps|zWYU)Q%M{p362fJb^2H2 zksnVr#;Hv|zjo#hP6~i{B*|z;ZH5n_cPKK2F=n*R`INQ}wRctJUNTPlz$AIO!|N0x z8478<9s=)6=?YLZDyv1w??qWr)b)3F5_k-X4Yrj%o0$%N=5hE5S$msz@Bk}cVC`jm zJ*0)jz&t2wjWhxyM4@_){;R<|h6ZN`hvK4fmdZ5qy)p({_Qjz5B2+S7ed4-LK=&Bu zdIDPz5UDe!9T*ijMI=bQkFZ6h2{n#eDuHD=47&`gXiww9O|Jl`f&_J`#fUZL_bcBrR+5fdDt6|8L!7W%q4 z-PFvvZ$Z|b^92MW97?k_o{DKCc~X^}KvIP}i_@cQ?l1ZnO+3*y(lw0#rn27Pz+;y` zUVE)iaL{Oq(<;wrz(s59kXfOh)l`zej%JH^n7q;vGn^Y!bU2?&N;nD}b)X|s+tZ?s z4}#o+K!5qC{};Z%AD}X$)@|NUR2q=@e`*kAx6}G1C@w3329KtP!O+GSHu9IRboZ+-c=lYq`002$!EYj18P@2CT}P! zHstx3C;X)(oxROq^)l~*vqCD!P=PF0{Uft1@YYaU;vSi1)39+kZEcRq2VH@LPf#EzY(WgXqf7g( zywtaO(xdeCb9L_Sf?TNLizFHlQl>`Bzb3vsFy6y8`iet)pS5`PV>G*JRwEBRXaxC4 ztTQ&a*a{C_0S$Er;$XOF&;Ap-6lmQP9GnUsmLBs9enol^%l8o)6k~Qt#20zepP<9Sv-DN;K zR^vIvP55=<<=EmiG4aiOJ_j<5JIriDG(O0k@KS_6lvZ13zk>Gx*e2@9UU%?Xf z3y!gKTiXi{V(m9j^d~PT#`j3AWZzLQ@{)gj%TCex_)Rj90v1f3ZmPr_D@|)hc=`j) zX+ra!_wwji*j@qOtI-r?&R!el*pQKoek$hEc%_GV0IAmJum0w(Aav>Jgc!wiG6eMRFWQ&F@1ixBl5%kMMA(6& zg?7114M7FO2eNA{C<#CGzjp1b^>FUnFV}H{83DxkvI*P#SnV7u#dtEyUN#X8w6V8p zf9K^qAJKwJXFOz2Y?KStT6lfQYzcoevO`>0(Ow~ZQvxv%oj#1LdrmVR%ROk57CK2SPT7LULxU7eY)Y9WnYgcMV>Wk(r*?3 zeP!SGAP*=?=~5oHkQ{-dvsXx}GnV{wkc(P$Atz9kP0;oc+2XA(dc!jshj|wmqv+lq zlcT9Lvs=l0!6cnpvaNZiBNqOKq!FP=T(& zBI|&oW0juIlos=e`6K?fFnl(e+KgE- zXLWY{375KJgQHcpjO#XUE<0xfRW4;+TZq!-7E#x$+JP2jGtiEx_;8nt%Qw6fo%4#s z?uNt$GAM7b<2?)(30@~p>hUD-DUCM59xQ3~X1OfGKaiW+NZHAKFiucEzlP6LSiTb1@e?u%%=A3&nevvhaXvmBZOF&$5T3_G~yg;k%ERq2MA1 zfV$`hy2Ryz46ilVpt5^6Eem0OF;Kk2kq;+AGpw&tQ6}A)E87Cd0HDiHN0;(~g6H-V z_NI$<((<*t(@4h#_a4!xlsPMXVB#+0AS{tIab>W8G-~t z_W_@#U!z}q0%rrEd!Qs1e8`V~XPjkQu}r}NtJlcIY#T_1t;F2efWJc9VlAO|d=oXU zpC?r{uHoRw)!J}Kd8Edah6MJEGSdb~TK8oOY5yBHe17xc{pZyveu?XH@0q}B#_6o@ z>UU706aS{4e z4^^HvRLY8{ZcpcpoW?aybs>uhec_Rlzw%VDC# zyu0~iq;Nz$UeA^Jmx@Abt2i1-HeScAi6n`TpvMh7>h%M8B7upIBnd>T1SbGP$Epbi zE!D!-aa4C^>Lk9!#R^aiC9*IFPUzfeZfltY>Y|bN&V@np6OFmJ@Qa=|ORa6tlxHwzG-c z{g`;AC8*vp=H*ZlQoM9Bl_Ocu@FlsDBUk->eR47o(FaB5SZu{r?c>>lVQ!Fq-6yw? zpo9mo&LpY_u^ti97;jgN?mn{$wT#}Ut*U)hW46*G;pgT(rqHi?xa?~yM7@JRsfzaje6P-LMZRE5CF$wW1XO6 z6g5sno>TDp76V5S6*5bc2ZL>Q5j2!3kBUQ)$`&lWdQOz@5#JN&x(7RpEa z9Q>5bKd(k{NpVTVywJx}6;a2^rn4_g;ao=`crC~eX)`0oIf%3T8r6{cU5-ya;s@TD(XTWgGuESSJ{vHs`9F(pIiNAY z+%$Jw)T(Azckg{#jP7uo>-7#iNtE0*VY^J$gQuC6N6I%OxnRyOvDLGPwqCBGieFGl zOVdK7f8%Ji*cPb|>U!$HO?OUH9Z_j(ggtm_WM?;~?+I0g+tJ4$%mjzU=^1REW1dTw zmpO{oJ61@JnXiwJ84ot6i>YKiNn6L2&kuPJoOER>0}(t=(IpGinGd@*XJ))3a+MzN@?I_Awi@7Btf zX7H_?AbiXq>Qe~DyQVpp*QRh#W@~K8GX9$rC@L>x5nDhuG@e|lgVpznh~k9Tyg=#= zZSegaC$=EJ&(h9Fb*bdvg3|A0 zQdmsi-k#TA&(tnCqF&5JG`9>LYFUy8)eHMfv*nxbi9upVmoiobAAt*Xf?q`E zv*j;+TVPA3l?8%{xY&#ZViQRyVyI{d(PzE1cnS}N3#@utXYWUJ54)1&A|s)=FR(#L zX&hkTjZXPlp?+g|7w1vo+tfur&{!giN0o!pmYNh%y+y(G9|iOh=JKM8)_6{@zg&#~ za}{?(nByT7WawgwblQ``yz9f?`YTf6OkZ~6Z(+PdbOzTebP~sO)`N`lg2|gDKw@x^>erwwU1L?Ayem|qU9i}D8xD&aCLV^~%vJlU z0S{16h#(p}%F(wIfeH&f0$-e-`++9sx}eo)-t3Y?Hcw=fNms*nm-5iSbW%j8j@60< zh~Y#h7=rZSG#l`3{s$=KZBO6_@mz zetfy=IoTCI$W^NDQs@X@m?8c!J?B5zPkxeAH%JizD(8?!{qn8b2k2|iTL1=AC zVh$}vccIhkjFmogZbpUWJU>q|ierG^vvFuz2_QjWYvsN|6B*JocLA3(QDQW?uM|$v zNXf+J!Cx^i9cb)H*uu65`4C?2O+2I{NJ^{xBhwGC+*=c4kA38&Kf8PMdmIb*kL!D7 z2T6Q#%8ZNd{t#FnBg{^;5FO8h`uUJOvH+wv^oLfY_6vKzwA>!Y;q$IhGA zT+Di9fqL%E6BvMT6!o;CQunEz zFZk>RqVDJu5GM7O*cS}hMWQK&tz|e-F0f%zE!%kEQv671^4nu{eKufe2 z@QfEwRVd|G)V9&w-$Ekm*+ydD=l9pslsdXDoQ!tj4-8Z{(=d}FoAPN#riIT0RuBvV z{c^edL(30sac7)Zxxpl{af!D;uj4W{{EI;4>gN*e!)QI;W7%uL-C^3oc-zBF1Efdi zDr>h7Fb~Q+T5)`vkb0Sj{n2|K`b#;Q&;sE{c~p-0i!hH6v0kBU4!+W7+-fJ}QyI>_ z#6d|`B{RzzJh$!p6milPKc*k_xCTByXd>3oD|dM~WXrHsnafwS2b8UpcQy~gA<5WA zFu9}aL-n{t=r8SK>do&F0g{o?fcJ+ZyRmal%XnT)ef{&(fel~Erd8GWDKf%iP%$kY zn>_Q1t`{cI0zvJes&Ch_Guy70)G73}7rynE_be74P&Op{JWYDRmkb5EsN3+wd*HOK zy-ERnjcaG}-6< zW3|1?g`^*7Td(|1_kEX&2Hp)B{!UhQplvyFd{4p1S$C!8-%}U?*IEQ z$J6Vh7qX|5h|6n}w2S*6wORgdx0<7|Vf8agL!K@0kCGkBKf~0Ra#tmm)*0~mcIuhN zUCGa(Tg0o{9GLHb(MKWIC2mEFLXL<^(&LQzOAOt55pu;xbu2RnpCBRL3JSBEWC*2! z#6*8=F|*hwf$Ba=h*^ZH&y%)_F+It#yw}U^wi)b#!!)NQXaIa6;MnV4K73#dS$-DJ zkU%8sg_dKw;JzkQ{=Ca~QVNSj0S}}qN@T4STSmc5-)RH;I}p69jz;ylM`KH^V`hv% zZ@!@BDvET5SBJFs76Mxw%DG^)ueKMvUM;HR4&wYZ?qwe0tVR4Z1l~g!`2yh@%M^D> z{EgqpaXkQ&#^(|fp^-VUzg+FtX{rMTd_>CP-LWL_c~mbVT6P@x0oo7@fw!%b0x9K= zf2Tv?Prx5HAQQLs52o=1rkQ$Sz-NGxH%j)fi*WR^JBg3FH9B4Wc|9E~d10*=lu>U> zS&-?LpPgdYtZS08F$Sdu0yNW?9KTfuNTVG<37-57;OTMwmzQPy9F&B@=$aLef&p-u zv8bS^c&%JCG$NGq&pW;~FK&f5SoH>aU~8R~M2rBjr9hS3{`6w%T4_snXEdG5STYIB-(}?$JTPN`2?vK30Mpz2Ic_4q5263ZSGFdFfK>z%ZJkRj2XBDT!G&L6ovNq5S&Rru~hvXWo--~ zm{RR0ll>y@TPTV*n*N42#!%SK{|&Y_s7@dJb*I(&3P#P({(9Vts5Px?eEtSYLKNQO zf5Vs95CmuDzk|@n_iO(5?GsK)J$^dkX^Yf$=?5?H;$Wm-M9UXFd~O5O0{q9ZSjma? zUyjI1PVy}xP>XVc^iGiVydNrJC@Cn-3{1ouj9CP=0f;gm^bLDX>qZXxh6R^aP{&0- zdh?`^cbo`I?6Uxd*X%6SoaYz*4`?h>XiB2~H2q}-oY}lD`|jOrn*^gfj~nK2f2gRq zh#u58r%6qM)+Bofw-$O0_2LLE8`<4F-lC{(TX&LyWa8HsEyYa->uwIpl~^2}Q;s`I z8K!W|NNIG?nI}x3_4AKFT^>1EekR))&JI~)g^(5Fn|{ne%ZRO>(ufya4Y^)vX=>8v zO&qd1k=T5(zZkY{Ep%%03Ka?8v3qj;qNz#bO;*ydDwx`vu>8rvJKQpcOs5a(h}nC? zPnsHaIa7D~E8HonFG=E`hf;w7y~quekHEeI)e_pOwp~of6lHT+{Jg?s;b^%}ry2;wdAHxy|jsiL&4OaSq-J$qy^l;R_BFH_%OO? zt~AU_bBJCARyhfMDG_HQs7uqGNr68%XhR&3G0)H&R7DBKhAiMa36UI(^mX<^GS+By z%35E{?Xon9y{~X9cofh5rJUKGO+?)TB(!EvFP>ZCmd!mq560g67&05k(&h;g_Mx#5 zO#^{$3Et;$KB%m9|72BQd#+(B5&B5?A#2YjzM@sy>*nC4FRTw7O61sxLQ90@9MxHR z(^86Qw8B-CyyuQX5(#NI2tXSClsR!L`(|_}bS_is1)ruL)#C%-jEW@R!v~zDOVEBh z`!dJVTu}_m7Gb*B=(%Z*E6Ecs-gjkzWyUfzbjDMXK_Ee8rr_92{N4Q$-@L}ImYv2k zRLo>fJPA#?+y8U>F}>kGaa_Bi++5SWU_I6SqU?7pCVA=6 z?xba*uu-8ex#58de_k7Wnb=oaaX9(|?MWT$b^4>Z3c0890HF52(un_?N8SI=)bPJF z>iu_qKJ4*-w^XYa9XkhVF#hw)YrHfn9> zLd3=*m!bhFGjKBowF$KdF)(i_lEPUzUbtbs|WF{a%MvU<7#25 zM-i!>i8L3QWJb*|>E@(V;LBoA;>bvqBFf0nl&wpOMU~C9@kgV_ z%#p5BMG0x!s^A9Ya&RI?t!Ru85yePlOUiaps`t{D#?sajJ_C%q0`4^9kfKB8;`~SN zH>T3bpltb=v{AZ3)7~4al@25pY8C||%#=A$LMVofk|KdI2y*_^$|~R$;bduBJcmJ3CD<^*wVDLyQy)E^u@gmVxkcv$z{(ByCoIN;b>Rd%^cEr1m7?My zQO3RT+h|6v7Py#86nEanzjEZ~P(M>Sa`Js}!*AA4H5=VCHrzOYc*!{vqknu5nqe8X ztq;8a24$Uh#ykKe9i}tU^kGn>sqY(m5e9Tmt7BV7 z5R5ru;~cm+FA|-0cCUs>{9Pc-ULkB`LFa6WN4G(5XS|-qxlnfq58T-|;ww&OMlfwR zeoX_|-#n|YN+jp6u6dl(+s^+8y7thK;i$Y^jITAPz^ZL%^ogn1!&^$vyvo>Iq4OVb z4PKZ%cpH84p%WAj1jVY+-nL9*Ln3EXr+a!(={nf%XEPXOBMtB3el`$!>I-g3(Kx zen!5wvufRHv|o4k3kDP)Kt<0o=4_<)L~PBCBw6F%aEHf@twqAXa_*iPs)KgUA9I4Z z^g7sVSf0{jb*1YhyoRwRq55Wr;Dzk9GjwGH z?ZpQ%*ML+1DjW|v zU+)gYFt(4g41`2lMp4W~XzcBU@w9V-d9&R3DVLo6%e^yJBf?O|PcH^IDaYZhAhsz|DAVF85LZ2vboivRUC=%|0RK+l zN3&=FkRl79Z^^k6LOiC=JpX=eHK9?&!y=v{f4)M3Qjot#Cf%Qfm0{Sk$H0_E3Mwr6 z)L~gaRmLOLPTj?RzvP+FSDI--PimwyuBJwq0-tUIq>vWBUYEGRmTFp@$#OhCEyfga zm%%y0;M?Iv=QPZ<-CJJpw&#R2P}K#JejrS1BXy0Q{ZJ%>BR`?QLByLAyTV1gjMT}6 zmwZ+3E69lG`=HhPK1KQ0_9qi6JxA=Ghfa<9F!yyrMD`9~Ck_jwZ}S1{4B#XTj)mqI z-DhPLvx|V_$D*QeF2{8S1DMHTQP$#RAkIS5f97GrK6vNC+zb2+8%!j z1*HfmF+TAx9u1W`(SJBMRO-t=M|rHc5y{cWrq>|`W|X_(`659iVlY^4{F9E&w-uu= zrbUK5IR!bih90>+RkdW(qUbk|e5E1m_0&t+DA0H3YPyi0a8TLSx*iEcK5JOpI|@et z-(r#%xd-5tUI_b4?yGsb>O>38)yG4G86HoBWswJ8GjkFkw!po&S#f&t@H!TeP3@fe?lIXlJ9x=OcR(o# z_))tgB*`mS!;dXD4^Z-pHmF@x0odVNaaE6Uks-t{G85 ztC-BTG#8cT{PXKKk9|YN>!}3And}?-)rplNki+(bkbQu#5S{i1KS;&3W+B8g0dFs} zeIVk9;QVP{J<_ZrJny9-y{L^1%&j|}`8Ild#)-^2F$6C?d%<)#(cvXNBd;E~AW1j9 z)H+@*x3&L%a)QcI9DG|@WmkD?O6SVQ>}akbbmIlMbqFAaG_wxcr`uLlq&b_4aMJ`% z)!5I|XkzEd6E`IqAfE-%ci&o)QhRw?Yhv?;oiBAD;u&9K|( zp+MWL9Q9G4AoN8ySQWD&Ha}D#9anlmTJ)*8!>Vziix2*gM;p!1j;u$sdi0mkRjjtd zd{!G6CWUX>Wa#9C*Ihk$r@aMdM|n0#}kKaaTWONXg{D)%DQC}bq?MbQ(v_+ zL)sajh5o>CG5Ydi*H!V*=OA|iS>m!wx{Wo`)x$0VU{hk^0CgtDGD6?`|hYHmu24}Nl-u$P@*7l1_4EK4kFBu zhnz(~;s67ZqX>%RFo2RI=O8fTq+|xkc^GoeNl-xa&EC52*=L`7&)IL?x86GUz5U;; zuWP!xs=L3hy1MFDb(PEsTJ8L`iAH={h=P@6dMCwB#Uql{!<)|hHRj>~cuqmpvlcHf z;U1NB)!}n@Bzi}EvIfnlox^pNwL7eO+m%~h-bQC4KHfD@h;(XBTf4+TIFDWfItoE# zHO9KfVPgb<|J)A8S_gk!wf+98?H}UO{-{BasE@*IG`$V8oE1GhG$!YIx~zw!X%D%z z=7+;?ir%uh8wMB%v-mCE<(Ej{{HZ^yM26R zB>utqXG1a;VD0~*A!W|p$;X||b#&;C%ZWAVNd&m%u^0K4(4$JMXY`nsQ2+Arx z#ktrNg6-5|(i!99oSvgAiYt41_0@*1Dh+HZ1z-~3$3S96C-|zs1fvpH4d`|&6}=#o zMtBmB{$3l^(Ycz9)9CcRl-U=d`JyT}`TQ;JnIhNLyx}x>PrHLu&z{je*XXgTUW?jS z`d6Bl(`=N~G&ot{J5&$H>Upr>z7F;>E$@0ABjbYg6xLpyNQoW}J_P}pxITJq6{9mH@;$F0*T1ZTty zt8^d(BNH%*K8xFewDY&(as6u+?f$P5Y1VhQgIyfbGxS)P=8vz+Fm#sPPv4+Z;HY#s z*xstIZTQ&WBa~CSBzEUTY&OvPCYw$FJiIYwEm#TjX_UPxes=y7r!%(`lVRJ7#qGRm zV^0L-0yo8C53^wfg@kOt%1%e@wr+M6!&pVH{+V>BYzo|36;ue-f!@8i@H%oHN+0!m-~6?77bV-Yt_K(gB2|{|S!& z_v^WltN0n;%9er|m$i(Ywq$&jhZ_7LOqq)DiT@AK$h*;(w4{$xv{HUX@tK^`d1@Gq zJ0~?%S-GZ4R@J~=GuJ~sPa1uriN)cy*PAOeU+=Ikkr0Xt9hBA4blO#Sj#BrjpgwXL zF+SOIjJLe&%H0EhAh^PeqvZUgUtD%@lGy?`GsV2K*qxhYp;`0hyZhI8HjJfg9gLe^ zn8K=N%<~D@Yz#tW;Gc=nA_oBVFHGw6dRHGVk|)KF)VVx!Ha9V}8mKPE@2zNM&OgUN z)+PkNi6P+kU?#HJOpo!7rMN9t9uSzRUZGC@@j z1IRcRSULAp?{bn|gKS*ACs{y)$_7#00KCwoK>y`}bk2EC40Ta8>d7``f9*@@Fh}#j%%Qm!}dtB-5F!=F= zQR_41cH($YUEfm&yGMeuA&}M{4DnI`;ce{q9>lYFl#jUjk`l4~JOB=jzvj}VN(Y&u zW8t8Azf)LuWqgA|(s;mnBwO*+T_Vf4NXOYb3q~CloH=4)IFXkp^E_`(doRUpGclHA zcD?6o8eb|3K>ChMu|z^M0~uT?SEO!O0&C!#%BQL9tNzd=Cc-BeU6DJw#vk!gHl}jV zTh1S5-P7&$AS%%XEwDqT>|&R!xitlbRy(^%FRBWYo$VA1(x(~;pto^ps=(#?tj4bR za1Zz@^vuoe@yk%kY!@@RR0F7@m^A5BG1_E0ecskrb~ibV$8Cd_(YxlRU{6uZN8$Jy z8P`QgNc{JLh4ONsc{V$;bjnFK$og2+jCdWVuHR4f>}YpMp0rs^ zIkM~48Nv^Uf1F-M%c45NO>3yG=WrF-sI=^ z3`xUPV^~%&ClR7QbctrS=25wW$yKeyuxD__**udRpzs}nn~QF3kQgc@(4hh3Oa(yn zKt$+RxwQ(AeZhMqc0002|DKDR7_F`4UZ(I(B9t8YB^0YfRZ7kSi$?)2($0L_$P5s4 z7NXt68jIW5RoeLJ)h(Viq+L0h-?;iaz-*AHvA3atw1Hr8gPqQqSmRo@u;=Rq=@-?g zhfZW)(*#guad%vH!Et%TWd^sYW3N~}DKw300bPd5e3)sd4;bO;g-}#+&&OL?K8U7@ z6qb0pW_e`US5u8M1?qV(@n)yABNyqL3^hExS7Z9bSoe}^{mD^ZCYU)8l?qh?ip4Ym zVm>HD(T0WgN7o%H@vXMG_wu~`tO*PMDkF|!&SGd?>JVcFxdUPvI(qPK2drG&R_LwU zp?&(R6t<@*6D)cdP4x_aPP0h4GW=x8__6VrM1ePfSO}CN*I1Fo2H%G{qP6Zq-Qw?7 zkKTs#|NIBz7x8QH{^9YW6(o@|8ruRiP->~IE2hIz^|p6SfDj$hza?<|qmlU}45Um` zq8!XTc}S}>?Z{sdoPH80^j!0$=~fPF_7>d+A)G;Ap!41O#XT;)Elw{l`WIPsEY^vs z_&LN72*OEtm!2#4em4p0CUJuB+2-xtt7zETq~PaEtwx8$VtR9tg?N$o*_GYv)(;tN z1|rcY?df*y4)HSQ06Nn1bVvZ5B)^W9+S~RyJ^sk}`<~jgej8L!+MuJ%$Fx1*)#?|- z^b}gr_veg~C7yh8sT3d(CwidT)8HjNZC$ zc8lg-9OD{y4Aqzvz2Iit$J$~Tx3Cz(iBExEtEot4(|aaO0yRzi@=a_`rs$RG=q;2? zYakso5FxWoWn`c=5{mUIB4gv(3`1Dp`07DHsWN_VfCZYox`cC~_e=;mk-6XEi0XzNVp0@iECpr7w7EzC+nK}bQE7J&9zkbH2T z;=vTzI}@$Ikw_}m`ix*+!i8(KY3$Wgqe-FSYfrkGF-fsUV9T!L=LRozg^JjJ(?-i z={|b>jJuQWEqyxzu|&?VMXByAYzY)=;Idhz_ZvkS0`l27@R#Pt6Mr1&Epipzy;Ga# zB(DG??{<7ZR&o;oT)Ig}h&gCFrG@dW@IPzN!SoSyx<-?8kUmNFm^IwX^>Jw8T1_>pkXovoH?l3>$CYgw?{xs<>ri1 z9_Dp}!0LnhLO#(Eu?$SH#3c9@6pBn1R&c6Ax1!NR!BQ0rFNH? z7Kq5^=EhO~-}^^6i3wewWO@12mCno=y^5BvmJ8zG=_H28DY(Z~1Vv1Lh8J&5(AM>F z@4|Y&)^Phw+-2dllom=kGJ62SUz>j(8uE9(6`T^t)E^#TO9=vf7GoFlkl)1&HHO61$lxP!1rQP#Kg1C(RZDlNo`DzFXfC0v8s&<5hoIpR@1;#vjON!`-mJSHWt=GF|9kk<3RwefJunQ@>rhy1Me@6!LCd);ldw<9*ia!v?IG7VES&ri6)Yu&`zLsBl`SlU0 zg6rpGIg}yHDwz;Q$R?&h82L?L^#-R;`gBr4h*#wD(CfqH=2eu|Vya&2I6V{+n|bLl zmTOP?&7nD(a{aobvbStbD9hqzsv*Jj{dc<)M7#P>WS*yTpwd z9ttSDToZAcpze+T=7ut~Iav)ctL}7)2;!HS1^4d?eEz-Yx=vZfv<=Xq?(kh!{@4V}?0rO_ z^yXY^PW`W9=lvKihI4RKG*5f8No6?(*mTI_&7RUbcYCeE zcI@{W$CiH4edE1sbfO}X*epL_F;?_@Ik6RK%6>3(;q?hi>nYOxy<(GtsZ4%l$p3@8 zPU-JgoDd19@mGd6Ke(3|{}|4X{v9Z_J%435@`Jme@Afzm@RBZH;Fn_ ziUC=T*(bC~Kh#|*|B|gPq84e!v?nH4ksM5c%;93=Ix9L-Y+2+w(Kace{g0o>@N#VB zuf{_uNrIY$%NAjX&P4@7M?Mp{Hx9@U0`YMuS&MMH%pNmxlBT&j$1uS<9gT7Agu8zN zPUr+ehOt@9GI%h!`EuCrw^PQFsH?iW>MmW2b-DzeiA2A4>nY4+i#{$r?0)J!$So}R;(o1XeNk7* z6mx+@-`lLOLI>Z5CeNi@@^X`mT}CewbF!o^%b{;W>?_XiqU9orLbWLPNS&fq*tcxQ z5-trdY9VEx7BDU6@m#U06c*_8(rQUnJlFOUh_Evl*IFE{Vq|*|w{f=mQ zQ#_att1QJ2jb+wofvje%_Iop_$DP>p@GBFoIEy%4)`b?^qEIh_x2j*reA8xZZ4Ea^ zHFCrUK{D2<0MNOYg533X@-T( zQ*P<=9)XyGI=NXa<>@^emx^xQ%%6ppl7lNcA_T81dETdozggX3vsGWO2HHKN zGa_3sn!pq3{D%I5tAPJ(m68zs9|1%EG-dTKd+Zl`th?@#KzV#nFa;v~)=-H*N!=mm zankY6z$lHX*jPp?mt~DD{{5o2x1;n+=X%s`)80Tts!lcYYM3mk+H@!+Gi3I+sgllz zXNj?~a$2=i)RyK(SiCISz55AMz_+6;nVY2;t66=h*JuvQB<zFZlFiH}V}vFNOvrdw zoQ><@Fw03K^+PKj+5<5P#iQ|+vYQQk;l?}T@e(vY#zGD$ibQ+_mpdUe30Te!IX z7PzLUv=Z&-GW2+QzRt;~Biy2S@v1b?CjBA0@sY+s1zQ{GSz{Y+hPN~ z5iU<~jf0tAU5gS9=H%ei&~Vxg$B7nt{`{GqqtV;V3kgz+(i7La!*7)>fxLoye4Dj` z>$0z4R^@u))1 zt%=G1?q9x~e?9nh!~R^F|2z8VVO(4qRv)qfa30`~8vof0C+bp_Pb)3jhjF+)uJF!< z+C&BqWJx?_27w-WvtM{ZyT&^0!EVyxlVr1rm;BS9#|Dlvim^5K7X?K}Q>SYtK+8)S z?}vS|-#5+B!MkBOooS>THZoi?s%~BfX;fn+=%pCLC$(=1Zptk;jsTzBw*SBxo_n-Z z?S&|Ym|ZpgqXv)_G0rsUw@Y)h%O&@t5$IbM`Lt3!lf-C`wiSVBg{%7 zyxGB2I|qj^c6Tsbfg9ZZunOlG-_iRS5H0T{3{-s_g_s?gHWy*R9ycre+qp@W@bh#H zUJSbv0N=Yuwp@4h`8Hf-^z~lV^OIZ8)%L%f=#}`YRMZ)?XzO=6DSrpFIjFeGYUWSm z>Kl{-DMG6;)-;BKMh?+6JkTjh&k8-Kj(>2jL0foID$@_bk@2LX7C` zYDTE1IC8_zwPeQ%2JbG9MkPcvQXuAI5c{QKy}8BjY)e$EP2SdKx@bHuA*)9#Dnb*Z zahR-}%F?5{A!P)9JsGeGCGN-vO{z@FP2FyHI5&gx)OCV`zVt#G)gdn~;zD1DNz!UGyH~bE?N}2}61q~Sc z*d1hNJwOdtQHdaBPu%XNkTrcN=(BPgIZ-jre*&_f*sZjO z8EQJ-+aLDq-XX2nb%IjxT@PQIkajVAkua)Lq~tCZpYx^M2uKWXjrK7^tcajhA09-5 zZkHJ(UF4P4%WyZ{Z@V#Cf~bTG?uHz}*o2|COoqtm?SoP*%p)|c|71QAIgg*bf2O~s z-=~`I0J*mBfEv5c1b(DUS7qkib-hOXQH=bgT(qQzZ9K~BA9K<_SwSiG72k8%d8q;h zXv6aZf1d|cGO?q?tBY}^PESJ9I;#m(43o)OtA!#0J?6Ix&SVaN18{}NCzJNM^x zpkM`bDEoNl{2^L^puT@!PB-89+9H@B>n)093qL;ZW91~Mto91dvnhx=_0DBCq~4l{ zb9|vI``cPU_N#>p3>*8RmNtR?W?*z#)~(eXClAJ zqLu7mome+%RymRgLxQ*$x`K_tpUFd<+rw^{2hEHxS=LT2HBcq8=r5{l5OG1H)3?&Z zK&>dPOA31VTY^m$A4XH&sKvetZo1E)qDo|a9ho=dbR;Kvw{>L7XdFRkggne}Fjts* z>;YY{sMA+Jy3;^CK;bAEFPw*sB;JceH9E(IKktfAN2W9wDIKJ`HqUOC38vPL*zd4) zUEOtGrb^#l5+|Qgr%V?UtH*@LTO z54%PBZE=@S_+Y^qz1|w)39t~3{R0{brmCC3dqyCjAuJWaO4@@bLT{Hm9I#d260fM6 zmiiQFwD#_}RgS_!Kee(Q)|k;=3$3&M+(ki|lE)%Zb&ErHJs*RVUIC3ZOdslcstg;~ zE4vNSfZ3EF441_hdJdc^?seA1bCSi->Dx33J@I2qbN2199v$2(|2SFjh?|tz+gk6k z4%u)3y)&P?Q&|nnG;3+f6_57mtVK(Hsl93}&lP55o17zXF$j@=R0PtXGvQfi^L4RR zXSjV_x}8Y32OQB??u$^>7=)k!;fiu;r!5@^P;d9$QpZ1smGfU)`2PwX>K|_xgs&tW zRYxHdkf6`qUx{w;=)q!k)u$5~h&&g2L#9cfw9vT4<2`z=cY4RL&@EKZG##4MrlzEJ zJrcc5q*PpD5nQk{1tHhqCtO{>Yl$gUHk`kkx7_@08paWd@G4wmi8VH4yyHo1T(xTA zB+xI!)h^pJg#%$>)Y;zeQ?q7}x@Od9s=)w>>?@cuYEz`S$pcL1^LRubb+>echxXQB z$cXed-^WL-CU~6(3iM9>+Y@1?Roo&wm7m_e@FV1zm^z=SffXSO6Yh=AN)j&X@8_a- z9j8n0%|rl$6J{)jf|=#=Lnr#Zwm!=zST}Ce-muEzf63M*hmxj~VUrWa-@9E)ei$9W zkZNq^no9?BXmv?}QsC&&-|QD&VsCmib=sM~h%CCORMV$l9!mZQli*cH0X;A`s2BQp zFfy%t-C)qeyO|Y4n==}MCD(2r z%-Kd{XA>}db%N`?S>8FV@#eRKeVgXvo_R(7Fy-nJEU*;}=Yj-4J>Z=}s{uK~^^C6v z_vy!JVQYCZE&4oA#>lF7iF7!fXlSo339!+RNqw4 ze|zkU`pp7;uRCSQQYW@wfkT!)Q`n3m-395#IhN}c&Gz2`Q$_p#{=Iq|@(SX>^RAK` z#47BRQ4;31@I(#YsxF#4ev5YGSf0kCP%MeDb zwNWA$%rHg4pwnhw5m;4x0uYMRd?{_0ot?kFwT@SoA@W6?W?S@R@6lKdgKWD;RCg^^ z!6(t!>q}Um4JUWU(v0r_hNq}Et=sO|`FVtvor;7E+MV{)VA1EA@)cbM2Cm15_*x-B z02@T|5!vSEbrI_7s$8?VH_b-Z31;GjYs}M&k~!$-6~$g$l*GD3o*mh?$@g|v_@=d$ zeF=JMSJ|IgMqN2JcC@Lt^A>ek30*wI9rI~rMdhK*o%E9N&=9QE*IZWv@=5Ppxpd)} zNQVBz&fKNRB>pbvPJ!WrUCgCaJ-8BVuJZUY8Vu9V%^Hm^zF^+~d2HXw_!Vzob}OXxG4D z4-|t~^{^6PZ>JVwpxlQEOKxtN$+VPb+$5DZ9!%gF4arCw=pRxv&BwI}Z}!^;X+=we z0BL}vbFJWd$$HnYzUF!Ei2SzlWF`f1r*{U1KCAHBFQ-$xOVBest#@I&Z>swp?s%Mp ze7@!Hi(y+l;$jB%-)qD_m`hsIO6M8$l(n7UX@$|kL)2kv4R_==l$;umO16OzxEvEs zz$GE`(yIBbpsJhm41jBHbvT>O*f^xTLI4YoTt-Mpt9c z#gGQo8{a{UxBiLLY}&emMDh8pJq@j*2Wqx9zen}{4(J(|9zQ(Z{;rE0UI0G ziiPnLgd{pD^iki4#J&T@dHgRVq`m{3Yj?}#rxmyK>wTx-%db1wc-b;jk=S(fB`i`F z$%U_JTpbp_Ikm=rFbW&88k!$oc%D&vQn~HlB=J*(%mSSd$Ui=W;Fs0YMcGMqosaw! zVH1Q%|EGjw=W*lQw1K;AUXgmQs9?f5cf@JvY~}@{uAJ!JDXr`7jS|SuED5uz<$dkr z!fdU&7ZJZZ^VOCwbBj*#(0MhZw$tq!{uA{cGm~r5Xb_m|pqZM8(YDWd2$R%R@zIa_ z0u()c*$}*+Y=>$ED|VTDbtqz`ap|mR%(q@CkguImkfNvbDC|6Ra_Mq^ayHfUv6f5O zD7J*QdiZfY$;Xf`3wwqygkrV3+xePDJn3_J#-Jrc>V;tkbaTW1Td7!!q{0rBQenuz zzl@*EV0=Qa|5~Ew0*y4R1C0u)IE}v!2H{1C*~!n;u|cks#8*>Cll8m3Qh#`F#bGaB z&*_HC17Q`xJO=kz>Yiz=A&T{a7v!=TI~jYsClCvupS|G&=JT0|P=s6mPx2dYjN-Q8MM#1Q?KV|&A=vz&}o->4RyIEpEW$87>U)=+I&BKSW!m# zmHvuTvD*b{K}lzKwjC?jxmUm?y@8L$7gqZY_Zc)Xo@q&p(9PMJjIuwXF7>+ms3vrq z(y(2gUHtWyY<8yXIHCl9se_KtZ{MqkRf5#ZA8g`^KRH}7v@?i)Jpl<_#BVQIuG+I@ zBAtG;=C`?Xd3pBhSFRNOq>p{_eXER!0<8N3p>L27K$5ta2hxEGU4pR^eCKdU(4ZGrP3HR&eZOP&%oJII@*- z(eAM>(thp{d9GJx;|I5IfElhLdR89`H$RGG_D~D;j(GgI&m-R2Ym0V7V*Lue7Zfu-YQl`By;sC!3haUL*QSF3@Hx}W`E?brLvmh@{5W^; z#tGshpw<#h41z{Zae3f)j7TeIdhRIY0Z}w7IERn zgY*>f)87H}*QG8VNa;vz=6%Jr`VPo{x!HB0MR+yG?K|KL{=j#D2KiXjS;2Pz;LUfy zCi7W2wk9l2JD1L)Voebt-TU(`Y?-I`?EYsVe#QP{CJf)y%HLX7uVx;oUJP}A2kfN_ zvlS`Z+HG4|=zZEjul7Qw;N9vNUrynEO7*zs5$Br%YIgraBQJf}1O5*QJow+ z)~i{*SSr!v?y!lynu;h{-id^JbZWV$Tv@HEMfh1%>g|b!9?s;p*~i0E=Gm?-vDjGP z#Cl_KDa49Q>}e9F!%j{QCK$EcFRSsj0ITo<$wWK+iYpEu@4js|DorS$b&bcEzm9rm zKdtNnxoJQ#?{*$k+S#FBs1f;?PL?k0KH`*h zxJqv6C}maOStFfxQ7_A4Bs)er6aET^WYVNlD1{5YSBCwBTqAfRxON6BXd|G*A)9uK zVeu64ft_#6g{P$xYOCtkiPY=Ol#7s^!<;6QqQ_dYr$q+4eJvg6srGchSPLT>n?`dg zOAkz>*^#Mf_cO|ie>ZY%nuoh>5)rX{TI9T!CqVXA6WF5XkdV30mLbX&?h;@wCich* zi*RaPHtg1~kZ3tE;>fZc`4u(z}wTh{lo^N5hyHsctJM2tj$x0wx+8&tE z(%hn~vJAblLEfmE7BTSfUg&KDk?eE}MmS!|fzi#*$+_@|<7cI^3zhck;8Z)7th$?x zMGnU527#fdlS7>pGtd{Nn21jp&aYk1r<55{7m{nt`wTUi%x<00@%)*a+?G5=$*Om? z=;wBp)Y(NiDjI0^c?4;tfQqT|M}0UJj|Fk1)$!5Ort0fcULn1E{Qy*S+iw?qF3q|C^0KyE%+TQ{DaIBm3n56lG=~e+&By{{4%Xh$s&bjY^ zQP4Sof9czAWP;xT-!gP3xh`DYNu1(F!K{=m&z{R+V=`c%m7f%Ob-kuNO(!psgDKJ4y+0|`ToPL1ymv^2NPx(q#Ge#t zYux_qpV71ql$tiWygw-<`VGFR>>;tk$sH~=`AOlOQuQ@?&Kf&=Dmg|?KPi42gi~hn z-{M_6evYEJyx8qaMGP}SnEH9NQc!3DXG{)IABa=<>>kUb>wBT|_s6t-gTKn3vzMk( z)Lsy>>=1}^IG>TJp-J>I6W_Ijv{;Oo`Cs$vDas4lHI?Z(mr*0eG(V3!2`pWN3DsRk zJ0>LCqM$^N!C;pT)d{|0$IUroJ5`h8^OTyI*U1$9h^HRD+|S?!Zz88Sx~y18^l7?F zc^q^p?%G4iQ$LuBvp98%<9?G;5x9diLbBfEJ^Wm_47S1V@`Aj^CM|(AS6Cyn5U~;h z6Ba~Kdg50X(ORsPeQXis0nv8}=_l515n=8SjS;8GDZmIMLiQ`{+~`I@bjC?J@| zeRcloZ2t#zS2GzZ^NXziAn&}=x{T(Rb8=Eo%tM{Jl{2!4RBtJ|lH?69*DTDk| zzB`>4{eFUTdhVQoDk=~MhtTSUOtHD_hIPFGbDhallvBLj4gBVY=Qc!eFbI|wRf$Y@ zqR>W?sgaJ|Sm1hfQ&5EPJD{KBa%#6zdC#qHPSBnQ#VQQh7E;!gWvxB6nC-_a!_U`O z%^UiF-UM_a>THC?*J3)eOovZ|JVhSwSw2V(YBtMJ_Pwv37POgw8@v2CqU2c$K`{MV zVWS6R=@VrXQbn8%&cWWn!gBsWiPB*f>6aWXAdorP@Vee+eTLcPmMh6)Gq9ooG$2}| z$rFqg!Dh&cDO+FmtA3TY(p!Hwz5?@dwf+nb$Q4PQAVb6^q7SI`iH1i^<+l)F`~qVd z&f5o#N?C#^yB7}oPfE2ACzF4~g2$+r?24FJwyoDO_B2w0% zUH0LBx+s%h{MG)?HtnC#PYVu+)^C$UJ2v60VUzc_y<``i zv5}wq{x=&!rOjrcNe+NaEoTP3?_Pb?@xX}T8J}~u5c4t&2T>>*g%g_e^OX<$`RdTB z7&2U4^x5$*Dow$trIsDLj>Y|lH}#tFr3|uZm>M?_N0s`P>d?-69RhEx<@vJl zg535Tcx$uN{||4s0&8UdGc58gx#!vKu@k7&m{AvW?nYsia>FqSs(BlXpOIfgj1BDu zSmgt_y#FAClI)xR!*c!^UHloE&9Pk9fZ-?s_6guJ7)GKXoZlcF49D1!|G_ZKNeGiO zs+gmx?M|8LG26f)c6tqD3c|hqECQqA9oMV<&4?pY%2~r^8CdlUhY1n$yoIiYw(z(4 z`E3#&zq#e!C?9WYWZmrv(V#a30`QVbetl;OJpGO4r{19*l&jIrzKpjUGe*5_H?V~o zKVl0ta&`S2m%+%gQSP)!=$+xKMymZodct!$9n{8BPTIK}_Vvv43!zvnu>W6&u>VLsC|jc^hE@A}1H0sczsHwf)j*l773fz+bvt25V@a+eE% z)3{dMTfcl1^MZbSWR)oWlD`99iE?a=f9n|dH1anVrf;xT+^@FzFIL?zc2W6X-TtM7 zf6tXL5r3GNsXnFNM6fiU@se25k-FvnZ_D=A-Eqqr_&DrIQ4@>$=4s6nv?fe6oldqk zC>uZg^KPGAx3Eglpab@2}N39Mj8%TknWHW z>24T)dq%(a-uKUU{BY*Xj&t^$y`H_EXRSS+4xLUxRJRmW6d@Q4f?(hubUF&jL4da&k_?85fPIi&XAm?Afu$9AR{NIq^3VdNkvCZPJWK@9NqZ~3=9mEG)yc^ z7g*>oFkHZ01O~bi6A_aV6O&$`BB#3W|NC{?2vH;8A;f4ni~}N|hQX;}r(Yo!FjN=> z2L~`%{0jyVz=?CE-vAe7#@+4j z>)$=`7bPMlL4exp)L;ZK@Vww8gfODNL!u^t)36g>yQz7P=v*K%hme*_+B$9Q$L{;| zoLo)6PDdazIOs$Tr-o3_5skiRdQFg2Y^GOI&+}C}x-rfGBH9}tl%QKzK2>UFBAGsD z>p6!L|5+{krHmH4vzn%iroFQo9^jZZZt^)Mz_IX`rY!z#+~i9jG%i=ON{h5|W5j)e z-^lyquIIcX9MHMA%F@RXLyUXl3?T0xGy2dCfv$i1n0!vJ2~oaY9p=W!N63^1Zku)% z-@`gUOw%z#1;H?`P#t+j(H-2qF~-V`VsTZZulSB*`uH9r2i%Q*Q5cQymCg&A7tZK2 zQ&}p|FRltJ5HIrgkJ%C7Z?_YyWGv>J}}&F zS}XT4@U9#HZW-dNnGA-S_i2!(307Sb;XY=K`zGL;*3Ylktg>`BP6gk{s;-GN2gUq- z*42Ty<{!X)`amcA+lOITWS`OBW8S9;1eqi5re7c{a95EiE@Cmw;~%FG?OT-YmpeV8 zBkd^NP!k7V@iQvm~sf zmUPa6($WLIsZ(xq3eBBD#1F8#4rZ%eO>uL!%&Awby{|ON=>FW*3v4-st~@E))0=8K z5F9B#5}W$gm3HU%fsxa45y620IKt=sZi5D9ji|Yle(cgIxTcbw#$v_+)K zri~nf;`8l!QrnVY#W(IW+_&5$wMDtc;JP)mzg6EhyKH~d-FqB7;y;#Yxh&9&onZ%K zz0r_B@B61-+h8vv!<(|~6#AqrBkv%#N`o6`-GQdnB+Y?_NJR3izJ&5{w@(S^8S9HSDdtmCCsr>DP4i zfV0eh`v8nI<>eedrYr(?sMQeJRJCqc*dd)w=@wi)38w+t?d*=~D+3*U&xREp+f;ZEIZ$~eA+bSt&ZncR>YfbY2m-KzFq-eM(1fqRhyKpxV&M{S7IrbP@kCJKvsd^@5{Fr)mLTnv|~I zCn)_|Rs2!o{GS57J;uq$b?0;a=8RTxkGi*47hjJ%YS3=~xVv{5)HCH^|J4unisI|( zQToy6{;a6@wHU32W$Xx!E%|mo`ct-xef9CMj};ugdZb?M^M6#Zr}|xlZ5)iQW+=PD zt1b#vru6Mm!BFPLT@<`cePWSxyKIhf<6>a9n$KQppl8ZYZJCH4H;ql7Jg0PXy?x{$ zqH6t1O*~n){Ov(W$39bJ3g* zsItR84`yR;Iw=PKY3b*n=F(a%ws+}{=Semg=ijo14b~qT#3mEpq}NgnPizHc%0VZ- zF>etm1}D8;TR*O`yAz%(MFU@JcL$g5d+;xw^rL7d3dT`ioeaGvrOOY8!cq)R>@`Rl zjP_32|L`Z} zHNVBOr00`IeeLBs3$PVDAuG0f-@OgTx0nV}mT)E9^SY;y(YepN{dGAQJEM_T6L-fB z`Wptmviu$Uc2`DFV&>KkwcRoQxb6S7*WHOFSlSQEKq|*k3ghJ^}2*f+#fDj_+ zIjj-!j+Ovr&WM(aLWU5c(aNBjy^$*B9T$>|`dwon1t|d`D2qiX1ah|qrST!Csp6H< z%Ir;N&|GM_AR0vYEq1aqgz`vPeIy4vFJVYf6E&?|P?LEOJCcKgfV~Ok8^T2>uMf^3 z3IrOVj39i@!GR19M}(shssxR4LG%Qy@A%}o_z01F^72sEJlkclGRNaGlihBO?<)&N za#zHE)t^EPY~4ScQ5dN-?G>MoOPjlnoRkO!dK!}`Opf8`hf0J8>+cq)f&%+~Iwys! zNy4sn2du43**^XKp2?h3=)0=fxI=358@7XIRs-$*r%+;a`=-IgAtTIUUORuZ5vkdy zFzkJswOuB&?d}70RZjc!g8dD@SyREPRpvjJb_%^Rv@#LxJrVZPjAexv=2+l&S zj%sXo`!w{V?m46$S|7jV5^Zulc`&ydJE&TE3aOp%_`y_|V6{hmAmc%-I3!?}Usv7D zwr2v`c6IoYpF)p^{ohBo3mGt(S1wNSKR(~3TzVAFI=p;@T>vM?dF-(jq<*=(4OWwT zYM3!a(}Cr!`HbO{uYLw`U=R(O-a4hxN9nz*8G<&uF5q$_zF+;!@KtKwDy7MfBT&_r zD-=IuycCMYQNkT-WDOpm+bu8b57cU&Q}w+iZ{bs@dc|+ypn0$M`q~2uS;FrS|gvI_(s1cFmJ`SOnY1MNCG^`OpnOqv)+3N&t_MaZtOg^)&EOZ&3Zyz9Lk%n!S zzxPx-@YeV$mbHSS(UMlpdgr$UmyQZND$}^ufbUV4J55zXpH!=;||Em^e0}NC%4INu$*7ycubLbl0D6mc~a#O zG`{t*TPBO-m36AlxwI9Ym5#4BRF3R59d{N+#QGY)^p-j0D5;KUjDqX*?aobWj(dqI z9A-_izUWR}oPJzapy9BOu|wI!JXuri#^x}1=|=BZFM1oMj8cqbS0ZmX$^XHA7L@<9 z1WHY8Umq;r09CwmeUH^x%P;jNUudT>e!%^5^CWHf1{3+Mr2KiA=FB&bM?2S=Qr{Wt z3qbKt5B)?X*^NO!UD0kht z;mG&}`uj$LB@-RTl&E>S`j25rgNJSO5$;8n6(-yMfng(JOl6BJ*y8z4zwbPKvMYvMDd7{7XuMA!UP0h z`M!xbOGQP?g}xFV3#D-wH_8)&A{t=?IwT~BN+f_CMnFTw#f9byM~A}Raj-X1(-R`l znhJf&;WS)Gd3jAnLN02m^l;^H4uN#Sa7`FBwH#Q+uW~6+pK(J*e2Af@hEdHtBc(nM zgJ^K8I|q`T19uTPAr~(pSmn_Qec_1eSeP3J5=jHGLnN(OozaI`bC0Yo8hkM)u_xO8 z7grQaC)*tx^rUvq<~7Y6($g5GwXxmup-c?p^1T?F|5T1UhB@)}YzaKJEh!N_Aa~If zZO8qRJ4Tp+ED`N$a6PV-LRp9belrD$mVnwUNZ4bL^k3NH>~SZ|0T$4J9FhSEF486! z!}nB~sugaI92SM-4X(#92NI)i23rsk#SkD{Er`wlqgubxadGbC;$i-^d&J(3>2Z+kY$-I7N_tcklOD#@axN5(r@aU$f>H?GAF z{+!zTmyK%Lc)p2dPVZZC9bA8t#VE3{>$Y$=kEG>ErZy&+#K!K@E%kw5SMMIjB-cs4 zdS)}CDWfpv?X7l?_UBa@BR%hv<|I}Rb4fkdTy313qeVymy!}I6Uy-Mnznq}jCVMWP zoiJnM)2DoejbE<|#b@8DtpE7^#PNDWX{D{_q);#1)ei>+WCE(CQRk-wq{-i; zzTMRp$S}=(m!lp(gJ{>D=b9s*=i_O*H+^YlA0uP+GfMJtj@HC-jx1A;YvQcdgwtD( z-zW1NoQ2n}(OtK=bC#T1nUL@fBLS^G6+7C-m4H_H4pqQA8g>|$EU@1oS%m;|F9x0qG&(;WT*iBP)NC`iwl6JT*0~G$JMjpyrHuI6J@s z6^tFia~>M@`y7n{^n?L~faC;(hwF!jeSjgT)5z0`w7l_%ws3BkO4OFaKTaG7s zVD?k?2fb5h-fCy$pkq3ECNDR+qHoddM1lEdyjiqbN12f!Jkbf88!AchBnx{ zR(gPiX+n4$4kWGyWQqC|syL(|iSXzN07Xwd0)ZWtz++l1N(dOBR>oFhO&nbK0%qX> z&@2;QFbv`#k%EK_V67H#qy*nnGzpS{EMOKP1e&+W8VI2cKp9anLF+jmjg;E;sD}fT zPZzy&N|gV&d5!L5D9AWH+cuC3l5Fd!?B|-KVgBY@QWX5VGY7!D4AXhbl33_T%qo53O=5PSk}h`>M##D#MN z&_&D5b3*XLD5=O-(OeG1Z)6Dq8iwUGZ0|;!O8x2Re3LvctXW2Or;hG9R zL&7=O!3=V6EE4j@1X5E`Qy}z<^;c=Y3I{3{W#yr%@O0cc#Lhwbj2?lgjzT~o&k1NC zK9_%gXCSIr2p$9(4}QgpXx$xf;!4pjv~4(9#-8i=E<$P8fb92mW2@K2}Shr}vO!2)&(t`;Es~^$;)@C6DSqtIjRgV4dK_uo6 zZk~dOE6U2;LKdH?B&fHUFA$W{E0#U*M;nu#lU^WRmm4yg~EZ#!b z#lo*=_buf<>GixJx%pM&dci<@Bp3Ek)9Zw=AK6W9RrLJ?c2gw+UUk-Wfh(Z~j9a!}E=>RqKb9 zfW2c!TO+jK%7{ChXxSKmTY>U8IM!k1hZzM zsbXN}%1}rELKPxJ0-xq=(kA=4cJ#A$=H$55#?v=jw1cLoXD87GYCXesin)iP<(9vQ z{h1bwt?&AOj+9lc&u*O)x7m~+=QQAZ4Zv0c9>Z*j6>w`Y0EmjNXltRrCA=l@xK;}x z;s68*c$5}|013>AXb1;g^v{2YBuYdJ!Q;pRzy!yECruQ?7h?f=trL(1T9g1AzUJAJ0*)MZE;Q?UdW9cUJQk zHmBTab!jAz^qrNg)DX;hF;C?#zLp?v6(V3!2A3myY0kbIXhF&z*9xPCThOZ_ZzI4w zGY1f(T^W$}B*?~qVD1!%@MavC_`txcZN%sj09U~nwtnh^Y>wIcQjzbvWo5xEW)fNx zB6?sjDp>+#BoeR>UgZG!a6%OhSYs^ojDtX!;3gv)DN7s!ge8~kO)ex-R+dnnn<^G| z223J14Gjn44&n^6&Z@(TjU5H4?>lRwPVO9U zHsW_Z(MH`aC<)W_oc~Bv&(o4E@=3p6g}>+X0;46Xy;tdzCDB8EBBO(ciw(ZfNqUEe z4Zfh|MYczLGQa(#ikxUZ*A!x^Gmd|F$(}-cqp3TV2cVvW^;b((YIK3p9kWW~Qz%dH zyD!fN>?!1i&2JFWbQ)Ygdd(J7prK`4PFe(dyfy61$B^Gz@PE~z&|o}efa2c%UD8k| z(9n0keB4roF>CPfMZD2YI_ly!w)_;*u0ERC3^p>Gz?k`HIt`Ep9%zl2xcPey9-}zT zF72!xCYTW&oaH}E%{TOF*+NjsvW21##&D&Lnj@mjIjS)@;Cb4k zisE-GI)$43K~T5qN#eJqgU!Q+6H&}z^d=p5T^lQl!;WdSC2GfP7A2VL z*7FOScO0tX!5ikH2kyZ*1dcdxU!RqmkPBXSybgBaP|GLeB+~RsbD>%acac}|yKn{Ta zVSu#}%ONEM%WyQ|b(>Wc~DzW0ivkRg)s1;!$gqFv7B~tbnDzF-aGm!;UxacNw zIPr8ml2X{4BQ^kwAy_J7 z+AN@FI3j~4Jb3LC@Y>v8S1Gm(zZwa@$Ux&A4Hd0&OhD|}-vmf&1fYXdiUFa}JL*OR zB4COTtw{hvS@I$5vhuWPg!rY1H=Ty-CL=f}APfml|04~5Ds;e^S2rSkvE^FM<6Lbs z*=V3Txk&6EJ_?m|bmO_8@F+foMoM~h)THmby~8hAnQ;w6Q|8$D!nHN0T&MA|KG6KR z@%iCaEu%@(6~|bcSG{>xa*8q(oR7K!=LIXe?tc32@IZZlGT+z6qoH-{RhxO4k=yE( zv3@s!E82Nt+u0X~o5h*w7?>$YASy>LevD7Ke*->hr;pC`jM| z*Mpk!>Hn2pU}glR^yKEO=jma=Q3d*1Pc3?6?8a|JmNB_1y~^?au!M1~=w**j;n6h{ zVmm*%3Sa4e5=5J7sdPPuC%pLze}n6_d!-n|P#yO|u?vqPbN!FI4q57>O9!g&uVjvU zzpu01GWH{_VrhnEvTr3{*34F?4&f-ypbLqNB7rP$gH$D6*N0!=V`?%dn<@rbum2sN zK3trPsnMJ)w)Scx1E)M_S$qL5&c_H^!bQnpKOKDbM6YW(?Rg5G+aeeHtu*ELvQp6f zz6jeXWXJX#CE8nmA71r-pQ-H|N#g{10MKCr*k7Z5F$+u|$p8ob7}&T~xCO9ka0&xr z6G>cxmz4O{I7^2E?+efX{40(O2jYn~EC!Hg3sR&!1979P8tB9h%nZ+!=bz!g0eBS& zJ8;s$_hMRUQ>X_f`p2?)R%I6BzEYy{#~clTU<0g5$KlQ3%iP1#pCfV@cvf%DG-8{+ z`>SSxZ{Hx{qsQX!J~2N{dkKvE7EN2yOLLsIB?T(h0ug|-YGP$70>GW+0GYy(ItXxq z+fW3K^r3^naQFZQ0)X9SM!RUd>F$26=y6SNif( z`NLJtphjp)84eF@oC0=Y9dK1&B&ZQ&%8T`(Ci4MO$m(B>9U<(03>+8`9ucr$jWaGX zOm(sewQTt@Y-+cPvzXhJw+;&%ZcUJ1%HQKGtnPObVQRSEn{%r`uZuaoX7o?gjMDIP z(F9I+f9^-_Kg71~Vd+lLi}{b$0XG!;f^;_4ptg$d&)uyo&P*knOL@LpxXd*wmCsc4p)+H; zG~~W6OF@E^g7b}PQfeh58w$jV4IdY~nY<^0r?JGA0r(|o_QfpFsRXU`I2i+IFB&l2 z|J61(i`!uIa=;59abhQ~m6`;N8yZ>-SiUQHfGBN+<%u_zmB{dSN-NtSdX%HpVbkUA&Z}}V|)_hYhIZJxF@@clfUWjiGi1S zk4%#<2xe)iaK5V>br2Z;H1fG%tY&6W=^IvQey0$(f;(PO&!6l`D(MvJcGB^#6x&(J zQtz2a&YwH{F)xa#l3|cKg;I6nlNP7*M>~>69Zs^_FVV;DTp4!DxNWj_3K66}+nfre zEHG_hc1YX3wl(5kqqN3$Vd~Bi-3n_(-Wp}f&y!w|;@&VSuK~@ zlsu+5BUh~VgvB9uKf}~L+v*e&H2cQlNJ7N&rmE+#r?CF(N@`&i`o$ZvTWj^I~H%=^U z>`x^4TJ(C~0Kz@(nM5x`kY$=NJL*#|Ee#rT9z?gjDN0N;KZSOvj=T6n$7;`f$oeJ- z(~{O`^TxDqegloz4_@qQXoa5X*Gl94X8EZ26_mQ-|F5jHDXGJfzHBJL_4HH|`u%s>|0LghRGzn|?%1<2l_;r`oo{zA&x%NBO@6Vb(c-&x>GsSA<+MtPoC*IH~YDvs7Id(pU z(i#$sK;B;~XOHuMLNDz5ih#^}KiKbnY_2KE;AW{cYLW7U7x3g#vo5SBD)NH=Zj?G3 z+uMP|91oIvdW&;YPe|=W`aA1t8@@S_oRG{#Ri+4)`ZMYK$-N9S8R|Q%@*sI|cL)MuOO*CF;(zUdHC$Ff!({i+*2$`Czz=x9rkXNS@*I!X6ze;4$Qm< zZ}+5PQ%}CNE_zFN3pR>zFPByvn7AG+80sBT0{M_Qpwm^5VWS1?8HC_I7)NEh?Oh%B z{l4m_)52O2_#RcFdKlLJMkA$bVdDDnx2|^!duGm#^}nC2*%VBI-3evDNt4A;DZ?8<$fV3h) z@I>x7O7gHX;fALoDHB#55_FOc5$BGCheU;O`)9>8e;@&8+y{}e9@ z{^LUEe{w+RpSuwH|I#GT4HvN>!Cf~1w{8PD0ku-7;`j*ufEOBneNl^+e!%Qhk(7M! zhyK+YHw|zc!2Mp|W8#Id#tAG&)-~(j=79en957=3& zC{auvcMNb@FfhE^m5oM1lMt#PqjlsOtKv_sNGKqU1I-Jlk8HplTr!*&c#Ep%gw_aU zK#Zt3m5E>!P!BScKo|ra(aPcE)Ptr=h7Xfclg4|s%<>{80JE9f9c%l|==L79p8zKR z)^P__v`u`X8F$~JQw2u3*ztO`j;&yS_s(QAm!Ge9zO7-fpS!!3%kH`H=pY%IZYKlZ z%JT_Ek)dV0N|}pZo9naD67K$`(f~I^bJVydVh_QtR91%G71Xwm1bv_kCBOp{nfj$zc+&@+%ulf}; z?_5?)dUWyeo2c?zA#8ed~ajK^UPSyd0$}H>x5JGV-K(6XzxUhV5*^yrCQvaf)g5Pa;;WSc$A`ZJNlMF5XifTzw*6VT7oNd?b&I*m||=A0!b#@sz-0@bE!0z>xMpKmh85$Lw8j zRzeg9KH@|M8pN9kgob82%>7N2e!n$-0kku18p!QA-D(# z>hC`S5~AC7aG|+iEKLnti22Y!!M7M*7OspA4}&f+pD{@Q&XRQsii7XzFC=m3*Thp)e$R?Q1+UXu1CR(O8nE61jhUCj$E#-@gFg*O#k^02EGlr=U>w=c>qiw zD2=D+2|0k7{}A!7_ytJl0;SD>#2;~byj2u<7J$Ex`FJSdK%*=?Bx)rEN8T^nU}k+_ zT$I3gY2)RMqu0N#x0&>9MH=jAa^=e`o%MKrW0Cr^lAa}nl8eec8oKQ-vH7o!BQCzS zlUe@@J)9J3wO|5*=c&hk(#7uofY83<9)Kxu5nz-!RRf#|kp$V(o5dnHcz~$;Q}o|_ zGv1MbB1l4LSk$r@Rg}@jq{!LEbkmTQI3R@j9hWs#KpLSmx(8&ULck`f7)B0S0`!O~ z|1Zwm+7UI5j%x|+1)9ERWlA?@sYeTMO2#kTb-Ubo_4)iIDb6nwD+P1LpW8!kcx)$~ zc={>bbn|ximOjdRlaRX6amhC+lztCWW0W~ErAGZ zGFL`~5x@eQ&Y}Sf$S6qR0Wbgs!eVg<*PIBrnMJkClCHSvP1^MRcM^*_ks^tm?~ims zc^eE^OtTARlgtXEN?KI=DuQzJ#BSJ$9X8himLBS_rT?ut!!u{X^L^?qom?4@Il75S zc`yF+mb}BAVwg=mG2>pl{;++$og0y2Dy6;rI;4h|N#FX6rF#1s?D5f3KY#vN7WjwQ z#zd%CE)e*zGmtsX#0eUwoQajY!3VB?lAi>Q3kiS&jDi#*1OTTeAqs&03wJ>m;PS-E z&S)#5z$RD{8~M^VQQx)L0b8MO49;B;>fESgl(&w81Wzc>mt`iULoT1Unj+E#&o#ej zVkpZF9nAj3a``T-Gbh=sGe`XO4o>XAY5YCsXx_wGEgk_J&3`Qz4;Vft_F60U7_r*F zMjdtALWs`^2b>7T31ICO`V(>I1LozwARSP~c|ERSfY<*Z2}W_nfDarw zaj0N7#Dg4QqpP+Ap7NLQG2vVZK4M0;f%htZk=ChFOa)Lv3^JK`o>gL5PsldpGV5>w zq;Kg|-T@(w!wE0B8_47-KmpGQIwdyY^3f7G4sOY1dfqm zm|f->lYVg&Fw^*U*$SMU#EAc3TFzd^SfVV*%$=P|Q%gF)Ut?>govZ4fk}=F9h|?%; zSAVtR6WNrK+nx{*1_)KkPtHdH2;g`cW2PWyE?6Kl4M7tZcL0vBUDc-n8z9XgRl*U~ zaO&p?oJ5J;O{S}~)B7pjngb8iP{tIUdv`;{8w@zxO&50x9qGz5d35trI1w)PKg#Q~ z%g$e@P*1LRxx8sC)4A)d#(A@($9g(as?T30|5Ll!`4fbs8k@!29Ht-la{E1-qs8TB zCQc#vpls4TmZ5wu8$LdYvt$$`1h60gtG~fDXY>V{#wjS2y%A3g;ejnQ|A52{ixWm$ zL;@-a93>ee5`<`R3@o4xzzlqDge8HEq0lwt)dLpM0{&6MXm4hTDv3Tsu|G-?(y zfP3yGZnVS#%Xeb-O>1w$krpLZ?6D7kuLOJ!$0X&DHYGR*=Zfn}akC!wl7Ah3#DNk9 zaWBPgZIHF#sER`hql2;xAKO$X*eX&P&s~h$fX75jdsj#*6h2VATk~c?#jC+H#qr3# z+$g_!{BsZG;(+B|>h>9@pN@_z_6tuidbEGO)MplZE{#|Aet0OVook{xauB=WV-`4N zJCNt1(^sVH_7EFCJ9>PZKEliP$RV%a>0@La;lX{TAHMnfc`GtCF0xsSm^If-V~Y&+ zE5TETEaK0VXl-thQWa?mn2=`~ppL&EQ>QIzruJ@s!UVA_rB1Mo3!`sk^_*m&+Ao)n z%#60If)pVaNAT!Gp-$SKL6(~rW;bU2;Z?4*wt4r7e$-c;olXB-e>?GSXPMivJHCug zE5hYWUAftN4MS7xatnsZ6COXqhB;NWKP@@2)keo_(Hboo7PxhGgVe9>j^qf6)<-T- z)v3yMtstoZuBsnxbO}q+s85@?ocK|xAWH*o`l@`%?(S;xeQJdog?s~JpKP7WuL+eOZrfR~9;JNRPO2g5CPn>r- zr+6!Kb*t~r`Ko?n(q(*HW!k+Vm1?O`AijUzQe!AOAcub`F~Q=%q-R68%ucU;5%X;J z#$~_H0-dZ~)3@R#zFBLVU3p{XG1&H}UT48~riC}Ft<8t)hu5UR-L(dgF5P9jI67wt z9{fu>N$u>tXx1!E*CLRy1X|CwynW8ix#9c$%wJ1>s>U+vY##Nu#vhMwNTpZT8W%b0 z9lRK6=sV9Z3-Z<9RKE22#MdpbeU-Uoc48zondK%fz?jgEef0c&_u{thgWNIaHJj%z z!}6k9ZC}1f;#JA|kz!ddx|aU*hHnT|U)A|%bC3tQlqN7$#A>}3lc-TDpkt~r-J z-||i7&0;R{p6Z>qM+%u|5wyp2eyOOns z8XM^==RT5`Sv!i-**gezvc$X^+%?!J(9<$2e18fhU>_!a%qhxN0KYY2j?VjTcO=+Y z+g^Q0**Il5@xDIo^Pdv z|KeGfg94{}n76vS44oBkmI^}8A5Vh(vVvLG`(FR=p{)l}6Gg=>#W~B@SB&mXW`R2j ziB{xB_w-0Ha<|=Cs$op|BG8;&(%J2xmfF)jx9|6GpwGr%RXJz(!Pa?h$*6X}4gv6Y zBdcuW%E5@5=?FsZ`j;i?^4(hi&ad_{wSDi*uCF`l>@)f5UdgSZB$g1Qee6))nNZ- z|Jxg#T~@OR{r$90*>S)V79T~M*UPIW77=ZT7E}oNl;^5W{~Wv9Y+mfGlf4{LWmhJ% zDO~28i1{(-P`}C4?(u@2(zU|dDr?KF#;G=Jwmw4eZGqv*tXPuUzDzwS%Yomrc(T1&>Jw6>K|Nqkb!z zHkmvwz&${wcR3zO^pp>G&mLtiH2l80OL=FZYVYZv%cdw;@St>P-yko~F4!cTE7)(f z0vdbDHZ8J#Tg=e3jW>Xr*<_N{6?F7XJ&&{ zD%k0q+^4s^wa(q79tA=Z9V@i+ERuO)E~|o%BWi37^3*l=Gd??rT)Neqd2S{usC%E~ zq@UNpL{Hb2LSI#iUL8B4(xWEayDBBq%d@cfGgX9-jhv}wk?L0HxUxWmglxiLkYx(V z{Ix-m519!g{^#t>1~4ns_hS!PRe9z+tjn_whpri^6nH-#VtZ#%;z3t-*nU`HFKiC_J21UY%-WeYNOOT-&sao-K<5#}#`)y&J;RE3O7o_eZa!R4676 z>n`1I+?C)90O!_$F!1GTjjg`{8KQAY+E$1L{n zdKIt};C%t{2JV|ZhM|L6lH7Ktv@A{|V#=K^J3K|zuqOM;;bTkT;>|Xp;yIAny(Udp z=0f{;msHt#=qj3e5?c)CrF_w&|&Y4~5F4`cxl z1Le2~EmWIn&0LGO<|wG>vby_1f$}u!+;hS0h<1GX4}qM) zmfR+DG@-LHDFOH)5-!R~t52ZpEEfO+#o^BSK>bBPazP-Zql_jY41t|FXCr`A$yGw0DTy@Rwx2E|rq=zw&y-a_YE$MK&dTl0IE4-UfMMjwDJ1d6urR;M7L&SI zI>(_+`O(hsyQl47>LQaNwz|r8&mY7Unu6$vYUs+m(N%WmWv*loyNj zL5*AaTjhp2Ti-q#2fv=8HL=}ZrCW0L{54&P-pW;z-;QmfJJd2yR$uOe?Bb(zGV)@h zm3~o-K|^}RBg6ROY9*3+CN+OI*`%bYF|!`yXiJyRd>f-Gft!fn=UaK z+^{3H#nAoUmwnSUx+7z- zLw8q?H5P~Mo4Zs~o`D2yYxJy3hegFhVME#7X$WWEP zOOM%^jxA=&-w(4JLM3LVnfmoEzuZ7&anDxXH*)&fW>pRLVO@Xjr?Pv|Y&?Mz%eZCM z7vVI|+EY2u_0`)b^zbl+XIrW^X4z=Y=5hUb8u7{4l_;MaQ^BpF^vSMz7uPCd?F}}` z8skD+kJP%3*3P!d&+pUNLY;QS=Vpx_vi+#;(>r)}3f)v_IJ4E!w{HSkqZ-W0wKq`C zm{_%B&+$GHqT_D=pT97%Ea(0#?k&w+@RN=3<16RC>DP}sr>fJ3S-HD`dH@Vr@XwNa z#Ed!*ZEcu=FS*|@AM0;`>MAsFY&XBJGEp9UzoT%Z9o=s^D~jnJVK}K++7i#2mGoD# zK0oS&oz1HJO={YIeuZqXokhp5_JqSvu5f?Vfrzal__*_uCuLuKRFy4O*rMS>`ee#+ zX8-vmy?}zOewmNHANJxlIrH;Zm*_QS&Tu|8mER5Z8Q0M8_w~Q2B_+I*tUmW^OeEFP z+{WfSCgO5@<|K2yRaN$EgbhDWL!G5FMp@gYf!2#fOr;=8ytmJO?v`=l<`lMP1jV17 z=0KhqUy!41SH))RB$^ofO_eeFK;utjQr^2ec?=UHck@_(ySr}JQ z`!4f#ApIl1sQ=?SRBoY{J$B)Qj-S)=;fWDrss+U*YbHU#p+&)2UaaSV(gy)AH;w9! zIgt^c$BQ7p{ZnYT4ue_VGGnx#PVBI+IWaP81>1&Ega?`?kBW1qI_9*ihqc$OR(pn7 zFuvE0!vO9cn(@$WJR_NAyF4?{lYT@xXUSU8tF)pu-tWcCu;b>HTm3gQ}r^jc5M`dx^$p(|Df4gx6fuRv#XDLc2vRF8cRY} z``U&wF8vrf9(RyQo)m-~sAQwE1>xJ5JOz6ud9{uyE_w3zre1#BxSq;eHh4HVb;VkE z@Ni~CwM{4Nmb19!Y;IQ;7s*S*U9btlG}O-P(MdYbdrVxOX*`(L%ascx(EK<100lI&6$s@J=^!KOTn8o4*^Te4|1w&(%Y)_WrD zZc^xjD^{kxpv@+GKJY=;=rUE*- zNltBZhjT7>DDx^ac49VI2i_jKY{;B1W{d-cUxw5ST&rHOMB7k0%5XQZ*?1)19ZI%z zt$L05SUo-GBKVbJ)KB_PCC0P2?&Ton$}P#yp
wDe0K`zY4k-;xm}xeV0v#T|BPBTBoxNZ zj#j>T7Dh`N1Aa|{lnVkR1^iA`cfdJ_T_G5io<_(1jF(PZ0jh83JdtT!ga7 z3p7*&)DZYjIjBYBrk!=GUYwgI;alY8EUIpZ6h3({Rp0+WU2s6E;CG(s;+&D;XSduR z(^rE!EItk|dd-@@nN<3cZGD*XvtK7y!pNXuKx9ed_{w2&bHvAe5azSQh53SJ5I)|M zQ{ETuzTk-0(x;W-zph-;Q$R<0>oWE4Nim*@$-8T+@W3X8AQ1H@1v}e>`c|(8 zo;c{_T&Y9R=54vmF<@`U7xs*-ig0{NwYSKOCb#;aQ(5@OSW$aX;a9UUqcnf=u`Tx! zY}CB?ezFdCQ&Mds{(Hf|;3r+d`1r3x1NsP!MVHi47&A1HoXaNJ=A5F4mk!@&9wH9- zwQO@@>_b*9j4hNQk%7GvXF2Py8(f^V%$W-PWM%=L^jmP6M%BshN8k~54asF-ChiTj zjHFIjbq82eCjX2sN7YgU?JoMx{Ao0&eEsv+N#^nG8hHp*Dtpl2R-}mt$N+?n)-60(!-6D+)=@t*>=v^;K1(Z&g;6KOp>B}Pfa{RrN+AaCkJrV zKGa5llH9!-J3-F6=Z8g3$KKvwd^I^Pr2V1GcqaGDQcPqhFC_O==&J3hf=-<-=r@(E z>u&>a2&=NyuPM z^6S59BIYFy;Omi-vhcqTW@he1k zEw+kE3Bg)CD)I$dV5=pRk;O{2=Xk~mho6E<4FqFn#q`nAP*Z^ZOC&iUmW?muuk=ILPdTl#hfG$$={b^XHDQqNr$FF{Xhv)(>!UntZ z#680tmTBQpY@5fic2~zA|xBMOaxi@pd-vf_yBV3RQcr$PS?)g6W#2ij8 z(Mv(cufWaO;-zq`r@N#s)pxO|cj3FVb~Ly#a3M2d<>($~O8||f#rE^_ zR8^gCDB2Mc@V~|icEkh7Dk(Gb4FyHq5EdMAw7kMctX;D(qi=<ED)Rp(9u>)@lK>pgEaJ`Bak-9|WK?EL)e>?)127%@`uJq0x~7Er3fnWq_oJU1 zf~I0;eo8VdFl(jUqF8ivScq+AUCkZJTU7B$PERY5N1#@ zgAfF0$U0EPPnz%48rrz`BQZW}UQ?z@H4g&QEY) z!sepPQIK2DjGGPqTkpY;8(6Qrb%mWE&QQevs0a`|@$kEN5L~>d|1(~!wS$=#VYmQnJL|7+@v`w08o9^GQ!yx_Fh>IR_glpvQY`<+lMN}L&xed8Ot92WBn7tQ%9D& z7NZ#V7KXuRHF}PAg(}6Xe=nxdo@#9Mx0Nk#0fx%LrzEA2?;&KwkO~EPMb77VfWBP; zuo0kPwNSxi1o++jp|?OLMMf-60U-e09$>OcApmBy0ExN*G5Wuj58!(fd~o>=`4C*8 zDE}mciKHSlSWrP8jO!1eW^@nE1SR=&j1a5`d=`i4+BPIhZ?0GkQn*wv=hhC>9h?|n z^ChUPPYJuIbHH=!S-+wvxD*fX`>@3NQ+~&S_n3%CKVi8ssu~c89R#}+*z5!$MsyH7 zti@V~f_zMq5GMfyiU9&uA~1&*#QQ+>UbzBdUEu%$rucFg=abzy1YG$H3;{2A00O{8 z;dfs`emyoK!b$KC%l%IEFa}rN=Bw~soCvxy4#OfC#Z*}9ijj+js6&QD{dROU{+c1h zNux+_%$tFzz*7C5OTRBuf6(@79IPX=#;683w+C03OG1aHEl2cdwMHgCyuNA<`7}L3 zjHI9JMgH7|8b8PM?MQR22bM(~(Z3EwCnD8cC}QWoyj(0W7WuuZDHJT|00GDiD#%M- zL7ZT4BN;ef?T9Lbsc?D0((J>P|K2BW2N}Rg`z|Q-eMJ8I0Axr|=y&y-K3qPkRMz;< z2@CkbJ~Y_;0Edp8obWbqkmMh<3CO;h{lwB-a!Hwi>JgUsFKaf8Msxp z`j4PDNy>tUS{94R#6WwnM1wvCQ<$5@=F^m^ z+%nc%R}+dJOZ;g`%omfHf+(L6d6U7(@zF5h$gra&Yg2jhLWVn*vqOaa06iFuxSm_$ zfe(8=J&bP}+cPK^A2zdOjPtuBv4tkVI@uC_)ORXbG7EoXRs84LS?*)~I$9i*9Z!RK zi3x%~RM=q+&xi~{2gEoQDj?#3qz+6F_%9YLw&w#VMj#4I0AtAn*y(gGSdk83G^Bu@ z{s|Ar4}=CoG$8o>&R4o<+itf5BL*j<)b+oCJ}QUlu5ICrqbF0pE1@dyMkrymspQA6 z;@K_F-zHV6bNDGaOPW7=m2gVG!Vsn+^jDk(1yke2 zyriOLRd$99(ct~hh1%T(CJB#4aQ0imlLYbO$3$jP`jzDyGAB-sHl|K=?N^irrH_{807g}wX7%AE>*Wh=eAhYAk5X=qm;JKJ@;x72C8$60LtMM#tH}O3LUNdyR6}zBButT$ zkeT`WZD46Yv_Rn|6O++DX%>PH&;OCGZeWk_xtQ+VON(4}(XWp-d|cAl20R|qIfj2! zcgejjEg>?na$?>-QsO8!Ey?4q{q|7hzhEs(7SS1RRyS3Gkm>bF%G-rA@MUJjfAiSk zpe?m}6e039oAR3DiumB^WR&?MZq$`XpP7TE>wl2j>v2j1oFyzni*+>!g=pAF0hRK< zg!Ej8g_~7A6ex7vR15>0c@hOx02sR!3@Sj_9lA=q3>!FROqQKTJ)b6-hC6#P7p z4HJ=lg)lSSpmT;WegA|*0E$w``y28Q;1(4ipkM{g5Q42=ts@}d>m!0t0skovtV$72 zeh6H{!&=@~p&=TQUR5X1QtG8B3Ajf+2+2(Y6sTj4A@L7IC!b|T9dNnn+zh1Q`_G&JY ze7d&iDZS)@jPwWzZOmz)N{z`eFA&^kzo@M{_aXn56Ht_K{E`Y#@xQPrkoX&b z-^a$8KpsZGjJ2xxyc@ShC*=RtN>suCfrUc`qA(Ls3Q`XKCk6BnCC)veZT5X7S?RvN z2L>3Mf2vq7*Lk;+9}@UI^6q(`LahG-CS#z%rFw>Vv7_B|;J~d@tbJ&f+R_>Uf-N*A z5ANYR);TDWjP6)yLq0&kOvU_)p<B|Hk^-H8U_t_Lkqpd> zLV-U6(G{C4nq$XA|15zN2kY!`u<8z^kC-0Gr44-BJ}=oSlG8(%GZuZ{o7$>W55n`= zj)^@el|3)NUh}BF*V&@=Ua}=J;`)|GtLF7X{d3KCS2R-gX7OW_yt*omYRiTebEBGLE9$d&W&I6OKL4_sg z()9h}lf%@XM|Y`{^irsD&N^w+j#G)lezQYeR=>;}&m-A4=~If~rt`zINIEtD$ubw| z0AH~*rQR_2SoLhSozTTlsRO0m=71PcYGGEnOg<#=3> zq<}jDtK!q^&p4liA=XsiaG5$kQRDH_K=7Ep_bIdg`um#w7qRNTX#9tL(GU08I0*@{ z47%qDSYq8LLej$<3c$ovvA}ReATdS$KbajC-n`&vy}6O%MTXic<}SYQEVjxc!$xhh zf&9(qYO)+Ze2|<4oAkFjj~vp&6z0g0I$C6*-nB{rsZ&L2=iLnHaC@2+C4;wmd={HD z%L2RQXwP%eU*~&`W0eL4AbeVZ@F|nYO;&I}NvB)Ugl~7lS0+FGP2sf8b92+Be@UK8 zqz2Xf(835hLXd@m8i^T1d#on|8R>^lAcO^nfj)vFwqbCSeN>(jbO%5$54SKLhx7wB z3;cjJG*xV72O52fOrQmJ%a-*cg!W>;_0m+)=aEf+k|yiLru^qd=bXJQD`~*gOf*8qO$4;*5hCAF z5+I6Wg@drhZA1!T<^?|*Pn_bWB!o2e`3)8b$lM@`@;DPAAn-Clz7GlBV8NBUAq=4+ z&V2?#5xD;26A{i=V245q-*G~|;)>(S;~9Zq3)~>^e|Rz2l4TuY{ka09{`sI15dKb7 zo&9*Jxf1RM`cfm&%j}(xF?NzTXV-A}%!UWOgCja+GwuBO<t{~Tz#(8N$KqxQE#?|W0K0f#Ancx z(Gtb(ps^FENN^!w_uzk`H9(e#QX8Bi4rLDpxgD@VW;Cz^`8i^q;$lA@$vQ9n z9OW3!hJn1f`#R{rTXlh^HMEgqhuca0Va*K7&1qL6FXWU0Zk=8dtz=lumN@tEzhbOI zUnu3;3HzB_>D5+gCE06@r?yK^E{F*5UogI&d4qiMqGs%gZq(lLDPw=6JS*r^5*cTw{7rQf${txm~v+QpseI~M{+sn63c57D_FzU#} zW;0djGwR%e9!U;*%7q)BmijqF!3xc^Gr5HGl9r?MzonhfX+?;}XMcIzUz2h#db^j~ zq7F14!yA%|^+jJtmCfUo(L`DhS<9`J#{5iZ7%Co#IvdE;=bQh1so}9K?!Cj#rPM5K zV;OViTU9IFW;%L&qBQO}d)vd0YcS;;7SPgNJ4x@L(vYJ*7Y=vgMzT*f6!!f%kV_ZO zsOIOM9-Yg-?*3JY6nyhB$ic+NQ`m_a3G(knb!eq{Yfy0DqzJL69|GZVe_d1@)u;s zkW_i>`o(4f5Xir!LImmg`VvnQ7xG#Y_b;B&OHc)!;e35b)TbO0lHxLumL?%jB2m09 zT&{~G86Gz)Vn2QS+sSNW!aVG^nRSC~!YN~(I=2Q9`iP!CY^ABdBrRcmLS&b%eHvcf zro~V>^Sn_<9%3V|$ni)(yPH&Z&3zGVvLB1HpMyQ#`J z4t0LaeQWBkxNN+TU&65W|59W;wkMxnk$k3pZS!mT5x=}!S4L9|w`(`+R3{_u;WV4bz@!EnhwKZr#^r;-xm zY(Ood7EsJvSsvZlfVP9)(|sfdKM(gPL?y(SR4>f4ORxde)7rLP<-s=|w}slgC?D`3 zJ@vM~B2pP(J8!Q*SxTCFA8KgUOJR1_t=Ra&T_HNw?`>D2pO)Yt0hyg1C5 zv9tPCE6PH64N3d9$MaDUB8y)~lm7Up_M}p+=)p9jxUZAUl*S{sLE;fQFR5K0kw+I} zF8uW~o>n%Vl=ii?PD5$#Yj5YaFhwoBkFIU|FH(%puS5QWkkEKgai^=5j5i`X4n;Mp z-ktXB1%+JTB^qjg^lnfov%2IyIefK#Yt%1AoE0N!RY1BX@q70`Pn6`f_DGpbi^S+@TA^mI8Q zx_)HKyy@$fW2VgPY`402x2K@@tgY4Yi`9GE)wFnX0XAw6t7axxPiCK5vR7qk*Wy}!&FSySHXfa?GAxcBN&1rcC=PDi z;}5!f3a$Gol;QrZi|4?G>_Y4DVZ;I6$Qz0{+Nnn)btRc zil20X_0riwvOI_seX8F@<$H*YpDKivEy1W!jM~ii1RArCYlL5Hu^T7ql?WE>(JHev7{1TXr zY>)^;@KMKakbnMPRtOowmboB7ybHl0c!_(HJs1?NFXcf!p$fSPA&`gM#f9LpL2mpn zn7qP94apEVld>iRLf`=*A_OTus79*)UtM|QJ5F#gwKXqfP=^9y^C=i7rRw(xL{=l+ zNWXxyms_nwc#s~su3xgo6Lo(@D+B3afje96JO-yOQ~b<;H~75ko~@sQZ<-mD&%$=u zz_q@nQ*zc&4fJhe`W@^sG2Zo1>19^#`aO!WI;0h+HI}o8)X0ghz%sw;Hi5SM(@_HD~x!+OD%hH#Y zy{vi>f$^Yhf0k}A7iK?q5Dx^ze)=E~n*LMCsRxyZj{dv#Wo;SSPS6^gIafY&Kb52d z+M3?gy0{DS>ee`(qPV5R>vvJUr|9A1Sfyf>76!43QA(cF6VgI^wID~zy%5g28k1_N z;a@c-0qeUxXh3g+Y5oUcz3lgFO!Jo#z~iiIG(pI@{Hl>OU*A<<(=l6AyQCFaYm5x~ z^W^d&0B0$+TzYWJI6K-4OAP~&wLF2;t@(?7KAk=PP2=*=#QU{Zgcj2mF~!qfV?gTc z)uLN%>9GjVfg95w2n!-Pxu0@HWo#aVs5OmOz~){SViQPkETXLAL9w`VdV21gWT}_CEpe~jbTg2HWw4> zZlBX;(RbRWL&e#?mWK9aVq$(iR!eFjJGJ?1YF}4qVM^Jb(P`0Qh{nP9&nF6WrN(G4 zv>pW%srz&pg$x#=)oga?SAVZd?T6YE(}`j?dca{pV-=qjckZMQg;pqPPxy>gt=PXIx6v z>DOmGXad%aHGQfu0-^z3DLHCK%40{AGrR?q>9R`)wle{@kKdzXiW^DljkOZ{vt4|e zFtRIdZ_nZOHBIQSq|N5plSc=hp7w|)G)&6t@YAQ3E9QN z*S>cN0oEZQi+#ykP&9$-wAhc|t#*Td)<8 zTja;GP@oa=V@8I%;2T2}a;wrt@|~1zzTUBsuTXxV6-5&I=T-}?E~D5kegQc7Ez^ZQ z0*(e!7BY;&*A5i2Jh-AA_h~LQ> zTphuXh#_b2$>>_bpbLQhhEGkGnRM~~Tb2*bJF&^WNt*`#RXl?6t)?$P-U1I=XF5@x znTg`IH|gA#GWB+zOO=p&&65t)0vt?Av)+H5wLg!64opp@BgMu}*ms-=8Gki7)i&n; z*{o`7MRICK*-wp0cZK?;`t7`aJZS&Zbk;F0HNfN=p`rOXw0inYb#WhY^F)xl?11dS zzUi#t^HF|V7e6nP7USk#kB5Kfrv?+rQ!FUwJbfPyIHW82mI|8-E$tV3U(p2^wfSe? zI<6XgEFn22`TOq$3|4nkQe-|{U9K~gXnJHtu}rkzw;osyMrSfUoyVcf`o4 zY=qkAkr5EW%@&}mpEcB7x+-rz4dUdnYk34s4noyV=btUlxJXU)E@fP(#|o;w(voiF zII2zk3t*#f1pZ!Y!1fwudkYtvvww(%UgJBsR5tN=JQ*&pnsWbo?nxNPxiY*KVOVhB zoc{5uC95sgT}Gu{61m~=>RaS0$H%yPjK!w|ZRzTp)wQ}Y67K0x?SM;3I?6}UZ`pD( z82{+(TUF>FCNEP)ma1Al|0r0_$gj87mbJ=1S2X?+SD7jx^ zP$$E)t3>O|R_!9>l%q$sjqD3Q2fFTMPB#nNkW)$1pCixaB%XP`dc}TKWmB?|78AxZ zD~U;7x!op&Aavq53n&Pxnu?iAMLyMzhDlbB7#TUmu|F8Q7+lQcuoLOO4bO@znw=Cq zU{`=gCZ3c%yV9`n|B5zM)1mRqDW5~N!n!iKpxl{Is>Kdsi+ka|7LRpTZN<#tKm9HS zPXinUB7?M0TACw^R4b`VeXY8TDRR8N6G4y7`z~~4htg-%WLldea|Sv5zmx4QQw-e67ik4`Yv(&EyR6#`&E{7X$lVMU4lgL0~JIBhA|lg=n9MvJ_q537Q4a1iXj6Spy_c~jrH+~T??!K{FgbVc&? z`4W~0qxT#ZVnO=#Xx3#kGFs?TuI(h~_0|=k$yWXsN10i)XVTZ*{cm4me_g+r4B~8@ z>oaaomMN)js*WZ9R<4XN<6m#)V7zy9m#6RbfZ0$~TRj{KZ=Z$J#I#kGqkZAJG%IV9 zk3~pE759esQ@^27{C7sX^v-NfEEh97?-mX5bn#1O5+HdCxSaR?nt3?)Ro7&N=if87 z;{QveQ>xs;t`laY|9(YWP0btrY5Ehk`A-VEy^v238psVg2n7U;qyRyc5CU3YEKGQ~ z@BaTr7)XlnKsz163nBdQ67sMA3NjyDdAmM{fAEkX9>_f)q}XOBL>^mPgNXulqWpKF zG&~5nf(x2d-IDeik&>E*$~ggA_hPpu`qNfpPk-m8b+CQEoU7q!5n8LN*YdcWEYp$V z(Ky`uc{tG+K$}KhV33w!q3k{TRJ~z@Te8}#==shjfZvis9`~U z<5Eoyzn4F%`x5j&R{5dc2y>CG*VCjJuCU68SAOyl=hLenpCCV}s6iVR+Yuv)l#YQv z`U|`N@Du2LrMeJMP`c9pIbk8a`|E|u#e zViMuJ>yJK}97AglQS$zn-Z+?v(Y)x<6b8!dL9~{5dYiRD9y?E73Qj5M) zk%qpz&@qC2Ga1QiMZV~LTblVBsp;CWcntRy9rcB})-=W0co0)X^=7xQCQK`hibgKG zj(NI2-kkW~9CXLFR!n6`PQ&SGO=H4Qnf+e+sh2>ARBYL_ZF#wiL|HpY$yr;)7S|ic zG~;satn~c4O87)$m^~Ygnz?i-&SGlW(A1g>8N8R;vD-4HJ9q* zV>KF+evg6i#Q*>Y*R44)dC}v8;23(oDI*~=5Y{~-TJtxR3PADSTwL}hu0z{DA?mtn z!?TU`ET}X9Pb2c(nt6Go?eO}!^WtIFPewX`cN=A zIOXweMm^7WsXM~g`x5YSrTucs%Wk7Xdx0b8px$U1;`dk2WYPC)G0SCbB`-L5Jti;S z9F#xFITGB(&(ZBBuv++&JC`eYIDxX|^?c{BQT6MbOl9>rYEkSgaTv|7?@%75tj@Q! zT&-jAY$X<@7Vx;^TxgM>QtZxI-|$IP@=EvA$n|@elOH`IaJZj7Kfpvz`atH1Yjg2J z8_Sl6Nbh!AH0hM-;fHM@H1>Lg8RM6?=;0MN%DsZ?<(A?yeVAM{<0=p)R+rM$C{p3| zOqo5$6TEqpYH5QT5}JTke-Z zabf3P^)B+d!+!dhb2H6vHNKDDN;6evWUr>wg5NV)wg;~9qERCT5mZ@lrB+?HwKsa$ z;K#wzf!X&sSKC5ji)2= zziMmmWcrjx+nn7+wAt!aHw1{M8TeYa8Q|Vo)Xmjz-E+Z`=m7|?YeTNiKoAa^gA*h-HRQSc$S~THj zL9a2@+fokh{Tc=%tjI}|nf4k72@Ma!=oe3EGYdbr^Gy-lMuSemov6cdtzgg?9MemEpM~%X*YGeB|R0 z6Vn8*F!a+_Z(q7b5Q>j{s|vrekE7emq7MQ$-=q7w+zv+LF}s3qYqFnezlEWRF_Es< zd^qvqI{N71ah;UZQ@fhGqWc@DHn>36(xO_Yir?5qhd?%kPk-)y+_#}A$yOVra;cZ5 z%n>mvDLu76De=g(qob!!gU)@gkN45^Y=|%RD|w& z%1UDq%IVZ5aXqKXON@FJ=DaZP^W{uwIH^c|iNZz4WoylZ__q1pytnDx)}D+={?HKM zREr+kNG7=GM%9#4bE%X)?k`(jbFe?!lh*fBxd@EQMP0~c$@u}*>R#_F^lm zCP0ML*j{?D;}UsBIdas`ziB>4!K%D>^Ajq=LUt5tcDWal9E zX{5=>Gl|;(R#drAMd#7C^84C*zon+m)RJKRT=Xb>P~CB?F*>Qhb>!S=k!mer@aIOs z_&^HdneJemVMOVvt=6c+Pk*VpEbq9Iwl5+wc$?}Yd;;91XpaMmi3ikWE(lUF$nk76d zl$|<1PGjYEZ6bnK%~>3wuJM4Yj=FP2cz0w+#xbUp4?10e{4l05W~=38Y0!(&v;TqQ z-xY5)+}2c&4|B{JwR!LZIim)TpS3s16hGF-6zOU9>SP*=oHo;&9~H;)JaI+`jCwS| zwnp6}(>zKgKlOQOo7Qmif2^79UUD7ldd<&(i8H0MeXafTKZsPT?;hNw`friv7pwxl zm8;ntg%7${^X$4Q`;x;B zO8C@l{Oi=udR^F6dNXllP&;4A>$It;ub#|AvMY4-w~!Cf2GmFkvz%xScY# zwI(qdCaqO{)->xH7cM!bF8md)4!A=*J6%RE@Xx*e#$8kUr@eXGVbZ5PSuA5}P_E_M zf`O~)r_Y%sSy$eQ=n<&n*mt^o8q}d&UUjYK^wvLP{Q-Zq?7z@Hu?A82N+lV_n{TT0RDaAb=~@=^2G+H`to z>X?6~I5bmOI5g(C)QlOFHq<;l`EgD6W&W-PW+m%UkXIEdeyW9~R;~&-w`9XVwi<2) zhVP^}^7SpAd;tt8n<#p3Mz;~PK<`19InEE9EBRKF7ofS`ax%h(}x4xv{M~w*+j%1YaQZP#BtY70B zJNj{IdHO^rc4)msR!*->wOLKG^`%Qx2B@LP$r(ZssI#10N#Bu;ggls|7eEqlj zt+z~z>4=;t6BuMSWp6u7~c`{E6N7DzK2m!t-EO;h%*xSilYQOFZL$@Ge)} zr8w5ttiL7|T9!g+0RG`-=q;RF<`bOTEI0}_)eRYLMXApv*oEV=5bli`56X1R2Fg~i zRAei81H33Bx%MO-T9nwXtje258_v6p6_T$jy;hI)X0DKJ>5NeWKMZ5cX#G4f2kMr&kN*2B8NpYrLjG9kgc8H3}|()F%n z8khg6uE*u@lHXG`d~_4L!wQ%e-M-0rGjQnsjSpcs9^_2qV3RK{un(A`ICkR{8k`%X z!Z#IpK2s2YZE^_Y>kM7qb~g}0fEJ0;WXB}Xo;%jNN}`XEv!M&cXSCFqBoPHQ#BU~N zpHkk-p#w>o?jd>qIUALF>GmkuZ{0WI;)V}HHmV2fKS-YB>jreaNY*_3I7vXSF}xw` zzHt51X5F0;4QpFbaTdpg``#rfj0j2nI((kjitjJ)16Y$!T%%LlwGITHs<|SK{kb&` zWFQM9QtL&Vt21Wh2h_fj+pf(G76ZO4?I$>E{(aBQ${nsr&MDK;QOVm{wPcrZN?~AE z`t4WJWn67BawjsxTLDS5pN^l?<7YTLI(N6&uenkg^!R*5L$aM9 zHP=hB{=ek@a=Di;w!HdKXo^`piKupUtW-xVqvTcNHe<<_Llw~-h&ZqCQ935nZKQcr>AO<;Ju}U z^*!Eo}#)sb=`Boi@x&w+XSMtz+?C>Rbv|m54Z}*U36K7kLE;Joj00y%b${QnisO8Fs&~ z{_N2O1gy>Z1dhR0@Lv^9>Xjl#eZ&5yxo>;XSw7f?8ZBKUEV+AlHoY#gPxi1cOIF_@ z-$-;GY{ria)>e9_*XUXH^l3!**vlsmD zn2fzGZcHlRT^mmiI7ieGKQ)tpO*cgc{_hk5IQ22#Qt^U zn3GI4hxLG}Ob4m7C~4-~k$29k*47WbSIB7MrG3u=VvCpj#I&@G2J{aL_ZDZiJb|3B z6LL<{t9m;}Jss@UkUx@mR(aua%(^x+;a+@EO0&N>gtYz8n{;Kp$D7wQp}!~F|JVsk zusDrPV72HeM$z<)dViDEtJN~mt68(-6DfgQ17yrBItjsLwcPA?1kyd45B}!#{uP7n zzYpe`nh5?(Ix+zLO#O^JLo)pgl-xN1&Uxd|4fUnV7IPX=P&z(5-X%2B zl`l1T&^P{P$n$8CZ3>pP{5M&q998MaSv;kE#8`uo6RevGoGVUt@K*7#sc#O%UpKu| zV{qtUB9rVD!QRpxzv+LzznfMTgc23zNT3}AY}ICulfD5WQiUHel-c%DCxls z5-xq=S)J$|PMv~j;Uolu>q*saU%xXEX1u66+6*Y0l?k;u<54?!8$w#~7 zE%WZlU}9>8w~=pq9tJu(7QIpAh9+?vrf+A`1Qez0JKAt@9{TH~F8I!-Psx02sEe?v z5*72XN2Q=keZ++qt{k^9cb3|4-I6o}k4KM}rZo9FQ+xyBM(wsKMzt%pPFPYMbTq9z zc@F;i$dG466xpBaucqc~cxor>8-$s7=@fI-uhgs4HT%U4i#qjO4{W@6?r|51%0!LT zsIr@gQR-w;j5;S-9U98s$axWACyjVB7ok5j(hxTL(~wF#!6mLUJHwvV%@T=zz>i_Q zyWx&e3E7XTH1(*OI=Y6_XC|BTdmtn|w$NO<(Wbq<8jse-US2M4nnbgnYj77Ccd_aS z>@67m;5%O(+i2DDTvJcUuA$*WT#2@rB*t>WR_Lti|U{k(q1Z6t9N2 zH68dh`#v?PwD>D`RHvd$S`d=D>gbf6K?MH;rimAHU^Ab=Ue8XQ@fuEt#s|4I^~EBz z^?K^b61K_D_O;sDzN~O<&8}wp(F^D{R%JFXobZe!S&wME>sWr@QC=P!s~tTkvpIR# z$REXDwhHzvS-P5E539ip%uS*ib99K4jg<3f(P_Px)O}648XFxyuib|{<2NoI7Z~-f zIu6wqXz>fgvZO!$U~dP%FK8L^33x5g>p0jus}H8Xq(eYqy;lm@0P>KFQBm)RO(pzu;cX z)=KdTXLg#(8;q|(AP27}Y9p}bG%{iF{C+LA1(2aLnPyHSE$unC0?uc|Pe;Yp1d-{O z$@tg4q8D`0cx|W{Cse!IHXg{q=DAvK91Er2s=FITz*+L!H^V1g* z+5vbjeM&&2s_IP3{vu+99*&`8e_G%xqh<5iS5xWGc-A+t|AKv^iDN4&A!k^I-(mf; zul2;R3_hYH`x`^Tx4>Vs?9!Vfwp*Wl{jU;HQkxuGU%WiEN;DePeF`iEY`%C39cyV! zPs$edAzz;ZX{j`jXbNqu%nYi}!qL&}$>!8>8>L-Zw0UpDBa49BS59b59uJEf;< zQ(aP>1bo(KZ_}@L)WV0sr?WuXYBLnSZeGfNXgQ_3^V9_zG8~_(Zr<&`&5RM+X2pQ1 zoq0x6$VjToY?JCmw3MPL7ph*Po{oLSi+I?J+B2%&7@E_0g13p0vbAf9mQtGG;~J~J zcu!KlHCtZaS3{v&>M&!A|7!bLlcm+dYvYq9Czn)507qhy?REseVcxlNA6K@(2_v-b z7iG1I|NERCAKX$vdpAzv`j&C3Qb@b?s=JZoGvbHVEADyp8No|0T_xX0iQ?1Z#{c-@ z{D}(JucR*u1Muj=*xyQ273)5ldM~+P#-?BfUlR#)7vOC4u*_O*C=dDjcALMtXs*>( zLz-Iv9T>%lfE+)eda?%?s#Q-`Rj}gH|I^p zkRPys9_z?s_hOJIPu%_@JzKK`P+sa{)ZOIps@0GT!Qv!0jx6?2Wlh{wkPg!VY!|i_ z&Hvi!7m5>gAIPW&e{7#T>H2;GtZWN zFRyvjC5z1@V8ZvE@lou%1D&-^I(o!hj%O*T*h^K0-A`JVC+z&#l~LM_Z7%fUYT|r?=>i~&u>sxtN~5sjv7+g&g7o7UT@MDE=v~c|g+18e7rTC_ z7Xzb)#ga>Tj56Js3)E$hVsFNB#$c<7U5rXr3b`wu?zoaMKAtL8r(IiJSNx*-+EMbk zYvL)JuZ7Tz3L8?LEXrRm#ef9m8T2c5-2Pjbqqa86agKI7>qxm}*epwG@?5FlQ)cYf z)C@EGRH4OG?%q4?kHlzv)Rm38y$)VePb$QS!=}gJO=3S@i+6WskI^_K#(YE}IZfXj zLhk~ztTwDF_s(TGgTO^HrFvQ zDa~|{+!PN}UF=&NF85_yHmivqu<)2RnCeZ{W7R>Ze=4enl9@3!|E-wdV|Kd}RrL17 z%ILms;U3j7Or7t1*5c<%0rybe3PE30T8CeQvLCNrzs5O!4{$)>GxVHRy!@+<|%wyM9BF!P(o$)uhC=j)v?38qtQr^~O*E_lqJ> z=Nuw?Ixk5qOncvs*h>mkG_<-V_Gb1P6^mc+n|J@pN-r5f?@yoxIOv)+_YQ&DK-=t5 zbz^HHe~PfkZhGtpm^r2H?^1gcb9EAO?T#dq9kbyno$A`WHWlRzZ#7FLtMRiL znYI0+SR{{cQH^9`fwpS~+p@repU3A~Niwc;x3lwR?M}>o=ygy3$PN@??OPm>jhnE7 zdF2|G7TbAwxntn!Qb~(!`-_<~-pkpnldx;u%c9<7#0vNEaBYU}wu7<^f;%d9vs-uq zO7dQgwst?>b0OKmR5IZhF7b4CDIj4#gvzNmD>(}#TkXH{b}g^L&3t7xE)*H`)$dL% zg`eZMq$3)a7^x4}v8%Jv$8OzPi~b{$EtUJa-_rG+BvF+GHh$q_{nhua=VTMi*t?sr zh5lFwq^-8kNHqn>X4+Rp*cxQ2Hx%f*_nPUNuOItnU)pTzkEff6Nju3Ik((_@yK&Osts_SgH-I8eI?oW9yF8BoGPRXQC7W-)FwuIU*KdLL( z)X#3`N$t#%El=jzEHa6%`Ji6v@S$fU`Br^W`f`T!#eSeiP!P;yUovc@z{TYGOlSLG zgOkhBgwG&M+f~IVMkN5qSSWFQ{LIva$jK^?u!cdQC(YlF1rec zzhsvy?ya}#miX$69lmmDp-H?lWBe2QCwL<(;>ke{$iukB=iWa>GoL^YgAaD(;iyOp zTmgj$RiKLs21~=F3~oc;%!>XAuXPOTPxRL}SgbxCJ|gl9Bav9&tm)lf36hZFd*a$s zek!-_E6u;qj*<5n6dPUcJ%*pQ3!LMvmNdMuUDjksbd~uv@YnBC&RShfSZ|9M7;&vk zZ~kAqLF)%8iRil@B7qHBj&ai7`D-$SZYm@h~-^J-j|e9M}OR_ zi8|H)4`O*Zkef{_5s*wa1N0=%@TW)t_@RW<>8D+QYyA; zheBJ0C*%Bwab_PK^`f!%;y_czetVqm1cA-<~WZyR-??c zFV5r^I7_>4I(Kd&P$Zz<(ZWDzVegPxBeq-kAB5#F_G_EfV0*#EZN8lV4&wd8WZ8os zr3Oc`qWZEuxrd#-YqsY8bQVY8^S;XI0;;+@D(?@+lm^CEg1Cc54H{zIfqw2AZ6}Yl z(Tic975;+`-@vrhVu2V5l7e*)B>P@y8vqZsWukUSB=Wow)fDs1?0db8T2t3YTHBu( zMh3V}BAm3E_cqPy{L5eghmH=yK<0A(fJw&IL8}GdLGwuZ+gR?IUetvqpVn#c;eh)^ zhcEs5K~};^YF^t1lBb_#7mj4QpKS1-k$_|3gV7T_Cikp_Yv#kaX5z&zrsaEuhmkvs zs(#GMB#^!fgOuXNp+JMPRpXM3Obd7kso6+5-0zArA1v7VXAl1fq$ zRJHl^rM+QmO@)SyykDAMeS7EmG0D81sav^`>77Djw%!8sClY%#_=N|B1mJuz|3~QF zC$Y5zdOb92figQ!fWPZ9P1sg@~~|$ zjbDjB+uiND_wz@_jzHa&q(DBbd@ZjAm7aA(H}RF?y=hjVBu!?WRw)N&tDpqAlG;XX z6U=$&bd_D%@K!`AhjWN{q2k0~t}$pqFJi=_}@4HI7bz2V8rIFsGP&nYo2VMkg2Rp!=Vr>)S}^ zl3iG01PjerC(^8|f=4Ekyh3>o_tqCkR6Y7|_}NawS)X59zkHqEmMN}beP~g*@zvZP zs}kq8&Wb9Zg_t!;m+BvtvVI=E(B7Co{95s4&V=}h0u%X9dzBP8+ywrFj$iY<@-I?f zwS3iAsbVV6OlWoDB5RE63D?q5Z!w3Qo@mocIlF6bu=ALIn~0oN4qNc7s1u0JpERu; zIU1&5Jo@@+iJqq?-?L67dyff4uXKkTr>pX_Dr4OH^N*>e9+u+N9Eh|%tRbBs=cc}} z%gKJrFBkmiEAQ*R1C{$EZ|rk;;aOq19RDwpIyu-E%jMybB`Lh>{>z{)&N5a)BGy?a zc5u6dhkJndx0&iMF2!gCdU*5aM;8>Fy&hisp({%7QdHP;i0!Fy^fjm4B>z|au6Z`* z{pa(FeAC?1Da&$EL@ORt~8r=Up!wmI!vJ$dMYbK?;)vTz{nlf;!Gv86L9>F-$|r1d}Z{O<9}{kx%ygz&Fjeod<_ zmEXjEITRi~s-OJWJ^5*>j(>`IW9|9C8ToI-dh;v6r^luSWTtCIEpE)7`mL`ar^5AN zb@okT{#$#m!X$s<13#ni{?Vsd1`00y$sh1}d$oUauCwO&V29cr*rw}JmzNTF)plmF zen|M4o-uh{tfKyi_Hm6KS7>;BE@D~pd-C`t`Mfr_($41x#ahw|3e2+dUDe`l$9p#n zF87;XS=sho^~QtVnWe^q5$Sm&C#tKn_=m3+6l;jDJ32-GYI|Kuk>JHiV0abVrHqB%t%-Si-1hy@^iVuq@vD{M)ccnnM~b z94avNiB(6S?Fb)oIF&D;ja4nZ+~Wb$?U9}py?STE%q8A4esetu?&)&bYl24dtSvK6 z-qK7`Pt!eAo-%##^;dPjn|J)J{{*Y`(m&opTXpZ2M|2?5wlUg^d8{i!5tjRT*6#WD zc}pUbV|R~F?_KW=pS-G)t9tewRlt8Tc2Il+jY(_1+AdK9TNgTHo?%V%HC1nGdmS3+*mFC z?UZr3ZxDJx58s&c6~kW_Z_nFm&viHLO@31<&-c*NeMj|;ubdPd)69g)U0G7$s&+<` z{4cW~p8zm%^!Ked%iNDfj{TOn@$j7@_q%S{*{L6!_IIzYFMPT1r;XMqasjQdn%5Bf@xvefI zD_-=S=JWU+(fZG`%)93?>P>?BBg;jRVW%f9_w`TeejW%tDt30q*{SbydPd)WY33jA z@V0L9E{wP{kR5Fjs;ZNgVBo@)$1C+YM#VB&*@OQ$<}sY?Xr_n>TFu^ zhcK-d{)u7!&f8*@J3|zw2Y3!fE=4W-=P!ut8@6;6@pacxNcv^?a8|^bzFc@{B|TMV z)+qOjNnw~tyX%6c6@T$!NJLsPtA!K?@Ea@MA%X?UHZ`dqrOmFbzWNd??dcL!Q+i1C z33p^&c-72+y}$d9(Bng|=-C&P!jtWV$iWTBt!S<1g=Yo#TLo1-k3Ry;L&5xDo%R(K ztI+2$<5vgGf5NKV5|xpnrHZfo{9{?211d+8S9PvG?8qMDvbH)JE@=1FrtOer=~=_1 zjM~SM6Y8E`iUFA=Z!e8#dW8g3Ir;U^e{-nsUJ2;b$ypP3>%XB`+;ru4#LMA2q*}&Y7%n1g z?G%XT1EPnULKX*p2#2FlDTKhSMg%V4hlGvjobXZ#1#82G=j6m}MNrxZsG&XLxF9Ah zV}J8hsq$K>Y=`Nu3*6_tyg4%ktuuDZy;X|`2}$-!K5aHt_5UK{njxp%wy&hW9yaURw|&2$Q%Py*{`~4p zH52!%7apC>^-B+)bSU7gsapIhGqtsL-nLHB@3W}Sz42eC>sLl27mD(xy|)?+N@xs! z=;Xe{Hwww5;ZVY>TdF#Sd*yo#LmdB@mZ-(O@EQAO#LG*;Zd4>?T3d@vGIPsan5<`u*=#Xj&e8@h_rk-Z3p-{>tRikw0E50l&p=_bO#|l`-=h>Xo>u zRAl_p`gz$Qw`y2<$3mowu2FVYY*N8J`F93l_vL~Y7H1RO4PZiK@4gd{yQjr%x|33_ zw3Zh9m{-kK3Yit{{%W>%drMaNZN5rVm9}iNZ!T=23xgUDf2MaVeJjpR`8^$$DIj|} zKaIyWZ!S&e;A8T`#u9gbA49{u%hB0$U)Op<-dBvy|G3u||1j9w_LNd`&*RFHDn$i_ z&4uo-U+*m49=&#YT=!M;pKA$nYu;_%vEebjeo4-o?)03}8UM_GduR8f&=`*|t@Eck zx2+yB9^QZO>5%owIOnK%zL|nrdllYJ$?L5VrH5aQC1_18%l$swu`cMZ**vxEYVb<; zVd_}s6@#&`fRr)0Ih${9eMSq0*UuY4s}+volBw4^HNbaCEO z&&SW}q}^Bld?|h85GmB^!qiNmN$IqvWJd% zD$6eo$K**pEiJH(Yt+>_@iu4gg_D`>s@qCpzZie_I~wuJ;h>z=uvJiT$uC9Tt`a24NMS9`x|-^8PVvfV&Po^AE7y)fUiuJ&2!dG5g zK6*YU?1+6z`?IMuEw8QmqY0S}bM9;2o+2ky9i*F=t1S%@a(hoMtt<8n?7Ml=Nkec& zbJ)g!&dGo8;m@dq<_wqxjz&5lWzmIiEfDVTIBF|c9s%*gg@8>W7Tk#Nv1o$$Ei4X$ zE}?eFCtO7XAoNH;*UMozsF*-0D%unQ8j>N*R0x!eso(5mWalk}CFmY6Ix$>}{gxWp z4o7W=2N;?sI$0bO%7SzxL93#}&L$>?Hv>MVv@^YMJ=n6}}S~`+bsPiJ^8)pZH<3yTwfx2jx3D_BWKo4)APis3~9;a7d|# z36_@*Ip;~Zu2u1f`tKNQ;3?KMpOl$!eCkAuqa?^(tlb@b*uhjPkJ)lc%Hg#8wP-)R zjuMr^L7jbx{nqc!t2~6pRUcHuw7v7MjO&_@c;;VOj%XSSUn%ka$Ku458#->EUyXM@ z)*h;VZDw5N)2KG|-T2eqKUzP$6#c8>8XT>R!w*XZZ+h52nJnj$n`4=~@9oEMZK>1D zT}uaY>G>~&UHNGCBDZv-Hi@gFt=*s-2o~+Y0IdlrSYaBDOb$m-|H%!xi8#1U@UeR! zNr5&EG#t!G2?Q}=te->wEl6hqH(|(X6r3;_8o&DEE<>w4!+D3VNVh2IT3m46cjxN< z83o>dPUhH-HSYK^X=)&)JeAcyb0fBXHoE`Qb!Ely;aKmJqTieU1P?1mbREpS8Bt`H z>R5cDF8hZ>?@#R;L-VSz!LP&{?7#L* zAME?65UVnxpFNZcEZQ#TzRG_@l47MKKE1TIdi7|f{>W)^U3t-!qxJh9&5fsQzn1%H z&M@)!m!gXUkM9p$?w5CI*HgNXBUb+-fvNaiexvE3C-W8)SHh3bMq~|2kJg+QoLYBN zdpTxWf9h1%hZNsJ>#a8``p#Ra*O$1s;LS4)tcuEQYdF|`ERWy7_x5>il9vB=>FgiA z^3#1Y`>QJ)XC3QKDulMEHvsP;nfJZ5y>P0@YuEYu{;uV8p`&fr5|&@MxP&%pSf=Gp z8C8cJ;#YjA{c3Y)hz;k5gLj>l+Vx4Fd)`^)DQ@$)eMmI>%kG6!Q*)}@tNN?&l*D@O zdu{uCv}vfS+V4=QtXf? z_ga5Zu2c2H@GQ&Qu7;lCAB87t3mn$uvmd^=7bW%nA;?nZS*WsYj&#NO{R+aB+YhL_e?dzJFmfrbC z_x*C?g_-KpyH6)Rnr&a;->225P&D*(xX0Wi%+}$R=b?7VCGLYN@d?*m`PGbSi}n^j zkv0x+n5in3yxK0}lsDW{oh4wD!{2vJ+4$S6NQ3{(%Ez3YO=i9oF=LY_VePPnjB!Ya zw)$D2e}>A81x{#2>^=V8cf{~sMA!Jt-cnY113m}E2`st>A5kOY)DY-WgujM>5~JN5 z)R;Lr2{DnSN8C&~pc5?QbaI}Qm; zlEr~n!XX$MvWcemxTsw&*atkO1>#fh5w6tuG5Yr*SZ`RZOtabFWSF` zhBerPjM*;sW=9{1-lMKY6wl<|oS4%Bbvk@0Q5;7mt0T*(dx3Kr6zfpjMu46Oj(DR@ z7{N;DjLL;k!{WqmWANaGh@3E+jT;w4!yB=wQ7C9vzdZorz}t|B0Nmhg(smrqAzYB8 zsKOLuPEO_(Xpo0C`=L@99m{0IH7;LHIeCBeMR8VchCX(evvcbD|gM(;<8Va*XzUN@;BNIS0Z^LR2DxBl?ja;T#7Syl;7W;cPK>Xv!R!= z+}%5!{RcF>UJSo&sH%xcrB!yGukbqM*VhXx*ym>DQ{@b^3;vlJsP49olQ51@;{0)@ zqjrZ3ZI)$f)-A0}^iA~?G_rDDP;9T6md$FPF8?63IFc~z>MHU!S$VEVic4vWbdT%Z zr>cEzkF|Tu_H3V1HZ5G{{CxG%+CydKnhy0K|E1O4Eg$k)cYmc*7_A`~2P~Oii-z2$ za&jWF1aUn1CYm=+LT?%NEllDtA{Z=fCmp6}X7~jrVWG7pLPmHqU@=v5D<`NY6O#rx z4Aw|t8FM)?cJM-2RfX`$HY45fNOP|&0x6~n*|7zEVPcHNDY(SwWd;fglRfQk?SQsK zx0ocK*X|B`UW<9}^6s*qL1E5o%jGyenK+&3YsMk%4x;I~(|oTDyVRdXidkAWy;&S} z;@gszV7PQ(tZ=Zz%-Rl?HkYJuwJv zGDHx_ApIA&qPZb12}7*9?eRZm?aVPY98rWoCmio%&46eI4Las==;OaZ45vP~8_)=v z5B#44^ku|_?@TAEST-6IeqRfl-6xmd6i~#}=G^l^=$TVd<#1bBt;o-`PVA;Um z%#SP0x&8=;6)nA%xvZH}ZF^s7F6og__eh!X4$dDZjwA@(*&Si`2&&i!I2UH3C2oYxo({FDfq7+oj zo|wrEzcP(L*&l5=gUG%{5E&3VKN&ejWg;N5t4XXKNNX-TW<3GCVt-6`~M zR>lniFyh0iv1|xOoSBnRwTEZBW5~dI0^pz%@EMatH7Z)gKZb{`Z=tag6{`DeoN$Q{ z&9He=P-uSCUtuuM6VPr;8x8>V0(%e={GextKYQ%HGQrwvG4G&zL)tX;MS!Mez`7Z0 z)PQ}+iGYtC7wg9Q_jUe@(0utluGJLGEAFR9xSUM)8;T2v%Plh%2)Vgz+~w5XubGj# zXna=Cj>EI%LtsZOH&GaeCBw)<5J1I(BN{`WLU2e?fn-!jWaNQ3jbUjx8nT0#fG6N6 zm6>?3M1g%dWE zbEftHj-yf_1+Jq4-waZMEsq({j!eQrH%nm(HVC&dlj3+j=>Jw@3~C+dVGrgP0@rBN zAjM#cb&J-2V=t90!s$suS?JLBWPV&QG&p+%x)~vHSNiy!Jfe!kx4^DxuJ>;iJ}fHz z$&xk=9A91?@5xeJkBIyuc)hXaXSzXm=MGce)W|!ML}+~+a5qm1|1Ez(Q!}p=|IKj$ z9Ny)LD^i;59QG}lsCW^`6X1Zr2f(0bBqEUu0r!^3iRjVboq2g4L60VcIgo;7r;T4dE2VYh-k=f*SAQJDA(*5Pup6Rg6FeYu7)^7ecfZamop#$A}~1TAVo z5NZf+&D=%9EF6|lVLdwG z*S}M6%d_{LNAufn9sQN?B<+ex0*ki5HfOQWy}x-^W>OAaU2}OGQRbWn`#(p2JNi25 z&TJS@EvN1H6W{!~$puaQbB!mxPu7y2s^sr(r@vrI^Z-@gcw@mm2nb+9GAuaMJ8dLy zn=??gC*Ti&jLt}IB61O1@!Vu$3;K`<@gxvccj4xC5|JZA5)mZsKr}3J?eMsrhzx?Y zLvV(Pm}mkj0l-nPA{anZryBg)NrmgOPeMPV5eLoP*ppR7SMWfo2T37X2>_k-CWpgY|<$INpW3PvSy+5wILi9yL%;5-2JMFO#f_ zbMe{*zc?Sv;M>QhX)ZAMzWefM6&=nd?^}hMT`3jy0;>=r@*F;fY6-QvZX0XL;#dp) z^QEFIE7a1?yzHTE;n5^bW4p$aUQ2a5IG4$vts{2sZn^*0`@#7C!qf0L1Z1Mfh9J;6 z6OrA4fEK_~jo>JT|6v@Mz)o-o$n6XfB!vKQC2t^2+$yiUJxF94*~pb4*=C5T8G4oO zKwpS9m_f}2Y4)WecrXWsM`95r^9FnTH}ud!Tid}e=tJ{zfH5iQsR0cGJ`CDAzp%|SW!TA8v+XM5_ z&|wBo1)Y^qcf}ZWI0)OoHwTH)Li7I%&d6PYjEW|N0F{DteupQ412!=5r%e*%dd5^u z)2vJq-UvrxiW}E2jj6u8{+jP_O>X0>n~$&m(pOV{U9q&EbU0^g;P&Z@gE+4~dcQ>} ze%6UE$o~KY=MYx$|K5X>v1T107vKRHaa7q6aD^~-U=jrEBo-le-2%&CZNVYbV^mH8 z6Sx}~_AV;C8f^h9iRJ;Tfe5HH3!)bs2w+7L`YYqnF$aA|$mp2EtH1(S7y@7tZUOT+ z)65Qj44Nux3?+InMJtuNddLW9MDd}X|0Y3Rix8NJfllY*xML`emIq}fQ@df@fg?p- z5rJD&AHydrjsw`ujjIj3Pr|4PI-lebERWMZr@TJeuXh z7o|dY-WUeR{0UrK92pGJFa!cEIN6T{zK|O){wX(;-+8D0J*%njPr0^flMh0@r&ZV& zl2RHixHRu1OdTv0nCR?YdNiJ*!usXl;|>*`Q{w>fj8ff7804~DU2=r{OAMy>^;&%#jdom2v}O9F`yuo&Ul4+aafjh%rXrg^Ge-Rs3Yow~{ZHi~V`GM%){M|B+o4N`ea@&8pzF_l zc9ESnB5pS)=01ZS=?cW$7ipmyXXC-u5anrqoj!Q~Kt92Jh7QSbM;5h`%aNUUG9nXR|Ss~SbX>`Ph}mNlU0h3u33q$Y{IY0 z8Ca77Tr0lbuq>1QsI@LM&4$Y%10;gBvJie@jLGExf-RXGn#n^ZkTAq)B17llpx#GiZ>MJgAm}ts^@m_v zSM=UlaSdB>xa7|6|Sj(X;j3zy;Huiw)jo&h1TEs=B>{xPU9=#_h57 zpWlsNPjr?q-e2=YIEkE4m}48a^Fnr?zVa0VXwvlk!#7n@zgOwPTtk6n?fJuPDwaFy zUJQGz3$^yhTzFS`V%T5WN3+^i%*FMFizoMFe^^uRe1YR|qxee1w%-OW%5EomibVFl zmip}CaNp`jn%%Y6$-~3mUEvbFRitj5#_)?|bV8DtBd5aI`PVPxt;Z43<^ zcQXiE_h1+}7fWD7b_m`Wl2itT!raA>{szjTvK!JXLb9~&0~byGII2Ib6?^Wt9PW`0 za1jgds~#ALeS2Q`1;@+vMJ2PxN|qgK&a>eq7YpRCCcVjV^K=l?@pEqp<*Mq1nxXe$6+XqB{i+Th`9D1t4s%TCu74b4t8)3K?N&m2Ns(AaK;_n~=e0qTl1xAGn8Z)V9}#-#Sd(B{dPwGA6Kig!TElc@-^2FA!t85~PpiXAFTHg0RN1L& zITU`Pv$)S*vNDpn)I#l0QGff<_24mc12!enrw+o83Ca~G)c6rj9x|4Zfe|Me3W4Lt zq22%vWH{V|iWy4Iv(IA?;yC~#AZk3IH-@TUe9(x7Z$&ercJKzP2$~gqdb`HaRE|P`DAD+mT8( zxkxyf)-<$=B7~ z>4TMVY?c(S^1zen`0}yXY4t;6olhU@n)r+ljX52vS-E16lEdvLc~~o9X~tPi_drtg z4>=1}uJRFnjWOK-{hYU*_F1m|yY3VWI;r}+bia3L>>J5%AR)GNYHx4!4W>!iY&;d9 z0K-Z_`5=~4Ko)wC5C+mvXto&*MD{BHjTtbE-6D!aLa+>Az(IvPq!l394#bd72Z*-H zdxrcX5zBF(9f2nOqU2^a`T;b7+#qEkPzAz0Sb!Tr4uH>k!AL`R6FPt@L(#!GvWF(U z@|;q5#x@Oo5;I$@W2PFKYXlP1jKE0&yfUIPycGkC2?3AAr=gC}7+8&hdNl@n(vrc6 z!4NI=UO|lA4j#&@=+HcyZ|El44@5QC0dvT7M!Nv9>%QH~i%ByNB z_p~xB6tC^Qt)BL=)oE?iRkmrOJpNj)f7P6;0`7~6ZN;~3IHlC(<>#fsy^2LeYrQvf zO!RYCo9@=$35HdxF)61#lmcq=ybjIhvBvGqPLh$B+3T{k!<*acl+x#{6P0C+^|xxv z>`F?`ZWqp8llvH-XgO}xZJ`jNTNv0baNaT~-}G_`&`0k+jjdXLDp{PjzlX zexZNTd&{PN3HnJ1b*aVJjuAh_U9Z!sqMaj-%)h^CKD=cvS;SI8M>@jLC1>T(-~sn^ z_Ah}m(o6N%Q)PNn72*UgUb=nF!(f?PO%aHzetwhaefd{lOdFlo*+nkRFUkJekON0Z2RXCt$Si5FCZ6|>f!@C3A z2Kobx-QaOaaFZCq0hSGCh(UxT!Jk541h0o?5xk@`&3Gy%h6FUlwh0z(Iኀ&Gs zCi{FR6Tn_}GA0qaOaIS3w!*gIt$a)x2|^?qw_}?;R@6AB1yquZDkIhias@~Uh(W-* zamm~Dplm8xjY_7VDKQg8myWrQ!Km4F@f^27z-umJgegowy$NYhaFA7i7lkAZ^_3)U zEE$ceXjX=WM2w+}X!Z;V5hF;Dz?BlY;9E-t0i;5R6pT<|-%P=n(dB4OCn?Q#^C-c{ zSi@x3!&8U^0L0%QNzZk%+P2ATJ;zTtUiY%ean#g}ij)eaOHE3|J(n@;O;N*H*f zkaEJM;MKkOp9wd+iu!upgoa-%s_Inti@*J+_Fw|j3m2U?3coBpA~}v|o>0yYZ>-PZ zwa*rntaGuBm2j?zbq`CsdoD{_(Z6NBdXR_j_U9Ero+p9@7faLkS*e@2KRYk=TzM~M z_b?GnqA6tNJ|F=hj>{s%7Bq*&Tn3dy;2#NrfQ?3ONSd&s2!$PBmLjMxB>RDj1n{fu z;8YPBpmk3AndZP4mcOVEFg;T~s<3uRU-C zR8mGj>mNR7Wv~t0UzS#D2B)keGv1C&&l07piRvRj1^}Rg{~;K8J6|LO1K`~Vzl3F? zfHy_mXgIGLXe1Rb0B$D&mxE`!x(7of?3reI4q+P59wbh8jX~u&v7m-rXe4FGy)q2gD0?E#lU@K4*FGDEOo3(Jb*0fmZI+9lhoJG-oFW1UKEbpRryB1e*eUy zbh>%P^;kcda}WrE|3B`<6ovQr{f-huzqJ@R`Td#f--3 z+z2erzKX&IG zo*z}p?Yx+toZnXj0mB?U&ArM0}h9q@!#x|5}lgfr{ z0qmZ91R@!Uj_Yk&cn_V|bcD0Tii_Hv%+c0#*zSYzUwflocc5 zw%$tw;($@cV$lLe!yq1kF@`Y7&I=rIJ8a0)F5j_?Ws%&8$U=PVm2FO>pFy6`Pz<;A z-Zo(IVLK)8GiVCgOO)eCyUL6nyM$AMH0d?AGm7U zpIJ(})M7PqxK`b6Nq$w(RL)#vmz9p$Qs3F_&1>{~yNEO>*JNP+`(PM2LpNJO7_<)j zhKJh_G!UFYEmn_Ohatc}%m#r8cDa)YQhZzs%H+s0xE-{}NCq&&K*o%u9>vOx9Is^~ z-`~jQb^@QN$AB`cu+iUN0ntG~fJ_)EJk(8?C@|k>n>e`}6%^SqkWGUIe6lcE9LO^? zlY=xri>v{bjD~{@VHO$it0QK<#4!he}e)_Te7hlV+DvPK%qdU2I8Us zW`?-INZd9kH3*7mWYn;8eEtqm@huCSAMooeFH+&_;HaLn=Kb>Rv|MAoANrfc&Sy)& zwHkVOT*UsVraxi=-G+G3qDe=6i$#qBuZ?1XeQag@hF~IzY5zBC!}9Ae_N` z2Sa<2h(Jjf-P7Z5!g8OP&l=Dl3ACP3Jj9X7J-~25Z3^9nj8R#nq8vTAOHqBI5QJ^` z>>0pTq9GZmN-{Z$N)3Sk4fK9%#vZTOY!xp$%!NJVsDvM6e8C1T_(q*aFgvfy|N36Ubj084YOn z5U5iTuooyuV!R|07#t5pdz1{rFkm~7pX@sW5j`q`1~3G1aJ@e>@VL_3c|@&8>f5x+ z&Mf^_>ClPmzP4n)+`g>G`$;`9YjriTAEnu&J!o@P(j9tf6^;!_vv1mZW~&-g8ZwWk zyWv-2|15b#ik?cQbc7--=d2TRFIXpPZWJVYh+R}_638e?R(#7IStkT~^E0z;U@DTG zj=ZQkhVcv<5kpWX2hk7^Jz5)Ma3zVb%N#2aL=AN_h<)fqsM&PW8Au8ae=!xnsSQkp z0h|RG4OcTXF1(_Y5qJiB&SMvaNY-h)AKb) z_0|fO7?zd4cFwUw|GzgSY$ao-(JTX$4QGv77wr}i1t4S`8Ao9UehQ2X0U}zmg_Cdq zzNYLd4Gm6&15#89f+K6F19yFai~xf0OZQA$ShNoeH%N=UE^daS_`Y&QXb#g`APKhV1Kxh z!`|xwf?dVOk~l^Y&oD+s=s3sTrqCYHrPiUC0q_k`{~YJ-GJt z&CA=VSy^ghw-5K^Xha*|J}hz@`awy3tg_p>B&aU&je%$+` z3A@3Rd4I|}Q}W4jx;o*L-0`3r-?@uj&C&~BZaxy7F_Pu}Z1R<7nvnC#al79cGtndO zMEmz1?i-tREl_{9L|WFZLLeH8$c$_It*$8kqe{JZl z8k(pOF;Y}Ag@~-~-Y&?LW*wPU^q%#VX{x{Xbj4iZo2CzI#T;p~U`W3o$u-scL6Kj# zFQK`u!Sv(1K_}_!4s|_3A)HFWjsBT8=k(Kl>7L5u+Sfhoq3M2{-e<}9m2bb&t5WFD z4z?8QPdIo^6b(-hkKlkX&TfstS0Y1r#OXl{x&>L@e@G6+1u6j9uwXNQi_)#2g@@wgSiu2jHEFaFJ0PV1YZa;~B^vF6&WuZpNYr?+g)XrfqedNDJHNqgQ>! z;%&|m=}6{vL(HclbzQUH4qv^LZ+U!llUHAjd06(J-oaJnzN}myjb4{NC$;QR z-b1}UX9Er^z5A`_7*}a1)vwUc`uV!1gT_i6<&k1vM*cv=^>s^+pIWQy0dnzM|5?8? zJwFw&KQGPNJy!Fp%1$Z_U?4& ziIMR^c7c7p=9@m>4(N943JDY7!PV}j56%gxaa(4JKT+7OXdJ)IXF8JQx3X%=je;9$ z?Z*orjTX4O%k9clRjvGU(eK*I-GZVL8{@cQk-Y=6nS9Y5X|}#n-#R9%LhA+;JLMMg^x7}u`JO>3$KJ3#hke5z9m2hpo2h9Ti^+XAYuokD%*&*lqobDNJLDgf z_vZhwPOp<0i}|6Rl80|QC*yA&uqRzVt0dhx?qjvvH@$a0vr3w~-PgQmH>_6ACS)rm zUwl($bI9FtY`8f2OH!=RUWZv#g2T)oo3fJp{bCLmza*K>pE~^nUwFAQ@MrBBze?xg zOPlUs5BZLEdiQSn1Y@SfEiJvI4YVAXO%B?^Nn%n9%VeTZh>m)hWXEe*qP%z;S2$3b z+dBZmF}QZ4L=Bjx2X-NjGe5v6{YULV87L@9Gm0|;H(?Ga4o$Fd0H3>&z^w!Vq*Umv z&y83ktlov<>c7~joWj(FM?n;Lcp7`_JUZ1vq-E9lfr?_s*V5B0{>v5vqblrLQ>E|Z z-Z;yyb2qVOfRP+3Xy>-Gx7B$!ILtG5x^JCT|8!>X>YvoFE>(hFoNEGU4|P3L=&J|4FLKDYBJ4?NP|5?X$FMHyQ3K|OJ<}&tQ~qw zz`UFy$gooZWRqCPvcxkuax+54+#;MYj?GjMZ+)r#N#V81;>xZf<<0Gp(c8<1Ur)%d zUwGqO756LRN3wPPT!F$s%D{P%ewC%jSB80peQxsgKgUiDTrGT6rXccm&~#2FZP)$e z-oAE2@pj&VyAwxGKm4Wt;FsYE^05EG+~k1dx3@Ku%sE#kPku7l+URbbq_;S0^YYYH zXNUJU>?~LHx|c^;yR2NZPEDKb&z-H>?CE)2`Fq>gt)JIYm8!S?;V*U8`8d%pvFn>p zZ_VJPf?nHC{C_sT^_!{F4*u{Ts7#9qTJHGmJfp|J3>`TD8P4(nlt1B0;yLX5{+N$m|wfAGJ=y*x0mW4HvFdjYnR+ zwxv9;EkfF3UXA3;9*S`j8(55ZGBc2vXWh0dy0$#~tMzMdo$(@%-vQ5_x{3}=C+l8m z^frEaGk(c&@4+=z4(9_6BCk(OdYokrFkc5?dy4-S9w|DTP~Qlsq3vB#nrFW3`B zl#%;X*m?gV%7p2ZB@WoAu+KE%WOxt^8F2nx-yYv*HUHnF&LBYg6eW2mMdB zslbMy=OhQnTCx@xY>OT4mPiyAdpsE^Z?KYbUJS9-jx7pugX2VqGR=Vx2w+bv&BsNe zS~%gGkdH+XB=3-EvKH7D0{TLN>_vno%Znwk2dbPo7DbedC&L}F*cJ@IP8O#g)v|+p zi>M#SLcX`-SRm%$mcRqV76PXoey3eI`2fH`i*bGG0sH6{&ghG>xaM0#J^U^#+(#70 zj!8@q-Jh61D$=g7u%oCYZx^L7#XB9OFgLumrz4-@7fDQ39Ud*T-4FHYNP$c{N!;U= z&wE;6`|ffg|L48sq}$pN76h&s{?8O_vV1ZL+s-(^!&4lma$AC!(aSp)cQ?SzLN3x+ zC2JB19E)(7x#5)$9aCIBW=B5S;hl!3o=kL;gdj9(fOmDII>j-+z)PFP81Qf>Y#1*iCAof$NSOyx5NG2WZ;i1d?Xv_ z82gFx7V4oyHzoKacNR(5u`?4b)bRpQ`uF~raSH1TGsED18}08TrmTjTXG*yhV*lvQ zl4KuDSG#}qxbd>Ht2p^7Ntk0{sEO4@yMZKR6j_*(&NQ7TN-L8MLfB!e>oWl# zA>Q^f`tFi&ioQ?#6Zl?Yq84`KUFe5pK|dUB0xmO2mM>5+4GZwOnS3@r91OXeQp`no zd@BY%;aT{ydicP*vkVcKV&Fx{L~AvL#bd$1P?*enW(01;UQT2gJMKfrvoLNF&w*Y+-*VT+ pA)Z1$_)L= 2x --waypoint-tolerance " + "or the corridor check cannot discriminate " + "route-following from goal-beelining. Default: open " + "30 m square (3 corners) climbing 10 m above " + "takeoff altitude, so the route clears scene " + "clutter (e.g. AirSim Blocks) — this test judges " + "route-following, not obstacle avoidance.") + parser.addoption("--waypoint-tolerance", default="15", + help="Pass distance (m) to each intermediate waypoint in " + "test_waypoint_flight. Calibrated to stock droan_gl " + "plan-following, which trades deviation for path " + "progress 1:1 and cuts corners deeply (7-10 m " + "observed in Isaac). Default: 15") + parser.addoption("--goal-tolerance", default="2.5", + help="Pass distance (m) to the FINAL waypoint in " + "test_waypoint_flight: NavigateTask goal tolerance " + "(1.5 m) plus tracking-point lag margin. Default: 2.5") + parser.addoption("--waypoint-timeout", default="120", + help="Per-waypoint time budget (s, odometry clock) in " + "test_waypoint_flight. Default: 120") def pytest_configure(config): diff --git a/tests/pytest.ini b/tests/pytest.ini index dc8c939de..6f4547c0c 100644 --- a/tests/pytest.ini +++ b/tests/pytest.ini @@ -8,6 +8,7 @@ markers = sensors: Sim and robot sensor topic rates, LiDAR validation, sim RTF takeoff_hover_land: End-to-end takeoff / hover / land action tests autonomy: Fixed-pattern trajectory path-tracker benchmark (test_fixed_trajectory.py) + waypoint_flight: Ordered-waypoint navigation judged on the odometry track (test_waypoint_flight.py) testpaths = . addopts = -v --durations=0 --import-mode=importlib cache_dir = /tmp/.pytest_cache diff --git a/tests/system/test_waypoint_flight.py b/tests/system/test_waypoint_flight.py new file mode 100644 index 000000000..1ad501b4b --- /dev/null +++ b/tests/system/test_waypoint_flight.py @@ -0,0 +1,372 @@ +"""Ordered-waypoint navigation flight tests. + +Per (sim, num_robots, iter): ready → takeoff → navigate waypoint route → land. + +After takeoff the test sends the route to the local planner's NavigateTask +action (``/robot_N/tasks/navigate``) as a dense ``nav_msgs/Path`` and +captures odometry throughout. Pass/fail is judged by the standalone +``tests/waypoint_checker.py``: the odometry track must pass within +``--waypoint-tolerance`` of every waypoint **in order**, each within +``--waypoint-timeout`` seconds of the previous arrival, and additionally +end within ``--goal-tolerance`` of the final waypoint. Judging on the +odometry track (rather than the action result) keeps the success criterion +independent of any particular planner implementation — swap the global or +local planner and the same test still judges the flight. + +Tolerance calibration: the stack's navigation contract is "reach the goal +precisely, follow the route corridor loosely" — stock droan_gl scores +candidate trajectories with cost = deviation - path_distance, which cuts +corners deeply (7-10 m observed in Isaac). Hence the loose default +intermediate tolerance (15 m) with a tight final goal tolerance +(2.5 m = NavigateTask's 1.5 m + tracking-point lag; capture continues +after action success until the drone is stationary, since the action +succeeds on the tracking point, which leads the drone by up to the +look-ahead distance). Route legs must be >= 2x the intermediate +tolerance for the corridor check to discriminate route-following from +goal-beelining. + +Waypoints are given relative to the robot's pose at dispatch (x forward +along initial heading, z up from dispatch altitude) so routes are +spawn-point and sim agnostic. + +Route design constraint: NavigateTask declares success when the tracking +point is within goal_tolerance_m of the route's FINAL pose, so a route +that closes back on its start "succeeds" instantly without flying. +Routes must end away from the start; the default is an open square. +""" + +import math +import time + +import pytest + +from conftest import ( + current_test_id, + get_metrics, + get_robot_containers, + logger, + ros2_exec, +) +from system.test_fixed_trajectory import ( + ODOM_SCHEMA, + TARGET_ALTITUDE_M, + _action_message, + _action_ok, + _finish_captures, + _landing_one_robot, + _quat_to_yaw, + _run_parallel, + _stamp, + _start_captures, + _takeoff_one_robot, + _transform_to_world, +) +from waypoint_checker import check_track, parse_waypoints + +PX4_READY_TIMEOUT_S = 300.0 +NAVIGATE_GOAL_TOLERANCE_M = 1.5 # goal_tolerance_m sent in the NavigateTask goal +# The action succeeds on the TRACKING POINT (controller reference), which leads +# the drone by up to the look-ahead distance (~10 m observed) — keep capturing +# after the action returns until the drone itself stops moving. +MAX_SETTLE_S = 30.0 +SETTLE_POLL_S = 2.0 +SETTLE_STATIONARY_M = 0.3 + +METRIC_UNITS = { + "waypoint_success": "", + "waypoints_reached": "", + "navigate_action_success": "", + "route_time_sim_s": "s", + "ready_duration_sys_s": "s", + # Everything else defaults to "m". +} + + +def _record(robot_n: int, metrics_dict: dict) -> None: + """Record per-robot scalar metrics; unit inferred from METRIC_UNITS.""" + m = get_metrics() + tid = current_test_id() + higher = {"waypoint_success", "waypoints_reached", "navigate_action_success"} + for key, value in metrics_dict.items(): + if value is None: + continue + unit = METRIC_UNITS.get(key, "m") + direction = "higher_is_better" if key in higher else "lower_is_better" + m.record(tid, f"robot_{robot_n}.{key}", value, unit=unit, direction=direction) + + +PLAN_POINT_SPACING_M = 1.0 + + +def _densify(pts: list[tuple[float, float, float]], + spacing: float = PLAN_POINT_SPACING_M) -> list[tuple[float, float, float]]: + """Linearly interpolate between consecutive points at ~spacing intervals. + + The local planner walks the global plan by distance with a look-ahead; + sparse poses (e.g. 10 m apart) get skipped over and corners are cut, so + the dispatched plan must be dense like a real global planner's output. + """ + dense = [pts[0]] + for (ax, ay, az), (bx, by, bz) in zip(pts, pts[1:]): + d = math.sqrt((bx - ax) ** 2 + (by - ay) ** 2 + (bz - az) ** 2) + steps = max(1, math.ceil(d / spacing)) + for i in range(1, steps + 1): + f = i / steps + dense.append((ax + (bx - ax) * f, + ay + (by - ay) * f, + az + (bz - az) * f)) + return dense + + +def _build_navigate_goal(world_pts: list[tuple[float, float, float]], + frame_id: str, tolerance_m: float) -> str: + """YAML goal for a NavigateTask send_goal call from world-frame waypoints. + + frame_id must be a real TF frame: the local planner TF-transforms the + plan by its header frame and dies on an empty one. + """ + poses = ", ".join( + f"{{pose: {{position: {{x: {x:.3f}, y: {y:.3f}, z: {z:.3f}}}, " + f"orientation: {{w: 1.0}}}}}}" + for x, y, z in world_pts + ) + return (f"{{global_plan: {{header: {{frame_id: '{frame_id}'}}, " + f"poses: [{poses}]}}, goal_tolerance_m: {tolerance_m}}}") + + +def _snapshot_start_pose(robot_container: str, cfg: dict, n: int): + """World-frame (x, y, z, yaw, frame_id) of robot n, from one odom sample.""" + snap = ros2_exec( + robot_container, + f"timeout 5 ros2 topic echo --once --csv " + f"/robot_{n}/interface/mavros/local_position/odom", + domain_id=n, setup_bash=cfg["robot_setup_bash"], timeout=10, + ) + for line in snap.stdout.splitlines(): + parts = line.strip().split(",") + if len(parts) >= len(ODOM_SCHEMA): + try: + row = dict(zip(ODOM_SCHEMA, parts)) + return ( + float(row["pose.pose.position.x"]), + float(row["pose.pose.position.y"]), + float(row["pose.pose.position.z"]), + _quat_to_yaw( + float(row["pose.pose.orientation.x"]), + float(row["pose.pose.orientation.y"]), + float(row["pose.pose.orientation.z"]), + float(row["pose.pose.orientation.w"]), + ), + row["header.frame_id"].strip() or "map", + ) + except (ValueError, KeyError): + pass + return 0.0, 0.0, TARGET_ALTITUDE_M, 0.0, "map" + + +def _navigate_one_robot(n: int, robot_container: str, cfg: dict, + waypoints_rel: list[tuple[float, float, float]], + tolerance_m: float, goal_tolerance_m: float, + budget_s: float) -> None: + route_timeout = budget_s * len(waypoints_rel) + 30.0 + + x0, y0, z0, yaw0, frame_id = _snapshot_start_pose(robot_container, cfg, n) + world_pts = _transform_to_world(waypoints_rel, x0, y0, z0, yaw0) + logger.info("robot_%d waypoint route (frame %s): %s", n, frame_id, + [(round(x, 1), round(y, 1), round(z, 1)) for x, y, z in world_pts]) + # Dispatch a dense plan from the current pose through the waypoints + # (mirrors real global-planner output); judge only the user waypoints. + plan_pts = _densify([(x0, y0, z0)] + world_pts) + + streams = _start_captures(robot_container, cfg["robot_setup_bash"], + n, route_timeout + MAX_SETTLE_S + 10, "waypoints") + goal = _build_navigate_goal(plan_pts, frame_id, NAVIGATE_GOAL_TOLERANCE_M) + result = ros2_exec( + robot_container, + f'ros2 action send_goal --feedback /robot_{n}/tasks/navigate ' + f'task_msgs/action/NavigateTask "{goal}"', + domain_id=n, setup_bash=cfg["robot_setup_bash"], + timeout=int(route_timeout + 15), + ) + # Record the drone catching up to the tracking point: poll its position + # until it is stationary (arrived and hovering) or MAX_SETTLE_S elapses. + settle_deadline = time.monotonic() + MAX_SETTLE_S + prev_pos = None + while time.monotonic() < settle_deadline: + px, py, pz, _, _ = _snapshot_start_pose(robot_container, cfg, n) + if prev_pos is not None and math.dist((px, py, pz), prev_pos) < SETTLE_STATIONARY_M: + break + prev_pos = (px, py, pz) + time.sleep(SETTLE_POLL_S) + odom = _finish_captures(streams) + + action_success = _action_ok(result.stdout) + _record(n, {"navigate_action_success": 1.0 if action_success else 0.0}) + if not action_success: + logger.warning("robot_%d navigate action did not succeed: %s", + n, _action_message(result.stdout)) + + if not odom: + pytest.fail(f"robot_{n} waypoint flight: no odom samples captured") + + rows = [(_stamp(r), + r["pose.pose.position.x"], + r["pose.pose.position.y"], + r["pose.pose.position.z"]) for r in odom] + verdict = check_track(rows, world_pts, tolerance_m, budget_s) + + reached = sum(1 for w in verdict["waypoints"] if w["reached"]) + goal_error_m = verdict["waypoints"][-1]["closest_approach_m"] + metrics = { + "waypoint_success": 1.0 if verdict["success"] else 0.0, + "waypoints_reached": float(reached), + "route_time_sim_s": verdict.get("total_time_s"), + "worst_closest_approach_m": max( + w["closest_approach_m"] for w in verdict["waypoints"]), + "final_goal_error_m": goal_error_m, + } + _record(n, metrics) + for w in verdict["waypoints"]: + logger.info( + "robot_%d waypoint %d: reached=%s closest=%.2fm elapsed=%ss", + n, w["index"], w["reached"], w["closest_approach_m"], + w.get("elapsed_from_prev_s", "n/a")) + + assert verdict["success"], ( + f"robot_{n} reached {reached}/{len(world_pts)} waypoints in order " + f"(tolerance {tolerance_m}m, budget {budget_s}s/waypoint); " + f"closest approaches: " + f"{[w['closest_approach_m'] for w in verdict['waypoints']]}" + ) + assert goal_error_m <= goal_tolerance_m, ( + f"robot_{n} final goal error {goal_error_m:.2f}m exceeds " + f"{goal_tolerance_m}m (NavigateTask tolerance " + f"{NAVIGATE_GOAL_TOLERANCE_M}m + tracking margin)" + ) + + +# ── test class ───────────────────────────────────────────────────────────── + +@pytest.mark.waypoint_flight +@pytest.mark.timeout(2400) +class TestWaypointFlight: + """Full takeoff → waypoint route → land chain, judged by waypoint_checker. + + Route, tolerance, and per-waypoint budget come from --waypoints, + --waypoint-tolerance, and --waypoint-timeout. + """ + + @pytest.fixture(scope="session") + def _failed_envs(self): + return set() + + @pytest.fixture(autouse=True) + def _chain_guard(self, request, airstack_env, _failed_envs): + """Skip tests whose env was poisoned by an earlier failure. + + A waypoint-flight failure does NOT poison the env — landing always + runs after a successful takeoff. Takeoff or landing failures do. + """ + env_id = (airstack_env["sim"], airstack_env["num_robots"], + airstack_env["iteration"]) + if env_id in _failed_envs: + pytest.skip(f"earlier waypoint-flight test failed in {env_id}") + yield + rep = getattr(request.node, "_rep_call", None) + if rep is not None and rep.failed: + if "test_waypoint_route" not in request.node.name: + _failed_envs.add(env_id) + + @pytest.fixture + def waypoints_rel(self, request): + return parse_waypoints(request.config.getoption("--waypoints")) + + @pytest.fixture + def tolerance_m(self, request): + return float(request.config.getoption("--waypoint-tolerance")) + + @pytest.fixture + def goal_tolerance_m(self, request): + return float(request.config.getoption("--goal-tolerance")) + + @pytest.fixture + def budget_s(self, request): + return float(request.config.getoption("--waypoint-timeout")) + + @pytest.mark.dependency(name="wpf_ready") + def test_px4_ready(self, airstack_env): + """Wait until MAVROS is connected and local_position/odom publishes.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + num_robots = airstack_env["num_robots"] + + started = time.time() + connected: set[int] = set() + pending = list(range(1, num_robots + 1)) + deadline = started + PX4_READY_TIMEOUT_S + + while pending and time.time() < deadline: + for n in list(pending): + if n not in connected: + r = ros2_exec( + robot_container, + f"timeout 5 ros2 topic echo --once --csv " + f"--field connected /robot_{n}/interface/mavros/state", + domain_id=n, setup_bash=cfg["robot_setup_bash"], timeout=10, + ) + if any(line.strip() == "True" for line in r.stdout.splitlines()): + connected.add(n) + else: + continue + r = ros2_exec( + robot_container, + f"timeout 5 ros2 topic echo --once " + f"/robot_{n}/interface/mavros/local_position/odom", + domain_id=n, setup_bash=cfg["robot_setup_bash"], timeout=10, + ) + if r.returncode == 0 and "pose:" in r.stdout: + _record(n, {"ready_duration_sys_s": + round(time.time() - started, 2)}) + pending.remove(n) + if pending: + logger.info("px4_ready: connected=%s pending=%s elapsed=%.0fs", + sorted(connected), pending, time.time() - started) + time.sleep(2.0) + + if pending: + pytest.fail( + f"robots {sorted(pending)} not ready (MAVROS connected + odom) " + f"within {PX4_READY_TIMEOUT_S:.0f}s" + ) + + @pytest.mark.dependency(name="wpf_takeoff", depends=["wpf_ready"]) + def test_takeoff(self, airstack_env): + """Take off to TARGET_ALTITUDE_M at a fixed velocity of 1 m/s.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + _run_parallel( + airstack_env["num_robots"], + lambda n: _takeoff_one_robot(n, robot_container, cfg, TARGET_ALTITUDE_M), + ) + + @pytest.mark.dependency(name="wpf_route", depends=["wpf_takeoff"]) + def test_waypoint_route(self, airstack_env, waypoints_rel, tolerance_m, + goal_tolerance_m, budget_s): + """Send NavigateTask with the route; judge odometry with waypoint_checker.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + _run_parallel( + airstack_env["num_robots"], + lambda n: _navigate_one_robot(n, robot_container, cfg, waypoints_rel, + tolerance_m, goal_tolerance_m, budget_s), + ) + + @pytest.mark.dependency(name="wpf_land", depends=["wpf_takeoff"]) + def test_landing(self, airstack_env): + """Land the drone; runs even when test_waypoint_route fails.""" + cfg = airstack_env["cfg"] + robot_container = get_robot_containers(airstack_env["robot_pattern"])[0] + _run_parallel( + airstack_env["num_robots"], + lambda n: _landing_one_robot(n, robot_container, cfg), + ) diff --git a/tests/waypoint_checker.py b/tests/waypoint_checker.py new file mode 100644 index 000000000..7a7683157 --- /dev/null +++ b/tests/waypoint_checker.py @@ -0,0 +1,164 @@ +#!/usr/bin/env python3 +"""Standalone ordered-waypoint track checker. + +Judges whether an odometry track visited an ordered list of waypoints, each +within a distance tolerance and a per-waypoint time budget. Used by +``tests/system/test_waypoint_flight.py``, but deliberately dependency-free +(stdlib only) and runnable outside the AirStack harness: success is defined +purely on the odometry track, not on any AirStack-specific interface, so the +same checker can judge waypoint flight on any ROS 2 system that can dump +odometry to CSV:: + + ros2 topic echo --csv /robot_1/interface/mavros/local_position/odom > odom.csv + python3 waypoint_checker.py --odom-csv odom.csv \ + --waypoints "10,0,10; 10,10,10; 0,10,10" --tolerance 1.5 --budget 120 + +Exit code 0 iff every waypoint was reached in order within tolerance and +budget; a JSON verdict is printed to stdout either way. + +Input CSV format: the flattened ``ros2 topic echo --csv`` output of +``nav_msgs/Odometry`` (header stamp in columns 0-1, position x/y/z in columns +4-6). Lines that do not parse (ros2 banners, partial writes) are skipped. +""" + +import argparse +import json +import math +import sys + +# Column indices in `ros2 topic echo --csv` output for nav_msgs/Odometry +# (all primitives flattened in declaration order). +_COL_STAMP_SEC = 0 +_COL_STAMP_NSEC = 1 +_COL_POS_X = 4 +_COL_POS_Y = 5 +_COL_POS_Z = 6 +# A full Odometry row has 4 header/frame fields + 7 pose + 36 cov + 6 twist +# + 36 cov = 89 columns; require at least position columns to be present. +_MIN_COLUMNS = 7 + + +def parse_odom_csv(path): + """Parse a ros2 ``--csv`` odometry dump into [(t, x, y, z), ...] rows.""" + rows = [] + with open(path) as fh: + for line in fh: + parts = line.strip().split(",") + if len(parts) < _MIN_COLUMNS: + continue + try: + t = float(parts[_COL_STAMP_SEC]) + float(parts[_COL_STAMP_NSEC]) * 1e-9 + x = float(parts[_COL_POS_X]) + y = float(parts[_COL_POS_Y]) + z = float(parts[_COL_POS_Z]) + except ValueError: + continue + rows.append((t, x, y, z)) + return rows + + +def parse_waypoints(spec): + """Parse ``"x,y,z; x,y,z; ..."`` into [(x, y, z), ...].""" + waypoints = [] + for chunk in spec.split(";"): + chunk = chunk.strip() + if not chunk: + continue + parts = [p.strip() for p in chunk.split(",")] + if len(parts) != 3: + raise ValueError(f"waypoint {chunk!r} is not 'x,y,z'") + waypoints.append(tuple(float(p) for p in parts)) + if not waypoints: + raise ValueError("no waypoints given") + return waypoints + + +def check_track(rows, waypoints, tolerance_m, budget_s_per_waypoint): + """Check that the track visits every waypoint in order. + + A waypoint counts as reached at the first sample (searching forward from + the previous waypoint's arrival) within ``tolerance_m`` of it, provided + that sample's time is within ``budget_s_per_waypoint`` of the previous + arrival (or of track start, for the first waypoint). Times are whatever + clock stamped the odometry (sim time in AirStack runs). + + Returns a verdict dict; ``verdict["success"]`` is the pass/fail judgment. + """ + verdict = { + "success": False, + "num_odom_samples": len(rows), + "tolerance_m": tolerance_m, + "budget_s_per_waypoint": budget_s_per_waypoint, + "waypoints": [], + } + if not rows: + verdict["error"] = "no odometry samples" + return verdict + + start_idx = 0 + prev_arrival_t = rows[0][0] + all_reached = True + + for wi, (wx, wy, wz) in enumerate(waypoints): + # arrival = FIRST sample within tolerance (preserves ordering + # semantics); closest_approach = true minimum over the whole + # remaining track, so the report reflects how near the drone + # actually got, not just the tolerance boundary crossing. + arrival_idx = None + closest = math.inf + for i in range(start_idx, len(rows)): + t, x, y, z = rows[i] + d = math.sqrt((x - wx) ** 2 + (y - wy) ** 2 + (z - wz) ** 2) + if d < closest: + closest = d + if arrival_idx is None and d <= tolerance_m: + arrival_idx = i + + entry = { + "index": wi, + "target": [wx, wy, wz], + "closest_approach_m": round(closest, 3), + "reached": arrival_idx is not None, + } + if arrival_idx is not None: + arrival_t = rows[arrival_idx][0] + elapsed = arrival_t - prev_arrival_t + entry["elapsed_from_prev_s"] = round(elapsed, 3) + entry["within_budget"] = elapsed <= budget_s_per_waypoint + if not entry["within_budget"]: + all_reached = False + start_idx = arrival_idx + prev_arrival_t = arrival_t + else: + all_reached = False + # Keep evaluating later waypoints from the same index so the + # verdict reports closest approaches for all of them. + verdict["waypoints"].append(entry) + + verdict["total_time_s"] = round(rows[-1][0] - rows[0][0], 3) + verdict["success"] = all_reached + return verdict + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + ap.add_argument("--odom-csv", required=True, + help="ros2 topic echo --csv dump of nav_msgs/Odometry") + ap.add_argument("--waypoints", required=True, + help="Ordered waypoints 'x,y,z; x,y,z; ...' in the odometry frame") + ap.add_argument("--tolerance", type=float, default=1.5, + help="Pass distance to each waypoint in meters (default 1.5)") + ap.add_argument("--budget", type=float, default=120.0, + help="Time budget per waypoint in seconds of odometry " + "clock (default 120)") + args = ap.parse_args(argv) + + rows = parse_odom_csv(args.odom_csv) + waypoints = parse_waypoints(args.waypoints) + verdict = check_track(rows, waypoints, args.tolerance, args.budget) + print(json.dumps(verdict, indent=2)) + return 0 if verdict["success"] else 1 + + +if __name__ == "__main__": + sys.exit(main()) From ae2e94202901796d1fc543c8c66526ad50925556 Mon Sep 17 00:00:00 2001 From: Andrew Jong Date: Tue, 4 Aug 2026 17:20:23 -0700 Subject: [PATCH 11/21] Add feature-notebook workflow: per-feature design specs + test results feeding PRs (#381) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Add feature-notebook workflow: local design specs + test results per feature Every feature a coding agent implements now gets a numbered entry under notebook/ (gitignored, local-only): a design_spec.md written before coding (problem context from the session, proposed implementation with per-section DESIGN/TODO / WIP / DONE status labels, lettered test plan) and a results/ tree with per-section raw artifacts plus a self-contained results_summary.md (embedded tables + figures) that populates the feature's PR description. - New skill .agents/skills/use-feature-notebook with SKILL.md and design_spec / results_summary templates - AGENTS.md: skill registry row, notebook-first Agent Workflow Example, new "Feature Notebook" section - .gitignore: /notebook/ Co-Authored-By: Claude Fable 5 * Bump version to 0.19.0-alpha.10 Co-Authored-By: Claude Fable 5 * Document the feature notebook workflow under Development docs Adds docs/development/intermediate/feature_notebook.md (directory layout, 5-step workflow, status labels, local-only rule, notebook → PR flow), wires it into the mkdocs nav under Development > Intermediate Tutorials > Contributing, and lists it in the Development index. Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- .agents/skills/use-feature-notebook/SKILL.md | 90 +++++++++++++++++++ .../assets/design_spec_template.md | 58 ++++++++++++ .../assets/results_summary_template.md | 32 +++++++ .env | 2 +- .gitignore | 4 + AGENTS.md | 32 +++++-- CHANGELOG.md | 1 + docs/development/index.md | 1 + .../intermediate/feature_notebook.md | 67 ++++++++++++++ mkdocs.yml | 3 +- 10 files changed, 282 insertions(+), 8 deletions(-) create mode 100644 .agents/skills/use-feature-notebook/SKILL.md create mode 100644 .agents/skills/use-feature-notebook/assets/design_spec_template.md create mode 100644 .agents/skills/use-feature-notebook/assets/results_summary_template.md create mode 100644 docs/development/intermediate/feature_notebook.md diff --git a/.agents/skills/use-feature-notebook/SKILL.md b/.agents/skills/use-feature-notebook/SKILL.md new file mode 100644 index 000000000..7a2f2dc8a --- /dev/null +++ b/.agents/skills/use-feature-notebook/SKILL.md @@ -0,0 +1,90 @@ +--- +name: use-feature-notebook +description: Maintain a local, gitignored notebook/ directory that records the design spec and test results for every feature an agent implements. Trigger at the START of any feature-implementation task (create notebook/NNN-feature-slug/design_spec.md before writing code), while implementing (keep the spec's per-section status labels DESIGN/TODO / WIP / DONE current), whenever tests for that feature produce output worth keeping (store under results/
/), and when opening the feature's PR (populate the PR body from results/results_summary.md). +license: Apache-2.0 +metadata: + author: AirLab CMU + repository: AirStack +--- + +# Skill: Use the Feature Notebook + +## Purpose + +Every feature implemented by a coding agent gets a **notebook entry**: a numbered folder under `notebook/` at the repo root that holds the design spec written *before* implementation and the test results produced *during* validation. The notebook is the agent's lab journal — it captures the session context that would otherwise be lost when the conversation ends, and it is the source material for the feature's PR description. + +`notebook/` is **gitignored and local-only**. It never lands in a commit. Each developer's machine has its own copy. What *does* leave the machine is the distilled content: the PR body is populated from `results/results_summary.md`, and figures/tables from `results/` are attached to the PR. + +## Directory Layout + +``` +notebook/ +├── 001-add-new-planner/ +│ ├── design_spec.md # Written BEFORE implementation +│ └── results/ +│ ├── results_summary.md # Written AFTER tests; feeds the PR +│ ├── a-planner-core/ # Raw artifacts for test section (a) +│ │ ├── run1_metrics.json +│ │ └── trajectory_plot.png +│ └── b-planner-hyperparameters/ # Raw artifacts for test section (b) +│ └── sweep_table.csv +├── 002-fix-lidar-filter/ +│ └── ... +``` + +Naming rules: + +- **Feature folder:** `NNN-short-kebab-slug`, where `NNN` is zero-padded three digits. Pick the next number by listing `notebook/` and incrementing the highest existing prefix (start at `001` if empty or missing — create `notebook/` yourself, it is not committed). +- **Results subfolders:** one per lettered test section in `design_spec.md`, named `-` (e.g. section "(a) Planner core" → `results/a-planner-core/`). The letters MUST match the test-plan section letters in the spec so a reader can navigate spec ↔ results directly. + +## Workflow + +### 1. On starting a feature — write `design_spec.md` + +Before writing any implementation code, create `notebook/NNN-feature-slug/design_spec.md` from [assets/design_spec_template.md](assets/design_spec_template.md). It must capture: + +- **Problem context** — what the developer is trying to solve, in the developer's own framing from the session: motivation, constraints, prior attempts, and any decisions already made in the conversation. This is the section that preserves context which exists nowhere else. +- **Proposed implementation** — the design: affected packages, new/changed nodes and topics, algorithms, data flow. Diagrams (mermaid) welcome. Split into subsections if the implementation has multiple parts. +- **Test plan** — lettered sections `(a)`, `(b)`, `(c)`… each describing one validation axis: what is run (unit test, system test mark, sim scenario), what is measured, and what outcome counts as pass. These letters define the `results/` subfolder names. + +If the design changes materially mid-implementation, update the spec — it should describe what was actually built, with a short note on what changed and why. + +### 2. While implementing — keep the spec's status labels current + +`design_spec.md` carries an implementation status at two levels, using the values **`DESIGN/TODO`**, **`WIP`**, or **`DONE`**: + +- **Overall status** in the header block — the least-advanced status of any implementation section (all sections `DONE` → overall `DONE`; anything in progress → `WIP`; nothing started → `DESIGN/TODO`). +- **Per-section status** on each Proposed Implementation subsection heading (e.g. `### 2.1 Cost-map integration — \`WIP\``) — so when the implementation has multiple parts, a reader can see exactly which parts are designed, in progress, or finished. + +Update the labels **as you work**, not retroactively: mark a section `WIP` when you start writing its code and `DONE` when it is implemented and building. A spec whose statuses lag reality misleads the next agent that picks up the feature. + +### 3. During validation — store raw results + +Every test run that validates the feature drops its artifacts into the matching section folder, e.g. `notebook/001-add-new-planner/results/a-planner-core/`: + +- Metrics files (`metrics.json`, CSVs), copied from `tests/results//` when using the system test harness +- Plots and screenshots (cross-track error curves, Foxglove/RViz captures, sim screenshots) +- Relevant log excerpts — excerpts, not full container logs + +Keep raw artifacts as-produced; interpretation belongs in the summary. + +### 4. After validation — write `results/results_summary.md` + +Create `results/results_summary.md` from [assets/results_summary_template.md](assets/results_summary_template.md). One section per test-plan letter, mirroring the spec. The summary must be **self-contained**: embed the quantitative tables and qualitative figures directly in the document (markdown tables; images via relative paths like `![xte](a-planner-core/trajectory_plot.png)`) so a developer can understand the results all at once without opening the raw artifact folders. End with an overall verdict: which spec sections passed, which didn't, known limitations. + +### 5. On opening the PR — populate it from the notebook + +The PR body for the feature is built from the notebook, since reviewers cannot see `notebook/` itself: + +- **Motivation / context** ← `design_spec.md` problem context +- **What changed** ← proposed implementation (as-built) +- **Validation** ← `results_summary.md`: paste the summary tables, upload the key figures as PR attachments, and state the per-section verdicts + +## Pitfalls + +- ❌ Writing the spec after the code — the spec exists to record intent and session context before they're lost. +- ❌ Stale status labels — a spec still marked `DESIGN/TODO` (or a section marked `WIP`) after the work shipped misleads the next reader; update statuses as you go. +- ❌ Committing `notebook/` or referencing `notebook/...` paths from committed code, docs, or tests — it doesn't exist on other machines or in CI. +- ❌ Results subfolder letters that don't match the spec's test-plan letters. +- ❌ A `results_summary.md` that just links to raw files — embed the tables and figures. +- ❌ Confusing this with [capture-discovered-knowledge](../capture-discovered-knowledge): the notebook records *per-feature* design and evidence locally; durable repo-wide knowledge still goes to AGENTS.md/skills, and module documentation still follows [update-documentation](../update-documentation). diff --git a/.agents/skills/use-feature-notebook/assets/design_spec_template.md b/.agents/skills/use-feature-notebook/assets/design_spec_template.md new file mode 100644 index 000000000..ecb184ba2 --- /dev/null +++ b/.agents/skills/use-feature-notebook/assets/design_spec_template.md @@ -0,0 +1,58 @@ +# Design Spec: + +> Notebook entry: `notebook/NNN-feature-slug/` · Date started: YYYY-MM-DD · Branch: `` +> +> **Status: `DESIGN/TODO`** + +## 1. Problem Context + + + +## 2. Proposed Implementation + + + +### 2.1 — `DESIGN/TODO` + + + +### 2.2 — `DESIGN/TODO` + + + +### Affected packages + +| Package | Change | +|---------|--------| +| `path/to/package` | ... | + +### Interfaces + +| Topic / Service / Param | Type | Direction | Purpose | +|-------------------------|------|-----------|---------| +| | | | | + +## 3. Test Plan + + + +### (a)
+ +- **What is run:** +- **What is measured:** +- **Pass criteria:** + +### (b)
+ +- **What is run:** +- **What is measured:** +- **Pass criteria:** diff --git a/.agents/skills/use-feature-notebook/assets/results_summary_template.md b/.agents/skills/use-feature-notebook/assets/results_summary_template.md new file mode 100644 index 000000000..5f3b0977f --- /dev/null +++ b/.agents/skills/use-feature-notebook/assets/results_summary_template.md @@ -0,0 +1,32 @@ +# Results Summary: + +> Spec: [`../design_spec.md`](../design_spec.md) · Date: YYYY-MM-DD · Commit tested: `` + + + +## (a)
+ +**Setup:** + +| Metric | Value | Pass criterion | Pass? | +|--------|-------|----------------|-------| +| | | | | + +![description](a-section-slug/figure.png) + +**Interpretation:** + +## (b)
+ +... + +## Overall Verdict + +| Spec section | Verdict | +|--------------|---------| +| (a) ... | ✅ / ❌ | +| (b) ... | ✅ / ❌ | + +**Known limitations:** diff --git a/.env b/.env index c9527e1f8..020228fa2 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.9" +VERSION="0.19.0-alpha.10" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/.gitignore b/.gitignore index 4868b5c74..d9dd6a07b 100644 --- a/.gitignore +++ b/.gitignore @@ -94,6 +94,10 @@ simulation/ms-airsim/assets/scenes/* # Test results tests/results/ +# Per-feature agent notebook (design specs + test results) — local-only, feeds PR descriptions. +# See .agents/skills/use-feature-notebook +/notebook/ + # Local-only — embedded sibling repo, not part of this branch common/rayfronts/ diff --git a/AGENTS.md b/AGENTS.md index 1579006ec..88d524483 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,17 +100,37 @@ For detailed step-by-step instructions, refer to the **`.agents/skills/`** direc | [configure-multi-robot](.agents/skills/configure-multi-robot) | Setting up multiple robots, ROBOT_NAME namespacing, and ROS_DOMAIN_ID isolation | | [bump-version-and-release](.agents/skills/bump-version-and-release) | Bumping `.env` VERSION and CHANGELOG before merge to clear the version-check gate | | [capture-discovered-knowledge](.agents/skills/capture-discovered-knowledge) | After long context-discovery / surprising findings, persist to AGENTS.md or a new skill so the next agent doesn't redo the work | +| [use-feature-notebook](.agents/skills/use-feature-notebook) | At the start of EVERY feature implementation: create `notebook/NNN-feature-slug/design_spec.md`, store test artifacts under `results/`, write `results/results_summary.md`, and populate the PR from it | **Agent Workflow Example:** -1. Study reference implementation for module type -2. Follow `add_ros2_package.md` to create package structure -3. Implement algorithm with proper topic interfaces -4. Follow `integrate_module_into_layer.md` to add to bringup -5. Follow `update_documentation.md` to document -6. Follow `debug_module.md` and `test_in_simulation.md` to verify +1. **Create a notebook entry** — follow `use-feature-notebook` to write `notebook/NNN-feature-slug/design_spec.md` (problem context, proposed implementation, lettered test plan) before writing code +2. Study reference implementation for module type +3. Follow `add_ros2_package.md` to create package structure +4. Implement algorithm with proper topic interfaces +5. Follow `integrate_module_into_layer.md` to add to bringup +6. Follow `update_documentation.md` to document +7. Follow `debug_module.md` and `test_in_simulation.md` to verify, saving artifacts under `notebook/NNN-feature-slug/results/-
/` +8. Write `results/results_summary.md` (embedded tables + figures) and populate the PR body from it Also see: [AI Agent Quick Guide](docs/development/ai_agent_guide.md) +## Feature Notebook (`notebook/`) + +Every feature an agent implements gets a numbered entry under `notebook/` at the repo root — a local lab journal that survives the agent session: + +``` +notebook/001-add-new-planner/ +├── design_spec.md # BEFORE coding: problem context from the session, proposed implementation, lettered test plan +└── results/ + ├── results_summary.md # AFTER testing: self-contained doc with embedded tables + figures, per-section verdicts + ├── a-planner-core/ # Raw artifacts per test-plan section (letters match design_spec.md) + └── b-planner-hyperparameters/ +``` + +`notebook/` is **gitignored — local-only on each developer's machine**. Never commit it or reference its paths from committed code. Its content leaves the machine one way: the feature's PR description is populated from `design_spec.md` (motivation, what changed) and `results_summary.md` (validation tables, figures uploaded as PR attachments). + +**Full workflow and templates:** [.agents/skills/use-feature-notebook](.agents/skills/use-feature-notebook) + ## Reference Implementations Study these well-structured modules as examples for different types: diff --git a/CHANGELOG.md b/CHANGELOG.md index 43940f0b7..5a6673928 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Feature notebook workflow (`use-feature-notebook` skill): every agent-implemented feature gets a local, gitignored `notebook/NNN-feature-slug/` entry with a status-tracked `design_spec.md` (written before coding) and `results/` artifacts + self-contained `results_summary.md` that populate the feature's PR description - Battery and telemetry display in GCS RQT control panel (voltage and percentage per robot when MAVROS battery topic is bridged) - `TARGET_ARCH` build arg (default `x86_64`) in `Dockerfile.robot` to arch-parametrize `LD_LIBRARY_PATH`; `docker-compose.yaml` passes `TARGET_ARCH: aarch64` to the `voxl` and `l4t` real-robot image builds - `ros-${ROS_DISTRO}-mavros-extras` in the robot image (provides the vision_pose plugin used for external-pose deployments) diff --git a/docs/development/index.md b/docs/development/index.md index a1ccd4663..9ddd13678 100644 --- a/docs/development/index.md +++ b/docs/development/index.md @@ -34,6 +34,7 @@ Welcome to AirStack development! This guide will help you extend and customize t - **[Docker Build Profiles](intermediate/docker-build-profiles.md)** - Robot image build args and platform profiles (`robot-desktop`, `robot-l4t`, etc.) - **[Contributing](intermediate/contributing.md)** - Contribute to AirStack - **[Documentation Guide](intermediate/documentation.md)** - Write great documentation +- **[Feature Notebook](intermediate/feature_notebook.md)** - Record design specs and test results per feature; populate PRs from them ### 🚀 Advanced Tutorials diff --git a/docs/development/intermediate/feature_notebook.md b/docs/development/intermediate/feature_notebook.md new file mode 100644 index 000000000..10dc776a9 --- /dev/null +++ b/docs/development/intermediate/feature_notebook.md @@ -0,0 +1,67 @@ +# Feature Notebook + +Every feature implemented with a coding agent gets a **notebook entry**: a numbered folder under `notebook/` at the repository root that records the design spec *before* implementation and the test results *after*. The notebook is a lab journal — it preserves the session context (problem framing, design decisions, validation evidence) that would otherwise be lost when the agent conversation ends, and it is the source material for the feature's pull request description. + +!!! warning "Local-only" + `notebook/` is **gitignored**. It never lands in a commit, and committed code, docs, and tests must never reference `notebook/...` paths — the directory doesn't exist on other machines or in CI. Its content leaves your machine one way: distilled into the feature's PR description. + +## Directory Layout + +```text +notebook/ +├── 001-add-new-planner/ +│ ├── design_spec.md # Written BEFORE implementation +│ └── results/ +│ ├── results_summary.md # Written AFTER tests; feeds the PR +│ ├── a-planner-core/ # Raw artifacts for test section (a) +│ │ ├── run1_metrics.json +│ │ └── trajectory_plot.png +│ └── b-planner-hyperparameters/ +│ └── sweep_table.csv +├── 002-fix-lidar-filter/ +│ └── ... +``` + +- **Feature folders** are named `NNN-short-kebab-slug` with a zero-padded three-digit prefix. Pick the next number by incrementing the highest existing prefix (start at `001`). +- **Results subfolders** are named `-`, one per lettered test-plan section in `design_spec.md` — section "(a) Planner core" maps to `results/a-planner-core/` — so a reader can navigate spec ↔ results directly. + +## Workflow + +### 1. Before coding — write the design spec + +Create `notebook/NNN-feature-slug/design_spec.md` with three parts: + +- **Problem context** — what you're trying to solve, in your own framing from the session: motivation, constraints, prior attempts, decisions already made. This is the section that preserves context which exists nowhere else. +- **Proposed implementation** — affected packages, new/changed nodes and topics, algorithms, data flow. Split into subsections if the implementation has multiple parts. +- **Test plan** — lettered sections `(a)`, `(b)`, `(c)`…, each describing what is run, what is measured, and what counts as pass. + +### 2. While implementing — keep status labels current + +The spec carries an implementation status at two levels, using **`DESIGN/TODO`**, **`WIP`**, or **`DONE`**: + +- An **overall status** in the header — the least-advanced status of any implementation section. +- A **per-section status** on each implementation subsection heading (e.g. `### 2.1 Cost-map integration — WIP`), so a reader sees exactly which parts are designed, in progress, or finished. + +Update labels as you work, not retroactively — a spec whose statuses lag reality misleads the next person (or agent) who picks up the feature. + +### 3. During validation — store raw results + +Each test run drops its artifacts into the matching lettered section folder: metrics files (e.g. `metrics.json` copied from `tests/results//`), plots, sim screenshots, and relevant log *excerpts*. Keep raw artifacts as-produced; interpretation belongs in the summary. + +### 4. After validation — write the results summary + +Write `results/results_summary.md` with one section per test-plan letter. It must be **self-contained**: embed the quantitative tables and qualitative figures directly in the document (markdown tables; images via relative paths) so a developer can understand all the results at once without opening the raw artifact folders. End with an overall verdict — which spec sections passed, which didn't, and known limitations. + +### 5. Opening the PR — populate it from the notebook + +Reviewers can't see `notebook/`, so the PR body carries the distilled content: + +| PR section | Source | +|------------|--------| +| Motivation / context | `design_spec.md` problem context | +| What changed | Proposed implementation (as-built) | +| Validation | `results_summary.md` — paste the tables, upload key figures as PR attachments, state per-section verdicts | + +## Templates and Agent Skill + +Fill-in templates for both documents, and the full agent-facing workflow (including pitfalls), live in the [`use-feature-notebook` skill](https://github.com/castacks/AirStack/tree/develop/.agents/skills/use-feature-notebook) under `.agents/skills/`. Coding agents are instructed via `AGENTS.md` to follow this workflow at the start of every feature implementation. diff --git a/mkdocs.yml b/mkdocs.yml index e75d85bc7..a36d4c75f 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,9 +73,10 @@ nav: - CI/CD Orchestrator: tests/ci-cd-orchestrator.md - Frame Conventions: docs/development/intermediate/frame_conventions.md - Docker Build Profiles: docs/development/intermediate/docker-build-profiles.md - - Contributing: + - Contributing: - docs/development/intermediate/contributing.md - docs/development/intermediate/documentation.md + - Feature Notebook: docs/development/intermediate/feature_notebook.md - Advanced Tutorials: - AI Agent Guide: docs/development/advanced/ai_agent_guide.md - AirStack CLI Tool: From 9788b14a867af672f5a112bc23ac16b7e5c30500 Mon Sep 17 00:00:00 2001 From: Andrew Jong Date: Tue, 4 Aug 2026 17:43:15 -0700 Subject: [PATCH 12/21] Remove stray files --- --gui | 0 --num-robots | 0 --sim | 0 --stress-iterations | 0 --trajectory-types | 0 -v | 11 ----------- 6 files changed, 11 deletions(-) delete mode 100644 --gui delete mode 100644 --num-robots delete mode 100644 --sim delete mode 100644 --stress-iterations delete mode 100644 --trajectory-types delete mode 100644 -v diff --git a/--gui b/--gui deleted file mode 100644 index e69de29bb..000000000 diff --git a/--num-robots b/--num-robots deleted file mode 100644 index e69de29bb..000000000 diff --git a/--sim b/--sim deleted file mode 100644 index e69de29bb..000000000 diff --git a/--stress-iterations b/--stress-iterations deleted file mode 100644 index e69de29bb..000000000 diff --git a/--trajectory-types b/--trajectory-types deleted file mode 100644 index e69de29bb..000000000 diff --git a/-v b/-v deleted file mode 100644 index fa52f4143..000000000 --- a/-v +++ /dev/null @@ -1,11 +0,0 @@ -access control disabled, clients can connect from any host -============================= test session starts ============================== -platform linux -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0 -- /usr/local/bin/python3.12 -cachedir: /tmp/.pytest_cache -rootdir: /home/pranavkumara/Desktop/AirStack/tests -configfile: pytest.ini -plugins: dependency-0.6.1, timeout-2.4.0 -collecting ... collected 0 items - -- generated xml file: /home/pranavkumara/Desktop/AirStack/tests/results/2026-05-28_14-14-04/results.xml - -============================ no tests ran in 0.00s ============================= From 234587aa05b1011d5f20bf8c0a2857433eedf3fc Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Wed, 5 Aug 2026 14:56:21 -0400 Subject: [PATCH 13/21] Robot deployment fixes: bag recording + adding warning for robot-identity failure (#377) * make RECORD_BAGS actually reach the bag recorder LOG_CONFIG selects which topic set in logging_bringup/config to record, default log.yaml. Co-Authored-By: Claude Opus 5 * warn when the robot identity fails to resolve Co-Authored-By: Claude Opus 5 * chore: bump version to 0.19.0-alpha.12 Co-Authored-By: Claude Opus 5 * fixed comments and documentation * fix the bag recording status bridge direction It was bridged gcs -> robot, the same direction as the command it answers, so status never reached the GCS and the rqt Recording: label stayed blank. Co-Authored-By: Claude Opus 5 * fix the exclude flag so the main bag section records ros2 bag record renamed --exclude to --exclude-regex, and the old name is now an ambiguous prefix of four options, so argparse rejected the command and any section using exclude: recorded nothing. Co-Authored-By: Claude Opus 5 * restore the bags .gitignore files #318 dropped robot/bags/.gitignore and gcs/bags/.gitignore while moving a dozen others; nothing has covered recorded bags since. Co-Authored-By: Claude Opus 5 --------- Co-authored-by: Claude Opus 5 --- .agents/skills/configure-multi-robot/SKILL.md | 57 ++++++++++++++----- .env | 2 +- CHANGELOG.md | 5 ++ .../bag_record_pid/bag_record_node.py | 7 ++- .../logging_bringup/launch/logging.launch.xml | 10 ++-- docs/robot/docker/robot_identity.md | 22 ++++++- gcs/bags/.gitignore | 11 ++++ overrides/l4t-px4-realrobot.env | 6 +- robot/bags/.gitignore | 11 ++++ robot/docker/.bashrc | 15 +++++ robot/docker/robot-base-docker-compose.yaml | 1 + .../onboard_all/config/domain_bridge.yaml | 4 +- 12 files changed, 120 insertions(+), 31 deletions(-) create mode 100644 gcs/bags/.gitignore create mode 100644 robot/bags/.gitignore diff --git a/.agents/skills/configure-multi-robot/SKILL.md b/.agents/skills/configure-multi-robot/SKILL.md index c70ba8dde..640b183bd 100644 --- a/.agents/skills/configure-multi-robot/SKILL.md +++ b/.agents/skills/configure-multi-robot/SKILL.md @@ -103,17 +103,35 @@ docker exec airstack-robot-desktop-1 bash -c 'echo $ROBOT_NAME $ROS_DOMAIN_ID' If you need a non-default name (custom hostname scheme on a physical robot, or you want `drone_alpha` instead of `robot_1`), you have two options: 1. **Write a mapping YAML** in `robot/docker/robot_name_map/` and point `ROBOT_NAME_MAP_CONFIG_FILE` at it. Preferred when the name should be derived from the machine (hostname/container) — keeps the resolver in charge of `ROS_DOMAIN_ID` co-assignment. -2. **Pin `ROBOT_NAME` directly** in a per-deployment override env file. `.bashrc` honors a pre-set `ROBOT_NAME` (guard: `if [ -z "${ROBOT_NAME:-}" ]`) and skips the map lookup. This is the clean shortcut for a **single real robot** whose hostname doesn't match `robot-` (see [Real robots and the `unknown_robot` fallback](#real-robots-and-the-unknown_robot-fallback) below): - - ```bash - # overrides/.env — single robot, named directly - ROBOT_NAME=robot_1 - ROS_DOMAIN_ID=1 # set alongside — pinning ROBOT_NAME skips the map's domain co-assignment - ``` - -**Only pin `ROBOT_NAME` in an *override env file*, never on the shared `robot-desktop`/`robot-l4t` *service* in compose.** The service is reused for every replica; a hardcoded `ROBOT_NAME` there collapses all robots onto one name/domain and silently breaks multi-robot. And when you pin it, set `ROS_DOMAIN_ID` too — the resolver is what normally co-assigns the domain, and skipping it leaves the domain at whatever the environment defaults to. - -For a one-off override (e.g. ad hoc debugging): +2. **Rename the device** so the default map resolves it. On real hardware + (`ROBOT_NAME_SOURCE=hostname`) the OS hostname *is* the identity, so + `hostnamectl set-hostname robot-1` is a complete, one-time fix — and it scales to a + fleet, since `robot-2` and `robot-3` then resolve on their own. + +!!! danger "Setting `ROBOT_NAME` in an env file does nothing" + No compose service declares `ROBOT_NAME` or `ROS_DOMAIN_ID` in its `environment:` + block, and Docker Compose only injects a variable into a container if some service + names it there. Putting `ROBOT_NAME=robot_1` in an override `.env` sets it for + **compose's own interpolation**, not for the container — `.bashrc` sees it unset, + the map lookup runs anyway, and there is no error. The robot simply comes up under + the resolved name instead of yours. + + `overrides/l4t-px4-realrobot.env` used to ship `ROBOT_NAME` / `ROS_DOMAIN_ID` on + this basis; they never had any effect and have been removed. Use a hostname or a + map file instead. + + The general lesson applies to **any** deployment knob: it needs a declaration in + the service's `environment:` *and* a consumer that reads it. Always + [verify](#verification-commands) rather than assuming. + +**Never hardcode `ROBOT_NAME` on a service in compose either.** `robot-desktop` and +friends are reused for every replica, so a pinned name there would collapse all robots +onto one name and domain and silently break multi-robot. Identity must come from +something that differs per container — the container name in sim, the device hostname on +real hardware — or from a map rule that derives it. + +For a one-off override (e.g. ad hoc debugging), pass it to the shell directly, which +does work because `docker exec -e` sets it in the process environment: ```bash docker exec -e ROBOT_NAME=robot_5 -e ROS_DOMAIN_ID=5 -it airstack-robot-desktop-1 bash @@ -349,16 +367,25 @@ On VOXL/Jetson the service uses `ROBOT_NAME_SOURCE=hostname`, so the **OS hostna Pick whichever fix matches your topology (see [Configuring a Single Robot](#configuring-a-single-robot)): -- **One robot, quickest:** pin `ROBOT_NAME=robot_1` + `ROS_DOMAIN_ID=1` in the deployment's override env file. The `.bashrc` guard honors it and skips the lookup — no hostname change, no map file. -- **One robot, machine-derived:** rename the device hostname to `robot-1` so the default map resolves it automatically. -- **A fleet:** name each machine `robot-` (default map handles it) **or** ship a mapping YAML that matches your hostnames and point `ROBOT_NAME_MAP_CONFIG_FILE` at it. Do **not** pin a single `ROBOT_NAME` on the shared service — every robot would collide on it. +- **Quickest, no config:** rename the device — `hostnamectl set-hostname robot-1`. The default map resolves it to `robot_1` on domain 1, and a fleet named `robot-2`, `robot-3`, … resolves the same way with nothing further to maintain. +- **Hostnames you can't change:** ship a mapping YAML matching them and point `ROBOT_NAME_MAP_CONFIG_FILE` at it. Needs no code change — the variable is already forwarded and `robot_name_map/` is bind-mounted into the container — and keeps the resolver co-assigning `ROS_DOMAIN_ID`. -Verify on the device: +Setting `ROBOT_NAME` in an override env file is **not** an option: nothing declares it in +compose, so it never reaches the container. See the danger note under +[Configuring a Single Robot](#configuring-a-single-robot). + +Verify on the device — do this every time, especially after pinning `ROBOT_NAME`, since +a pin that never reached the container fails silently: ```bash docker exec bash -c 'echo "$(hostname) -> ROBOT_NAME=$ROBOT_NAME ROS_DOMAIN_ID=$ROS_DOMAIN_ID"' ``` +If it still reports `unknown_robot` after you set `ROBOT_NAME`, the variable did not +reach the container. Check that the service (or the base compose file it extends) +declares it in `environment:` — see the warning under +[Configuring a Single Robot](#configuring-a-single-robot). + ## Pre-Merge Checklist Before merging a change that touches anything robot-namespaced: diff --git a/.env b/.env index 020228fa2..04de90a2d 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.10" +VERSION="0.19.0-alpha.12" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/CHANGELOG.md b/CHANGELOG.md index 5a6673928..68f37e9b0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Robot name resolution now honors a pre-set `ROBOT_NAME` (e.g. injected via docker compose) instead of always overriding it from the container/hostname mapping (`robot/docker/.bashrc`) - Robot name-map catch-all fallback now maps to `unknown_robot` (valid ROS namespace token) instead of `unknown-robot` (`default_robot_name_map.yaml`) - l4t robot image: replace dustynv's `/ros_entrypoint.sh` with a passthrough so its prebuilt source-ROS libs (older `fastcdr`) no longer shadow the apt Jazzy runtime and crash apt-built nodes like MAVROS +- `RECORD_BAGS=true` never brought the bag recorder up on a robot: `logging.launch.xml` hardcoded `record_bag=false` and `onboard_autonomy_all.launch.xml` includes it with no arguments, so the variable was forwarded into the container and read by nobody (only `gcs.launch.xml` consumed it). With no `bag_record` node running, the GCS control panel's `set_recording_status` toggle had nothing to reach despite being bridged in `domain_bridge.yaml` / `dds_router.yaml`. It now reads `RECORD_BAGS` and selects its topic set via `LOG_CONFIG` +- Falling back to `unknown_robot` / domain 0 now logs a warning naming both fixes (rename the device `robot-` on the host, or supply a `ROBOT_NAME_MAP_CONFIG_FILE` matching your hostnames). The fallback itself is unchanged — it deliberately keeps an unidentified robot out of every real robot's namespace — but it used to resolve silently, so the symptoms surfaced far from the cause +- Dropped `ROBOT_NAME` / `ROS_DOMAIN_ID` from `overrides/l4t-px4-realrobot.env`: no compose service declares either, so an env file could never set them and the lines were inert +- `bag_record/bag_recording_status` was bridged GCS -> robot in `domain_bridge.yaml`, the same direction as the command it answers, so recorder status never reached the GCS and every recording indicator stayed blank +- `bag_record_node` passed `--exclude` to `ros2 bag record`, which Jazzy renamed to `--exclude-regex`. It is now an ambiguous prefix of four options, so argparse rejected the command and any section using `exclude:` (including `log.yaml`'s `airstack` section, i.e. everything but the cameras) recorded nothing — surfacing only as a usage dump in the node's stdout. Multiple `exclude:` entries are now alternated into one regex instead of repeating a single-valued flag, which had silently kept only the last ## [1.0.0] - 2024-12-19 diff --git a/common/ros_packages/logging/bag_recorder_pid/bag_record_pid/bag_record_node.py b/common/ros_packages/logging/bag_recorder_pid/bag_record_pid/bag_record_node.py index d3d746d99..a815b6047 100644 --- a/common/ros_packages/logging/bag_recorder_pid/bag_record_pid/bag_record_node.py +++ b/common/ros_packages/logging/bag_recorder_pid/bag_record_pid/bag_record_node.py @@ -128,9 +128,10 @@ def add_topics(self): exit() self.commands[section_name]['suffix'].append('--all') - for topic in section_config['exclude']: - self.commands[section_name]['suffix'].append('--exclude') - self.commands[section_name]['suffix'].append(topic) + # --exclude-regex takes a single regex, so entries are alternated. + excludes = [str(t) for t in section_config['exclude']] + self.commands[section_name]['suffix'].append('--exclude-regex') + self.commands[section_name]['suffix'].append('|'.join(excludes)) self.get_logger().info(str(self.commands[section_name])) else: for topic in section_config['topics']: diff --git a/common/ros_packages/logging/logging_bringup/launch/logging.launch.xml b/common/ros_packages/logging/logging_bringup/launch/logging.launch.xml index 6c6df0502..7b56ca1b9 100644 --- a/common/ros_packages/logging/logging_bringup/launch/logging.launch.xml +++ b/common/ros_packages/logging/logging_bringup/launch/logging.launch.xml @@ -1,13 +1,15 @@ - - + + + + - + - + diff --git a/docs/robot/docker/robot_identity.md b/docs/robot/docker/robot_identity.md index 494b5daae..dde74696f 100644 --- a/docs/robot/docker/robot_identity.md +++ b/docs/robot/docker/robot_identity.md @@ -107,8 +107,24 @@ The serial port for MAVLink (`/dev/ttyTHS4`) is hardcoded for real-robot profile export FCU_URL="/dev/ttyTHS4:115200" ``` -!!! warning "Hostname convention is required" - The hostname must match a rule in the mapping config file. If no rule matches, the script exits with an error and `ROBOT_NAME` / `ROS_DOMAIN_ID` will be unset. Make sure every physical robot has a hostname that matches a rule before deployment. +!!! warning "Hostname convention is required — and it fails quietly" + The hostname must match a rule in the mapping config file. With the stock + `default_robot_name_map.yaml` the failure is **silent, not loud**: its final `.*` + catch-all matches anything, so a device named e.g. `airlab-jetson-42` resolves to + `ROBOT_NAME=unknown_robot`, `ROS_DOMAIN_ID=0` with no error and a clean boot. The + symptoms surface later — topics under `/unknown_robot`, per-robot config lookups + keyed on `ROBOT_NAME` finding no profile, and containers pinned to another domain + (`zed-l4t` hardcodes `ROS_DOMAIN_ID=1`) no longer seeing the stack. + + Make sure every physical robot has a hostname that matches a rule before deployment. + Run the following to set the hostname on a device: + ``` + hostnamectl set-hostname robot-1 + ``` + Then verify the mapping in a shell inside the container: + ```bash + docker exec bash -c 'echo "$(hostname) -> $ROBOT_NAME / $ROS_DOMAIN_ID"' + ``` ### Fallback (anything else) @@ -139,4 +155,4 @@ If you need to override the robot identity for testing, you can set the variable docker exec -e ROBOT_NAME=robot_5 -e ROS_DOMAIN_ID=5 -it bash ``` -Or add them to your project's `.env` file and pass them through in `docker-compose.yaml`. +Or add them to your project's `.env` file and make sure to pass them through in `docker-compose.yaml`. diff --git a/gcs/bags/.gitignore b/gcs/bags/.gitignore new file mode 100644 index 000000000..cc6955f65 --- /dev/null +++ b/gcs/bags/.gitignore @@ -0,0 +1,11 @@ +# list of all ros2 bag files to ignore +*.mcap +*.mcap.index +*.mcap.meta +*.mcap.bag +*.mcap.bag.index +*.mcap.bag.meta +*.db3 +*.db3.index +*.db3.meta +*.yaml \ No newline at end of file diff --git a/overrides/l4t-px4-realrobot.env b/overrides/l4t-px4-realrobot.env index 271726801..6da88cf8d 100644 --- a/overrides/l4t-px4-realrobot.env +++ b/overrides/l4t-px4-realrobot.env @@ -14,9 +14,9 @@ AUTOLAUNCH="true" NUM_ROBOTS="1" # --- Robot identity ----------------------------------------------------------- -# Shortcut to name only a single agent. -ROBOT_NAME="robot_1" -ROS_DOMAIN_ID="1" +# Resolved from this device's hostname: name the Jetson robot-1 on the HOST +# Run the following: ``hostnamectl set-hostname robot-1`` +# resulting in robot_1 on domain 1. # Launches entire robot autonomy stack AUTONOMY_ROLE="full" diff --git a/robot/bags/.gitignore b/robot/bags/.gitignore new file mode 100644 index 000000000..cc6955f65 --- /dev/null +++ b/robot/bags/.gitignore @@ -0,0 +1,11 @@ +# list of all ros2 bag files to ignore +*.mcap +*.mcap.index +*.mcap.meta +*.mcap.bag +*.mcap.bag.index +*.mcap.bag.meta +*.db3 +*.db3.index +*.db3.meta +*.yaml \ No newline at end of file diff --git a/robot/docker/.bashrc b/robot/docker/.bashrc index b3903a144..a62e5f18b 100755 --- a/robot/docker/.bashrc +++ b/robot/docker/.bashrc @@ -103,6 +103,21 @@ if [ -z "${ROBOT_NAME:-}" ]; then export ROS_DOMAIN_ID=$existing_robot_domain_id fi fi + + # Warn about unknown robot mapping. + if [ -z "${ROBOT_NAME:-}" ] || [ "$ROBOT_NAME" == "unknown_robot" ]; then + echo "WARNING: could not resolve a robot identity from '${name_to_map:-}'" \ + "using $ROBOT_NAME_MAP_CONFIG_FILE." + echo " ROBOT_NAME='${ROBOT_NAME:-}' ROS_DOMAIN_ID='${ROS_DOMAIN_ID:-}'" + echo " Topics will not be namespaced under /robot_, so nothing will reach" + echo " the rest of the stack. Fix by either:" + echo " - on the HOST (not in this container): hostnamectl set-hostname robot-1," + echo " which the default map resolves to robot_ on domain ; or" + echo " - adding a mapping YAML under robot/docker/robot_name_map/ that" + echo " matches your hostnames and pointing ROBOT_NAME_MAP_CONFIG_FILE at it." + echo " If ROBOT_NAME is empty rather than unknown_robot, check stderr above" + echo " for a resolve_robot_name.py error (missing or malformed map file)." + fi fi diff --git a/robot/docker/robot-base-docker-compose.yaml b/robot/docker/robot-base-docker-compose.yaml index a4ed7e1bb..8cb714e18 100644 --- a/robot/docker/robot-base-docker-compose.yaml +++ b/robot/docker/robot-base-docker-compose.yaml @@ -12,6 +12,7 @@ services: - QT_QPA_PLATFORM # Record bags - RECORD_BAGS=${RECORD_BAGS} + - LOG_CONFIG=${LOG_CONFIG:-log.yaml} # docker compose interpolation to env variables - AUTONOMY_ROLE=${AUTONOMY_ROLE:-full} - URDF_FILE=${URDF_FILE} diff --git a/robot/ros_ws/src/autonomy_bringup/onboard_all/config/domain_bridge.yaml b/robot/ros_ws/src/autonomy_bringup/onboard_all/config/domain_bridge.yaml index b60935c87..34e7e1d26 100644 --- a/robot/ros_ws/src/autonomy_bringup/onboard_all/config/domain_bridge.yaml +++ b/robot/ros_ws/src/autonomy_bringup/onboard_all/config/domain_bridge.yaml @@ -33,8 +33,8 @@ topics: # bag recording status bag_record/bag_recording_status: type: std_msgs/msg/Bool - from_domain: $(var gcs_domain) - to_domain: $(env ROS_DOMAIN_ID) + from_domain: $(env ROS_DOMAIN_ID) + to_domain: $(var gcs_domain) # ============= Incoming to Robot ================ From 5cf595523e3ebec1b7b7fd8ad3c20c271c25af3a Mon Sep 17 00:00:00 2001 From: pvkumara <99618405+pvkumara@users.noreply.github.com> Date: Fri, 14 Aug 2026 18:09:53 -0400 Subject: [PATCH 14/21] ci: land OSMO ephemeral runners and system-test harness on develop (#382) * ci(orchestrator): migrate ephemeral CI runners from OpenStack to NVIDIA OSMO Replace the OpenStack-Nova spawn/reap backend with OSMO workflow submission. The GitHub side is unchanged (self-hosted/airstack-ephemeral labels, single-use JIT runner tokens, same-repo fork guard) and the one-job-per-worker destroy-after model is preserved; only the spawn target moved from creating a Nova VM to submitting an OSMO workflow. orchestrator.py: submit/query/cancel/list via the osmo CLI, job_id -> workflow_id state, re-login-on-auth-failure, orphan sweep via osmo workflow list; drop floating-IP/boot-volume/placement/keypair/security-group logic. runner.Dockerfile + runner-entrypoint.sh + runner-workflow.yaml.j2: prebaked privileged docker-in-docker + GPU GitHub runner image/task (replaces cloud-init.yaml.j2). config.example.yaml, setup.sh, airstack-orchestrator.service, requirements.txt: OSMO service-account token auth, install the osmo CLI, drop openstacksdk. Docs (AGENTS.md, tests/README.md, orchestrator README) updated to OSMO. Co-authored-by: Cursor * ci(orchestrator): pin AirLab OSMO JSON keys and runner image path Resolve uuid/live name after submit (OSMO returns name-only + suffix), default config to the Keycloak-backed airstack pool and Harbor runner image, and add scripts to build/push airstack-ci-runner on OSMO DinD. Co-authored-by: Cursor * docs(ci): document the OSMO-backed CI/CD pipeline Fills in the empty ci_cd.md stub with an end-to-end guide to how CI runs the full AirStack stack on ephemeral OSMO GPU pods: architecture and job lifecycle diagrams, runner pod anatomy, the three trigger paths, what each pytest mark catches, the metrics regression gate, the security model, and layer-by-layer troubleshooting. Adds the page to the mkdocs nav (it was previously unreachable) and cross-links it from tests/README.md and the testing index. Co-authored-by: Cursor * fix(ci): repair Docker builds on OSMO ephemeral runners Every build_docker and build_packages test failed on the OSMO backend because the inner dockerd kept its data-root on the pod's overlayfs rootfs. Linux rejects a directory on overlayfs as an overlay upperdir, so image pulls still succeeded -- containerd unpacks layers with plain writes -- while every build step needing a real mount died with "mount source: overlay ... err: invalid argument", surfacing as unrelated-looking apt-get and WORKDIR failures. runner-entrypoint.sh now picks a storage backend by attempting a real overlay mount rather than trusting the filesystem type, preferring a loopback ext4 data-root (real overlay2, sparse, dies with the pod) and falling back to a pod-mounted filesystem, fuse-overlayfs, then vfs. vfs is a last resort only: it copies the whole filesystem per layer and would exhaust the storage request on the sim images. Also bumps the GitHub Actions runner to 2.336.0, since 2.334.0 stops being able to run jobs on 2026-08-10. Co-authored-by: Cursor * fix(ci): seed PR Docker builds from a floating cache tag Versioned cache_from entries always miss on PRs because VERSION is forced up; add a stable cache_* tag published only by docker-build.yml so system tests can reuse layers without writing the shared cache. Co-authored-by: Cursor * ci(docker-build): retag unchanged images on VERSION bump Skip full compose rebuilds when a service's content fingerprint matches the previous versioned image label; registry-retag instead and only rebuild services whose Docker inputs changed. Co-authored-by: Cursor * fix(ci): parse quoted .env values before inline comments docker_image_plan was feeding NUM_ROBOTS with a trailing comment into compose config, which broke strconv.Atoi for deploy.replicas. Co-authored-by: Cursor * ci(docker-build): build/push services sequentially Publish successful images even when a sibling (e.g. isaac-sim) fails, and still cosign whatever was retagged or pushed in the same run. Co-authored-by: Cursor * chore: bump VERSION to 0.19.0-alpha.8 for retag validation Seeded gcs/ms-airsim/robot images carry content-fingerprint labels; this bump should registry-retag those digests without rebuilding. Co-authored-by: Cursor * fix(ci): unblock isaac-sim PX4 apt and robot colcon pytest Isaac's PX4 ubuntu.sh fails dpkg configure on the NVIDIA base; pre-fix ca-certificates, drop software-properties-common, and skip NuttX/Gazebo like ms-airsim. Pin pytest<8.1 and disable launch_testing for colcon unit tests so ROS Jazzy's outdated pytest hook no longer aborts CI. Co-authored-by: Cursor * fix(ci): pass colcon --pytest-args as separate tokens A single quoted blob made pytest treat "-p no:launch_testing" as part of the -m expression, which broke lidar_point_cloud_filter colcon tests. Co-authored-by: Cursor * fix(ci): quote colcon pytest args through bash -ic Nested single quotes around 'not linter' terminated the outer bash -ic string early, so pytest saw 'not' as a path. Use shlex.quote for the whole command and list-form pytest_args in the YAML. Co-authored-by: Cursor * fix(ci): pass colcon pytest flags via PYTEST_ADDOPTS colcon --pytest-args is a single nargs='*' option, so repeating it dropped -p and pytest treated no:launch_testing as a file path. Set PYTEST_ADDOPTS with docker exec -e instead. Co-authored-by: Cursor * fix(ci): rename helper so pytest does not treat it as a hook conftest functions named pytest_* are registered as hooks. pytest_addopts_env caused PluginValidationError and exit code 3. Co-authored-by: Cursor * ci: skip image-build for build_packages reruns Pull and retag cache_* images instead of baking isaac/airsim on every colcon/pytest iteration. /pytest --no-image-build does the same for other marks. compose up --no-build when AIRSTACK_NO_IMAGE_BUILD=1. Co-authored-by: Cursor * fix(ci): disable pytest plugin autoload for colcon tests -p no:launch_testing is applied after setuptools entrypoints load, so pytest 8.1+ still crashes on launch_testing's path= hook. Set PYTEST_DISABLE_PLUGIN_AUTOLOAD so cache_* robot images (unpinned pytest) can run lidar tests without a rebuild. Co-authored-by: Cursor * fix(ci): skip lidar ament linters in package pytest config PYTEST_ADDOPTS -m not linter never reached ament pytest, so copyright / flake8 / pep257 still ran after the unit tests passed. Ignore those modules in setup.cfg and collect_ignore. Co-authored-by: Cursor * ci: default system tests to isaacsim only PR-open and bare /pytest were sweeping both sims. Default --sim to isaacsim; msairsim is opt-in via --sim msairsim. Co-authored-by: Cursor --------- Co-authored-by: pvkumara Co-authored-by: Cursor --- .../skills/bump-version-and-release/SKILL.md | 12 +- .agents/skills/run-system-tests/SKILL.md | 5 +- .env | 2 +- .github/orchestrator/README.md | 320 +++---- .../airstack-orchestrator.service | 17 +- .github/orchestrator/build-and-push.sh | 28 + .../orchestrator/build-runner-on-osmo.yaml | 63 ++ .github/orchestrator/cloud-init.yaml.j2 | 71 -- .github/orchestrator/config.example.yaml | 139 +-- .github/orchestrator/orchestrator.py | 795 +++++++++--------- .github/orchestrator/requirements.txt | 1 - .github/orchestrator/runner-entrypoint.sh | 178 ++++ .github/orchestrator/runner-workflow.yaml.j2 | 48 ++ .github/orchestrator/runner.Dockerfile | 67 ++ .github/orchestrator/setup.sh | 51 +- .github/workflows/docker-build.yml | 249 ++++-- .../workflows/scripts/docker_image_plan.py | 544 ++++++++++++ .github/workflows/system-tests.yml | 91 +- .gitignore | 5 + AGENTS.md | 26 +- CHANGELOG.md | 6 + airstack.sh | 71 +- .../development/intermediate/testing/ci_cd.md | 569 ++++++++++++- .../testing/end_to_end_testing.md | 6 +- .../development/intermediate/testing/index.md | 2 +- gcs/docker/gcs-base-docker-compose.yaml | 2 + mkdocs.yml | 1 + robot/docker/Dockerfile.robot | 6 + robot/docker/docker-compose.yaml | 16 + .../lidar_point_cloud_filter/setup.cfg | 10 + .../lidar_point_cloud_filter/test/conftest.py | 7 + .../isaac-sim/docker/Dockerfile.isaac-ros | 22 +- .../isaac-sim/docker/docker-compose.yaml | 5 + .../ms-airsim/docker/docker-compose.yaml | 2 + tests/README.md | 48 +- tests/colcon_unit_test_packages.yaml | 6 +- tests/conftest.py | 8 +- tests/harness/__init__.py | 3 +- tests/harness/commands.py | 8 +- tests/harness/discovery.py | 42 +- tests/system/test_build_packages.py | 18 +- 41 files changed, 2708 insertions(+), 862 deletions(-) create mode 100755 .github/orchestrator/build-and-push.sh create mode 100644 .github/orchestrator/build-runner-on-osmo.yaml delete mode 100644 .github/orchestrator/cloud-init.yaml.j2 create mode 100644 .github/orchestrator/runner-entrypoint.sh create mode 100644 .github/orchestrator/runner-workflow.yaml.j2 create mode 100644 .github/orchestrator/runner.Dockerfile create mode 100755 .github/workflows/scripts/docker_image_plan.py create mode 100644 robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py diff --git a/.agents/skills/bump-version-and-release/SKILL.md b/.agents/skills/bump-version-and-release/SKILL.md index 3c056b774..18792a965 100644 --- a/.agents/skills/bump-version-and-release/SKILL.md +++ b/.agents/skills/bump-version-and-release/SKILL.md @@ -64,11 +64,13 @@ Three workflows in `.github/workflows/` interact with `VERSION`: - **Trigger:** push to `main` or `develop` whose changed paths include `.env`, **and** the `VERSION=` line in `.env` differs from the previous commit. Also runs on manual `workflow_dispatch`. - **Behavior on tag change:** 1. Runs on a self-hosted ephemeral GPU runner (`[self-hosted, airstack-ephemeral]`). - 2. `docker compose build` for profiles `desktop,isaac-sim,ms-airsim`. - 3. `docker compose push` to `${PROJECT_DOCKER_REGISTRY}` (set in `.env` — currently `airlab-docker.andrew.cmu.edu/airstack`). - 4. Keyless `cosign sign` of every pushed image digest via GitHub OIDC. - 5. `cosign verify` against the workflow's certificate identity. + 2. Plans per service via `.github/workflows/scripts/docker_image_plan.py` (content fingerprint vs previous versioned image label). + 3. **Unchanged image inputs** → registry retag of the previous `v${PREV}_…` digest to `v${VERSION}_…` and `cache_*` (no rebuild). + 4. **Changed inputs** (or missing/unlabeled previous image, or `force_rebuild=true`) → `docker compose build` / `push` for those services only, labeling the new digest with `org.airstack.content-fingerprint`. + 5. Keyless `cosign sign` of every published image digest via GitHub OIDC. + 6. `cosign verify` against the workflow's certificate identity. - **Skip behavior:** if the merge commit on `main`/`develop` does not actually change `VERSION=`, the build job is skipped (the check-changes job sets `tag-changed=false`). +- **Docs-only VERSION bumps:** still required by `check-version-increment`, but publish should retag rather than rebuild once fingerprints are on the previous images. First publish after this feature lands (or `force_rebuild=true`) must rebuild to write the labels. ### 3. `deploy_docs_from_release.yaml` — versioned docs @@ -76,7 +78,7 @@ Three workflows in `.github/workflows/` interact with `VERSION`: - **Behavior:** runs `mike deploy --push --update-aliases latest`, publishing the docs site under the release tag and pointing the `latest` alias at it. - Companion workflows publish unversioned docs from `main` (default alias `main`) and `develop` (alias `develop`). -So the full release path is: bump `VERSION` → PR → merge to `main`/`develop` (rebuild + push + sign) → cut a GitHub Release matching that VERSION (versioned docs go live). +So the full release path is: bump `VERSION` → PR → merge to `main`/`develop` (retag unchanged images and/or rebuild changed ones + push + sign) → cut a GitHub Release matching that VERSION (versioned docs go live). ## Choosing the Bump Type diff --git a/.agents/skills/run-system-tests/SKILL.md b/.agents/skills/run-system-tests/SKILL.md index 3923b6842..22ce20520 100644 --- a/.agents/skills/run-system-tests/SKILL.md +++ b/.agents/skills/run-system-tests/SKILL.md @@ -98,6 +98,7 @@ The `system-tests.yml` workflow's `Parse pytest args` step automatically prepend - `/pytest -m takeoff_hover_land` → effectively runs `-m "build_packages or takeoff_hover_land"` - `/pytest` (no marks) → pytest defaults (everything) - `/pytest -m build_docker` → unchanged (the build_docker tests rebuild from scratch anyway) +- `/pytest -m build_packages` → **pull-only** (retag `cache_*`, no `image-build`, no Isaac). Add `--no-image-build` on other marks to skip the bake. This guarantees that ROS 2 workspaces are built inside the containers before any launch/liveliness test tries to source them. If you intentionally want to skip `build_packages` (e.g. you trust the prebuilt images), include it explicitly: `-m "liveliness and not build_packages"` would work, but the simpler path is to run locally where the prepend logic doesn't apply. @@ -157,7 +158,7 @@ The `airstack_env` fixture is parametrized over `(sim, num_robots, iteration)` t | Flag | Default | Affects | Becomes | |------|---------|---------|---------| -| `--sim` | `msairsim,isaacsim` | `airstack_env` | One env-tuple per sim | +| `--sim` | `isaacsim` | `airstack_env` | One env-tuple per sim (`msairsim` opt-in) | | `--num-robots` | `1,3` | `airstack_env` | Cross-product with sim | | `--stress-iterations` | `1` | `airstack_env` | Up/down cycles per `(sim, num_robots)` | | `--stable-duration` | `120` | `system.test_liveliness::test_stable` and `system.test_sensors::test_sensor_streams_stable` | Total seconds polled | @@ -356,7 +357,7 @@ If multiple tests need the same setup, add a fixture in `conftest.py` (not in yo - **Running on insufficient hardware**. `liveliness`, `sensors`, and `takeoff_hover_land` require an NVIDIA GPU plus nvidia-container-toolkit; without them the sim container won't get GPU access and topic Hz checks will time out. If you only have a CPU, scope to `-m "build_docker or build_packages"`. - **Expecting interactive sim feedback**. `airstack_env` runs headless by default (`MS_AIRSIM_HEADLESS=true`, `ISAAC_SIM_HEADLESS=true`, `QT_QPA_PLATFORM=offscreen`). Don't add stdin prompts, GUI dialogs, or `input()` calls to test code — they will hang in CI. For local visual debugging only, pass `--gui`. - **Not capturing metrics in a new test**. If a test fails silently (no metric recorded) the regression report has nothing to compare. Always record at least one scalar via `MetricsRecorder` so the test shows up in `metrics.json`. -- **Letting parametrize cardinality explode**. Defaults `--sim msairsim,isaacsim --num-robots 1,3` with `--stress-iterations 3` multiply stack bring-ups for each selected mark (`liveliness`, `sensors`, `takeoff_hover_land`, …) — expensive. Override locally to a single tuple while iterating. +- **Letting parametrize cardinality explode**. Default `--num-robots 1,3` (and `--sim msairsim` if you opt in) multiplies stack bring-ups for each selected mark (`liveliness`, `sensors`, `takeoff_hover_land`, …) — expensive. Override locally to a single tuple while iterating. `--sim` defaults to `isaacsim` only. - **Hardcoded container names**. Always use `find_container`, `get_robot_containers`, or `wait_for_container` — replica suffixes (`-1`, `-2`, `-3`) and compose project prefixes change. - **Asserting on stdout instead of using `read_log_tail`**. The conftest captures each subprocess's combined stdout/stderr in memory; assertions should reference it via `read_log_tail()` (`f"airstack up failed:\n{read_log_tail()}"`) so failures attach the relevant context to the JUnit XML. - **Trying to SSH into a CI runner mid-job**. Workers are ephemeral OpenStack VMs destroyed within ~30s of job completion. Re-running the job creates a fresh VM. For genuine debugging on the runner, see `.github/orchestrator/README.md` (also exposed at `tests/ci-cd-orchestrator.md`) — but in 99% of cases, reproduce locally with `airstack test`. diff --git a/.env b/.env index 04de90a2d..4aa2502f1 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.12" +VERSION="0.19.0-alpha.13" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/.github/orchestrator/README.md b/.github/orchestrator/README.md index c10da3383..f2df1ff22 100644 --- a/.github/orchestrator/README.md +++ b/.github/orchestrator/README.md @@ -1,131 +1,128 @@ -# AirStack CI Orchestrator +# AirStack CI Orchestrator (OSMO backend) -This describes how to use a self-hosted OpenStack VM to run GitHub Actions jobs on truly ephemeral workers. The orchestrator is a Python service that continuously polls GitHub for queued workflow jobs, spawns a fresh OpenStack instance for each one with a single-use JIT runner token, and reaps (deletes) the instance when the job completes. This allows us to run CI workloads on GPU-equipped VMs without sharing any state between runs or exposing long-lived credentials on the worker. +This describes how a small always-on orchestrator service runs GitHub Actions jobs on truly ephemeral GPU workers scheduled by [NVIDIA OSMO](https://nvidia.github.io/OSMO/). The orchestrator is a Python service that continuously polls GitHub for queued workflow jobs, submits a fresh **OSMO workflow** for each one (a single-use JIT runner in a privileged, GPU-enabled container), and reaps it when the job completes. Each CI job runs on a clean pod with no state shared between runs and no long-lived credentials on the worker. -The orchestrator VM is the only host that holds the GitHub PAT and the OpenStack credential; the workers are destroyed after a single job. +This is a drop-in replacement for the previous OpenStack-Nova backend. The GitHub side is unchanged — `system-tests.yml` still uses `runs-on: [self-hosted, airstack-ephemeral]`, the single-use JIT runner config, and the same-repo fork guard. Only the *spawn target* changed from "create a Nova VM" to "submit an OSMO workflow", so the one-job-per-worker, destroy-after semantics are identical: when the runner's `run.sh` exits after one job, the OSMO task completes and the pod is torn down. + +The orchestrator host is the only machine that holds the GitHub PAT and the OSMO service-account token; workers are destroyed after a single job. ## Architecture ``` ┌─────────────────────────────────────────────────────────────┐ -│ Orchestrator VM (airstack-ci-cd-orchestrator) │ +│ Orchestrator host (airstack-ci-cd-orchestrator, no GPU) │ │ │ │ airstack-orchestrator.service → orchestrator.py │ │ spawn loop (every 15s): │ │ • GET /repos//actions/runs?status=queued │ │ • POST /repos//actions/runners/generate-jitconfig│ -│ • openstack server create (image, flavor, user_data) │ -│ • record (job_id → server_id) in state.json │ +│ • osmo workflow submit runner-workflow.yaml --pool ... │ +│ • record (job_id → workflow_id) in state.json │ │ reap loop (every 30s): │ -│ • job completed → openstack server delete │ -│ • job age > N min → force delete (straggler) │ -│ • owned but not in state → orphan reap │ +│ • job completed → osmo workflow cancel (if live) │ +│ • job age > N min → osmo workflow cancel (straggler) │ +│ • our-named but not in state → orphan cancel │ │ │ │ /etc/airstack-orchestrator/ │ │ config.yaml │ │ github-pat │ -│ /home/orchestrator/.config/openstack/clouds.yaml │ +│ osmo-token (OSMO service-account token) │ │ /var/lib/airstack-orchestrator/state.json │ +│ /var/lib/airstack-orchestrator/.config/osmo (CLI session) │ └─────────┬─────────────────────────────────┬─────────────────┘ - │ Nova / Neutron API │ GitHub REST API + │ osmo CLI (submit/query/cancel) │ GitHub REST API ▼ ▼ ┌──────────────────────────────────┐ ┌──────────────────────┐ -│ Ephemeral worker (per job) │ │ GitHub Actions │ -│ Image: Ubuntu-24.04-GPU-Headless│ │ workflow_job queue │ -│ cloud-init: │ └──────────────────────┘ -│ install docker + nv toolkit │ -│ download GH runner │ +│ OSMO CI GPU pool (privileged) │ │ GitHub Actions │ +│ Ephemeral runner pod (per job): │ │ workflow_job queue │ +│ Image: airstack-ci-runner │ └──────────────────────┘ +│ start dockerd (DinD) │ │ run.sh --jitconfig │ -│ shutdown -h +1 │ +│ exit → task done → pod reaped │ └──────────────────────────────────┘ ``` Key properties: -- **Truly ephemeral**: every job runs on a clean VM. No Docker layer cache pollution, no leftover networks, no carry-over from prior runs. +- **Truly ephemeral**: every job runs on a clean pod. No Docker layer cache pollution, no leftover containers, no carry-over from prior runs. - **PAT isolation**: the GitHub PAT lives only on the orchestrator. Workers receive a single-use [JIT runner config](https://docs.github.com/en/rest/actions/self-hosted-runners?apiVersion=2022-11-28#create-configuration-for-a-just-in-time-runner-for-a-repository) — a base64 token bound to one runner registration, valid only for a short window. -- **Application-credential auth**: the orchestrator authenticates to OpenStack with an application credential (revocable, scoped, no password), not the user's `openrc.sh`. -- **Crash-safe reaping**: every server we spawn is tagged with `airstack-role=ephemeral-runner`. The reap loop force-deletes any owned server not present in `state.json`, so a crashed orchestrator can't leak instances. +- **Service-account auth**: the orchestrator authenticates to OSMO with a shared, non-personal [service-account token](https://nvidia.github.io/OSMO/main/deployment_guide/appendix/authentication/service_accounts.html) (the analog of the old OpenStack application credential). CI runs never route through an individual's account, so PRs don't consume anyone's personal GPU quota and nothing breaks when a person leaves. +- **Crash-safe reaping**: every workflow is named `gha-runner--`. The reap loop cancels any active workflow with that prefix not present in `state.json`, so a crashed orchestrator can't leak workflows. ## Prerequisites -- OpenStack instance already setup for the orchestrator VM. The orchestrator itself is lightweight and doesn't need a GPU. 1 vCPU, 2GB RAM, and 20GB disk is sufficient for the orchestrator service. Make sure you can ssh into it and that it has outbound internet access. -- An OpenStack flavor with GPU passthrough and enough disk to run Docker + the tests. The orchestrator spawns workers from this flavor, so it must have a GPU and sufficient disk (or `boot_volume_size_gb` must be set) to run the workloads. It's common for GPU flavors to have `disk=0`, which means they boot from an ephemeral disk — in that case, you must set `boot_volume_size_gb` to a value large enough for the OS + Docker images + test assets (e.g., 40GB). If your OpenStack setup supports it, you can also boot from a Cinder volume sourced from an image; in that case, pre-bake Docker and the NVIDIA toolkit into the image to speed up boot time. + +- **Orchestrator host** — a small always-on VM (no GPU). 1 vCPU, 2GB RAM, 20GB disk, outbound internet to `api.github.com` and your OSMO URL. This is the only long-lived piece; the GPU compute is ephemeral pods on OSMO. +- **An OSMO service account + dedicated CI pool.** Ask your OSMO admin to: + 1. Create a service account (e.g. `svc-airstack-ci`) and a long-lived access token — `osmo user create` + `osmo token set`. On IdP-backed deployments (e.g. auth tied to the CMU Andrew directory) this is a non-personal identity, so it survives people graduating/leaving. If policy forbids OSMO-native service accounts, use a *functional/departmental* identity, never a personal one. + 2. Grant that account a role whose policy allows `workflow:Create/Cancel/Query` **scoped to a dedicated CI GPU pool** (e.g. `pool/airstack-ci`) that has its own allocation, so CI doesn't contend with researchers' interactive jobs. + 3. **Enable "Privileged Mode Allowed"** on that pool's platform. The AirStack tests run `airstack up` (docker compose) inside the worker, which requires an inner Docker daemon → a privileged container. Without this, submissions are rejected. + 4. Confirm the API gateway (Envoy) accepts OSMO access tokens for the API (not only interactive IdP logins). +- **A prebaked runner image** pushed to a registry the pool can pull (see below). ## One-time setup -### 1. Create OpenStack application credential +### 1. Build & push the runner image -On your local workstation (not the orchestrator VM): +The worker image bakes in Docker CE + compose, the NVIDIA container toolkit, and the GitHub Actions runner (what cloud-init used to install at boot on the VM), so pod start is fast and the JIT token can't expire mid-bootstrap. ```bash -source ~/.airlabcloud/openrc.sh -openstack application credential create airstack-orchestrator \ - --description "AirStack CI orchestrator — spawns ephemeral test runners" +cd .github/orchestrator +./build-and-push.sh +# or manually: +# docker build -f runner.Dockerfile \ +# --build-arg RUNNER_VERSION=2.334.0 \ +# -t airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0 . +# docker push airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0 ``` -The output prints `id` and `secret`. Build a `clouds.yaml`: - -```yaml -clouds: - airstack: - auth_type: v3applicationcredential - auth: - auth_url: https://airlab-cloud.andrew.cmu.edu:5000/v3/ - application_credential_id: - application_credential_secret: - region_name: Airlab - interface: public - identity_api_version: 3 +No local Docker? Submit the one-shot OSMO builder (needs your Harbor creds in OSMO): + +```bash +osmo workflow submit .github/orchestrator/build-runner-on-osmo.yaml \ + --pool airstack --priority HIGH ``` -### 2. Stage credentials on the orchestrator VM +Set `runner_image: airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.334.0` in `config.yaml` (step 4). Keep `RUNNER_VERSION` in sync with an [actions/runner release](https://github.com/actions/runner/releases). -```bash -# clouds.yaml: install for the orchestrator user (created in step 3) -scp clouds.yaml ubuntu@:/tmp/clouds.yaml +**Pool note (AirLab):** use the Keycloak-autosynced `airstack` pool (`privileged_allowed: true`). A hand-created `airstack-ci` pool is wiped by `synchronize_osmo_team_pools.py`. Ephemerality is per-job OSMO workflows, not a separate pool. + +### 2. Stage credentials on the orchestrator host +```bash # GitHub PAT: needs `Actions: read/write` and `Administration: read/write` # (fine-grained) or classic `repo` scope. -scp ~/.airlabcloud/airstack-github-pat.txt \ - ubuntu@:/tmp/github-pat +scp ~/airstack-github-pat.txt ubuntu@:/tmp/github-pat + +# OSMO service-account token (from `osmo token set`, provided by your admin). +scp ~/svc-airstack-ci-token.txt ubuntu@:/tmp/osmo-token ``` ### 3. Run setup.sh -On the orchestrator VM: +On the orchestrator host: ```bash git clone https://github.com/castacks/AirStack.git /tmp/airstack sudo bash /tmp/airstack/.github/orchestrator/setup.sh ``` -`setup.sh` creates the `orchestrator` system user, builds the Python venv, copies `orchestrator.py` and `cloud-init.yaml.j2` into `/opt/airstack-orchestrator/`, scaffolds `/etc/airstack-orchestrator/`, installs the systemd unit, and consumes `/tmp/github-pat`. - -You still need to put the `clouds.yaml` in place under the orchestrator user's home: - -```bash -sudo install -d -o orchestrator -g orchestrator -m 0700 \ - /home/orchestrator/.config/openstack -sudo install -o orchestrator -g orchestrator -m 0600 \ - /tmp/clouds.yaml /home/orchestrator/.config/openstack/clouds.yaml -sudo shred -u /tmp/clouds.yaml -``` +`setup.sh` creates the `orchestrator` system user, installs the `osmo` CLI, builds the Python venv, copies `orchestrator.py` and `runner-workflow.yaml.j2` into `/opt/airstack-orchestrator/`, scaffolds `/etc/airstack-orchestrator/`, installs the systemd unit, and consumes `/tmp/github-pat` and `/tmp/osmo-token`. ### 4. Fill in `/etc/airstack-orchestrator/config.yaml` -Edit the placeholders the example ships with: - | Field | What goes here | How to find it | |------|---------------|----------------| -| `flavor_name` | OpenStack flavor with GPU + enough disk | `openstack flavor list` | -| `network_name` | Network the workers attach to | `openstack network list` | -| `keypair_name` | SSH keypair for break-glass access | `openstack keypair list` | -| `security_group` | Outbound 443 must be allowed | `openstack security group list` | -| `availability_zone` | Optional AZ for the spawned instance; leave empty to let Nova pick | `openstack availability zone list` | -| `boot_volume_size_gb` | Set >0 if your flavor has `disk=0` (common for GPU flavors) — boots from a Cinder volume of this size sourced from `image_id`; leave 0 for direct image-boot | `openstack flavor show ` (check disk field) | -| `floating_ips` | Pre-allocated FIP pool, rotated through sequentially — each spawn picks the first free one. `max_concurrent` is capped at `len(pool)`. Leave empty to skip FIP attachment | `openstack floating ip list` | +| `osmo_url` | Your OSMO web service URL | from your OSMO admin | +| `pool` | Dedicated CI GPU pool | `osmo pool list` / the OSMO UI | +| `platform` | Optional hardware type within the pool; empty = pool default | `osmo pool list` | +| `runner_image` | Image from step 1 | the registry you pushed to | +| `cpu` / `gpu` / `memory` / `storage` | Resource request for the worker | size for full stack + sim | +| `privileged` | Must be `true` (docker compose inside the pod) | — | +| `priority` | `HIGH` \| `NORMAL` \| `LOW` | — | | `repo` | `owner/name` of the repo to poll | from GitHub URL | -| `runner_version` | Version tag from [actions/runner releases](https://github.com/actions/runner/releases) | check before each major upgrade | +| `runner_version` | Runner version baked into `runner_image` | matches step 1 | +| `max_concurrent` | Max simultaneous in-flight workflows | — | +| `max_job_minutes` | Straggler cancel ceiling | exceed the longest job | ### 5. Start the service @@ -134,7 +131,7 @@ sudo systemctl enable --now airstack-orchestrator.service journalctl -u airstack-orchestrator.service -f ``` -You should see `orchestrator started: repo=... labels=... max_concurrent=N` and then periodic poll activity. +You should see `orchestrator started (OSMO backend): repo=... pool=... max_concurrent=N`, an `osmo login succeeded` line, and then periodic poll activity. ## End-to-end verification @@ -142,147 +139,168 @@ You should see `orchestrator started: repo=... labels=... max_concurrent=N` and # Trigger a fast build-only run. gh workflow run system-tests.yml -f marks=build_docker -# Within ~30s, a server should appear: -openstack server list --metadata airstack-role=ephemeral-runner -# or if your OpenStack setup doesn't support metadata queries: -openstack server list --name '^ephemeral-' +# Within ~30s, a workflow should appear in the CI pool: +osmo workflow list --name gha-runner- --pool airstack-ci # Watch GitHub → Actions → Runners — the ephemeral runner should appear, # pick up the job, then disappear. -# Within ~30s of job completion, the server should be gone: -openstack server list --metadata airstack-role=ephemeral-runner -openstack server list --name '^ephemeral-' +# Within ~30s of job completion, the workflow should be terminal / gone from +# the active list: +osmo workflow list --name gha-runner- --pool airstack-ci --status RUNNING PENDING WAITING ``` ## Operational notes -- **State file**: `/var/lib/airstack-orchestrator/state.json` is the in-flight job tracker. Wiping it triggers an orphan sweep on the next reap iteration — owned servers will be force-deleted. Don't wipe it while jobs are mid-flight unless that's what you want. -- **Stuck instance**: any server older than `max_job_minutes` (default 90) is force-deleted regardless of GitHub job status. Bump this if liveliness/autonomy runs grow longer than ~75 minutes. +- **State file**: `/var/lib/airstack-orchestrator/state.json` is the in-flight job tracker (`job_id → workflow_id`). Wiping it triggers an orphan sweep on the next reap iteration — active `gha-runner-*` workflows will be cancelled. Don't wipe it while jobs are mid-flight unless that's what you want. +- **Straggler**: any workflow whose job has run longer than `max_job_minutes` (default 48h) is force-cancelled regardless of GitHub job status. +- **OSMO token rotation** (tokens expire — default 31 days): mint a new one and restart. + ```bash + # (admin) osmo token set svc-airstack-ci-token-2 --user svc-airstack-ci \ + # --roles osmo-user --expires-at 2027-12-31 + sudo install -o root -g orchestrator -m 0640 /tmp/osmo-token /etc/airstack-orchestrator/osmo-token + sudo systemctl restart airstack-orchestrator.service # re-runs `osmo login` + ``` - **PAT rotation**: `sudo install -o root -g orchestrator -m 0640 /tmp/new-pat /etc/airstack-orchestrator/github-pat && sudo systemctl restart airstack-orchestrator.service`. -- **Pause spawning** (e.g. for maintenance): `sudo systemctl stop airstack-orchestrator.service`. Already-spawned workers will still complete their jobs and self-shutdown; on restart, the reap loop deletes them. -- **Logs**: `journalctl -u airstack-orchestrator.service -f`. Cloud-init logs from individual workers are visible only via `openstack console log show ` while the worker is running. +- **Pause spawning** (e.g. for maintenance): `sudo systemctl stop airstack-orchestrator.service`. Already-submitted workers still complete their jobs; on restart, the reap loop cleans up. +- **Logs**: `journalctl -u airstack-orchestrator.service -f`. Per-worker logs come from `osmo workflow logs `. ## Debugging a failed job -When a GitHub workflow run fails or stalls, the failure can be in any of four places: the orchestrator (didn't spawn), cloud-init (didn't bootstrap), the GH Actions runner (didn't register or crashed), or the workflow steps themselves. Each has a different inspection path. +When a GitHub workflow run fails or stalls, the failure can be in one of four places: the orchestrator (didn't submit), the OSMO task (didn't schedule/pull), the GH Actions runner (didn't register or crashed), or the workflow steps themselves. Each has a different inspection path. -### 1. Find which worker ran the job +### 1. Find which workflow ran the job -`state.json` is the authoritative job ↔ server ↔ floating-IP map: +`state.json` is the authoritative job ↔ workflow map: ```bash -sudo jq -r '.jobs | to_entries[] | "\(.key)\t\(.value.server_id)\t\(.value.floating_ip)\t\(.value.runner_name)"' \ +sudo jq -r '.jobs | to_entries[] | "\(.key)\t\(.value.workflow_id)\t\(.value.workflow_name)"' \ /var/lib/airstack-orchestrator/state.json ``` -Pick the row for your failing `job_id` (visible in the GitHub Actions URL). Save the values: +Pick the row for your failing `job_id` (visible in the GitHub Actions URL): ```bash JOB_ID=73286176852 # from the GitHub UI -SERVER=$(sudo jq -r ".jobs[\"$JOB_ID\"].server_id" /var/lib/airstack-orchestrator/state.json) -FIP=$( sudo jq -r ".jobs[\"$JOB_ID\"].floating_ip" /var/lib/airstack-orchestrator/state.json) +WF=$(sudo jq -r ".jobs[\"$JOB_ID\"].workflow_id" /var/lib/airstack-orchestrator/state.json) ``` -If the job isn't in `state.json`, the orchestrator never spawned for it — see step 2 below. +If the job isn't in `state.json`, the orchestrator never submitted for it — see step 2. -### 2. Did the orchestrator spawn at all? +### 2. Did the orchestrator submit at all? ```bash sudo journalctl -u airstack-orchestrator.service --since "30 min ago" --no-pager ``` -What you want to see for a healthy spawn: +Healthy submit looks like: ```text -spawned server for job () -attached floating IP to server (job ) +submitted workflow for job () ``` -Common things that block a spawn (and how to spot them): +Common things that block a submit (and how to spot them): | Log line / symptom | What it means | Fix | |---|---|---| -| `find_queued_jobs failed: 401 ...` | PAT expired / wrong scope | Rotate the PAT (see Operational notes) | -| `spawn failed for job ...: Block Device Mapping is Invalid` | Flavor has `disk=0` and `boot_volume_size_gb` is 0 | Set `boot_volume_size_gb > 0` | -| `no free floating IP in pool` | All FIPs in `floating_ips` are already in use | Wait for an in-flight job to complete, or expand the pool | -| `floating_ips configured but not found` | Pool addresses don't exist in the project | Double-check `openstack floating ip list` | -| Job is queued in GitHub but no `spawned` log | Runner labels in the workflow's `runs-on` don't match `runner_labels` in config | Make them match | - -### 3. SSH into a running worker +| `find_queued_jobs failed: 401 ...` | GitHub PAT expired / wrong scope | Rotate the PAT | +| `osmo login failed ...` / `auth error` | OSMO token expired/invalid, or Envoy rejects access tokens | Rotate the OSMO token; confirm gateway accepts access tokens | +| `osmo workflow submit failed ... privileged` | Pool platform doesn't allow privileged | Ask admin to enable "Privileged Mode Allowed" on the CI pool | +| `osmo workflow submit failed ... pool` / permission | Service-account role lacks `workflow:Create` on the pool | Fix the role's pool-scoped policy | +| Job queued in GitHub but no `submitted` log | `runs-on` labels don't match `runner_labels` | Make them match | -If the worker is `ACTIVE`, the floating IP is attached and you can connect directly. The keypair was injected during spawn — use the matching private key: +### 3. Inspect the workflow / worker ```bash -ssh -i .pem ubuntu@"$FIP" -``` +# Status and scheduling detail. +osmo workflow query "$WF" --verbose -If your workstation can't reach the FIP subnet, jump through the orchestrator (which is on the same network): +# Scheduling / lifecycle events (image pull, start, evict, ...). +osmo workflow events "$WF" --task runner -```bash -ssh -J ubuntu@ -i .pem ubuntu@"$FIP" +# Combined stdout of the runner task — shows dockerd start, run.sh, and the job. +osmo workflow logs "$WF" --task runner +osmo workflow logs "$WF" --task runner --error # error stream +osmo workflow logs "$WF" --task runner -n 300 # last 300 lines ``` -### 4. SSH into a SHUTOFF worker +### 4. Break-glass shell into a running worker -Workers shut themselves down after `run.sh` exits (whether the job succeeded, failed, or the runner crashed). The orchestrator only deletes a server once GitHub reports the job `completed`, so a SHUTOFF worker is preserved while you debug. +If the workflow is still `RUNNING`, exec into the pod (replaces the old SSH-via-floating-IP path): ```bash -# Optional but safer — keep the orchestrator from reaping mid-session. -sudo systemctl stop airstack-orchestrator.service - -openstack server start "$SERVER" -# Wait ~30s, then SSH using the FIP from state.json. -ssh -i .pem ubuntu@"$FIP" +osmo workflow exec "$WF" runner # /bin/bash in the runner task ``` -When done, delete the worker manually and resume the orchestrator: +Once inside: ```bash -openstack server delete "$SERVER" -sudo jq "del(.jobs[\"$JOB_ID\"])" /var/lib/airstack-orchestrator/state.json \ - | sudo tee /var/lib/airstack-orchestrator/state.json.new >/dev/null -sudo mv /var/lib/airstack-orchestrator/state.json.new /var/lib/airstack-orchestrator/state.json -sudo systemctl start airstack-orchestrator.service +# GitHub Actions runner diagnostics. +ls -lt /home/runner/actions-runner/_diag/ +tail -300 /home/runner/actions-runner/_diag/Runner_*.log +tail -300 /home/runner/actions-runner/_diag/Worker_*.log + +# Inner Docker daemon (a frequent failure point for `airstack up`). +cat /var/log/dockerd.log +docker info 2>&1 | head +nvidia-smi ``` -### 5. What to read once you're on the worker +### 5. Common failure patterns at the worker -```bash -# Combined boot + cloud-init output. Most useful single file: shows every -# line our airstack-runner-bootstrap.sh printed, including run.sh's exit. -sudo less /var/log/cloud-init-output.log -sudo tail -300 /var/log/cloud-init-output.log - -# Cloud-init's structured log — quick way to surface errors. -sudo grep -E 'WARN|ERROR|FAIL' /var/log/cloud-init.log - -# GitHub Actions runner diagnostics. The Worker_*.log corresponds to the -# actual job execution; Runner_*.log covers registration and dispatch. -ls -lt /home/ubuntu/actions-runner/_diag/ -sudo tail -300 /home/ubuntu/actions-runner/_diag/Runner_*.log -sudo tail -300 /home/ubuntu/actions-runner/_diag/Worker_*.log - -# Sanity-check Docker came up cleanly — a frequent failure point. -sudo systemctl status docker -docker info 2>&1 | head +| Symptom in `osmo workflow logs` | Cause | Fix | +|---|---|---| +| `dockerd did not become ready` | Pod not privileged / DinD blocked | Enable privileged on the pool platform | +| `nvidia-smi unavailable` / no GPU | GPU not requested/passed, or toolkit missing | Check `gpu:` request, platform GPUs, privileged | +| `Could not connect to api.github.com` | Egress blocked from the pool | Allow outbound 443 from the CI pool | +| `Bad credentials` / `Invalid ... runnerEvent` | JIT config TTL elapsed before `run.sh` started | Prebake the image (already done) so start is fast | +| `Cannot connect to the Docker daemon` during tests | inner dockerd crashed | Read `/var/log/dockerd.log` via `osmo workflow exec` | +| Runner registered, then `pytest` failed | A normal test failure | Read the GitHub Actions log — the canonical view | +| `No space left on device` | `storage` too small for images + sim assets | Bump `storage` in `config.yaml` | +| `failed to solve: ... mount source: "overlay" ... err: invalid argument` | Docker data-root landed on the pod's overlay rootfs | See "Nested DinD and overlayfs" below | + +### Nested DinD and overlayfs + +The single most likely way to break every Docker build at once. The pod's root +filesystem is overlayfs, and **Linux refuses to use a directory on overlayfs as +an overlay `upperdir`** (it returns `EINVAL`). A dockerd whose data-root sits on +the pod rootfs looks healthy — `docker info` works, image pulls succeed, because +containerd unpacks layers with plain writes — and then every build step that +needs a real mount fails: + +```text +failed to solve: process "/bin/bash -c apt-get ... " did not complete successfully: +mount source: "overlay", +target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/buildkit1459786452", +fstype: overlay, ... err: invalid argument ``` -### 6. Console log fallback +That signature took out all four `build_docker` tests and all four +`build_packages` tests in one run, with each failure looking like an unrelated +`apt-get`/`WORKDIR` problem. -Some flavors on this cloud don't expose the serial console (`openstack console log show` returns *Guest does not have a console available*). For those, the SSH path above is the only option. Where it does work, the console log persists across SHUTOFF and is faster than restarting the VM: +[`runner-entrypoint.sh`](runner-entrypoint.sh) handles this before starting +dockerd. It picks a storage backend by **performing a real overlay mount** to +test each option rather than trusting the filesystem type, and falls back in +this order: + +| Order | Backend | Notes | +|---|---|---| +| 1 | Loopback ext4 image mounted at `/var/lib/docker` | Preferred. Real `overlay2`, self-contained, dies with the pod. Sparse, so it only consumes what Docker writes. Sized to free space on `/` minus 20 GiB, or `DOCKER_LOOP_SIZE_MB`. | +| 2 | A real filesystem already mounted in the pod | Kubernetes `emptyDir`/`hostPath`/PVC volumes live on the node disk, not the overlay rootfs. `/osmo/data/output` and `/osmo/data/socket` are skipped — the OSMO ctrl sidecar owns them. | +| 3 | `fuse-overlayfs` driver | Stacks where the kernel driver won't. Needs `/dev/fuse`. | +| 4 | `vfs` driver | Always works, copies the whole filesystem per layer. Too slow and too large for the sim images — **reaching this is a red flag**, not a working state. | + +The chosen backend is logged at startup, so confirm it in the job log before +debugging anything else: ```bash -openstack console log show "$SERVER" | tail -200 +osmo workflow logs "$WF" --task runner | grep -E 'runner-entrypoint|storage driver' ``` -### 7. Common failure patterns at the worker +Backends 3 and 4 also set `features.containerd-snapshotter: false`, because +`storage-driver` is only honoured by the classic image store. -| Symptom in `cloud-init-output.log` (near end) | Cause | Fix | -|---|---|---| -| `Could not connect to api.github.com` / DNS errors | Security group blocking egress, or no NAT for the network | Allow outbound 443; if behind NAT, ensure FIP networking covers egress | -| `Bad credentials` / `Invalid configuration ... runnerEvent` | JIT config TTL elapsed before `run.sh` started — bootstrap took too long | Pre-bake Docker + nvidia-container-toolkit into the image to shrink bootstrap | -| `nvidia-ctk: command not found` or NVIDIA driver mismatch | Image's driver doesn't match the toolkit version | Use a different image, or pin a compatible toolkit version | -| `apt-get update` fails | Image's apt sources are unreachable from this network | Check network/security-group; or pre-bake packages into the image | -| Runner registered, then `pytest` failed | A normal test failure | Read the GitHub Actions log — that's the canonical view of the workflow output | -| `No space left on device` | `boot_volume_size_gb` too small for Docker images + sim assets | Bump `boot_volume_size_gb` | +The one-shot [`build-runner-on-osmo.yaml`](build-runner-on-osmo.yaml) builder +sidesteps the same problem differently — `vfs` plus `DOCKER_BUILDKIT=0` — which +is fine there because it builds one small image. diff --git a/.github/orchestrator/airstack-orchestrator.service b/.github/orchestrator/airstack-orchestrator.service index 7123232eb..0c5c1843d 100644 --- a/.github/orchestrator/airstack-orchestrator.service +++ b/.github/orchestrator/airstack-orchestrator.service @@ -1,5 +1,5 @@ [Unit] -Description=AirStack CI Orchestrator (spawns ephemeral OpenStack runners) +Description=AirStack CI Orchestrator (submits ephemeral OSMO runner workflows) Documentation=https://github.com/castacks/AirStack/tree/main/.github/orchestrator After=network-online.target Wants=network-online.target @@ -10,17 +10,20 @@ User=orchestrator Group=orchestrator WorkingDirectory=/opt/airstack-orchestrator -# Application credential lives in the orchestrator user's home so openstacksdk -# finds it via the default cloud-config search path. -Environment=HOME=/home/orchestrator -Environment=OS_CLIENT_CONFIG_FILE=/home/orchestrator/.config/openstack/clouds.yaml +# The `osmo` CLI persists its login session under $HOME/XDG dirs. Point them at +# the (writable) state dir so ProtectHome can stay read-only. setup.sh creates +# these directories owned by the orchestrator user. +Environment=HOME=/var/lib/airstack-orchestrator +Environment=XDG_CONFIG_HOME=/var/lib/airstack-orchestrator/.config +Environment=XDG_CACHE_HOME=/var/lib/airstack-orchestrator/.cache +Environment=XDG_STATE_HOME=/var/lib/airstack-orchestrator/.state ExecStart=/opt/airstack-orchestrator/venv/bin/python \ /opt/airstack-orchestrator/orchestrator.py \ --config /etc/airstack-orchestrator/config.yaml \ --pat /etc/airstack-orchestrator/github-pat \ --state /var/lib/airstack-orchestrator/state.json \ - --template /opt/airstack-orchestrator/cloud-init.yaml.j2 + --template /opt/airstack-orchestrator/runner-workflow.yaml.j2 Restart=always RestartSec=10 @@ -33,6 +36,8 @@ KillSignal=SIGTERM NoNewPrivileges=true ProtectSystem=strict ProtectHome=read-only +# ReadWritePaths re-grants write access under ProtectSystem/ProtectHome so the +# OSMO CLI session cache and state.json can be written. ReadWritePaths=/var/lib/airstack-orchestrator PrivateTmp=true diff --git a/.github/orchestrator/build-and-push.sh b/.github/orchestrator/build-and-push.sh new file mode 100755 index 000000000..aacefb002 --- /dev/null +++ b/.github/orchestrator/build-and-push.sh @@ -0,0 +1,28 @@ +#!/usr/bin/env bash +# Build & push the AirStack CI ephemeral-runner image to AirLab Harbor. +# Run on a linux/amd64 machine (or buildx --platform linux/amd64) with: +# docker login airlab-docker.andrew.cmu.edu +# +# Usage: +# ./build-and-push.sh +# RUNNER_VERSION=2.336.0 ./build-and-push.sh + +set -euo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REGISTRY="${REGISTRY:-airlab-docker.andrew.cmu.edu/airstack}" +RUNNER_VERSION="${RUNNER_VERSION:-2.336.0}" +IMAGE="${REGISTRY}/airstack-ci-runner:${RUNNER_VERSION}" + +echo "==> Building ${IMAGE}" +docker build \ + -f "${ROOT}/runner.Dockerfile" \ + --build-arg "RUNNER_VERSION=${RUNNER_VERSION}" \ + -t "${IMAGE}" \ + "${ROOT}" + +echo "==> Pushing ${IMAGE}" +docker push "${IMAGE}" + +echo "==> Done. Set in /etc/airstack-orchestrator/config.yaml:" +echo " runner_image: \"${IMAGE}\"" diff --git a/.github/orchestrator/build-runner-on-osmo.yaml b/.github/orchestrator/build-runner-on-osmo.yaml new file mode 100644 index 000000000..a44f21ab5 --- /dev/null +++ b/.github/orchestrator/build-runner-on-osmo.yaml @@ -0,0 +1,63 @@ +# One-shot OSMO job: build + push airstack-ci-runner to AirLab Harbor. +# Uses the existing privileged DinD workspace image (same as airstack-dev). +# +# Prereq: your OSMO profile has airlab-docker-login (+ auto REGISTRY cred). +# +# osmo workflow submit .github/orchestrator/build-runner-on-osmo.yaml \ +# --pool airstack --priority HIGH +# +# Watch: +# osmo workflow logs --task build +# Cancel when done if it hangs: +# osmo workflow cancel --force + +workflow: + name: build-airstack-ci-runner + resources: + build: + cpu: 8 + gpu: 0 + memory: 16Gi + storage: 100Gi + platform: default + tasks: + - name: build + image: airlab-docker.andrew.cmu.edu/airstack/airstack-osmo-workspace:latest + resource: build + privileged: true + credentials: + airlab-docker-login: + AIRLAB_REGISTRY_USER: username + AIRLAB_REGISTRY_PASS: password + environment: + AIRSTACK_REPO_URL: "https://github.com/castacks/AirStack.git" + AIRSTACK_BRANCH: "ci/osmo-orchestrator" + RUNNER_VERSION: "2.336.0" + REGISTRY: "airlab-docker.andrew.cmu.edu/airstack" + command: + - bash + - -lc + - | + set -euo pipefail + # Nested DinD: overlay-on-overlay breaks BuildKit. Use vfs + legacy builder. + mkdir -p /etc/docker + cat > /etc/docker/daemon.json <<'JSON' + {"storage-driver": "vfs"} + JSON + dockerd >/var/log/dockerd.log 2>&1 & + for _ in $(seq 1 90); do docker info >/dev/null 2>&1 && break; sleep 1; done + docker info >/dev/null 2>&1 || { cat /var/log/dockerd.log; exit 1; } + docker info | grep -i 'Storage Driver' || true + + echo "$AIRLAB_REGISTRY_PASS" | docker login airlab-docker.andrew.cmu.edu \ + -u "$AIRLAB_REGISTRY_USER" --password-stdin + + rm -rf /tmp/AirStack + git clone --depth 1 --branch "$AIRSTACK_BRANCH" "$AIRSTACK_REPO_URL" /tmp/AirStack + cd /tmp/AirStack/.github/orchestrator + IMAGE="${REGISTRY}/airstack-ci-runner:${RUNNER_VERSION}" + DOCKER_BUILDKIT=0 docker build -f runner.Dockerfile \ + --build-arg "RUNNER_VERSION=${RUNNER_VERSION}" \ + -t "$IMAGE" . + docker push "$IMAGE" + echo "PUSHED $IMAGE" diff --git a/.github/orchestrator/cloud-init.yaml.j2 b/.github/orchestrator/cloud-init.yaml.j2 deleted file mode 100644 index 921417c18..000000000 --- a/.github/orchestrator/cloud-init.yaml.j2 +++ /dev/null @@ -1,71 +0,0 @@ -#cloud-config -# Rendered per-spawn by orchestrator.py with two Jinja variables: -# encoded_jit_config - single-use base64 JIT config from GitHub -# runner_version - GitHub Actions runner version (e.g. 2.334.0) -# -# The base image (Ubuntu-24.04-GPU-Headless) already has NVIDIA drivers. -# This cloud-init adds Docker (with the compose plugin), nvidia-container-toolkit, -# downloads the GitHub Actions runner, registers it with the JIT config, runs -# exactly one job (the JIT config + --ephemeral makes the runner exit after one -# job), and shuts the VM down. The orchestrator then deletes the server. - -package_update: true -package_upgrade: false -packages: - - jq - - curl - - ca-certificates - - gnupg - -write_files: - - path: /usr/local/bin/airstack-runner-bootstrap.sh - permissions: "0755" - owner: root:root - content: | - #!/usr/bin/env bash - set -euxo pipefail - - # Install Docker (with compose plugin) from Docker's official channel. - # get.docker.com handles apt repo setup + nvidia-container-toolkit-compatible - # docker-ce, plus the docker-compose-plugin we need for `airstack up`. - curl -fsSL https://get.docker.com | sh - - # nvidia-container-toolkit is required for GPU containers (liveliness / - # autonomy tests). The base image has the NVIDIA *drivers* but we still - # need the container runtime hooks here. - distribution=$(. /etc/os-release; echo "$ID$VERSION_ID") - curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ - | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg - curl -fsSL "https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list" \ - | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ - > /etc/apt/sources.list.d/nvidia-container-toolkit.list - apt-get update - apt-get install -y nvidia-container-toolkit - nvidia-ctk runtime configure --runtime=docker - systemctl restart docker - - usermod -aG docker ubuntu - - # GitHub Actions runner. - RUNNER_VERSION="{{ runner_version }}" - RUNNER_DIR=/home/ubuntu/actions-runner - mkdir -p "$RUNNER_DIR" - cd "$RUNNER_DIR" - curl -fsSL -o runner.tar.gz \ - "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" - tar xzf runner.tar.gz - rm runner.tar.gz - chown -R ubuntu:ubuntu "$RUNNER_DIR" - - # Run exactly one job under the ubuntu user. The JIT config is single-use - # and ephemeral, so run.sh exits after one job completes. - sudo -u ubuntu --preserve-env=HOME -H bash -c \ - "cd '$RUNNER_DIR' && ./run.sh --jitconfig '{{ encoded_jit_config }}'" \ - || echo "runner exited non-zero (job failure or runner error)" - - # Backstop: power down. The orchestrator's reap loop is the authoritative - # deleter — it sees the GitHub job complete and calls Nova delete. - shutdown -h +1 - -runcmd: - - /usr/local/bin/airstack-runner-bootstrap.sh diff --git a/.github/orchestrator/config.example.yaml b/.github/orchestrator/config.example.yaml index 4a47bbe1f..ba1c64384 100644 --- a/.github/orchestrator/config.example.yaml +++ b/.github/orchestrator/config.example.yaml @@ -1,54 +1,66 @@ -# AirStack CI orchestrator configuration. +# AirStack CI orchestrator configuration (OSMO backend). # Copy to /etc/airstack-orchestrator/config.yaml and fill in placeholders. -# --- OpenStack target --- - -# Cloud profile name in ~/.config/openstack/clouds.yaml. -openstack_cloud: airstack - -# Ubuntu-24.04-Desktop (confirmed available on airlab-cloud). -image_id: 2ebb9061-8995-4238-a3cc-e230a3e863aa - -# OpenStack flavor with GPU + enough disk for Docker + sim images. -# Look up with: openstack flavor list -flavor_name: "gpu.rtxpro5000.1" - -# OpenStack network the ephemeral instance attaches to. Must allow outbound -# 443 to api.github.com (no inbound is required: the runner makes an outbound -# long-poll connection to GitHub). -network_name: "airstack.AirLab.Apps_group_network_gates" - -# OpenStack keypair injected into the instance for break-glass SSH access. -# The orchestrator never SSHes into workers itself. -keypair_name: "airstack-ci-cd" - -# Security group applied to spawned instances. Outbound 443 must be allowed. -security_group: "default" - -# OpenStack availability zone to spawn instances in (e.g. nova, gpu-zone-1). -# Leave empty to let Nova pick. -availability_zone: "gates" - -# If the chosen flavor has disk=0 (common for GPU flavors), Nova rejects -# direct image-boot with "Block Device Mapping is Invalid: You specified more -# local devices than the limit allows". Set this to >0 to boot from a Cinder -# volume of that size sourced from image_id (deleted on termination). Leave -# at 0 to boot directly from the image (only works for non-zero-disk flavors). -boot_volume_size_gb: 300 - -# Pre-allocated pool of floating IPs to rotate through for SSH access to -# workers. The orchestrator picks the first free IP from this list, in order, -# for each new spawn. When the worker is destroyed the IP auto-disassociates -# and returns to the pool. If non-empty, max_concurrent is capped at len(pool) -# so the orchestrator never spawns a worker it can't address. -# Allocate via: openstack floating ip create -# Leave empty to skip floating-IP attachment entirely. -floating_ips: [] -# Example: -# floating_ips: -# - 172.19.220.131 -# - 172.19.220.171 -# - 172.19.220.89 +# --- OSMO target --- + +# Path to the `osmo` CLI. The install script (see setup.sh / README) puts it on +# PATH as `osmo`; override with a full path if needed. +osmo_bin: "osmo" + +# URL of your OSMO web service (the control plane the CLI logs into). +# AirLab: https://airlab-share-01.andrew.cmu.edu +osmo_url: "https://airlab-share-01.andrew.cmu.edu" + +# File containing the OSMO service-account access token. This is the shared, +# non-personal "lab" identity — the analog of the old OpenStack application +# credential. The orchestrator runs `osmo login --method token --token-file` +# with it. Created by an OSMO admin via `osmo user create` + `osmo token set` +# (see README). It never leaves this host. +osmo_token_file: "/etc/airstack-orchestrator/osmo-token" + +# GPU pool for ephemeral runners. AirLab team pools are Keycloak-autosynced; +# use the stable `airstack` pool (privileged_allowed=true). A hand-made +# `airstack-ci` pool will be wiped by synchronize_osmo_team_pools.py unless +# it is added to Keycloak. The service-account needs workflow:Create on this +# pool (role osmo-airstack) plus osmo-user for cancel/query. +pool: "airstack" + +# Optional platform (hardware type) to target within the pool. Leave empty to +# use the pool's default platform. List options with `osmo pool list` / the UI. +platform: "default" + +# Scheduling priority: HIGH | NORMAL | LOW. +priority: "NORMAL" + +# --- Runner task (the per-job worker) --- + +# Prebaked image: Docker CE + compose + nvidia-container-toolkit + GH Actions +# runner. Build & push runner.Dockerfile to a registry the pool can pull from. +runner_image: "airlab-docker.andrew.cmu.edu/airstack/airstack-ci-runner:2.336.0" + +# Resource request for the runner container. Size for the full stack build + +# sim (Isaac Sim / ms-airsim + robot + gcs) running under docker compose. +cpu: 8 +gpu: 1 +memory: "32Gi" +storage: "300Gi" + +# REQUIRED for the AirStack tests: they run `airstack up` (docker compose) +# inside the pod, which needs an inner Docker daemon -> a privileged container. +# The pool's platform must have "Privileged Mode Allowed" enabled by your OSMO +# admin, otherwise submission/scheduling will be rejected. +privileged: true + +# Use the node's host network for the runner container. Usually not needed +# (the runner only makes outbound calls to GitHub); leave false unless the +# tests need host networking. +host_network: false + +# GitHub Actions runner version to bake into runner_image. Kept here for +# reference/traceability; it is a build arg of runner.Dockerfile, not consumed +# by the orchestrator at runtime. Must match a tag at +# https://github.com/actions/runner/releases +runner_version: "2.336.0" # --- GitHub --- @@ -56,23 +68,22 @@ floating_ips: [] repo: "castacks/AirStack" # Labels the orchestrator polls for. A queued workflow_job whose `labels` -# array is a superset of this list gets a server spawned for it. +# array is a superset of this list gets a workflow submitted for it. These are +# unchanged from the OpenStack backend, so system-tests.yml needs no edits. runner_labels: - self-hosted - airstack-ephemeral -# GitHub Actions runner version (must exist as a release tag at -# https://github.com/actions/runner/releases). -runner_version: "2.334.0" - # --- Limits --- -# Maximum simultaneous in-flight ephemeral instances. +# Maximum simultaneous in-flight workflows the orchestrator will submit. OSMO +# queues anything beyond the pool's capacity on its own, but this caps how many +# jobs we hand it at once. max_concurrent: 3 -# Hard ceiling for a single job. Past this age the reaper force-deletes the -# server even if GitHub still reports the job as in-progress. Must comfortably -# exceed the longest expected job (autonomy/liveliness runs). +# Hard ceiling for a single job. Past this age the reaper cancels the workflow +# even if GitHub still reports the job as in-progress. Must comfortably exceed +# the longest expected job (liveliness / autonomy runs). max_job_minutes: 2880 # 48 hours # --- Polling intervals (seconds) --- @@ -80,8 +91,10 @@ max_job_minutes: 2880 # 48 hours spawn_poll_interval_s: 15 reap_poll_interval_s: 30 -# How long to wait for a freshly-created server to reach ACTIVE before -# treating the spawn as failed. If Nova flips the server to ERROR within this -# window the orchestrator logs the full fault (code/message/details/host/AZ) -# and deletes the server so the next iteration can retry cleanly. -server_active_timeout_s: 300 +# How long to wait for `osmo workflow submit` to return before treating the +# submission as failed. +submit_timeout_s: 180 + +# Name prefix for submitted workflows. Also used by the orphan sweep to find +# workflows this orchestrator owns. Keep the trailing dash. +workflow_name_prefix: "gha-runner-" diff --git a/.github/orchestrator/orchestrator.py b/.github/orchestrator/orchestrator.py index 3e65e906f..4af62555b 100644 --- a/.github/orchestrator/orchestrator.py +++ b/.github/orchestrator/orchestrator.py @@ -1,56 +1,64 @@ #!/usr/bin/env python3 -"""AirStack CI orchestrator. +"""AirStack CI orchestrator (OSMO backend). Polls the GitHub API for queued workflow_jobs whose labels match this -orchestrator's runner_labels, and spawns truly ephemeral OpenStack instances -to execute them. Each ephemeral instance receives a single-use GitHub JIT -runner config via cloud-init; the GitHub PAT never leaves this orchestrator. +orchestrator's runner_labels, and submits truly ephemeral OSMO workflows to +execute them. Each workflow runs a single-job GitHub Actions runner in a +privileged, GPU-enabled container on an OSMO compute pool; the GitHub PAT never +leaves this orchestrator, and an OSMO service-account token (not a personal +account) is used only to submit / query / cancel workflows. + +This is a drop-in replacement for the previous OpenStack-Nova backend: the +GitHub side is unchanged (`runs-on: [self-hosted, airstack-ephemeral]`, the +single-use JIT runner config, the same-repo fork guard). Only the *spawn* +target changed from "create a Nova VM" to "submit an OSMO workflow". The +one-job-per-worker, destroy-after semantics are preserved — when the runner's +`run.sh` exits after a single job, the OSMO task completes and the pod is torn +down. Two cooperating loops: - - spawn loop: discover queued jobs, spawn one Nova server per job - - reap loop: delete servers whose jobs have completed, plus stragglers + - spawn loop: discover queued jobs, submit one OSMO workflow per job + - reap loop: cancel workflows whose jobs have completed, plus stragglers older than max_job_minutes and orphans not in state.json State persists in /var/lib/airstack-orchestrator/state.json so the -orchestrator can survive restarts without leaking instances. +orchestrator can survive restarts without leaking workflows. """ from __future__ import annotations import argparse -import base64 import json import logging import os +import re import signal +import subprocess import sys +import tempfile import threading import time from datetime import datetime, timezone from pathlib import Path from typing import Any -import openstack import requests import yaml from jinja2 import Template DEFAULT_CONFIG_PATH = "/etc/airstack-orchestrator/config.yaml" DEFAULT_PAT_PATH = "/etc/airstack-orchestrator/github-pat" +DEFAULT_OSMO_TOKEN_PATH = "/etc/airstack-orchestrator/osmo-token" DEFAULT_STATE_PATH = "/var/lib/airstack-orchestrator/state.json" -DEFAULT_TEMPLATE_PATH = "/opt/airstack-orchestrator/cloud-init.yaml.j2" - -# Metadata key/value applied to every Nova server we spawn. Used by the -# orphan reaper to identify servers we own even when state.json is missing. -ROLE_META_KEY = "airstack-role" -ROLE_META_VAL = "ephemeral-runner" -JOB_META_KEY = "airstack-job-id" +DEFAULT_TEMPLATE_PATH = "/opt/airstack-orchestrator/runner-workflow.yaml.j2" GITHUB_API = "https://api.github.com" log = logging.getLogger("orchestrator") +# ── file / state helpers ──────────────────────────────────────────────────── + def load_yaml(path: str) -> dict: with open(path) as f: return yaml.safe_load(f) @@ -76,6 +84,16 @@ def save_state(path: str, state: dict) -> None: os.replace(tmp, path) +def now_utc_iso() -> str: + return datetime.now(timezone.utc).isoformat() + + +def parse_iso(s: str) -> datetime: + return datetime.fromisoformat(s) + + +# ── GitHub API (unchanged from the OpenStack backend) ───────────────────────── + def gh_request(method: str, path: str, pat: str, **kwargs: Any) -> Any: url = f"{GITHUB_API}{path}" headers = kwargs.pop("headers", {}) @@ -156,267 +174,134 @@ def get_job_status(repo: str, job_id: str, pat: str) -> dict | None: return r.json() -def render_cloud_init(template_path: str, encoded_jit_config: str, - runner_version: str) -> str: - with open(template_path) as f: - tmpl = Template(f.read()) - return tmpl.render( - encoded_jit_config=encoded_jit_config, - runner_version=runner_version, - ) - - -def spawn_server( - conn: openstack.connection.Connection, - config: dict, - name: str, - job_id: str, - user_data: str, -) -> str: - flavor = conn.compute.find_flavor(config["flavor_name"], ignore_missing=False) - network = conn.network.find_network(config["network_name"], ignore_missing=False) - create_kwargs = dict( - name=name, - flavor_id=flavor.id, - networks=[{"uuid": network.id}], - key_name=config["keypair_name"], - security_groups=[{"name": config["security_group"]}], - user_data=base64.b64encode(user_data.encode()).decode(), - metadata={ - ROLE_META_KEY: ROLE_META_VAL, - JOB_META_KEY: job_id, - }, - ) - - # Flavors with disk=0 (typical for GPU flavors on this cloud) cannot boot - # directly from an image — Nova rejects with "Block Device Mapping is - # Invalid: You specified more local devices than the limit allows". When - # boot_volume_size_gb is set, boot from a Cinder volume sourced from the - # image and delete it on termination. Otherwise fall back to direct image - # boot (works only when the flavor has a non-zero root disk). - boot_volume_size_gb = int(config.get("boot_volume_size_gb") or 0) - if boot_volume_size_gb > 0: - create_kwargs["block_device_mapping"] = [ - { - "uuid": config["image_id"], - "source_type": "image", - "destination_type": "volume", - "boot_index": 0, - "volume_size": boot_volume_size_gb, - "delete_on_termination": True, - } - ] - else: - create_kwargs["image_id"] = config["image_id"] - - az = config.get("availability_zone") - if az: - create_kwargs["availability_zone"] = az - server = conn.compute.create_server(**create_kwargs) - return server.id - - -def delete_server(conn: openstack.connection.Connection, server_id: str) -> None: +# ── OSMO CLI output parsing ─────────────────────────────────────────────────── +# +# The exact JSON keys returned by `osmo workflow {submit,query,list}` can vary +# slightly by OSMO release, so these parsers try a set of likely keys and fall +# back to scraping the human-readable text output. Verify the keys against your +# deployed version with `osmo workflow submit --dry-run` / `--format-type json` +# once and simplify if desired. + +# Live AirLab OSMO 6.2.x returns workflow_uuid on list/query; submit may use +# workflow_id / id / name. Prefer uuid-like keys before "name" so we don't +# accidentally treat the human workflow name as the id when both are present. +_WF_ID_KEYS = ( + "workflow_uuid", "workflow_id", "workflowId", "id", "uuid", "name", "workflow", +) +_STATUS_KEYS = ("status", "state", "workflow_status", "phase") +_KNOWN_STATUSES = { + "RUNNING", "PENDING", "WAITING", "COMPLETED", "FAILED", + "FAILED_EXEC_TIMEOUT", "FAILED_SERVER_ERROR", "FAILED_QUEUE_TIMEOUT", + "FAILED_SUBMISSION", "FAILED_CANCELED", "FAILED_BACKEND_ERROR", + "FAILED_IMAGE_PULL", "FAILED_EVICTED", "FAILED_START_ERROR", + "FAILED_START_TIMEOUT", "FAILED_PREEMPTED", +} +# Non-terminal statuses the orphan sweep considers "still alive". +_ACTIVE_STATUSES = ("RUNNING", "PENDING", "WAITING") + + +def _loads_or_none(text: str | None) -> Any: try: - conn.compute.delete_server(server_id, ignore_missing=True, force=True) - except Exception as e: - log.warning("delete_server(%s) failed: %s", server_id, e) - - -def list_owned_servers(conn: openstack.connection.Connection) -> list[Any]: - """List all Nova servers that carry our role metadata.""" - owned = [] - for s in conn.compute.servers(details=True): - meta = getattr(s, "metadata", None) or {} - if meta.get(ROLE_META_KEY) == ROLE_META_VAL: - owned.append(s) - return owned - - -def find_free_floating_ip( - conn: openstack.connection.Connection, pool: list[str] -) -> Any: - """Return the FloatingIP resource for the first address in `pool` that is - not currently associated with any port. Returns None if all are in use. - - Iterates `pool` in order so attachments rotate through it sequentially. - Logs a warning for any pool member that doesn't exist in this project. - """ - if not pool: + return json.loads(text) # type: ignore[arg-type] + except (json.JSONDecodeError, TypeError): return None - pool_set = set(pool) - fips_by_addr: dict[str, Any] = {} - for fip in conn.network.ips(): - if fip.floating_ip_address in pool_set: - fips_by_addr[fip.floating_ip_address] = fip - missing = pool_set - fips_by_addr.keys() - if missing: - log.warning( - "floating_ips configured but not found in this project: %s", - sorted(missing), - ) - for addr in pool: - fip = fips_by_addr.get(addr) - if fip is not None and not fip.port_id: - return fip - return None - -def check_flavor_capacity( - conn: openstack.connection.Connection, - flavor_name: str, -) -> tuple[bool, str]: - """Pre-flight: ask Nova's placement API whether any host can satisfy this - flavor's resource request right now. - Returns (ok, reason). When ok=False the orchestrator should defer the - spawn iteration; reason is a one-line human-readable explanation - (e.g. "no host can satisfy {'VCPU': 8, 'MEMORY_MB': 32768, 'VGPU': 1}"). - - If the placement API can't be queried for any reason we return - (True, "") and let Nova make the call. The - pre-flight is a fast-path optimization, not a gate — Nova still has the - final say at create_server time (and ERROR-status fallback handles - anything we miss). - """ - try: - flavor = conn.compute.find_flavor(flavor_name, ignore_missing=False) - except Exception as e: - return True, f"flavor lookup failed: {e}" - - # Standard resources every Nova flavor expresses. - resources: dict[str, int] = {} - if getattr(flavor, "vcpus", 0): - resources["VCPU"] = int(flavor.vcpus) - if getattr(flavor, "ram", 0): - resources["MEMORY_MB"] = int(flavor.ram) - if getattr(flavor, "disk", 0): - resources["DISK_GB"] = int(flavor.disk) - - # Custom / specialized resources (VGPU, PCI_*, CUSTOM_*) come from the - # flavor's extra_specs as `resources:=`. This is how Nova - # itself learns to ask placement for GPU capacity. - extra = getattr(flavor, "extra_specs", {}) or {} - for k, v in extra.items(): - if not k.startswith("resources:"): - continue - rc = k.split(":", 1)[1] - try: - resources[rc] = int(v) - except (TypeError, ValueError): - pass - - if not resources: - return True, "flavor expresses no resources — skipping placement check" +def _first_str(d: dict, keys: tuple[str, ...]) -> str | None: + for k in keys: + v = d.get(k) + if isinstance(v, str) and v: + return v + return None - try: - result = conn.placement.allocation_candidates( - resources=resources, limit=1, - ) - if hasattr(result, "allocation_requests"): - candidates = list(result.allocation_requests or []) - else: - candidates = list(result) - except Exception as e: - return True, f"placement query failed ({type(e).__name__}: {e})" - - if candidates: - return True, "" - return False, f"no host can satisfy {resources}" - - -def wait_for_server_active( - conn: openstack.connection.Connection, - server_id: str, - timeout_s: int = 300, - poll_interval_s: float = 3.0, -) -> Any: - """Poll Nova until the server is ACTIVE. Raise with full context if it - enters ERROR or never reaches ACTIVE in time. - - Nova surfaces the actual reason for an ERROR via the `fault` attribute - (message + code + details), so we log it verbatim. We also include - task_state / vm_state / power_state because Nova sometimes leaves the - fault empty and these tell you whether the failure was at scheduling, - networking, or block-device-mapping time. - """ - deadline = time.monotonic() + timeout_s - last_status = "?" - last_task = None - while time.monotonic() < deadline: - s = conn.compute.get_server(server_id) - status = getattr(s, "status", "UNKNOWN") or "UNKNOWN" - task = ( - getattr(s, "task_state", None) - or getattr(s, "OS-EXT-STS:task_state", None) - ) - if status != last_status or task != last_task: - log.info( - "server %s status=%s task_state=%s", server_id, status, task, - ) - last_status, last_task = status, task - if status == "ACTIVE": +def _extract_workflow_id(stdout: str | None) -> str | None: + data = _loads_or_none(stdout) + if isinstance(data, dict): + wid = _first_str(data, _WF_ID_KEYS) + if wid: + return wid + wf = data.get("workflow") + if isinstance(wf, dict): + wid = _first_str(wf, _WF_ID_KEYS) + if wid: + return wid + m = re.search(r"Workflow\s*ID\s*[-:]\s*(\S+)", stdout or "", re.IGNORECASE) + return m.group(1) if m else None + + +def _extract_status(stdout: str | None) -> str | None: + data = _loads_or_none(stdout) + if isinstance(data, dict): + st = _first_str(data, _STATUS_KEYS) + if st: + return st.upper() + wf = data.get("workflow") + if isinstance(wf, dict): + st = _first_str(wf, _STATUS_KEYS) + if st: + return st.upper() + up = (stdout or "").upper() + for s in sorted(_KNOWN_STATUSES, key=len, reverse=True): + if s in up: return s + return None - if status == "ERROR": - fault = getattr(s, "fault", None) or {} - vm_state = ( - getattr(s, "vm_state", None) - or getattr(s, "OS-EXT-STS:vm_state", None) - ) - power_state = ( - getattr(s, "power_state", None) - or getattr(s, "OS-EXT-STS:power_state", None) - ) - host = getattr(s, "compute_host", None) or getattr( - s, "OS-EXT-SRV-ATTR:host", None - ) - az = getattr(s, "availability_zone", None) or getattr( - s, "OS-EXT-AZ:availability_zone", None - ) - raise RuntimeError( - "server " - + str(server_id) - + " entered ERROR: " - + f"fault.code={fault.get('code')!r} " - + f"fault.message={fault.get('message')!r} " - + f"fault.details={fault.get('details')!r} " - + f"task_state={task!r} vm_state={vm_state!r} " - + f"power_state={power_state!r} host={host!r} az={az!r}" - ) - time.sleep(poll_interval_s) +def _extract_workflow_list(stdout: str | None) -> list[dict]: + data = _loads_or_none(stdout) + if isinstance(data, dict): + for key in ("workflows", "items", "results", "data"): + if isinstance(data.get(key), list): + data = data[key] + break + items: list[dict] = [] + if isinstance(data, list): + for entry in data: + if not isinstance(entry, dict): + continue + wid = _first_str(entry, _WF_ID_KEYS) + name = entry.get("name") if isinstance(entry.get("name"), str) else None + status = _first_str(entry, _STATUS_KEYS) + if wid or name: + items.append( + {"id": wid, "name": name, + "status": status.upper() if status else None} + ) + return items + - raise RuntimeError( - f"server {server_id} did not reach ACTIVE within {timeout_s}s " - f"(last status={last_status!r} task_state={last_task!r})" - ) +def _is_terminal(status: str | None) -> bool: + if not status: + return False + return status == "COMPLETED" or status.startswith("FAILED") -def attach_floating_ip( - conn: openstack.connection.Connection, server_id: str, fip: Any -) -> str: - """Wait for the server to have a network port, then associate `fip`. - Returns the floating IP address.""" - for _ in range(60): # ~120s - ports = list(conn.network.ports(device_id=server_id)) - if ports: - break - time.sleep(2) - else: - raise RuntimeError(f"server {server_id} got no network port within 120s") - conn.network.update_ip(fip, port_id=ports[0].id) - return fip.floating_ip_address +def _looks_like_auth_error(r: subprocess.CompletedProcess) -> bool: + blob = f"{r.stdout or ''}\n{r.stderr or ''}".lower() + markers = ("401", "403", "unauthorized", "forbidden", "expired", + "not logged in", "please login", "authentication", + "invalid token", "token is invalid") + return any(m in blob for m in markers) -def now_utc_iso() -> str: - return datetime.now(timezone.utc).isoformat() +def _name_age_minutes(name: str | None) -> float | None: + """Age in minutes parsed from our `...-` name suffix, or None. + OSMO may append its own suffix after the name we submit, so we match the + first 10+ digit run (the unix timestamp) even when trailing chars follow. + """ + m = re.search(r"-(\d{10,})(?:\D.*)?$", name or "") + if not m: + return None + try: + ts = int(m.group(1)) + except ValueError: + return None + return (time.time() - ts) / 60.0 -def parse_iso(s: str) -> datetime: - return datetime.fromisoformat(s) +# ── orchestrator ────────────────────────────────────────────────────────────── class Orchestrator: def __init__(self, config: dict, pat: str, state_path: str, template_path: str): @@ -424,230 +309,308 @@ def __init__(self, config: dict, pat: str, state_path: str, template_path: str): self.pat = pat self.state_path = state_path self.template_path = template_path - self.conn = openstack.connect(cloud=config.get("openstack_cloud", "airstack")) + + # OSMO target. + self.osmo_bin = config.get("osmo_bin", "osmo") + self.osmo_url = config["osmo_url"] + self.token_file = config.get("osmo_token_file", DEFAULT_OSMO_TOKEN_PATH) + self.pool = config["pool"] + self.platform = config.get("platform", "") or "" + self.priority = str(config.get("priority", "NORMAL")).upper() + + # Runner task shape. + self.runner_image = config["runner_image"] + self.cpu = config.get("cpu", 8) + self.gpu = config.get("gpu", 1) + self.memory = config.get("memory", "32Gi") + self.storage = config.get("storage", "300Gi") + self.privileged = bool(config.get("privileged", True)) + self.host_network = bool(config.get("host_network", False)) + + # GitHub. self.repo = config["repo"] self.runner_labels = config["runner_labels"] - self.runner_version = config["runner_version"] + + # Limits / timing. self.max_concurrent = int(config.get("max_concurrent", 3)) - self.floating_ips: list[str] = list(config.get("floating_ips") or []) - # Cap spawns to FIP pool size so we never queue jobs we can't address. - self.effective_max_concurrent = self.max_concurrent - if self.floating_ips: - self.effective_max_concurrent = min( - self.max_concurrent, len(self.floating_ips) - ) - self.max_job_minutes = int(config.get("max_job_minutes", 90)) + self.max_job_minutes = int(config.get("max_job_minutes", 2880)) self.spawn_interval = int(config.get("spawn_poll_interval_s", 15)) self.reap_interval = int(config.get("reap_poll_interval_s", 30)) + self.submit_timeout = int(config.get("submit_timeout_s", 180)) + self.workflow_prefix = config.get("workflow_name_prefix", "gha-runner-") + self.stop_evt = threading.Event() + # Establish the OSMO session up-front for early feedback; individual + # commands re-login on demand if the session lapses. + self._login() + def stop(self, *_: Any) -> None: log.info("stop signal received; draining loops") self.stop_evt.set() + # ── OSMO CLI plumbing ──────────────────────────────────────────────────── + + def _run_osmo(self, args: list[str], timeout: int) -> subprocess.CompletedProcess: + return subprocess.run( + [self.osmo_bin, *args], + capture_output=True, text=True, timeout=timeout, + ) + + def _login(self) -> bool: + try: + r = self._run_osmo( + ["login", self.osmo_url, "--method", "token", + "--token-file", self.token_file], + timeout=60, + ) + except Exception as e: # noqa: BLE001 - startup best-effort + log.warning("osmo login raised: %s", e) + return False + if r.returncode != 0: + log.warning( + "osmo login failed (rc=%d): %s", + r.returncode, (r.stderr or r.stdout).strip(), + ) + return False + log.info("osmo login succeeded (url=%s, token_file=%s)", + self.osmo_url, self.token_file) + return True + + def _osmo(self, args: list[str], timeout: int, + relogin: bool = True) -> subprocess.CompletedProcess: + """Run an osmo CLI command, re-logging-in once on an auth failure.""" + r = self._run_osmo(args, timeout=timeout) + if r.returncode != 0 and relogin and _looks_like_auth_error(r): + log.info("osmo command hit an auth error; re-logging in and retrying") + if self._login(): + r = self._run_osmo(args, timeout=timeout) + return r + + def submit_workflow(self, workflow_file: str) -> tuple[str, str]: + """Submit a workflow. Returns (workflow_id, live_name). + + AirLab OSMO 6.2 submit JSON is typically only {name, overview, logs} + (no uuid), and the service may append a numeric suffix to the name + (e.g. ``...-1``). We immediately query to resolve uuid + live name so + state/reap stay consistent with ``workflow list`` (``workflow_uuid``). + """ + args = ["workflow", "submit", workflow_file, "--pool", self.pool, + "--priority", self.priority, "--format-type", "json"] + r = self._osmo(args, timeout=self.submit_timeout) + if r.returncode != 0: + raise RuntimeError( + f"osmo workflow submit failed (rc={r.returncode}): " + f"{(r.stderr or r.stdout).strip()}" + ) + submitted_name = _extract_workflow_id(r.stdout) or _extract_workflow_id(r.stderr) + if not submitted_name: + raise RuntimeError( + "could not parse workflow id/name from submit output: " + f"{(r.stdout or '').strip()[:500]}" + ) + live_name, uuid = submitted_name, None + q = self._osmo( + ["workflow", "query", submitted_name, "--format-type", "json"], + timeout=60, + ) + if q.returncode == 0: + data = _loads_or_none(q.stdout) or _loads_or_none(q.stderr) + if isinstance(data, dict): + if isinstance(data.get("name"), str) and data["name"]: + live_name = data["name"] + for k in ("uuid", "workflow_uuid", "workflow_id", "id"): + v = data.get(k) + if isinstance(v, str) and v: + uuid = v + break + return (uuid or live_name), live_name + + def query_status(self, workflow_id: str) -> str | None: + r = self._osmo( + ["workflow", "query", workflow_id, "--format-type", "json"], + timeout=60, + ) + if r.returncode != 0: + log.debug("osmo workflow query %s failed: %s", + workflow_id, (r.stderr or r.stdout).strip()) + return None + return _extract_status(r.stdout) or _extract_status(r.stderr) + + def cancel_workflow(self, workflow_id: str) -> None: + r = self._osmo( + ["workflow", "cancel", workflow_id, "--force", + "--message", "orchestrator reap", "--format-type", "json"], + timeout=60, + ) + if r.returncode != 0: + log.warning("osmo workflow cancel %s failed (rc=%d): %s", + workflow_id, r.returncode, (r.stderr or r.stdout).strip()) + + def list_runner_workflows(self) -> list[dict]: + """Active workflows (RUNNING/PENDING/WAITING) named with our prefix.""" + r = self._osmo( + ["workflow", "list", "--name", self.workflow_prefix, + "--pool", self.pool, "--count", "100", + "--status", *_ACTIVE_STATUSES, "--format-type", "json"], + timeout=60, + ) + if r.returncode != 0: + log.warning("osmo workflow list failed: %s", + (r.stderr or r.stdout).strip()) + return [] + return _extract_workflow_list(r.stdout) + + # ── workflow rendering ─────────────────────────────────────────────────── + + def render_workflow(self, workflow_name: str, encoded_jit_config: str) -> str: + with open(self.template_path) as f: + tmpl = Template(f.read()) + return tmpl.render( + workflow_name=workflow_name, + runner_image=self.runner_image, + cpu=self.cpu, + gpu=self.gpu, + memory=self.memory, + storage=self.storage, + platform=self.platform, + privileged="true" if self.privileged else "false", + host_network="true" if self.host_network else "false", + encoded_jit_config=encoded_jit_config, + runner_labels=self.runner_labels, + ) + + def _write_temp_workflow(self, name: str, content: str) -> str: + fd, path = tempfile.mkstemp(prefix=f"{name}-", suffix=".yaml") + with os.fdopen(fd, "w") as f: + f.write(content) + return path + + # ── loops ──────────────────────────────────────────────────────────────── + def spawn_once(self) -> None: state = load_state(self.state_path) active = len(state["jobs"]) - if active >= self.effective_max_concurrent: + if active >= self.max_concurrent: return try: queued = find_queued_jobs(self.repo, self.runner_labels, self.pat) - except Exception as e: + except Exception as e: # noqa: BLE001 log.warning("find_queued_jobs failed: %s", e) return - # Pre-flight capacity check via placement API. Every queued job uses - # the same flavor, so we check once per iteration. When OpenStack is - # out of GPUs / vCPU / RAM we defer the whole iteration — better than - # burning JIT tokens on creates that Nova will flip to ERROR. The - # next iteration retries automatically. - if queued: - ok, reason = check_flavor_capacity( - self.conn, self.config["flavor_name"] - ) - if not ok: - log.warning( - "deferring spawn — OpenStack capacity unavailable: %s. " - "Will retry in %ds.", - reason, self.spawn_interval, - ) - return - elif reason: - # Soft-skip path: placement check couldn't run (e.g. older - # Nova). Surface why so it's debuggable, then proceed. - log.debug("placement pre-flight: %s", reason) - for job in queued: - if active >= self.effective_max_concurrent: + if active >= self.max_concurrent: break job_id = job["job_id"] if job_id in state["jobs"]: continue - # Pre-check FIP availability before minting a JIT token so we - # don't burn one when there's nowhere to attach the worker. - reserved_fip = None - if self.floating_ips: - reserved_fip = find_free_floating_ip(self.conn, self.floating_ips) - if reserved_fip is None: - log.warning( - "no free floating IP in pool (%d configured); " - "deferring spawns until one frees up", - len(self.floating_ips), - ) - break - ts = int(time.time()) - runner_name = f"ephemeral-{job_id}-{ts}" - server_id: str | None = None + # OSMO workflow name doubles as the JIT runner registration name. + workflow_name = f"{self.workflow_prefix}{job_id}-{ts}" + tmp_path: str | None = None try: jit = mint_jit_config( - self.repo, runner_name, self.runner_labels, self.pat + self.repo, workflow_name, self.runner_labels, self.pat ) - user_data = render_cloud_init( - self.template_path, jit, self.runner_version - ) - server_id = spawn_server( - self.conn, self.config, runner_name, job_id, user_data - ) - # Don't move on until Nova reports ACTIVE. If it transitions - # to ERROR, this raises with the Nova fault details so the - # operator can see *why* the spawn failed (quota, scheduling, - # block-device-mapping, networking, etc.). - wait_for_server_active( - self.conn, - server_id, - timeout_s=int(self.config.get("server_active_timeout_s", 300)), - ) - except Exception as e: - # Tag capacity-related Nova faults so log-grepping for - # "capacity unavailable" finds both the pre-flight defer and - # the post-create fallback (e.g. PCI passthrough that - # placement doesn't track). - msg = str(e).lower() - capacity_markers = ( - "no valid host", - "insufficient", - "quotaexceeded", - "out of resource", - "no host can satisfy", - "no allocation candidates", - ) - if any(m in msg for m in capacity_markers): - log.warning( - "spawn failed for job %s — OpenStack capacity " - "unavailable (post-create): %s. Will retry in %ds.", - job_id, e, self.spawn_interval, - ) - else: - log.exception("spawn failed for job %s: %s", job_id, e) - if server_id: - log.warning( - "deleting failed server %s to release its volume / FIP", - server_id, - ) - delete_server(self.conn, server_id) + workflow_yaml = self.render_workflow(workflow_name, jit) + tmp_path = self._write_temp_workflow(workflow_name, workflow_yaml) + workflow_id, live_name = self.submit_workflow(tmp_path) + except Exception as e: # noqa: BLE001 + log.exception("submit failed for job %s: %s", job_id, e) continue - - floating_ip_addr: str | None = None - if reserved_fip is not None: - try: - floating_ip_addr = attach_floating_ip( - self.conn, server_id, reserved_fip - ) - log.info( - "attached floating IP %s to server %s (job %s)", - floating_ip_addr, server_id, job_id, - ) - except Exception as e: - log.exception( - "FIP attach failed for server %s; deleting to avoid " - "leaking a worker without external access: %s", - server_id, e, - ) - delete_server(self.conn, server_id) - continue + finally: + if tmp_path: + try: + os.remove(tmp_path) + except OSError: + pass state["jobs"][job_id] = { "run_id": job["run_id"], - "server_id": server_id, - "runner_name": runner_name, - "spawned_at": now_utc_iso(), + "workflow_id": workflow_id, + "workflow_name": live_name, + "runner_name": workflow_name, + "submitted_at": now_utc_iso(), "name": job["name"], - "floating_ip": floating_ip_addr, } save_state(self.state_path, state) active += 1 log.info( - "spawned server %s for job %s (%s)", server_id, job_id, job["name"] + "submitted workflow %s for job %s (%s)", + workflow_id, job_id, job["name"], ) def reap_once(self) -> None: state = load_state(self.state_path) now = datetime.now(timezone.utc) - # 1. Delete servers for completed jobs. + # 1. Cancel workflows for completed / purged jobs. for job_id in list(state["jobs"].keys()): entry = state["jobs"][job_id] + wid = entry["workflow_id"] try: job = get_job_status(self.repo, job_id, self.pat) - except Exception as e: + except Exception as e: # noqa: BLE001 log.warning("get_job_status(%s) failed: %s", job_id, e) continue + if job is None or job.get("status") == "completed": - log.info("reaping server %s (job %s done)", entry["server_id"], job_id) - delete_server(self.conn, entry["server_id"]) + # The runner usually exits on its own (task self-completes and + # the pod is torn down); only cancel if it's somehow still live. + status = self.query_status(wid) + if not _is_terminal(status): + log.info("reaping workflow %s (job %s done, wf status=%s)", + wid, job_id, status) + self.cancel_workflow(wid) + else: + log.info("workflow %s already terminal (%s) for job %s", + wid, status, job_id) del state["jobs"][job_id] continue # 2. Force-reap stragglers older than max_job_minutes. - spawned = parse_iso(entry["spawned_at"]) - age_min = (now - spawned).total_seconds() / 60.0 + age_min = (now - parse_iso(entry["submitted_at"])).total_seconds() / 60.0 if age_min > self.max_job_minutes: log.warning( - "force-reaping server %s (job %s age %.1fm > %dm)", - entry["server_id"], job_id, age_min, self.max_job_minutes, + "force-reaping workflow %s (job %s age %.1fm > %dm)", + wid, job_id, age_min, self.max_job_minutes, ) - delete_server(self.conn, entry["server_id"]) + self.cancel_workflow(wid) del state["jobs"][job_id] save_state(self.state_path, state) - # 3. Orphan sweep: any server we own that isn't in state and isn't - # in the brief just-spawned window. Catches state.json wipes and - # crashes between spawn and save_state. + # 3. Orphan sweep: our-named workflows still active but absent from + # state (catches state.json wipes and crashes between submit and + # save_state). Skip very fresh ones so we don't race our own submit. try: - owned = list_owned_servers(self.conn) - except Exception as e: - log.warning("list_owned_servers failed: %s", e) + listed = self.list_runner_workflows() + except Exception as e: # noqa: BLE001 + log.warning("list_runner_workflows failed: %s", e) return - tracked_ids = {e["server_id"] for e in state["jobs"].values()} - for s in owned: - if s.id in tracked_ids: + tracked_ids = {e["workflow_id"] for e in state["jobs"].values()} + tracked_names = {e["workflow_name"] for e in state["jobs"].values()} + for wf in listed: + wid, wname = wf.get("id"), wf.get("name") + if (wid and wid in tracked_ids) or (wname and wname in tracked_names): continue - created = getattr(s, "created_at", None) - if created: - try: - age_min = (now - parse_iso(created.replace("Z", "+00:00"))).total_seconds() / 60.0 - except Exception: - age_min = self.max_job_minutes + 1 - else: - age_min = self.max_job_minutes + 1 - # Only reap orphans that have lived past one spawn interval - # (to avoid racing our own freshly-created server). - if age_min < 2: + age = _name_age_minutes(wname) + if age is not None and age < 2: continue - log.warning( - "orphan-reaping server %s (not in state, age %.1fm)", s.id, age_min - ) - delete_server(self.conn, s.id) + target = wid or wname + if not target: + continue + log.warning("orphan-reaping workflow %s (not in state)", target) + self.cancel_workflow(target) def run(self) -> None: log.info( - "orchestrator started: repo=%s labels=%s max_concurrent=%d " - "(effective=%d, floating_ip_pool=%d)", - self.repo, self.runner_labels, self.max_concurrent, - self.effective_max_concurrent, len(self.floating_ips), + "orchestrator started (OSMO backend): repo=%s labels=%s pool=%s " + "platform=%s max_concurrent=%d", + self.repo, self.runner_labels, self.pool, + self.platform or "(pool default)", self.max_concurrent, ) last_spawn = 0.0 last_reap = 0.0 @@ -656,13 +619,13 @@ def run(self) -> None: if now - last_spawn >= self.spawn_interval: try: self.spawn_once() - except Exception: + except Exception: # noqa: BLE001 log.exception("spawn loop iteration failed") last_spawn = now if now - last_reap >= self.reap_interval: try: self.reap_once() - except Exception: + except Exception: # noqa: BLE001 log.exception("reap loop iteration failed") last_reap = now self.stop_evt.wait(timeout=1.0) diff --git a/.github/orchestrator/requirements.txt b/.github/orchestrator/requirements.txt index b69702b59..f710f7167 100644 --- a/.github/orchestrator/requirements.txt +++ b/.github/orchestrator/requirements.txt @@ -1,4 +1,3 @@ -openstacksdk>=3.0,<5 requests>=2.31 PyYAML>=6.0 Jinja2>=3.1 diff --git a/.github/orchestrator/runner-entrypoint.sh b/.github/orchestrator/runner-entrypoint.sh new file mode 100644 index 000000000..3026dbdf4 --- /dev/null +++ b/.github/orchestrator/runner-entrypoint.sh @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# Entry point for the AirStack CI ephemeral-runner container (an OSMO task). +# +# Starts an inner Docker daemon — the AirStack test harness runs `airstack up` +# (docker compose) inside this container — waits for it, then runs exactly ONE +# ephemeral GitHub Actions job via the single-use JIT config. When run.sh exits, +# the OSMO task completes and the pod is destroyed: one job per pod, same as the +# old OpenStack VM. +# +# Requires a privileged pod (dockerd) scheduled on a GPU platform; the NVIDIA +# container toolkit (baked into the image) lets the inner dockerd pass the node +# GPU through to the compose containers. +set -euo pipefail + +: "${ENCODED_JIT_CONFIG:?ENCODED_JIT_CONFIG must be set by the workflow}" + +log() { echo "[runner-entrypoint] $*"; } + +# --------------------------------------------------------------------------- +# Docker storage backend +# +# The pod's root filesystem is overlayfs, and Linux refuses to use a directory +# on overlayfs as an overlay `upperdir` (EINVAL). A dockerd whose data-root sits +# on the pod rootfs still *pulls* images fine — containerd unpacks layers with +# plain writes — but every build step that needs a real mount dies with: +# +# failed to solve: ... mount source: "overlay", +# target: "/var/lib/docker/buildkit/containerd-overlayfs/cachemounts/...", +# err: invalid argument +# +# which is what took out all of build_docker / build_packages. So put the +# data-root somewhere overlay actually works, and verify it by performing a real +# overlay mount rather than trusting the filesystem type. +# --------------------------------------------------------------------------- + +DOCKER_DATA_ROOT=/var/lib/docker +LOOP_IMG=/docker-data.img +# Headroom left for everything that is not the Docker data-root (the runner's +# _work checkout, logs, the loop image's own metadata). +LOOP_HEADROOM_MB=20480 +MIN_BACKING_MB=51200 # Below ~50 GiB the sim images can't fit regardless. + +# True if an overlay mount whose upperdir lives under $1 can actually be made. +overlay_upperdir_works() { + local base=$1 probe rc=1 + probe=$(mktemp -d "$base/.overlay-probe.XXXXXX" 2>/dev/null) || return 1 + mkdir -p "$probe"/{lower,upper,work,merged} + if mount -t overlay overlay \ + -o "lowerdir=$probe/lower,upperdir=$probe/upper,workdir=$probe/work" \ + "$probe/merged" 2>/dev/null; then + umount "$probe/merged" && rc=0 + fi + rm -rf "$probe" + return $rc +} + +free_mb() { df -Pm "$1" | awk 'NR==2 {print $4}'; } + +# Preferred: a loopback ext4 image mounted at the data-root. Self-contained +# (no dependency on how the pool exposes storage), gives real overlay2, and dies +# with the pod. The image file is sparse, so it only consumes what Docker writes. +setup_loopback() { + local size_mb=${DOCKER_LOOP_SIZE_MB:-} + if [[ -z "$size_mb" ]]; then + size_mb=$(( $(free_mb /) - LOOP_HEADROOM_MB )) + fi + if (( size_mb < MIN_BACKING_MB )); then + log "loopback: only ${size_mb}MB usable, need ${MIN_BACKING_MB}MB — skipping" + return 1 + fi + + # /dev/loop-control only exists once the loop module is loaded on the node. + [[ -e /dev/loop-control ]] || modprobe loop 2>/dev/null || true + if [[ ! -e /dev/loop-control ]]; then + log "loopback: no /dev/loop-control — skipping" + return 1 + fi + + log "loopback: creating ${size_mb}MB ext4 image at $LOOP_IMG" + truncate -s "${size_mb}M" "$LOOP_IMG" || return 1 + # No journal and lazy inode-table init: this filesystem never outlives the + # pod, so durability buys nothing and mkfs stays fast. + mkfs.ext4 -q -F -m 0 -O ^has_journal -E lazy_itable_init=1 "$LOOP_IMG" || return 1 + + mkdir -p "$DOCKER_DATA_ROOT" + mount -o loop "$LOOP_IMG" "$DOCKER_DATA_ROOT" || return 1 + + if ! overlay_upperdir_works "$DOCKER_DATA_ROOT"; then + log "loopback: mounted but overlay still rejected — unwinding" + umount "$DOCKER_DATA_ROOT" || true + rm -f "$LOOP_IMG" + return 1 + fi + return 0 +} + +# Fallback: a real filesystem already mounted into the pod. Kubernetes emptyDir, +# hostPath and PVC volumes are backed by the node disk rather than the overlay +# rootfs, so overlay works there. +setup_real_fs() { + local best="" best_free=0 mnt fstype opts avail + while read -r _ mnt fstype opts _; do + case "$fstype" in ext2|ext3|ext4|xfs|btrfs) ;; *) continue ;; esac + [[ -d "$mnt" && -w "$mnt" ]] || continue + [[ ",$opts," == *",ro,"* ]] && continue + # OSMO's ctrl sidecar owns these: /osmo/data/output is uploaded as job + # artifacts and the socket dir is its IPC channel. + case "$mnt" in /osmo/data/output*|/osmo/data/socket*) continue ;; esac + avail=$(free_mb "$mnt") + if (( avail > best_free )); then best_free=$avail; best=$mnt; fi + done < /proc/mounts + + if [[ -z "$best" ]] || (( best_free < MIN_BACKING_MB )); then + log "real-fs: no mounted filesystem with >=${MIN_BACKING_MB}MB free — skipping" + return 1 + fi + + DOCKER_DATA_ROOT="$best/airstack-docker-data" + mkdir -p "$DOCKER_DATA_ROOT" + if ! overlay_upperdir_works "$DOCKER_DATA_ROOT"; then + log "real-fs: overlay rejected under $best — skipping" + DOCKER_DATA_ROOT=/var/lib/docker + return 1 + fi + log "real-fs: using $DOCKER_DATA_ROOT (${best_free}MB free)" + return 0 +} + +mkdir -p /etc/docker +if setup_loopback; then + log "storage: overlay2 on a loopback ext4 image" + printf '{"data-root": "%s"}\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json +elif setup_real_fs; then + log "storage: overlay2 on $DOCKER_DATA_ROOT" + printf '{"data-root": "%s"}\n' "$DOCKER_DATA_ROOT" > /etc/docker/daemon.json +elif [[ -e /dev/fuse ]] && command -v fuse-overlayfs >/dev/null 2>&1; then + # fuse-overlayfs stacks on overlayfs where the kernel driver won't. Slower + # than overlay2 but nowhere near as bad as vfs. `storage-driver` only applies + # to the classic image store, so the containerd snapshotter has to go. + log "storage: fuse-overlayfs (no overlay-capable filesystem found)" + cat > /etc/docker/daemon.json <<'JSON' +{"storage-driver": "fuse-overlayfs", "features": {"containerd-snapshotter": false}} +JSON +else + # Always works, but copies the whole filesystem per layer. The sim images are + # large enough that this will likely exhaust the pod's storage request. + log "WARN: storage: falling back to vfs — builds will be slow and may run out of disk" + cat > /etc/docker/daemon.json <<'JSON' +{"storage-driver": "vfs", "features": {"containerd-snapshotter": false}} +JSON +fi + +# Start dockerd in the background (needs privileged). +dockerd >/var/log/dockerd.log 2>&1 & + +# Wait for the daemon to accept connections (~60s budget). +for _ in $(seq 1 60); do + if docker info >/dev/null 2>&1; then + break + fi + sleep 1 +done +if ! docker info >/dev/null 2>&1; then + echo "ERROR: dockerd did not become ready" >&2 + cat /var/log/dockerd.log >&2 || true + exit 1 +fi + +log "storage driver: $(docker info --format '{{.Driver}}' 2>/dev/null || echo unknown)" \ + "data-root: $(docker info --format '{{.DockerRootDir}}' 2>/dev/null || echo unknown)" + +# Non-fatal GPU sanity check — surfaces GPU/privileged/toolkit misconfig early. +nvidia-smi || echo "WARN: nvidia-smi unavailable (check GPU + privileged + toolkit)" + +cd /home/runner/actions-runner +# The JIT config makes this runner single-use + ephemeral; run.sh returns after +# one job, which completes the task and lets OSMO reap the pod. +exec ./run.sh --jitconfig "${ENCODED_JIT_CONFIG}" diff --git a/.github/orchestrator/runner-workflow.yaml.j2 b/.github/orchestrator/runner-workflow.yaml.j2 new file mode 100644 index 000000000..124210ddb --- /dev/null +++ b/.github/orchestrator/runner-workflow.yaml.j2 @@ -0,0 +1,48 @@ +# OSMO workflow rendered per-job by orchestrator.py (replaces the old +# cloud-init.yaml.j2). One workflow == one ephemeral GitHub Actions runner == +# one CI job. Jinja variables injected by the orchestrator: +# +# workflow_name unique name (gha-runner--); also the +# JIT runner registration name +# runner_image prebaked image (docker-ce + compose + nvidia-container- +# toolkit + GH Actions runner) — see runner.Dockerfile +# cpu / gpu / memory / storage resource request for the runner task +# platform optional OSMO platform to target within the pool +# (omitted -> the pool's default platform) +# privileged "true"/"false"; MUST be "true" because the AirStack +# test harness runs `airstack up` (docker compose) inside +# the pod, which needs an inner Docker daemon. Requires a +# platform with "Privileged Mode Allowed" (ask your OSMO +# admin to enable it for the CI pool). +# host_network "true"/"false" +# encoded_jit_config single-use base64 GitHub JIT runner config +# +# The pool is passed by the orchestrator via `osmo workflow submit --pool`, so +# it is intentionally not hard-coded here. +# +# Lifecycle: the container starts dockerd, then runs exactly ONE ephemeral job +# via the JIT config. When run.sh exits, the task completes and OSMO tears the +# pod down — same "destroy after one job" behavior the OpenStack VM had. +workflow: + name: {{ workflow_name }} + resources: + runner: + cpu: {{ cpu }} + gpu: {{ gpu }} + memory: {{ memory }} + storage: {{ storage }} +{% if platform %} platform: {{ platform }} +{% endif %} + tasks: + - name: runner + image: {{ runner_image }} + resource: runner + privileged: {{ privileged }} +{% if host_network == "true" %} hostNetwork: true +{% endif %} + environment: + # Single-use + ephemeral: the runner exits after exactly one job. + ENCODED_JIT_CONFIG: "{{ encoded_jit_config }}" + # The runner refuses to run as root without this; the DinD image is root. + RUNNER_ALLOW_RUNASROOT: "1" + command: ["/usr/local/bin/run-ephemeral-runner.sh"] diff --git a/.github/orchestrator/runner.Dockerfile b/.github/orchestrator/runner.Dockerfile new file mode 100644 index 000000000..18bcf8000 --- /dev/null +++ b/.github/orchestrator/runner.Dockerfile @@ -0,0 +1,67 @@ +# Prebaked image for AirStack CI ephemeral runners on OSMO. +# +# This bakes in what the old cloud-init.yaml.j2 installed on the OpenStack VM +# (Docker CE + compose plugin, NVIDIA container toolkit, the GitHub Actions +# runner) so pod start is fast and the single-use JIT token can't expire during +# a slow apt/bootstrap. Build it and push to a registry your OSMO pool can pull: +# +# docker build -f runner.Dockerfile \ +# --build-arg RUNNER_VERSION=2.336.0 \ +# -t /airstack-ci-runner:2.336.0 . +# docker push /airstack-ci-runner:2.336.0 +# +# Then set `runner_image: /airstack-ci-runner:2.336.0` in config.yaml. +# Keep RUNNER_VERSION in sync with the actions/runner release you want. +# +# GPU-in-Docker-in-Docker: the OSMO task must run privileged (see +# `privileged: true` in runner-workflow.yaml.j2) on a platform with +# "Privileged Mode Allowed" + GPUs. The inner dockerd uses the NVIDIA container +# toolkit installed here to expose the node GPU to the `airstack up` containers. +# The image is linux/amd64 (x86_64 runner tarball); rebuild for arm64 if needed. +FROM ubuntu:24.04 + +ENV DEBIAN_FRONTEND=noninteractive + +# Docker CE (+ compose/buildx plugins), NVIDIA container toolkit, and the tools +# the AirStack test harness / GH runner need (git, jq, python venv, ...). +# e2fsprogs + fuse-overlayfs back the storage-backend selection in +# runner-entrypoint.sh: the pod rootfs is overlayfs, which the kernel rejects as +# an overlay upperdir, so dockerd's data-root has to live on a loopback ext4 +# image (mkfs.ext4) or, failing that, use the fuse-overlayfs driver. +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates curl gnupg jq git sudo iproute2 \ + e2fsprogs fuse-overlayfs kmod mount \ + python3 python3-venv python3-pip \ + && install -m 0755 -d /etc/apt/keyrings \ + && curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ + -o /etc/apt/keyrings/docker.asc \ + && chmod a+r /etc/apt/keyrings/docker.asc \ + && echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \ +https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \ + > /etc/apt/sources.list.d/docker.list \ + && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \ + | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \ + && curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \ + | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \ + > /etc/apt/sources.list.d/nvidia-container-toolkit.list \ + && apt-get update && apt-get install -y --no-install-recommends \ + docker-ce docker-ce-cli containerd.io \ + docker-buildx-plugin docker-compose-plugin \ + nvidia-container-toolkit \ + && nvidia-ctk runtime configure --runtime=docker \ + && rm -rf /var/lib/apt/lists/* + +# GitHub Actions runner (self-contained; version pinned at build time). +ARG RUNNER_VERSION=2.336.0 +RUN mkdir -p /home/runner/actions-runner \ + && cd /home/runner/actions-runner \ + && curl -fsSL -o runner.tar.gz \ + "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" \ + && tar xzf runner.tar.gz \ + && rm runner.tar.gz \ + && ./bin/installdependencies.sh + +COPY runner-entrypoint.sh /usr/local/bin/run-ephemeral-runner.sh +RUN chmod +x /usr/local/bin/run-ephemeral-runner.sh + +WORKDIR /home/runner/actions-runner diff --git a/.github/orchestrator/setup.sh b/.github/orchestrator/setup.sh index 4803b4a33..adb3c078f 100755 --- a/.github/orchestrator/setup.sh +++ b/.github/orchestrator/setup.sh @@ -1,13 +1,17 @@ #!/usr/bin/env bash -# One-time orchestrator-VM setup. Run as root on the airstack-ci-cd-orchestrator -# OpenStack instance after cloning the repo. +# One-time orchestrator-VM setup (OSMO backend). Run as root on the +# airstack-ci-cd-orchestrator instance after cloning the repo. # # Pre-reqs (do these *before* running this script): -# 1. ~/.config/openstack/clouds.yaml staged for the orchestrator user -# (application credential — see .github/orchestrator/README.md). -# 2. /tmp/github-pat exists with the GitHub PAT contents. +# 1. /tmp/github-pat exists with the GitHub PAT contents. +# 2. /tmp/osmo-token exists with the OSMO service-account access token +# (from `osmo token set` — see .github/orchestrator/README.md). Optional +# at setup time; you can stage it later before starting the service. # 3. This repo cloned somewhere readable (this script copies code out of # its containing directory). +# +# The orchestrator host is lightweight and needs NO GPU — it only polls GitHub +# and submits OSMO workflows. 1 vCPU / 2GB RAM / 20GB disk is plenty. set -euo pipefail @@ -30,18 +34,33 @@ fi echo "==> Installing system packages" apt-get update -apt-get install -y python3 python3-venv python3-pip +apt-get install -y python3 python3-venv python3-pip curl ca-certificates + +echo "==> Installing the OSMO CLI" +if command -v osmo >/dev/null 2>&1; then + echo " osmo already installed ($(command -v osmo)); skipping" +else + # Latest client. Pin to a release from https://github.com/NVIDIA/OSMO/releases + # if you need a specific version. + curl -fsSL https://raw.githubusercontent.com/NVIDIA/OSMO/refs/heads/main/install.sh | bash + command -v osmo >/dev/null 2>&1 \ + || echo "WARNING: osmo not on PATH after install — check the installer output" >&2 +fi echo "==> Creating directories" install -d -o "$USER_NAME" -g "$USER_NAME" -m 0750 "$INSTALL_DIR" install -d -o root -g "$USER_NAME" -m 0750 "$CONFIG_DIR" install -d -o "$USER_NAME" -g "$USER_NAME" -m 0750 "$STATE_DIR" +# XDG dirs for the osmo CLI login session (see the systemd unit's HOME/XDG env). +install -d -o "$USER_NAME" -g "$USER_NAME" -m 0700 "$STATE_DIR/.config" +install -d -o "$USER_NAME" -g "$USER_NAME" -m 0700 "$STATE_DIR/.cache" +install -d -o "$USER_NAME" -g "$USER_NAME" -m 0700 "$STATE_DIR/.state" echo "==> Copying orchestrator files to $INSTALL_DIR" install -o "$USER_NAME" -g "$USER_NAME" -m 0755 \ "$REPO_DIR/orchestrator.py" "$INSTALL_DIR/orchestrator.py" install -o "$USER_NAME" -g "$USER_NAME" -m 0644 \ - "$REPO_DIR/cloud-init.yaml.j2" "$INSTALL_DIR/cloud-init.yaml.j2" + "$REPO_DIR/runner-workflow.yaml.j2" "$INSTALL_DIR/runner-workflow.yaml.j2" echo "==> Building Python venv" sudo -u "$USER_NAME" python3 -m venv "$INSTALL_DIR/venv" @@ -63,11 +82,14 @@ fi install -o root -g "$USER_NAME" -m 0640 /tmp/github-pat "$CONFIG_DIR/github-pat" shred -u /tmp/github-pat -echo "==> Verifying clouds.yaml" -CLOUDS_YAML="/home/$USER_NAME/.config/openstack/clouds.yaml" -if [[ ! -f "$CLOUDS_YAML" ]]; then - echo "WARNING: $CLOUDS_YAML missing." >&2 - echo " Create it (application credential) before starting the service." >&2 +echo "==> Installing OSMO service-account token (from /tmp/osmo-token)" +if [[ -f /tmp/osmo-token ]]; then + install -o root -g "$USER_NAME" -m 0640 /tmp/osmo-token "$CONFIG_DIR/osmo-token" + shred -u /tmp/osmo-token +else + echo "WARNING: /tmp/osmo-token not found." >&2 + echo " Stage the OSMO service-account token before starting the service:" >&2 + echo " sudo install -o root -g $USER_NAME -m 0640 /tmp/osmo-token $CONFIG_DIR/osmo-token" >&2 fi echo "==> Installing systemd unit" @@ -78,7 +100,8 @@ systemctl daemon-reload echo echo "Setup complete. Next steps:" -echo " 1. Edit $CONFIG_DIR/config.yaml — fill flavor/network/keypair/security_group." -echo " 2. Verify $CLOUDS_YAML exists with the application credential." +echo " 1. Edit $CONFIG_DIR/config.yaml — set osmo_url, pool, platform," +echo " runner_image, and resources." +echo " 2. Ensure $CONFIG_DIR/osmo-token holds the OSMO service-account token." echo " 3. systemctl enable --now airstack-orchestrator.service" echo " 4. journalctl -u airstack-orchestrator.service -f" diff --git a/.github/workflows/docker-build.yml b/.github/workflows/docker-build.yml index add5f5953..403864781 100644 --- a/.github/workflows/docker-build.yml +++ b/.github/workflows/docker-build.yml @@ -12,6 +12,11 @@ on: required: false default: 'desktop,isaac-sim,ms-airsim' type: string + force_rebuild: + description: 'Force a full rebuild of every service (skip retag)' + required: false + default: false + type: boolean env: DEFAULT_PROFILES: 'desktop,isaac-sim,ms-airsim' @@ -21,6 +26,8 @@ jobs: runs-on: ubuntu-latest outputs: tag-changed: ${{ steps.check-changes.outputs.tag-changed }} + current-version: ${{ steps.check-changes.outputs.current-version }} + previous-version: ${{ steps.check-changes.outputs.previous-version }} steps: - name: Checkout code uses: actions/checkout@v4 @@ -33,20 +40,22 @@ jobs: run: | # Get the current VERSION value CURRENT_TAG=$(grep "^VERSION=" .env | cut -d '=' -f2- | tr -d '"' | tr -d "'") - + # Get the previous VERSION value git show HEAD~1:.env > .env.prev 2>/dev/null || echo "" > .env.prev PREVIOUS_TAG=$(grep "^VERSION=" .env.prev | cut -d '=' -f2- | tr -d '"' | tr -d "'" || echo "") - + echo "Current tag: $CURRENT_TAG" echo "Previous tag: $PREVIOUS_TAG" - + echo "current-version=$CURRENT_TAG" >> "$GITHUB_OUTPUT" + echo "previous-version=$PREVIOUS_TAG" >> "$GITHUB_OUTPUT" + if [ "$CURRENT_TAG" != "$PREVIOUS_TAG" ] && [ -n "$CURRENT_TAG" ]; then echo "VERSION has changed from '$PREVIOUS_TAG' to '$CURRENT_TAG'" - echo "tag-changed=true" >> $GITHUB_OUTPUT + echo "tag-changed=true" >> "$GITHUB_OUTPUT" else echo "VERSION has not changed" - echo "tag-changed=false" >> $GITHUB_OUTPUT + echo "tag-changed=false" >> "$GITHUB_OUTPUT" fi docker-build: @@ -78,7 +87,6 @@ jobs: - name: Verify .env file and extract tag run: | - # Ensure .env file exists and is readable if [ ! -f .env ]; then echo "Error: .env file not found" exit 1 @@ -89,68 +97,184 @@ jobs: # Some compose files expect this file to exist, even if it is empty. mkdir -p simulation/isaac-sim/docker : > simulation/isaac-sim/docker/omni_pass.env - - # Display the current VERSION for debugging + DOCKER_TAG=$(grep "^VERSION=" .env | cut -d '=' -f2- | tr -d '"' | tr -d "'") echo "Building with VERSION: $DOCKER_TAG" - + if [ -z "$DOCKER_TAG" ]; then echo "Error: VERSION is empty" exit 1 fi - - name: Run Docker Compose Build + - name: Resolve compose profiles and previous VERSION + id: prep run: | - # Load environment variables and run docker compose build - set -a # Export all variables + set -a source .env - set +a # Stop exporting - - # Always override COMPOSE_PROFILES for all trigger types + set +a + if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" else export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" fi - - docker compose build - - name: Run Docker Compose Push + PREVIOUS_VERSION="${{ needs.check-docker-tag-change.outputs.previous-version }}" + CURRENT_VERSION="${{ needs.check-docker-tag-change.outputs.current-version }}" + if [ -z "$CURRENT_VERSION" ]; then + CURRENT_VERSION=$(grep "^VERSION=" .env | cut -d '=' -f2- | tr -d '"' | tr -d "'") + fi + + FORCE_REBUILD=false + if [ "${{ github.event_name }}" == "workflow_dispatch" ] && [ "${{ github.event.inputs.force_rebuild }}" == "true" ]; then + FORCE_REBUILD=true + fi + + echo "COMPOSE_PROFILES=$COMPOSE_PROFILES" >> "$GITHUB_ENV" + echo "PREVIOUS_VERSION=$PREVIOUS_VERSION" >> "$GITHUB_ENV" + echo "CURRENT_VERSION=$CURRENT_VERSION" >> "$GITHUB_ENV" + echo "FORCE_REBUILD=$FORCE_REBUILD" >> "$GITHUB_ENV" + echo "profiles=$COMPOSE_PROFILES" >> "$GITHUB_OUTPUT" + echo "previous-version=$PREVIOUS_VERSION" >> "$GITHUB_OUTPUT" + echo "force-rebuild=$FORCE_REBUILD" >> "$GITHUB_OUTPUT" + + # Content-aware publish: retag previous versioned images when image inputs + # are unchanged; rebuild only services whose fingerprint differs (or when + # force_rebuild / unlabeled previous images force a cold build). + - name: Plan retag vs rebuild + id: plan run: | - # Load environment variables and run docker compose push - set -a # Export all variables + set -a source .env - set +a # Stop exporting - - # Always override COMPOSE_PROFILES for all trigger types - if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" - else - export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" + set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" + + FORCE_ARGS=() + if [ "${{ env.FORCE_REBUILD }}" = "true" ]; then + FORCE_ARGS+=(--force-rebuild) fi - - docker compose push - - name: Sign pushed images with Cosign (keyless) - env: - COSIGN_YES: "true" + python3 .github/workflows/scripts/docker_image_plan.py \ + --version "${{ env.CURRENT_VERSION }}" \ + --previous-version "${{ env.PREVIOUS_VERSION }}" \ + --profiles "${{ env.COMPOSE_PROFILES }}" \ + --plan-out docker-image-plan.json \ + --override-out docker-compose.fingerprint.yaml \ + "${FORCE_ARGS[@]}" + + echo "Plan summary:" + jq -r '.services | to_entries[] | " \(.key): \(.value.action) (\(.value.reason))"' docker-image-plan.json + + - name: Retag unchanged images + run: | + set -euo pipefail + RETAG_COUNT=$(jq '[.services[] | select(.action=="retag")] | length' docker-image-plan.json) + echo "Services to retag: $RETAG_COUNT" + if [ "$RETAG_COUNT" -eq 0 ]; then + echo "Nothing to retag." + exit 0 + fi + + jq -c '.services | to_entries[] | select(.value.action=="retag") | .value' docker-image-plan.json \ + | while IFS= read -r row; do + IMAGE=$(echo "$row" | jq -r '.image') + PREV=$(echo "$row" | jq -r '.previous_image') + CACHE=$(echo "$row" | jq -r '.cache_tag // empty') + echo "Retagging $PREV → $IMAGE" + CREATE_ARGS=(--tag "$IMAGE") + if [ -n "$CACHE" ] && [ "$CACHE" != "null" ]; then + echo " also → $CACHE" + CREATE_ARGS+=(--tag "$CACHE") + fi + docker buildx imagetools create "${CREATE_ARGS[@]}" "$PREV" + done + + # Build/push one service at a time so a single Dockerfile failure (e.g. + # isaac-sim PX4 apt) does not discard successful siblings before push. + - name: Build and push changed images + id: build_push run: | + set -uo pipefail set -a source .env set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" - if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" - else - export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" + mapfile -t BUILD_SERVICES < <(jq -r '.services | to_entries[] | select(.value.action=="build") | .key' docker-image-plan.json | sort) + if [ "${#BUILD_SERVICES[@]}" -eq 0 ]; then + echo "No services require a rebuild." + echo "built_services=" >> "$GITHUB_OUTPUT" + exit 0 fi - IMAGES=$(docker compose config --images | sort -u) - if [ -z "$IMAGES" ]; then - echo "No images resolved from compose config; nothing to sign." + echo "Building services sequentially: ${BUILD_SERVICES[*]}" + FAILED=() + BUILT=() + for SVC in "${BUILD_SERVICES[@]}"; do + echo "::group::Build $SVC" + if docker compose \ + -f docker-compose.yaml \ + -f docker-compose.fingerprint.yaml \ + build "$SVC"; then + echo "::endgroup::" + echo "::group::Push $SVC" + if docker compose \ + -f docker-compose.yaml \ + -f docker-compose.fingerprint.yaml \ + push "$SVC"; then + CACHE=$(jq -r --arg s "$SVC" '.services[$s].cache_tag // empty' docker-image-plan.json) + if [ -n "$CACHE" ]; then + echo "Pushing cache tag $CACHE" + docker push "$CACHE" || echo "::warning::Failed to push cache tag $CACHE" + fi + BUILT+=("$SVC") + else + echo "::error::Push failed for $SVC" + FAILED+=("$SVC") + fi + echo "::endgroup::" + else + echo "::endgroup::" + echo "::error::Build failed for $SVC" + FAILED+=("$SVC") + fi + done + + echo "built_services=${BUILT[*]}" >> "$GITHUB_OUTPUT" + if [ "${#FAILED[@]}" -gt 0 ]; then + echo "Failed services: ${FAILED[*]}" exit 1 fi + # Sign whatever was published even if a sibling service build failed. + - name: Sign published images with Cosign (keyless) + if: always() && steps.plan.outcome == 'success' + env: + COSIGN_YES: "true" + run: | + set -euo pipefail + set -a + source .env + set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" + + # Retagged services are always published; rebuilt ones only if push succeeded. + BUILT_CSV="${{ steps.build_push.outputs.built_services }}" + IMAGES=$( + { + jq -r '.services | to_entries[] | select(.value.action=="retag") | .value.image' docker-image-plan.json + if [ -n "$BUILT_CSV" ]; then + for SVC in $BUILT_CSV; do + jq -r --arg s "$SVC" '.services[$s].image // empty' docker-image-plan.json + done + fi + } | awk 'NF' | sort -u + ) + if [ -z "$IMAGES" ]; then + echo "No published images to sign." + exit 0 + fi + for IMG in $IMAGES; do DIGEST=$(docker buildx imagetools inspect "$IMG" --format '{{.Manifest.Digest}}') if [ -z "$DIGEST" ]; then @@ -164,18 +288,25 @@ jobs: done - name: Verify Cosign signatures + if: always() && steps.plan.outcome == 'success' run: | + set -euo pipefail set -a source .env set +a + export COMPOSE_PROFILES="${{ env.COMPOSE_PROFILES }}" - if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then - export COMPOSE_PROFILES="${{ github.event.inputs.compose_profiles || env.DEFAULT_PROFILES }}" - else - export COMPOSE_PROFILES="${{ env.DEFAULT_PROFILES }}" - fi - - IMAGES=$(docker compose config --images | sort -u) + BUILT_CSV="${{ steps.build_push.outputs.built_services }}" + IMAGES=$( + { + jq -r '.services | to_entries[] | select(.value.action=="retag") | .value.image' docker-image-plan.json + if [ -n "$BUILT_CSV" ]; then + for SVC in $BUILT_CSV; do + jq -r --arg s "$SVC" '.services[$s].image // empty' docker-image-plan.json + done + fi + } | awk 'NF' | sort -u + ) for IMG in $IMAGES; do DIGEST=$(docker buildx imagetools inspect "$IMG" --format '{{.Manifest.Digest}}') REPO="${IMG%:*}" @@ -187,13 +318,21 @@ jobs: > /dev/null done - - name: Optional - Run Docker Compose Up (uncomment if needed) + - name: Fail job if any service build/push failed + if: always() && steps.build_push.outcome == 'failure' run: | - # Uncomment the following lines if you also want to start the services - # set -a - # source .env - # set +a - # docker compose up -d + echo "One or more services failed to build or push (see Build and push changed images)." + exit 1 + + - name: Upload image plan artifact + if: always() + uses: actions/upload-artifact@v4 + with: + name: docker-image-plan-${{ github.run_id }} + path: | + docker-image-plan.json + docker-compose.fingerprint.yaml + if-no-files-found: ignore notify: needs: [check-docker-tag-change, docker-build] @@ -203,8 +342,8 @@ jobs: - name: Notify build and push result run: | if [ "${{ needs.docker-build.result }}" == "success" ]; then - echo "✅ Docker Compose build and push completed successfully" + echo "✅ Docker Compose build/retag and push completed successfully" else - echo "❌ Docker Compose build or push failed" + echo "❌ Docker Compose build/retag or push failed" exit 1 - fi \ No newline at end of file + fi diff --git a/.github/workflows/scripts/docker_image_plan.py b/.github/workflows/scripts/docker_image_plan.py new file mode 100755 index 000000000..dcaf1a7ae --- /dev/null +++ b/.github/workflows/scripts/docker_image_plan.py @@ -0,0 +1,544 @@ +#!/usr/bin/env python3 +"""Plan retag-vs-rebuild for AirStack docker-build.yml publishes. + +For each compose service with a ``build:`` section under the selected profiles, +compute a content fingerprint of its image inputs. If the previous versioned +image carries the same ``org.airstack.content-fingerprint`` label, the service +is marked ``retag``; otherwise ``build``. + +Also writes an ephemeral compose override that applies the fingerprint as a +build label on services that will be rebuilt. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import subprocess +import sys +from pathlib import Path +from typing import Any + +FINGERPRINT_LABEL = "org.airstack.content-fingerprint" + +# Explicit roots so broad compose ``context:`` dirs do not hash the whole tree. +# Keys are compose service names after ``docker compose config`` resolution. +SERVICE_FINGERPRINT_ROOTS: dict[str, list[str]] = { + "robot-desktop": [ + "robot/docker/Dockerfile.robot", + "robot/docker/docker-compose.yaml", + "robot/docker/robot-base-docker-compose.yaml", + "robot/docker/custom_rosdep.yaml", + "robot/docker/wait_for_px4.py", + "robot/docker/.bashrc", + "robot/docker/robot_name_map", + ], + "gcs": [ + "gcs/docker/Dockerfile.gcs", + "gcs/docker/docker-compose.yaml", + "gcs/docker/gcs-base-docker-compose.yaml", + "gcs/docker/.bashrc", + "gcs/docker/resources", + "gcs/docker/Foxglove", + ], + "isaac-sim": [ + "simulation/isaac-sim/docker/Dockerfile.isaac-ros", + "simulation/isaac-sim/docker/docker-compose.yaml", + "simulation/isaac-sim/docker/fastdds.xml", + "simulation/isaac-sim/docker/.bashrc", + "simulation/isaac-sim/docker/omniverse.toml", + ], + "ms-airsim": [ + "simulation/ms-airsim/docker/Dockerfile", + "simulation/ms-airsim/docker/docker-compose.yaml", + "simulation/ms-airsim/docker/entrypoint.sh", + ], +} + +# Extra roots when DOCKER_IMAGE_BUILD_MODE=prebuilt (workspace baked into image). +PREBUILT_EXTRA_ROOTS: dict[str, list[str]] = { + "robot-desktop": [ + "robot/ros_ws/src", + "common/ros_packages", + "common/fastdds.xml", + ], + "gcs": [ + "gcs/ros_ws", + "common/ros_packages", + ], +} + +# .env keys whose values affect image tags or layers (included in fingerprint). +ENV_FINGERPRINT_KEYS = ( + "DOCKER_IMAGE_BUILD_MODE", + "PROJECT_DOCKER_REGISTRY", + "PROJECT_NAME", + "CACHE_TAG", +) + + +def repo_root() -> Path: + return Path(__file__).resolve().parents[3] + + +def parse_env_value(raw: str) -> str: + """Parse a .env value, honoring quotes and stripping trailing comments.""" + raw = raw.strip() + if not raw: + return "" + if raw[0] in "\"'": + quote = raw[0] + end = raw.find(quote, 1) + if end != -1: + return raw[1:end] + return raw[1:] + # Unquoted: drop an inline ` # comment` (space-hash) or a leading `#`. + if " #" in raw: + raw = raw.split(" #", 1)[0].rstrip() + return raw.strip().strip('"').strip("'") + + +def load_dotenv(path: Path) -> dict[str, str]: + env: dict[str, str] = {} + if not path.is_file(): + return env + for line in path.read_text(encoding="utf-8").splitlines(): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, _, value = line.partition("=") + env[key.strip()] = parse_env_value(value) + return env + + +def run(cmd: list[str], *, cwd: Path, check: bool = True) -> subprocess.CompletedProcess[str]: + return subprocess.run( + cmd, + cwd=cwd, + check=check, + text=True, + capture_output=True, + ) + + +def compose_config(root: Path, profiles: str, env: dict[str, str]) -> dict[str, Any]: + cmd_env = os.environ.copy() + cmd_env.update(env) + cmd_env["COMPOSE_PROFILES"] = profiles + proc = subprocess.run( + ["docker", "compose", "-f", "docker-compose.yaml", "config", "--format", "json"], + cwd=root, + env=cmd_env, + text=True, + capture_output=True, + check=False, + ) + if proc.returncode != 0: + raise RuntimeError( + "docker compose config failed:\n" + f"stdout:\n{proc.stdout}\nstderr:\n{proc.stderr}" + ) + return json.loads(proc.stdout) + + +def git_ls_files(root: Path, pathspec: str) -> list[str]: + proc = run( + ["git", "ls-files", "-z", "--", pathspec], + cwd=root, + check=False, + ) + if proc.returncode != 0: + return [] + return [p for p in proc.stdout.split("\0") if p] + + +def collect_tracked_files(root: Path, roots: list[str]) -> list[str]: + files: set[str] = set() + for rel in roots: + path = root / rel + if path.is_file(): + files.add(rel) + continue + if path.is_dir(): + for tracked in git_ls_files(root, rel): + # Skip local secrets / generated pass files + if tracked.endswith("omni_pass.env"): + continue + if "/.dev/" in f"/{tracked}/" or tracked.endswith("/.dev"): + continue + files.add(tracked) + return sorted(files) + + +def file_sha256(path: Path) -> str: + h = hashlib.sha256() + with path.open("rb") as f: + for chunk in iter(lambda: f.read(1024 * 1024), b""): + h.update(chunk) + return h.hexdigest() + + +def resolve_dockerfile(root: Path, service_cfg: dict[str, Any]) -> Path | None: + build = service_cfg.get("build") or {} + if not isinstance(build, dict): + return None + dockerfile = build.get("dockerfile") + context = build.get("context") or "." + if not dockerfile: + return None + # compose config usually resolves dockerfile to an absolute path + df = Path(dockerfile) + if df.is_absolute(): + return df + return (Path(context) / df).resolve() if Path(context).is_absolute() else (root / context / df).resolve() + + +def find_dockerignore(dockerfile: Path, context: Path) -> Path | None: + for candidate in ( + context / ".dockerignore", + dockerfile.parent / ".dockerignore", + ): + if candidate.is_file(): + return candidate + return None + + +def previous_image_ref(image: str, version: str, previous_version: str) -> str | None: + if not previous_version or not version or version == previous_version: + return None + # Tags look like ...:v{VERSION}_suffix — replace only the version segment. + needle = f":v{version}_" + if needle not in image: + # Fallback: replace first occurrence of the bare version in the tag. + tag_part = image.rsplit(":", 1) + if len(tag_part) != 2 or version not in tag_part[1]: + return None + return f"{tag_part[0]}:{tag_part[1].replace(version, previous_version, 1)}" + return image.replace(needle, f":v{previous_version}_", 1) + + +def cache_tag_from_build(build: dict[str, Any], cache_tag: str) -> str | None: + tags = build.get("tags") or [] + pfx = f":{cache_tag}_" + for tag in tags: + if isinstance(tag, str) and pfx in tag: + return tag + return None + + +def inspect_fingerprint_label(image: str) -> str | None: + """Return the fingerprint label from a registry image, or None if unavailable.""" + proc = subprocess.run( + [ + "docker", + "buildx", + "imagetools", + "inspect", + image, + "--format", + "{{json .}}", + ], + text=True, + capture_output=True, + check=False, + ) + if proc.returncode != 0: + return None + try: + data = json.loads(proc.stdout) + except json.JSONDecodeError: + return None + + # buildx JSON shape varies by version; hunt for Labels in common places. + def walk(obj: Any) -> str | None: + if isinstance(obj, dict): + labels = obj.get("Labels") or obj.get("labels") + if isinstance(labels, dict) and FINGERPRINT_LABEL in labels: + return labels[FINGERPRINT_LABEL] + for v in obj.values(): + found = walk(v) + if found: + return found + elif isinstance(obj, list): + for item in obj: + found = walk(item) + if found: + return found + return None + + return walk(data) + + +def compute_fingerprint( + root: Path, + service_name: str, + service_cfg: dict[str, Any], + env: dict[str, str], +) -> str: + build = service_cfg.get("build") or {} + roots = list(SERVICE_FINGERPRINT_ROOTS.get(service_name, [])) + + dockerfile = resolve_dockerfile(root, service_cfg) + if dockerfile and dockerfile.is_file(): + try: + rel = str(dockerfile.relative_to(root)) + except ValueError: + rel = str(dockerfile) + if rel not in roots: + roots.insert(0, rel) + + mode = env.get("DOCKER_IMAGE_BUILD_MODE", "dev") + if mode == "prebuilt": + roots.extend(PREBUILT_EXTRA_ROOTS.get(service_name, [])) + + # If service has no map entry, fall back to dockerfile directory. + if not roots and dockerfile is not None: + try: + roots = [str(dockerfile.parent.relative_to(root))] + except ValueError: + roots = [] + + tracked = collect_tracked_files(root, roots) + + h = hashlib.sha256() + h.update(f"service:{service_name}\n".encode()) + h.update(f"DOCKER_IMAGE_BUILD_MODE:{mode}\n".encode()) + + for key in ENV_FINGERPRINT_KEYS: + h.update(f"env:{key}={env.get(key, '')}\n".encode()) + + args = build.get("args") or {} + if isinstance(args, dict): + for k in sorted(args): + h.update(f"arg:{k}={args[k]}\n".encode()) + + context = build.get("context") + if context: + h.update(f"context:{context}\n".encode()) + ctx_path = Path(context) if Path(context).is_absolute() else root / context + dockerignore = find_dockerignore(dockerfile, ctx_path) if dockerfile else None + if dockerignore and dockerignore.is_file(): + h.update(f"dockerignore:{dockerignore.name}\n".encode()) + h.update(dockerignore.read_bytes()) + h.update(b"\n") + + for rel in tracked: + path = root / rel + if not path.is_file(): + continue + h.update(f"file:{rel}\n".encode()) + h.update(file_sha256(path).encode()) + h.update(b"\n") + + return h.hexdigest() + + +def write_override(path: Path, build_services: dict[str, str]) -> None: + """Write compose override that sets build.labels fingerprint for rebuilds.""" + lines = [ + "# Generated by docker_image_plan.py — do not commit.", + "services:", + ] + if not build_services: + # Valid empty mapping; compose merge ignores it when nothing rebuilds. + lines[-1] = "services: {}" + else: + for name, fingerprint in sorted(build_services.items()): + lines.append(f" {name}:") + lines.append(" build:") + lines.append(" labels:") + lines.append(f" {FINGERPRINT_LABEL}: \"{fingerprint}\"") + path.write_text("\n".join(lines) + "\n", encoding="utf-8") + + +def build_plan( + root: Path, + *, + version: str, + previous_version: str, + profiles: str, + force_rebuild: bool, + env: dict[str, str], +) -> dict[str, Any]: + config = compose_config(root, profiles, env) + services_out: dict[str, Any] = {} + cache_tag = env.get("CACHE_TAG") or "cache" + + for name, cfg in (config.get("services") or {}).items(): + if not isinstance(cfg, dict): + continue + build = cfg.get("build") + if not isinstance(build, dict) or not build: + continue + + image = cfg.get("image") + if not isinstance(image, str) or not image: + continue + + fingerprint = compute_fingerprint(root, name, cfg, env) + prev_image = previous_image_ref(image, version, previous_version) + cache_image = cache_tag_from_build(build, cache_tag) + + action = "build" + reason = "force_rebuild" if force_rebuild else "default_build" + prev_fp = None + if force_rebuild: + action = "build" + reason = "force_rebuild" + elif not prev_image: + action = "build" + reason = "no_previous_image" + else: + prev_fp = inspect_fingerprint_label(prev_image) + if prev_fp is None: + action = "build" + reason = "previous_missing_or_unlabeled" + elif prev_fp == fingerprint: + action = "retag" + reason = "fingerprint_match" + else: + action = "build" + reason = "fingerprint_mismatch" + + services_out[name] = { + "image": image, + "previous_image": prev_image, + "cache_tag": cache_image, + "fingerprint": fingerprint, + "previous_fingerprint": prev_fp, + "action": action, + "reason": reason, + } + + return { + "previous_version": previous_version, + "version": version, + "profiles": profiles, + "force_rebuild": force_rebuild, + "label": FINGERPRINT_LABEL, + "services": services_out, + } + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--root", type=Path, default=None, help="Repo root (default: AirStack/)") + parser.add_argument("--version", default="", help="Current VERSION (default: from .env)") + parser.add_argument("--previous-version", default="", help="Previous VERSION for retag source") + parser.add_argument( + "--profiles", + default="", + help="Compose profiles (default: COMPOSE_PROFILES or desktop,isaac-sim,ms-airsim)", + ) + parser.add_argument( + "--force-rebuild", + action="store_true", + help="Mark every service as build", + ) + parser.add_argument( + "--plan-out", + type=Path, + default=Path("docker-image-plan.json"), + help="Where to write the plan JSON", + ) + parser.add_argument( + "--override-out", + type=Path, + default=Path("docker-compose.fingerprint.yaml"), + help="Compose override with build labels for rebuild services", + ) + parser.add_argument( + "--github-output", + type=Path, + default=None, + help="Optional path to append GITHUB_OUTPUT keys", + ) + args = parser.parse_args(argv) + + root = (args.root or repo_root()).resolve() + env = load_dotenv(root / ".env") + # Prefer process env overlays (CI exports .env via set -a). + for key in ( + "VERSION", + "DOCKER_IMAGE_BUILD_MODE", + "PROJECT_DOCKER_REGISTRY", + "PROJECT_NAME", + "CACHE_TAG", + "COMPOSE_PROFILES", + ): + if os.environ.get(key): + env[key] = os.environ[key] + + version = args.version or env.get("VERSION") or "" + if not version: + print("ERROR: VERSION is empty", file=sys.stderr) + return 1 + + previous_version = args.previous_version + profiles = ( + args.profiles + or os.environ.get("COMPOSE_PROFILES") + or env.get("COMPOSE_PROFILES") + or "desktop,isaac-sim,ms-airsim" + ) + + plan = build_plan( + root, + version=version, + previous_version=previous_version, + profiles=profiles, + force_rebuild=args.force_rebuild, + env=env, + ) + + build_services = { + name: svc["fingerprint"] + for name, svc in plan["services"].items() + if svc["action"] == "build" + } + retag_services = [name for name, svc in plan["services"].items() if svc["action"] == "retag"] + + args.plan_out = args.plan_out if args.plan_out.is_absolute() else root / args.plan_out + args.override_out = ( + args.override_out if args.override_out.is_absolute() else root / args.override_out + ) + args.plan_out.write_text(json.dumps(plan, indent=2, sort_keys=True) + "\n", encoding="utf-8") + write_override(args.override_out, build_services) + + print(f"Wrote plan → {args.plan_out}") + print(f"Wrote override → {args.override_out}") + for name, svc in sorted(plan["services"].items()): + print( + f" {name}: action={svc['action']} reason={svc['reason']} " + f"fp={svc['fingerprint'][:12]}…" + ) + + build_list = " ".join(sorted(build_services)) + retag_list = " ".join(sorted(retag_services)) + if args.github_output: + with args.github_output.open("a", encoding="utf-8") as fh: + fh.write(f"plan_path={args.plan_out}\n") + fh.write(f"override_path={args.override_out}\n") + fh.write(f"build_services={build_list}\n") + fh.write(f"retag_services={retag_list}\n") + fh.write(f"build_count={len(build_services)}\n") + fh.write(f"retag_count={len(retag_services)}\n") + else: + # Also support GITHUB_OUTPUT env when set by Actions. + gh_out = os.environ.get("GITHUB_OUTPUT") + if gh_out: + with open(gh_out, "a", encoding="utf-8") as fh: + fh.write(f"plan_path={args.plan_out}\n") + fh.write(f"override_path={args.override_out}\n") + fh.write(f"build_services={build_list}\n") + fh.write(f"retag_services={retag_list}\n") + fh.write(f"build_count={len(build_services)}\n") + fh.write(f"retag_count={len(retag_services)}\n") + + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.github/workflows/system-tests.yml b/.github/workflows/system-tests.yml index 3ddebda58..2818af58c 100644 --- a/.github/workflows/system-tests.yml +++ b/.github/workflows/system-tests.yml @@ -15,8 +15,8 @@ on: default: "liveliness or takeoff_hover_land" required: false sim: - description: "Sim targets, comma-separated: msairsim,isaacsim" - default: msairsim,isaacsim + description: "Sim targets, comma-separated: isaacsim,msairsim. Default isaacsim; pass msairsim to opt in." + default: isaacsim required: false num_robots: description: "Robot counts, comma-separated (e.g. 1,3)" @@ -152,16 +152,28 @@ jobs: print(f'::error::Could not parse pytest args from comment: {e}', file=sys.stderr) sys.exit(1) + # CI-only flag: do not forward to pytest. + no_image_build = False + stripped = [] + for a in args: + if a in ('--no-image-build', '--pull-only'): + no_image_build = True + else: + stripped.append(a) + args = stripped + # Pull out --sim and -m so the image-prep step can scope profiles # and decide whether to skip (build_docker tests rebuild themselves). # When --sim isn't given we mirror conftest's default so prep covers # whatever pytest will actually exercise. - sim = 'msairsim,isaacsim' + sim = 'isaacsim' + sim_explicit = False marks = '' marks_idx = -1 for i, a in enumerate(args): if a == '--sim' and i + 1 < len(args): sim = args[i + 1] + sim_explicit = True elif a == '-m' and i + 1 < len(args): marks = args[i + 1] marks_idx = i + 1 @@ -174,6 +186,24 @@ jobs: marks = f'build_packages or {marks}' args[marks_idx] = marks + # colcon tests do not need a sim image. Default --sim would otherwise + # bake isaac-sim before a 1s colcon test. Pull registry cache tags + # instead; never image-build. + marks_norm = marks.replace('"', '').replace("'", '').strip() + args_blob = ' '.join(args) + heavy = any(m in marks_norm for m in ( + 'liveliness', 'sensors', 'takeoff_hover_land', 'autonomy', 'build_docker', + )) + only_packages = marks_norm == 'build_packages' or ( + not heavy and any(s in args_blob for s in ( + 'test_build_packages', 'test_colcon_', + )) + ) + if only_packages: + no_image_build = True + if not sim_explicit: + sim = 'msairsim' + skip_prep = 'build_docker' in marks quoted = ' '.join(shlex.quote(a) for a in args) @@ -181,10 +211,12 @@ jobs: f.write(f'pytest_args={quoted}\n') f.write(f'sim={sim}\n') f.write(f'skip_image_prep={"true" if skip_prep else "false"}\n') + f.write(f'no_image_build={"true" if no_image_build else "false"}\n') print(f'Resolved pytest args: {quoted or "(none — pytest defaults)"}') print(f'Resolved sim profile: {sim}') print(f'Skip image prep: {skip_prep}') + print(f'No image build (pull/retag only): {no_image_build}') PYEOF # Reply on the PR thread so the commenter sees their /pytest was @@ -200,7 +232,10 @@ jobs: const args = ${{ toJSON(steps.parse.outputs.pytest_args) }}; const cmd = `pytest tests/ ${args}`.trim(); const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`; - const note = `Note: \`build_packages\` is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.`; + const pullOnly = '${{ steps.parse.outputs.no_image_build }}' === 'true'; + const note = pullOnly + ? `Note: pull-only image prep (no \`image-build\`). \`-m build_packages\` does not pull Isaac Sim. Add \`--no-image-build\` on other marks to skip rebuilds.` + : `Note: \`build_packages\` is automatically prepended whenever any marks are specified, to ensure code is built before launch tests run.`; await github.rest.issues.createComment({ owner: context.repo.owner, repo: context.repo.repo, @@ -268,6 +303,10 @@ jobs: # inline cache (build_docker tests get layer-reuse speedup) and pre-pulls # before `airstack up` (other tests skip the implicit rebuild). When # secrets are absent both steps are skipped and behavior is unchanged. + # + # Read-only on purpose: AIRSTACK_REGISTRY_CACHE_PUSH stays unset here so a + # PR can consume the floating cache tag but never republish it. Only + # docker-build.yml (main/develop) writes it. - name: Log in to internal Docker registry id: docker_login if: ${{ vars.DOCKER_REGISTRY_URL != '' && env.DOCKER_REGISTRY_PASSWORD != '' }} @@ -284,26 +323,30 @@ jobs: - name: Ensure airstack.sh is executable run: chmod +x airstack.sh + - name: Disable compose image builds + if: ${{ steps.parse.outputs.no_image_build == 'true' }} + run: echo "AIRSTACK_NO_IMAGE_BUILD=1" >> "$GITHUB_ENV" + # The ephemeral runner starts with no local images. `airstack_env` in # tests/conftest.py fails fast if compose images are missing, so prep # them here. Profile-gated services (ms-airsim, isaac-sim) are skipped # by compose unless their profile is active, so we mirror the fixture's - # profile selection from the parsed --sim. Pull-only by default; fall - # back to a full build only if the registry doesn't have everything - # (e.g. new branch with no published image yet). Skipped when the - # marks expression contains build_docker — those tests build per-service - # themselves. + # profile selection from the parsed --sim. Pull versioned tags, then + # retag floating cache_* tags onto the VERSION name (PR tags never + # exist). Fall back to image-build only when --no-image-build is off. + # Skipped when marks contain build_docker — those tests build themselves. - name: Ensure Docker images present if: ${{ steps.parse.outputs.skip_image_prep != 'true' }} env: AIRSTACK_ROOT: ${{ github.workspace }} SIM_INPUT: ${{ steps.parse.outputs.sim }} + NO_IMAGE_BUILD: ${{ steps.parse.outputs.no_image_build }} run: | profiles=desktop [[ ",$SIM_INPUT," == *,msairsim,* ]] && profiles="$profiles,ms-airsim" [[ ",$SIM_INPUT," == *,isaacsim,* ]] && profiles="$profiles,isaac-sim" export COMPOSE_PROFILES="$profiles" - echo "Pulling images for COMPOSE_PROFILES=$COMPOSE_PROFILES" + echo "Pulling images for COMPOSE_PROFILES=$COMPOSE_PROFILES (no_image_build=$NO_IMAGE_BUILD)" # Pull from registry; tolerate per-image failures so we can detect # what's still missing afterwards instead of aborting on the first @@ -311,6 +354,25 @@ jobs: # still surface on stderr. ./airstack.sh --progress=quiet image-pull --ignore-pull-failures || true + # VERSION tags miss on every PR. Seed from floating cache_* tags. + cache_tag="$(grep -E '^CACHE_TAG=' .env 2>/dev/null | cut -d= -f2 | tr -d '"' || true)" + cache_tag="${cache_tag:-cache}" + while IFS= read -r img; do + [[ -z "$img" ]] && continue + if docker image inspect "$img" --format '{{.Id}}' >/dev/null 2>&1; then + continue + fi + # Replace :v_ with :_ (PR versioned tags never exist) + cache_img="$(python3 -c "import re,sys; print(re.sub(r':v[^_]+_', f':{sys.argv[2]}_', sys.argv[1], count=1))" "$img" "$cache_tag")" + echo "Versioned tag missing; trying cache tag $cache_img" + if docker pull --quiet "$cache_img"; then + docker tag "$cache_img" "$img" + echo "Retagged $cache_img -> $img" + else + echo "Cache tag pull failed for $cache_img" + fi + done < <(docker compose -f docker-compose.yaml config --images) + missing=() while IFS= read -r img; do [[ -z "$img" ]] && continue @@ -320,11 +382,16 @@ jobs: done < <(docker compose -f docker-compose.yaml config --images) if (( ${#missing[@]} > 0 )); then - echo "Pull did not produce these images; falling back to build:" + echo "Images still missing after pull/retag:" printf ' - %s\n' "${missing[@]}" + if [[ "$NO_IMAGE_BUILD" == "true" ]]; then + echo "::error::Pull-only mode (--no-image-build or -m build_packages) will not image-build. Run /pytest -m build_docker once, or omit --no-image-build." + exit 1 + fi + echo "Falling back to image-build" ./airstack.sh --progress=quiet image-build else - echo "All required images present after pull — skipping build." + echo "All required images present after pull/retag — skipping build." fi - name: Run tests diff --git a/.gitignore b/.gitignore index d9dd6a07b..37066a0de 100644 --- a/.gitignore +++ b/.gitignore @@ -103,5 +103,10 @@ common/rayfronts/ # Docker build cache (root-owned subdirs cause permission warnings on `git add`) robot/docker/cache/ + +# Ephemeral outputs from docker_image_plan.py (docker-build.yml) +docker-image-plan.json +docker-compose.fingerprint.yaml + .DS_Store gcs/.DS_Store diff --git a/AGENTS.md b/AGENTS.md index 88d524483..76987f8eb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -43,7 +43,7 @@ AirStack/ ├── tests/ # System tests (pytest) + metrics reporting ├── .github/ │ ├── workflows/ # GitHub Actions CI (system-tests, docker-build, etc.) -│ └── orchestrator/ # OpenStack-backed ephemeral self-hosted runners +│ └── orchestrator/ # OSMO-backed ephemeral self-hosted runners └── .agents/skills/ # Detailed workflow guides for agents ``` @@ -289,16 +289,28 @@ GitHub Actions workflows live in [`.github/workflows/`](.github/workflows/): ### Ephemeral Runner Orchestrator -GPU-required jobs (`runs-on: [self-hosted, airstack-ephemeral]`) execute on **OpenStack VMs spawned per-job and destroyed on completion**. The orchestrator service code lives in [`.github/orchestrator/`](.github/orchestrator/): +GPU-required jobs (`runs-on: [self-hosted, airstack-ephemeral]`) execute on **ephemeral pods scheduled by [NVIDIA OSMO](https://nvidia.github.io/OSMO/) — one per job, destroyed on completion**. The GitHub side is unchanged from the old OpenStack backend (same labels, JIT tokens, fork guard); only the spawn target moved from "create a Nova VM" to "submit an OSMO workflow". The orchestrator service code lives in [`.github/orchestrator/`](.github/orchestrator/): -- [`orchestrator.py`](.github/orchestrator/orchestrator.py) — Python service: spawn loop polls GitHub for queued jobs matching configured runner labels, mints single-use JIT runner tokens, creates an OpenStack server with cloud-init bootstrap; reap loop deletes the server when the job completes (or after `max_job_minutes`) -- [`cloud-init.yaml.j2`](.github/orchestrator/cloud-init.yaml.j2) — bootstraps Docker + nvidia-container-toolkit + GH Actions runner on the worker, registers with the JIT token, runs one job, then `shutdown -h` -- [`config.example.yaml`](.github/orchestrator/config.example.yaml) — flavor / network / keypair / floating-IP pool / runner labels / repo +- [`orchestrator.py`](.github/orchestrator/orchestrator.py) — Python service: spawn loop polls GitHub for queued jobs matching configured runner labels, mints single-use JIT runner tokens, and submits one OSMO workflow per job (`osmo workflow submit`); reap loop cancels the workflow when the job completes (or after `max_job_minutes`), plus an orphan sweep via `osmo workflow list` +- [`runner-workflow.yaml.j2`](.github/orchestrator/runner-workflow.yaml.j2) + [`runner.Dockerfile`](.github/orchestrator/runner.Dockerfile) + [`runner-entrypoint.sh`](.github/orchestrator/runner-entrypoint.sh) — the per-job worker: a **privileged**, GPU-enabled OSMO task (prebaked image) that starts an inner Docker daemon (the tests run `airstack up` = docker compose), registers with the JIT token, runs one job, then exits so OSMO reaps the pod +- [`config.example.yaml`](.github/orchestrator/config.example.yaml) — osmo_url / pool / platform / runner_image / resources / runner labels / repo - [`airstack-orchestrator.service`](.github/orchestrator/airstack-orchestrator.service) + [`setup.sh`](.github/orchestrator/setup.sh) — systemd unit and one-time installer -**Why ephemeral:** clean Docker cache per run, no leaked containers, GitHub PAT and OpenStack credentials only on the orchestrator host (workers receive a single-use JIT token bound to one runner registration). State map at `/var/lib/airstack-orchestrator/state.json`; logs via `journalctl -u airstack-orchestrator.service -f`. +**Why ephemeral:** clean Docker cache per run, no leaked containers; the GitHub PAT and the OSMO service-account token live only on the orchestrator host (workers receive a single-use JIT token bound to one runner registration). CI authenticates to OSMO as a shared, non-personal [service account](https://nvidia.github.io/OSMO/main/deployment_guide/appendix/authentication/service_accounts.html) scoped to a dedicated CI GPU pool, so runs don't consume individuals' quotas. The CI pool's platform must have **"Privileged Mode Allowed"** enabled (docker-in-docker). State map at `/var/lib/airstack-orchestrator/state.json`; logs via `journalctl -u airstack-orchestrator.service -f`. -**Setup, debugging a failed job, and SSH-into-worker procedures:** [`.github/orchestrator/README.md`](.github/orchestrator/README.md) (also exposed as [`tests/ci-cd-orchestrator.md`](tests/ci-cd-orchestrator.md) symlink for the docs site). +**Nested DinD needs a non-overlayfs Docker data-root.** The OSMO pod's root filesystem is overlayfs, and Linux rejects a directory on overlayfs as an overlay `upperdir` (`EINVAL`). A dockerd storing data on the pod rootfs pulls images fine but fails every build step that needs a real mount, with errors that masquerade as `apt-get`/`WORKDIR` failures: + +``` +failed to solve: ... mount source: "overlay", target: ".../buildkit/containerd-overlayfs/cachemounts/...", err: invalid argument +``` + +[`runner-entrypoint.sh`](.github/orchestrator/runner-entrypoint.sh) picks a backend by attempting a real overlay mount, preferring a loopback ext4 image at `/var/lib/docker` (real `overlay2`), then a real filesystem already mounted in the pod, then `fuse-overlayfs`, then `vfs`. Landing on `vfs` means builds will be slow and probably run out of disk — check the `[runner-entrypoint] storage:` line in the job log first when Docker builds misbehave. Details: [orchestrator README → Nested DinD and overlayfs](.github/orchestrator/README.md). + +**Docker layer cache is a floating tag, not the versioned one.** Every compose service lists two `cache_from` entries: the versioned image (`airstack:v${VERSION}_`) and a floating one (`airstack:${CACHE_TAG:-cache}_`). Only the floating tag can ever hit on a PR — `check-version-increment` forces `VERSION` up on every PR, so the versioned tag it builds under has by definition never been pushed. Reading and writing are separate switches: `AIRSTACK_REGISTRY_CACHE=1` (set by `system-tests.yml`) pulls and builds with `BUILDKIT_INLINE_CACHE=1`, while `AIRSTACK_REGISTRY_CACHE_PUSH=1` (set only by `docker-build.yml` on main/develop) also publishes both tags. PR runs stay read-only so an unmerged branch can't poison the shared cache or publish an unreleased version. If you add a service with a `build:` section, give it both entries or its builds will always be cold. + +**Publish retags when image inputs are unchanged.** `docker-build.yml` runs [`.github/workflows/scripts/docker_image_plan.py`](.github/workflows/scripts/docker_image_plan.py) on VERSION bumps: each service gets a content fingerprint (`org.airstack.content-fingerprint`). If the previous versioned image already has that label, the job registry-retags (`imagetools create`) instead of rebuilding; only changed services rebuild (and refresh `cache_*`). Use `workflow_dispatch` with `force_rebuild=true` to rebuild everything. PR `build_docker` tests still perform real builds. + +**Setup, debugging a failed job, and exec-into-worker procedures:** [`.github/orchestrator/README.md`](.github/orchestrator/README.md) (also exposed as [`tests/ci-cd-orchestrator.md`](tests/ci-cd-orchestrator.md) symlink for the docs site). ## Documentation Requirements diff --git a/CHANGELOG.md b/CHANGELOG.md index 68f37e9b0..8c99c905e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,12 +19,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Ephemeral CI GPU runners spawn via NVIDIA OSMO (not OpenStack); `system-tests.yml` / `docker-build.yml` still use `airstack-ephemeral` +- Default system-test `--sim` is `isaacsim`; pass `--sim msairsim` to opt in to Microsoft AirSim +- `-m build_packages` CI runs pull `cache_*` images instead of baking sim images +- `docker-build.yml` retags unchanged images on VERSION bumps (content fingerprint) instead of always rebuilding; floating `cache_*` tags still seed PR layer cache - `robot-l4t` compose service knobs are now env-overridable (`AUTONOMY_ROLE`, `FCU_URL`, and the rosbag path via `BAG_STORAGE_PATH`); `FCU_URL` unquoted so the literal serial path reaches MAVROS - `zed-l4t` image: ZED SDK 4.2 → 5.2 with the coupled ROS deps (`zed_msgs` 5.2.1, `point_cloud_transport(_plugins)` 4.x, add `backward_ros`) - Unit tests are defined by `tests/colcon_unit_test_packages.yaml`: `conftest.py` collects each listed package's co-located `test/` dir under `--import-mode=importlib` and marks it `unit` (ament lint files are skipped and run under `colcon test`) ### Fixed +- Isaac Sim image: PX4 `ubuntu.sh` no longer fails dpkg configure on the NVIDIA base (`ca-certificates` / `software-properties-common`); use `--no-nuttx --no-sim-tools` like ms-airsim +- Robot image: pin `pytest<8.1` and disable `launch_testing` for colcon unit tests so ROS Jazzy's outdated pytest hook does not abort `colcon test` - Robot name resolution now honors a pre-set `ROBOT_NAME` (e.g. injected via docker compose) instead of always overriding it from the container/hostname mapping (`robot/docker/.bashrc`) - Robot name-map catch-all fallback now maps to `unknown_robot` (valid ROS namespace token) instead of `unknown-robot` (`default_robot_name_map.yaml`) - l4t robot image: replace dustynv's `/ros_entrypoint.sh` with a passthrough so its prebuilt source-ROS libs (older `fastcdr`) no longer shadow the apt Jazzy runtime and crash apt-built nodes like MAVROS diff --git a/airstack.sh b/airstack.sh index 1c57c47cb..d42c5b6e1 100755 --- a/airstack.sh +++ b/airstack.sh @@ -824,6 +824,46 @@ function ensure_robot_l4t_stack_base() { run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${_ga[@]}" build "${build_opts[@]}" robot-l4t-stack-base } +# `docker compose push` only publishes each service's `image:`. The floating +# cache tags are declared in `build.tags`, so they need an explicit push. Read +# them back out of the resolved config instead of reconstructing the names here, +# so this stays correct as services are added. +function push_cache_tags() { + local -n _ga="$1" + local -n _sc="$2" + + if ! command -v jq >/dev/null 2>&1; then + log_warn "jq not found; skipping cache-tag push (floating cache will go stale)" + return 0 + fi + + # Service names only — drop any flags that were passed through to the subcommand. + local services=() + for arg in "${_sc[@]}"; do + [[ "$arg" == -* ]] || services+=("$arg") + done + + local tags + tags=$(run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${_ga[@]}" config --format json 2>/dev/null \ + | jq -r --arg svcs "${services[*]}" --arg pfx ":${CACHE_TAG:-cache}_" ' + .services | to_entries[] + | select($svcs == "" or (($svcs | split(" ")) | index(.key))) + | (.value.build.tags // [])[] + | select(contains($pfx)) + ' 2>/dev/null | sort -u) + + if [[ -z "$tags" ]]; then + log_warn "No cache tags resolved from compose config; nothing to publish" + return 0 + fi + + while IFS= read -r tag; do + [[ -n "$tag" ]] || continue + log_info "Pushing cache tag $tag" + docker push "$tag" || log_warn "Failed to push cache tag $tag" + done <<< "$tags" +} + function cmd_up { check_docker @@ -857,7 +897,12 @@ function cmd_up { fi log_info "Starting services..." - run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" up "${subcmd_args[@]}" -d + local up_opts=() + if [[ "${AIRSTACK_NO_IMAGE_BUILD:-}" == "1" ]]; then + log_info "AIRSTACK_NO_IMAGE_BUILD=1 → compose up --no-build" + up_opts+=(--no-build) + fi + run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" up "${up_opts[@]}" "${subcmd_args[@]}" -d log_info "Services brought up successfully" } @@ -877,9 +922,16 @@ function cmd_image_build { # Registry-cache mode (CI / opt-in): pre-pull existing images to seed the # local cache, build with BUILDKIT_INLINE_CACHE=1 so the resulting image - # carries layer-cache metadata, and push so the next run benefits. The - # cache_from declarations in each component compose file make BuildKit - # actually reuse the pulled layers. No-op when the env var is unset. + # carries layer-cache metadata. The cache_from declarations in each + # component compose file make BuildKit actually reuse the pulled layers. + # No-op when the env var is unset. + # + # Reading and publishing the cache are separate switches. A PR bumps VERSION + # (check-version-increment enforces it), so the versioned cache_from entry is + # guaranteed to miss and the floating CACHE_TAG entry is what actually hits. + # PR runs must not write that floating tag: an unmerged branch would poison + # the shared cache and publish an unreleased VERSION. Only trusted branches + # set AIRSTACK_REGISTRY_CACHE_PUSH=1. if [[ "${AIRSTACK_REGISTRY_CACHE:-}" == "1" ]]; then log_info "AIRSTACK_REGISTRY_CACHE=1 → pulling for cache seed..." run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" pull --ignore-pull-failures "${subcmd_args[@]}" || \ @@ -888,9 +940,14 @@ function cmd_image_build { log_info "Building services with BUILDKIT_INLINE_CACHE=1..." run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" build --build-arg BUILDKIT_INLINE_CACHE=1 "${subcmd_args[@]}" - log_info "Pushing built images for next-run cache..." - run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" push --ignore-push-failures "${subcmd_args[@]}" || \ - log_warn "Post-build push encountered failures; future runs may not benefit from cache" + if [[ "${AIRSTACK_REGISTRY_CACHE_PUSH:-}" == "1" ]]; then + log_info "Pushing built images for next-run cache..." + run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" push --ignore-push-failures "${subcmd_args[@]}" || \ + log_warn "Post-build push encountered failures; future runs may not benefit from cache" + push_cache_tags global_args subcmd_args + else + log_info "AIRSTACK_REGISTRY_CACHE_PUSH is not 1 → cache is read-only for this run" + fi else log_info "Building services..." run_docker_compose -f "$PROJECT_ROOT/docker-compose.yaml" "${global_args[@]}" build "${subcmd_args[@]}" diff --git a/docs/development/intermediate/testing/ci_cd.md b/docs/development/intermediate/testing/ci_cd.md index a5afaf265..a53f5926d 100644 --- a/docs/development/intermediate/testing/ci_cd.md +++ b/docs/development/intermediate/testing/ci_cd.md @@ -1 +1,568 @@ -# CI/CD Pipeline \ No newline at end of file +# CI/CD Pipeline on OSMO + +AirStack's continuous integration runs the **full drone stack** — simulator, +robot autonomy, and GCS — on a GPU for every change. Because that needs a +GPU, a Docker daemon, and a clean filesystem, jobs cannot run on GitHub's +hosted runners and should not run on a shared always-on machine. Instead, a +small orchestrator service watches the GitHub Actions queue and submits one +**ephemeral [NVIDIA OSMO](https://nvidia.github.io/OSMO/) pod per job**. The pod +registers as a single-use GitHub Actions runner, executes exactly one job, and +is destroyed. + +This page documents the whole system: the architecture, the job lifecycle, +what each test suite actually catches, how to trigger and read a run, and how +to fit CI into your day-to-day development loop. + +!!! note "Related pages" + - [`tests/README.md`](../../../../tests/README.md) — the test suite reference: marks, fixtures, metrics, CLI flags. + - [CI/CD Orchestrator](../../../../tests/ci-cd-orchestrator.md) — the lab-admin runbook: pool prerequisites, credential staging, `setup.sh`, rotation, break-glass debugging. + - [AirStack on OSMO](../../../tutorials/airstack_on_osmo.md) — the *interactive* OSMO dev pod (Remote-SSH + Isaac Sim streaming). Different workflow, same compute pool. + +--- + +## The short version + +| Question | Answer | +|---|---| +| Where do CI jobs run? | A fresh GPU pod on the OSMO `airstack` pool, one per job, destroyed after. | +| What triggers a run? | A PR being **opened**, a `/pytest` comment from a maintainer, or manual `workflow_dispatch`. | +| What gets tested? | Docker image builds, `colcon` builds, unit tests, stack liveliness, sensor rates, takeoff/hover/land, fixed-trajectory tracking. | +| How do I see results? | A metrics report comment on the PR, plus the `test-results-*` artifact (`summary.txt`, `results.xml`, `metrics.json`). | +| What fails the build? | Any failed test, **or** a metric regressing more than 20 % against the base branch's last run. | +| Who holds the secrets? | Only the orchestrator host. Workers get a single-use JIT token valid for one registration. | + +--- + +## Architecture + +Three planes, each owning one job. GitHub owns the queue and the logs. The +orchestrator owns the credentials and the job ↔ pod bookkeeping. OSMO owns the +GPU compute and the pod lifecycle. + +```mermaid +flowchart LR + subgraph gh [GitHub] + pr["Pull request / comment / dispatch"] + queue["Actions queue
workflow_job: queued
labels: self-hosted, airstack-ephemeral"] + api["REST API"] + pr --> queue + queue --- api + end + + subgraph orch ["Orchestrator host (no GPU, always on)"] + svc["airstack-orchestrator.service
orchestrator.py"] + spawn["spawn loop — every 15s"] + reap["reap loop — every 30s"] + creds["/etc/airstack-orchestrator
github-pat + osmo-token + config.yaml"] + state["/var/lib/airstack-orchestrator/state.json
job_id → workflow_id"] + svc --> spawn + svc --> reap + svc --- creds + spawn --- state + reap --- state + end + + subgraph osmo ["OSMO airstack pool (GPU, privileged)"] + wf["Workflow gha-runner-JOBID-TS"] + pod["Ephemeral runner pod
airstack-ci-runner image"] + wf --> pod + end + + api -- "poll queued jobs" --> spawn + spawn -- "mint JIT runner config" --> api + spawn -- "osmo workflow submit" --> wf + pod -- "register + long-poll for work" --> api + reap -- "osmo workflow cancel" --> wf +``` + +Key properties that fall out of this shape: + +- **Truly ephemeral.** Every job starts from the prebaked image with an empty Docker cache. No leftover containers, no dangling networks, no "works because the last run left something behind". +- **PAT isolation.** The GitHub PAT never leaves the orchestrator. The pod receives a [JIT runner config](https://docs.github.com/en/rest/actions/self-hosted-runners#create-configuration-for-a-just-in-time-runner-for-a-repository) — a base64 blob bound to exactly one runner registration, short-lived. +- **Non-personal OSMO identity.** The orchestrator authenticates with a service-account token scoped to the CI pool, so runs never consume an individual's GPU quota and nothing breaks when someone graduates. +- **Crash-safe.** Every workflow is named `gha-runner--`. The reap loop cancels any active workflow with that prefix that is missing from `state.json`, so a crashed or restarted orchestrator cannot leak pods. + +--- + +## Job lifecycle + +From "you comment `/pytest`" to "the pod is gone", in order: + +```mermaid +sequenceDiagram + autonumber + participant Dev as Developer + participant GH as GitHub Actions + participant Orch as Orchestrator + participant OSMO as OSMO scheduler + participant Pod as Runner pod + + Dev->>GH: open PR / comment /pytest / dispatch + GH->>GH: queue job with labels self-hosted + airstack-ephemeral + Orch->>GH: poll queued jobs (15s) + Orch->>GH: POST generate-jitconfig + GH-->>Orch: encoded_jit_config (single use) + Orch->>Orch: render runner-workflow.yaml.j2 + Orch->>OSMO: osmo workflow submit --pool airstack + OSMO-->>Orch: workflow name, then uuid via query + Orch->>Orch: record job_id to workflow_id in state.json + OSMO->>Pod: schedule privileged GPU pod + Pod->>Pod: start inner dockerd, nvidia-smi check + Pod->>GH: run.sh --jitconfig, register ephemeral runner + GH->>Pod: dispatch the one job + Pod->>Pod: checkout, pull images, pytest + Pod-->>GH: logs, conclusion, results artifact + Pod->>Pod: run.sh exits after one job, task completes + OSMO->>OSMO: tear the pod down + Orch->>GH: poll job status (30s) + GH-->>Orch: completed + Orch->>OSMO: cancel if still live, then drop from state.json +``` + +Two safety nets run on top of the happy path: + +- **Straggler reap.** Any tracked job older than `max_job_minutes` (default 48 h) is force-cancelled regardless of what GitHub reports. +- **Orphan sweep.** Active `gha-runner-*` workflows that are not in `state.json` and are more than two minutes old get cancelled. The two-minute grace window prevents the sweep from racing a submit that has not been recorded yet. + +--- + +## Anatomy of a runner pod + +The worker is a prebaked image — everything the job needs is already in the +layer cache when the pod starts, so a slow `apt-get` can never outlive the JIT +token's validity window. + +```mermaid +flowchart TB + subgraph pod ["OSMO task — privileged, 1 GPU, 8 CPU, 32Gi RAM, 300Gi disk"] + entry["run-ephemeral-runner.sh"] + dockerd["inner dockerd
+ nvidia-container-toolkit"] + runner["actions-runner run.sh --jitconfig"] + subgraph compose ["docker compose stack started by airstack up"] + sim["isaac-sim or ms-airsim"] + robot["robot-desktop x NUM_ROBOTS"] + gcs["gcs"] + end + entry --> dockerd + entry --> runner + runner -- "pytest tests/ runs airstack up" --> dockerd + dockerd --> sim + dockerd --> robot + dockerd --> gcs + end + gpu["Node GPU"] --> dockerd +``` + +| Piece | File | What it contributes | +|---|---|---| +| Image | [`runner.Dockerfile`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner.Dockerfile) | Ubuntu 24.04 + Docker CE + compose/buildx + NVIDIA container toolkit + pinned `actions/runner` (2.334.0) | +| Entrypoint | [`runner-entrypoint.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-entrypoint.sh) | Starts `dockerd`, waits up to 60 s for it, runs `nvidia-smi` as a non-fatal GPU sanity check, then `exec`s `run.sh --jitconfig` | +| Pod shape | [`runner-workflow.yaml.j2`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-workflow.yaml.j2) | Resource request, `privileged: true`, the JIT config and `RUNNER_ALLOW_RUNASROOT` env | +| Sizing | `config.yaml` | `cpu: 8`, `gpu: 1`, `memory: 32Gi`, `storage: 300Gi` — sized for sim + robot + GCS images plus Isaac assets | + +!!! warning "Privileged is mandatory" + The tests run `airstack up`, which is `docker compose`, which needs a Docker + daemon *inside* the pod. That requires the pool's platform to have + **Privileged Mode Allowed** enabled. Without it, submissions are rejected and + `osmo workflow logs` shows `dockerd did not become ready`. + +Build and publish the image with +[`build-and-push.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/build-and-push.sh), +or — if you have no local Docker — submit +[`build-runner-on-osmo.yaml`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/build-runner-on-osmo.yaml), +a one-shot OSMO job that builds the runner image inside an OSMO pod and pushes +it to Harbor. + +--- + +## Triggering a run + +### The three entry points + +| Trigger | When it fires | What it runs | +|---|---|---| +| `pull_request` (`types: [opened]`) | Only when the PR is first opened, and only for same-repo branches | pytest's `conftest` defaults — the full mark set | +| `/pytest` PR comment | Any time, from a user with `OWNER`/`MEMBER`/`COLLABORATOR` association | Whatever args you put on the first line of the comment | +| `workflow_dispatch` | Manual, from the Actions tab | The form inputs: `marks`, `sim`, `num_robots`, `stress_iterations`, `stable_duration`, `baseline_run_id` | + +Pushes to an open PR deliberately do **not** re-trigger. GPU pods are a shared +resource, so re-runs are opt-in via `/pytest`. + +### Comment syntax + +The first line is parsed with `shlex`; everything after it is free-form notes. + +```text +/pytest -m liveliness --sim msairsim --num-robots 1 --stress-iterations 1 + +Checking whether the DDS bridge fix holds under 3 robots — see thread above. +``` + +`-m build_packages` is **pull-only**: it retags floating `cache_*` images onto the PR `VERSION` tag and never runs `image-build` (and does not pull Isaac Sim). Use that when iterating on colcon/pytest failures. For other marks, add `--no-image-build` to skip the bake: + +```text +/pytest -m build_packages +/pytest -m liveliness --sim msairsim --no-image-build +``` + +The workflow replies on the thread with the exact `pytest` command it resolved +and a link to the run, and opens a **Check Run** pinned to the PR head SHA so +comment-triggered runs still show up in the PR's Checks tab. + +!!! tip "`build_packages` is prepended for you" + Whenever you pass `-m`, the workflow rewrites the expression to + `build_packages or `. Launch tests are useless against a stale + `install/` tree, and this removes the most common way to waste a 40-minute + GPU run. It is skipped when you already named `build_packages`, and when you + pass no marks at all (pytest then runs everything anyway). + +### What the job does, step by step + +```mermaid +flowchart TD + a["Resolve PR head — issue_comment only"] --> b["Parse pytest args
prepend build_packages, extract --sim"] + b --> c["Ack comment + open in-progress Check Run"] + c --> d["Checkout PR head with submodules"] + d --> e["Write omni_pass.env — guest Nucleus creds"] + e --> f["Create venv, install tests/requirements.txt"] + f --> g{"Registry secrets present?"} + g -- yes --> h["docker login, set AIRSTACK_REGISTRY_CACHE=1
read-only: PRs never republish the cache"] + g -- no --> i["Skip — build from scratch"] + h --> j{"marks contain build_docker?"} + i --> j + j -- yes --> l["Skip image prep — those tests build themselves"] + j -- no --> k["airstack image-pull for the active profiles
fall back to image-build for anything missing"] + k --> m["pytest tests/ with resolved args"] + l --> m + m --> n["Upload tests/results/ artifact, 90-day retention"] + n --> o["Finalize Check Run with the job conclusion"] + o --> p["report job on ubuntu-latest"] +``` + +The image-prep step is what makes runs on a cold pod tolerable: it pulls the +published images for exactly the compose profiles the selected `--sim` implies, +then falls back to a local build only for images the registry did not have (a +new branch that has not been released yet, for example). + +### Layer cache: the floating `cache_*` tag + +Every pod starts with an empty Docker cache, so `build_docker` is only fast if +BuildKit can import layers from the registry. Each compose service therefore +declares two `cache_from` entries: + +| Entry | Example | Who writes it | +|---|---|---| +| Versioned | `airstack:v0.19.0-alpha.7_isaac-sim` | `docker-build.yml`, per release | +| Floating | `airstack:cache_isaac-sim` | `docker-build.yml`, republished every build | + +The versioned entry alone cannot work on a pull request. `check-version-increment` +requires every PR to raise `VERSION`, so the tag a PR builds under is by +definition one that has never been pushed — the pull misses and the build runs +cold from the first `RUN` layer: + +``` +Image ...airstack:v0.19.0-alpha.7_isaac-sim Pulling +Image ...airstack:v0.19.0-alpha.7_isaac-sim failed to resolve reference +``` + +The floating tag is the one that actually hits. It tracks the newest build from +`main`/`develop` rather than any particular version, so a PR imports the layers +its base branch already produced and rebuilds only what it changed. + +Reading and writing the cache are separate switches, and PR runs get read only: + +- `AIRSTACK_REGISTRY_CACHE=1` — pull to seed, build with `BUILDKIT_INLINE_CACHE=1`. + Set by `system-tests.yml` whenever registry secrets are available. +- `AIRSTACK_REGISTRY_CACHE_PUSH=1` — additionally publish the versioned and + floating tags. Set only by `docker-build.yml`. + +Keeping the write switch off for pull requests means an unmerged branch can +neither poison the shared cache for everyone else nor publish an unreleased +`VERSION` tag. Override the tag name with `CACHE_TAG` (default `cache`) to keep +an experimental cache line separate. + +### Publish path: retag when image inputs are unchanged + +`check-version-increment` forces every PR to raise `VERSION`, including +docs-only changes. On `main`/`develop`, that would otherwise mean a full +multi-hour rebuild of every image for a no-op Docker change. + +[`docker-build.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/docker-build.yml) +therefore plans per service before building: + +1. [`.github/workflows/scripts/docker_image_plan.py`](https://github.com/castacks/AirStack/blob/main/.github/workflows/scripts/docker_image_plan.py) + hashes each service’s Dockerfile, compose-related files, build args, and + tracked fingerprint roots into `org.airstack.content-fingerprint`. +2. It inspects the **previous** versioned image’s label (from `HEAD~1`’s + `VERSION=`). +3. **Match** → registry-side retag with + `docker buildx imagetools create` (new `v${VERSION}_…` tag and floating + `cache_*` tag, same digest — no rebuild). +4. **Mismatch / missing / unlabeled / `force_rebuild`** → `docker compose build` + for that service only, with the fingerprint applied as a build label via an + ephemeral `docker-compose.fingerprint.yaml` override. + +PR `system-tests` / `build_docker` are unchanged: they still run real builds so +Dockerfiles keep being proven. Floating `cache_*` remains the layer-cache seed +for those rebuilds. + +Manual dispatch accepts `force_rebuild=true` to rebuild and relabel everything +(useful the first time after this lands, or to refresh `cache_*` from scratch). + +--- + +## What the pipeline tests, and what that catches + +Tests are selected with pytest marks. Collection order is fixed in +`tests/conftest.py` so cheap and prerequisite suites always run first — a +`colcon` break fails in minutes instead of after a sim bring-up. + +```mermaid +flowchart LR + u["unit
seconds, no Docker"] --> bd["build_docker
image builds"] + bd --> bp["build_packages
colcon build in containers"] + bp --> lv["liveliness
stack comes up"] + lv --> sn["sensors
streams flow at rate"] + sn --> th["takeoff_hover_land
flight chain"] + th --> au["autonomy
trajectory tracking"] +``` + +| Mark | Module | What it verifies | Bugs it is good at catching | +|---|---|---|---| +| `unit` | `tests/robot/`, `tests/sim/` proxies | Hermetic Python/numpy logic co-located with each ROS 2 package | Off-by-one and boundary errors in filters, converters, validators; regressions in pure algorithm code | +| `build_docker` | `system/test_build_docker.py` | Every image builds; records image sizes | Broken Dockerfiles, deleted apt packages, upstream base-image drift, accidental image bloat | +| `build_packages` | `system/test_build_packages.py` | `colcon build` inside robot, GCS, and ms-airsim workspaces | Missing `package.xml` dependencies, uninstalled launch/config files, C++ breakage on a clean tree | +| `liveliness` | `system/test_liveliness.py` | Containers reach Running, `/clock` publishes, tmux panes alive, sentinel ROS 2 nodes present, compute snapshot, stability poll | Launch files that crash on start, nodes that die after 30 s, `ROBOT_NAME`/domain-ID misconfiguration, runaway CPU or memory | +| `sensors` | `system/test_sensors.py` | Stereo and depth publish rates on both sim and robot side, filtered LiDAR liveness plus geometry sanity, sim real-time factor, time-series stability | Broken sim-to-ROS bridges, sensor Hz that silently halves, RTF collapse from a heavy new node, LiDAR filter range regressions | +| `takeoff_hover_land` | `system/test_takeoff_hover_land.py` | Four-phase chain per (sim, robots, iteration, velocity): PX4 ready → takeoff to 10 m → hover → land | Controller tuning regressions, altitude overshoot, hover drift, state-estimation bias against ground truth, PX4/MAVROS handshake breakage | +| `autonomy` | `system/test_fixed_trajectory.py` | Same chain with a Circle / Figure8 / Racetrack / Line pattern in the middle; records cross-track error and path RMSE | Path-tracker regressions, trajectory-library math errors, velocity/acceleration limit violations that show up as corner-cutting | + +### The flight chain + +Both flight suites run as an ordered chain per parametrization, so the drone +always ends on the ground before the next configuration starts: + +```mermaid +flowchart LR + r["test_px4_ready
MAVROS + EKF"] --> t["test_takeoff
within 10% of 10 m"] + t --> x["test_hover or test_fixed_trajectory"] + x --> l["test_landing
final altitude < 0.5 m"] + r -. "failure" .-> s["remaining phases skipped"] + t -. "failure" .-> s + x -. "failure still lands" .-> l +``` + +A failure in the middle phase (`test_hover` or `test_fixed_trajectory`) does +**not** skip landing — a bad tracker must not leave a drone stuck in the air +blocking the rest of the sweep. A failure in `test_px4_ready` or `test_takeoff` +does skip the remaining phases for that configuration. + +### Bring-up scope, and why mark selection costs money + +`airstack_env` is **class-scoped** and parametrized over +`(sim, num_robots, iteration)`. Each test class does its own `airstack up` and +`airstack down`. Selecting two suites with `or` therefore performs **two full +stack cycles per tuple**: + +```text +-m liveliness → 1 bring-up per (sim, robots, iter) +-m "liveliness or sensors" → 2 bring-ups per (sim, robots, iter) +--sim msairsim → opt in; both sims doubles all of the above +--num-robots 1,3 → doubles it again +``` + +Run one mark at a time unless you genuinely need both. + +--- + +## Reading the results + +### The PR comment + +After `run-tests` finishes — pass or fail — a `report` job on `ubuntu-latest` +downloads the current artifact plus a **baseline** artifact and runs +[`parse_metrics.py`](https://github.com/castacks/AirStack/blob/main/tests/parse_metrics.py) +in diff mode. + +| Run type | Baseline used | +|---|---| +| PR opened or `/pytest` | Latest `system-tests.yml` artifact on the PR's base branch | +| `workflow_dispatch` with `baseline_run_id` | That specific run | +| `workflow_dispatch` without it | Latest artifact on `main` | + +The comment has three sections per test module: a flat **Metrics** table, a +**Sim publishing rates** pivot (topic Hz aggregates from the `sensors` mark), +and a **Compute usage** pivot (CPU / memory / GPU per container). Regressions +are marked with a red circle, improvements with a green one, and the job +**fails** if any metric moves more than the 20 % threshold in the wrong +direction. That is the mechanism that catches slow degradation — the kind of +change where nothing throws but the tracker is quietly 30 % worse. + +### The artifact + +`test-results--`, retained 90 days: + +```text +tests/results/2026-08-06_14-30-00/ +├── summary.txt # human-readable per-chain summary — open this first +├── results.xml # JUnit XML: durations, pass/fail per test +└── metrics.json # every recorded metric, including time series +``` + +There are no per-test log files. Live output streams to the Actions log via +pytest's `log_cli`, and failed assertions embed the tail of the relevant +`docker` or `ros2` subprocess output directly in the failure message. + +Regenerate a report locally from a downloaded artifact: + +```bash +python tests/parse_metrics.py \ + --current path/to/current-run/ \ + --baseline path/to/baseline-run/ \ + --threshold 20 +``` + +--- + +## Using CI well while developing + +The pipeline is expensive at the far end and nearly free at the near end. Push +each class of failure as far left as it will go. + +```mermaid +flowchart TD + q{"What did you change?"} + q -- "Pure Python / numpy logic" --> u["airstack test -m unit
seconds, no GPU"] + q -- "Dockerfile / dependency" --> b["airstack test -m build_docker or build_packages
minutes, no GPU"] + q -- "Launch file / new node" --> l["airstack test -m liveliness --sim msairsim --num-robots 1"] + q -- "Sensor or bridge" --> s["airstack test -m sensors --sim isaacsim --num-robots 1"] + q -- "Controller / planner" --> a["airstack test -m autonomy --sim msairsim --trajectory-types Circle"] + u --> pr["Push branch, open PR"] + b --> pr + l --> pr + s --> pr + a --> pr + pr --> ci["Full suite runs on the ephemeral GPU pod"] + ci --> rep["Read the metrics comment"] + rep --> iter["/pytest with a narrowed mark to confirm a fix"] +``` + +Practical rules that follow from how the system is built: + +- **Reproduce CI locally with the same command.** `airstack test` and CI both call `pytest tests/` with the same flags. If a run fails in CI, copy the resolved command from the acknowledgment comment and run it on any GPU box — including an [interactive OSMO dev pod](../../../tutorials/airstack_on_osmo.md) if you do not have a local GPU. +- **Narrow before you re-run.** A `/pytest` with no args re-runs everything. `/pytest -m autonomy --sim msairsim --trajectory-types Circle` re-runs the one chain you are fixing, in a fraction of the time. +- **Never trust a green launch test against a stale build.** This is why `build_packages` is auto-prepended; keep it that way when writing your own `/pytest` line. +- **Read `summary.txt` before the raw log.** It groups each flight chain with per-phase wall times and status, so the failing phase is obvious without scrolling a 40-minute log. +- **Treat the metrics diff as a review artifact.** A PR that turns a metric red needs an explanation in the thread, even when every test passed. +- **Bump `VERSION` in `.env` when image content changes.** [`check-version-increment.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/check-version-increment.yml) gates the PR on a strictly-greater semver, and merging that bump is what triggers the release build below. + +--- + +## The release path + +`system-tests.yml` is not the only workflow on the ephemeral runners. +[`docker-build.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/docker-build.yml) +also requests `runs-on: [self-hosted, airstack-ephemeral]` and therefore gets +the same per-job pod treatment. + +```mermaid +flowchart LR + pr["PR merged to main or develop"] --> chk{".env VERSION changed?"} + chk -- no --> stop["No build"] + chk -- yes --> pod["Ephemeral OSMO pod"] + pod --> plan["Per-service fingerprint plan"] + plan --> retag["imagetools retag unchanged"] + plan --> build["compose build changed only"] + retag --> sign["cosign sign — keyless, GitHub OIDC"] + build --> sign + sign --> verify["cosign verify against the workflow identity"] +``` + +Signing is keyless via GitHub's OIDC token, and the same job immediately +verifies each published digest against the expected certificate identity, so a +published image that was not built by this workflow fails the check. Retagged +images keep the previous digest (and therefore an existing signature still +covers that digest; the job re-signs the same digest under the new tags’ refs). + +| Workflow | Runner | Purpose | +|---|---|---| +| `system-tests.yml` | Ephemeral OSMO GPU pod | Full test suite + metrics report | +| `docker-build.yml` | Ephemeral OSMO GPU pod | Retag or rebuild, push, and sign compose images | +| `check-version-increment.yml` | `ubuntu-latest` | Semver gate on `.env` `VERSION=` | +| `deploy_docs_from_*.yaml` | `ubuntu-latest` | Versioned MkDocs publish via `mike` | + +--- + +## Security model + +| Concern | How the design handles it | +|---|---| +| Cross-job state pollution | Fresh pod per job with an empty Docker cache; destroyed within ~30 s of completion | +| Fork PRs executing arbitrary code on a GPU node | `head.repo.full_name == github.repository` guard on the `pull_request` path, and an explicit fork check plus `author_association` gate on the `/pytest` path | +| Long-lived GitHub PAT on a worker | The PAT lives only on the orchestrator; workers get a single-use JIT config bound to one registration | +| Credentials tied to a person | OSMO auth uses a non-personal service-account token scoped to the CI pool | +| Privileged container is root-equivalent | Accepted deliberately — docker-in-docker is required — but bounded to a one-shot pod, on a dedicated pool, running only same-repo code | +| Orchestrator compromise blast radius | Systemd hardening: `NoNewPrivileges`, `ProtectSystem=strict`, `ProtectHome=read-only`, `PrivateTmp`, with a single `ReadWritePaths` for state | +| Leaked pods after a crash | Name-prefix orphan sweep plus a `max_job_minutes` straggler ceiling | + +--- + +## Troubleshooting + +A failed run can break at the orchestrator, at the OSMO pod, at the runner, or +in the tests themselves, and each layer has a different inspection path. Work +down the list. + +| Symptom | Layer | First thing to check | +|---|---|---| +| Job sits `queued` forever, no pod appears | Orchestrator | `journalctl -u airstack-orchestrator.service --since '30 min ago'` — look for `submitted workflow for job ` | +| `find_queued_jobs failed: 401` | Orchestrator | GitHub PAT expired or lost a scope; rotate it | +| `osmo login failed` / auth error | Orchestrator | OSMO service-account token expired (default 31 days); mint a new one and restart the service | +| `osmo workflow submit failed ... privileged` | OSMO pool | The pool's platform lacks **Privileged Mode Allowed** | +| Job queued but never claimed | Labels | `runs-on` labels must be a superset of `runner_labels` in `config.yaml` | +| `dockerd did not become ready` | Pod | Not actually privileged; check the platform, then `osmo workflow logs "$WF" --task runner` | +| `nvidia-smi unavailable` | Pod | GPU not requested or the toolkit is not configured on the node | +| `Cannot connect to the Docker daemon` mid-test | Pod | Inner dockerd crashed — `osmo workflow exec "$WF" runner`, then read `/var/log/dockerd.log` | +| `No space left on device` | Pod | Bump `storage` in `config.yaml`; Isaac assets plus all images are large | +| Runner registered, then pytest failed | Tests | A real test failure — the GitHub Actions log and `summary.txt` are canonical | +| Metrics report job failed with no test failures | Report | A metric regressed past the 20 % threshold; read the diff table | + +To map a GitHub job to its pod: + +```bash +JOB_ID=73286176852 # from the GitHub Actions URL +WF=$(sudo jq -r ".jobs[\"$JOB_ID\"].workflow_id" /var/lib/airstack-orchestrator/state.json) + +osmo workflow query "$WF" --verbose +osmo workflow events "$WF" --task runner # scheduling, image pull, eviction +osmo workflow logs "$WF" --task runner # dockerd, run.sh, and the job itself +osmo workflow exec "$WF" runner # break-glass shell, while RUNNING +``` + +Full runbook, including credential rotation and worker-side diagnostics: +[CI/CD Orchestrator](../../../../tests/ci-cd-orchestrator.md). + +--- + +## File map + +| Path | Role | +|---|---| +| [`.github/workflows/system-tests.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/system-tests.yml) | The test workflow: triggers, arg parsing, image prep, pytest, artifact, metrics report | +| [`.github/orchestrator/orchestrator.py`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/orchestrator.py) | The spawn and reap loops, GitHub polling, JIT minting, OSMO CLI plumbing | +| [`.github/orchestrator/runner-workflow.yaml.j2`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-workflow.yaml.j2) | Per-job OSMO workflow template | +| [`.github/orchestrator/runner.Dockerfile`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner.Dockerfile) | Prebaked worker image | +| [`.github/orchestrator/runner-entrypoint.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-entrypoint.sh) | dockerd bring-up, GPU check, single-job runner | +| [`.github/orchestrator/config.example.yaml`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/config.example.yaml) | Every tunable: pool, platform, resources, limits, poll intervals | +| [`.github/orchestrator/setup.sh`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/setup.sh) | One-time orchestrator host install | +| [`tests/conftest.py`](https://github.com/castacks/AirStack/blob/main/tests/conftest.py) | `airstack_env` fixture, collection order, `MetricsRecorder` | +| [`tests/parse_metrics.py`](https://github.com/castacks/AirStack/blob/main/tests/parse_metrics.py) | Report generation and the regression gate | +| [`tests/run_summary.py`](https://github.com/castacks/AirStack/blob/main/tests/run_summary.py) | `summary.txt` generation | + +## See also + +- [System Tests](../../../../tests/README.md) — marks, fixtures, metrics, and every CLI flag. +- [Unit Testing](unit_testing.md) — the co-location and proxy pattern for package-level tests. +- [End-to-End Testing](end_to_end_testing.md) — the fixed-trajectory benchmark in depth. +- [CI/CD Orchestrator](../../../../tests/ci-cd-orchestrator.md) — admin setup, rotation, and break-glass procedures. +- [AirStack on OSMO](../../../tutorials/airstack_on_osmo.md) — interactive GPU dev pods on the same pool. diff --git a/docs/development/intermediate/testing/end_to_end_testing.md b/docs/development/intermediate/testing/end_to_end_testing.md index 6cddf945b..794355ec9 100644 --- a/docs/development/intermediate/testing/end_to_end_testing.md +++ b/docs/development/intermediate/testing/end_to_end_testing.md @@ -64,7 +64,7 @@ Each run sweeps: | Parameter | CLI flag | Default | | --------- | -------- | ------- | -| Simulator | `--sim` | `msairsim,isaacsim` | +| Simulator | `--sim` | `isaacsim` (`msairsim` opt-in) | | Robot count | `--num-robots` | `1,3` | | Repeat count | `--stress-iterations` | `1` | | Trajectory type | `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | @@ -320,7 +320,7 @@ airstack test -m autonomy \ | Option | Default | Description | | ------ | ------- | ----------- | -| `--sim` | `msairsim,isaacsim` | Comma-separated sim targets | +| `--sim` | `isaacsim` | Comma-separated sim targets (`msairsim` opt-in) | | `--num-robots` | `1,3` | Comma-separated robot counts | | `--stress-iterations` | `1` | Repeat count per `(sim, num_robots)` | | `--trajectory-types` | `Circle,Figure8,Racetrack,Line` | Trajectory sweep | @@ -431,7 +431,7 @@ Action server: `/{robot_name}/tasks/fixed_trajectory` — see also [Tasks and Ta | PX4 ready timeout | Sim not running, GPU issue | Check `nvidia-smi`, Isaac `omni_pass.env` | | `trajectory_success = 0` | Tracker stall or timeout | Check trajectory_controller logs; rebuild the workspace (`-m build_packages`) | | Cross-track error >> 5 m | Wrong tracker params or frame bug | Compare launch params; check world-frame transform | -| Tests run for hours | Default `--sim` and `--num-robots` sweep | Pin `--sim isaacsim --num-robots 1 --stress-iterations 1` | +| Tests run for hours | Default `--num-robots 1,3` (and `--sim msairsim` if opted in) | Pin `--sim isaacsim --num-robots 1 --stress-iterations 1` | | Unknown mark warning `autonomy` | Mark not in `pytest.ini` | Harmless; filter still works | --- diff --git a/docs/development/intermediate/testing/index.md b/docs/development/intermediate/testing/index.md index c23195477..07c5fb887 100644 --- a/docs/development/intermediate/testing/index.md +++ b/docs/development/intermediate/testing/index.md @@ -87,4 +87,4 @@ airstack test -m "build_packages or autonomy" \ - [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, co-located tests, CI workflow - [Testing frameworks](testing_frameworks.md) — `colcon test`, rostest patterns - [Integration testing](integration_testing.md) -- [CI/CD](ci_cd.md) — pipeline overview +- [CI/CD Pipeline on OSMO](ci_cd.md) — how CI runs the full stack on ephemeral GPU pods: architecture, triggers, what each mark catches, and the metrics regression gate diff --git a/gcs/docker/gcs-base-docker-compose.yaml b/gcs/docker/gcs-base-docker-compose.yaml index c8ac5200e..8894bcb9f 100644 --- a/gcs/docker/gcs-base-docker-compose.yaml +++ b/gcs/docker/gcs-base-docker-compose.yaml @@ -7,8 +7,10 @@ services: dockerfile: docker/Dockerfile.gcs tags: - &gcs_image ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:v${VERSION}_gcs + - &gcs_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_gcs cache_from: - *gcs_image + - *gcs_cache command: > bash -c " ssh service restart; diff --git a/mkdocs.yml b/mkdocs.yml index a36d4c75f..4b77be708 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -70,6 +70,7 @@ nav: - Unit Testing: docs/development/intermediate/testing/unit_testing.md - System Tests: tests/README.md - End-to-End Testing: docs/development/intermediate/testing/end_to_end_testing.md + - CI/CD Pipeline: docs/development/intermediate/testing/ci_cd.md - CI/CD Orchestrator: tests/ci-cd-orchestrator.md - Frame Conventions: docs/development/intermediate/frame_conventions.md - Docker Build Profiles: docs/development/intermediate/docker-build-profiles.md diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index 3ff968750..ad73100eb 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -168,6 +168,12 @@ RUN pip3 install --break-system-packages --ignore-installed \ kornia \ typeguard==2.13.3 +# Keep pytest < 8.1. ROS Jazzy launch_testing still implements +# pytest_pycollect_makemodule(path=...), which pluggy rejects after pytest 8.1 +# removed the py.path hook argument (PluginValidationError on colcon test). +RUN python3 -m pip install --no-cache-dir --break-system-packages \ + "pytest>=7.4,<8.1" + # Install MACVO Python dependencies (skipped if SKIP_MACVO=true) RUN if [ "${SKIP_MACVO}" != "true" ]; then \ pip3 install --break-system-packages \ diff --git a/robot/docker/docker-compose.yaml b/robot/docker/docker-compose.yaml index 31b3e368a..cc2bbb6c2 100644 --- a/robot/docker/docker-compose.yaml +++ b/robot/docker/docker-compose.yaml @@ -17,8 +17,13 @@ services: ROS_DISTRO: jazzy tags: - *desktop_image + # Floating tag republished by docker-build.yml on main/develop. The + # versioned tag above never exists yet on a PR (check-version-increment + # forces VERSION up), so it can only ever be a cache miss. + - &desktop_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-x86-64_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *desktop_image + - *desktop_cache # we use tmux sd-keys so that the session stays alive environment: - ROBOT_NAME_SOURCE=container_name # see .bashrc @@ -126,8 +131,10 @@ services: ROS_DISTRO: jazzy tags: - *voxl_image + - &voxl_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-voxl_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *voxl_image + - *voxl_cache environment: - ROBOT_NAME_SOURCE=hostname # see .bashrc - AUTOLAUNCH=${AUTOLAUNCH:-true} @@ -158,8 +165,10 @@ services: DUSTYNV_IMAGE: dustynv/ros:jazzy-ros-base-r36.4.0-cu128-24.04 tags: - *l4t_stack_base_image + - &l4t_stack_base_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-l4t-stack-base_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *l4t_stack_base_image + - *l4t_stack_base_cache # =================================================================================================================== # for running on an NVIDIA jetson (linux for tegra) device @@ -184,9 +193,12 @@ services: ROS_DISTRO: jazzy tags: - *l4t_image + - &l4t_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_robot-l4t_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *l4t_image + - *l4t_cache - *l4t_stack_base_image + - *l4t_stack_base_cache # we use tmux send-keys so that the session stays alive ipc: host command: > @@ -246,8 +258,12 @@ services: L4T_MINOR: 4 L4T_PATCH: 0 IMAGE_NAME: dustynv/ros:jazzy-desktop-r36.4.0-cu128-24.04 + tags: + - *zed_l4t_image + - &zed_l4t_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_zed-l4t-36-4-0_${DOCKER_IMAGE_BUILD_MODE} cache_from: - *zed_l4t_image + - *zed_l4t_cache command: > bash -c "ssh service restart; tmux new -d -s zed_driver && diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg index 55f87f13c..7e02f4b35 100644 --- a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/setup.cfg @@ -10,3 +10,13 @@ python_classes = Test* python_functions = test_* markers = unit: Hermetic unit tests (no ROS stack required) + linter: ament copyright/flake8/pep257 (run separately, not via colcon test) + copyright: ament_copyright + flake8: ament_flake8 + pep257: ament_pep257 +# colcon test / PYTEST_ADDOPTS -m is dropped by ament pytest. Ignore linter +# modules here so only unit tests run. +addopts = + --ignore=test/test_copyright.py + --ignore=test/test_flake8.py + --ignore=test/test_pep257.py diff --git a/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py new file mode 100644 index 000000000..c985d5923 --- /dev/null +++ b/robot/ros_ws/src/sensors/lidar_point_cloud_filter/test/conftest.py @@ -0,0 +1,7 @@ +# Skip ament linter modules during pytest/colcon test. +# PYTEST_ADDOPTS -m is not forwarded by ament pytest; collect_ignore is. +collect_ignore = [ + "test_copyright.py", + "test_flake8.py", + "test_pep257.py", +] diff --git a/simulation/isaac-sim/docker/Dockerfile.isaac-ros b/simulation/isaac-sim/docker/Dockerfile.isaac-ros index 92ac63fdf..69bbd8117 100644 --- a/simulation/isaac-sim/docker/Dockerfile.isaac-ros +++ b/simulation/isaac-sim/docker/Dockerfile.isaac-ros @@ -154,11 +154,23 @@ RUN sed -i \ 's|param set-default IMU_INTEG_RATE 250|param set-default IMU_INTEG_RATE ${PX4_IMU_INTEG_RATE:-250}|' \ /isaac-sim/PX4-Autopilot/ROMFS/px4fmu_common/init.d-posix/px4-rc.simulator -# install px4 dependencies and build -# LD_LIBRARY_PATH= so apt/openssl use the system libcrypto, not isaac-sim's older -# bundled one (which breaks the ca-certificates postinst). Cleared for this step only. -RUN cd PX4-Autopilot && \ - LD_LIBRARY_PATH= DEBIAN_FRONTEND=noninteractive ./Tools/setup/ubuntu.sh +# Install PX4 host deps and build SITL. +# Match ms-airsim: skip NuttX + Gazebo — Isaac Sim is the simulator, and those +# toolchains are heavy. PX4's ubuntu.sh still apt-installs +# software-properties-common whenever /.dockerenv is present; on the nvcr.io +# Isaac base that package's configure step races with a half-configured +# ca-certificates/launchpadlib chain and fails with dpkg exit 100. Reconfigure +# ca-certificates first and drop software-properties-common from the script +# (add-apt-repository is unused on the --no-nuttx/--no-sim-tools path). +RUN apt-get update && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends ca-certificates && \ + update-ca-certificates && \ + dpkg --configure -a || true && \ + sed -i '/software-properties-common/d' PX4-Autopilot/Tools/setup/ubuntu.sh && \ + cd PX4-Autopilot && \ + DEBIAN_FRONTEND=noninteractive ./Tools/setup/ubuntu.sh --no-nuttx --no-sim-tools && \ + rm -rf /var/lib/apt/lists/* + # build px4 RUN cd PX4-Autopilot && \ make px4_sitl diff --git a/simulation/isaac-sim/docker/docker-compose.yaml b/simulation/isaac-sim/docker/docker-compose.yaml index dfd699aa4..50a9df16d 100644 --- a/simulation/isaac-sim/docker/docker-compose.yaml +++ b/simulation/isaac-sim/docker/docker-compose.yaml @@ -8,8 +8,13 @@ services: dockerfile: docker/Dockerfile.isaac-ros tags: - *image_tag + # Floating tag republished by docker-build.yml on main/develop. The + # versioned tag above never exists yet on a PR (check-version-increment + # forces VERSION up), so it can only ever be a cache miss. + - &cache_tag ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_isaac-sim cache_from: - *image_tag + - *cache_tag container_name: isaac-sim entrypoint: "" command: > diff --git a/simulation/ms-airsim/docker/docker-compose.yaml b/simulation/ms-airsim/docker/docker-compose.yaml index 3a4035374..9b49c9098 100644 --- a/simulation/ms-airsim/docker/docker-compose.yaml +++ b/simulation/ms-airsim/docker/docker-compose.yaml @@ -8,8 +8,10 @@ services: dockerfile: Dockerfile tags: - *ms_airsim_image + - &ms_airsim_cache ${PROJECT_DOCKER_REGISTRY}/${PROJECT_NAME}:${CACHE_TAG:-cache}_ms-airsim cache_from: - *ms_airsim_image + - *ms_airsim_cache container_name: ms-airsim entrypoint: "" command: /root/entrypoint.sh diff --git a/tests/README.md b/tests/README.md index 6172d1e98..558c3fe15 100644 --- a/tests/README.md +++ b/tests/README.md @@ -221,7 +221,7 @@ pytest tests/ -m sensors \ | Option | Default | Description | |--------|---------|-------------| -| `--sim` | `msairsim,isaacsim` | Comma-separated sim targets | +| `--sim` | `isaacsim` | Comma-separated sim targets (`msairsim` opt-in) | | `--num-robots` | `1,3` | Comma-separated robot counts | | `--stress-iterations` | `3` | Up/down cycles per (sim, num_robots) config | | `--stable-duration` | `120` | Seconds ``test_stable`` / ``test_sensor_streams_stable`` poll for | @@ -529,6 +529,11 @@ Regressions are flagged with :red_circle:, improvements with :green_circle:. ## CI/CD Integration +!!! note "Full pipeline guide" + For the end-to-end picture — architecture diagrams, job lifecycle, trigger + reference, what each mark catches, and how to fold CI into your development + loop — see **[CI/CD Pipeline on OSMO](../docs/development/intermediate/testing/ci_cd.md)**. + ### Workflow: `system-tests.yml` [`.github/workflows/system-tests.yml`](../../../../.github/workflows/system-tests.yml) runs on: @@ -549,7 +554,7 @@ Regressions are flagged with :red_circle:, improvements with :green_circle:. #### Jobs -**`run-tests`** runs on a freshly-spawned ephemeral OpenStack instance (`[self-hosted, airstack-ephemeral]`). The instance is provisioned per-job by the orchestrator described below and destroyed once the job completes. It installs dependencies, runs pytest, and uploads `tests/results/` as an artifact named `test-results--` with 90-day retention. +**`run-tests`** runs on a freshly-spawned ephemeral OSMO pod (`[self-hosted, airstack-ephemeral]`). The pod is submitted per-job by the orchestrator described below and destroyed once the job completes. It installs dependencies, runs pytest, and uploads `tests/results/` as an artifact named `test-results--` with 90-day retention. **`report`** runs on `ubuntu-latest` after `run-tests` (even if it failed). It: @@ -565,9 +570,9 @@ The workflow uses [`dawidd6/action-download-artifact@v6`](https://github.com/daw --- -## CI/CD Orchestrator (OpenStack-backed ephemeral runners) +## CI/CD Orchestrator (OSMO-backed ephemeral runners) -AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they execute on **truly ephemeral OpenStack instances** spawned per-job by an orchestrator. Each test job gets a fresh VM that is destroyed once the job completes — no Docker layer carryover, no leaked containers, no shared host state. +AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they execute on **truly ephemeral [NVIDIA OSMO](https://nvidia.github.io/OSMO/) pods** submitted per-job by an orchestrator. Each test job gets a fresh GPU pod that is destroyed once the job completes — no Docker layer carryover, no leaked containers, no shared host state. (This replaced an OpenStack-Nova backend; the GitHub side and the per-job-destroy model are unchanged.) ### Architecture @@ -576,19 +581,19 @@ AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they │ Orchestrator VM (airstack-ci-cd-orchestrator) │ │ • polls GitHub for queued workflow_jobs │ │ • mints single-use JIT runner tokens │ -│ • spawns / reaps ephemeral instances via OpenStack Nova │ -│ • holds the GitHub PAT and OpenStack application credential│ +│ • submits / reaps ephemeral OSMO workflows via osmo CLI │ +│ • holds the GitHub PAT and OSMO service-account token │ └────────────┬───────────────────────────────────┬─────────────┘ │ │ ▼ ▼ ┌──────────────────────────────┐ ┌────────────────────────────────┐ │ Ephemeral worker (per job) │ │ GitHub Actions queue │ -│ Image: Ubuntu-24.04-GPU- │ │ workflow_job status=queued │ -│ Headless │ │ labels: [self-hosted, │ -│ cloud-init bootstraps Docker │ │ airstack-ephemeral] │ -│ + nvidia-container-toolkit + │ └────────────────────────────────┘ -│ GH Actions runner; runs ONE │ -│ job, then is destroyed. │ +│ Prebaked airstack-ci-runner │ │ workflow_job status=queued │ +│ image: Docker + nvidia CTK + │ │ labels: [self-hosted, │ +│ GH runner. Privileged pod │ │ airstack-ephemeral] │ +│ starts dockerd, runs ONE │ └────────────────────────────────┘ +│ job (JIT), then the pod │ +│ is destroyed. │ └──────────────────────────────┘ ``` @@ -596,21 +601,22 @@ AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they | Concern | Mitigation | |---------|------------| -| Cross-job state pollution (Docker cache, dangling networks, leftover artifacts) | Each job runs on a fresh VM. Spent VM is destroyed within ~30 s of job completion. | +| Cross-job state pollution (Docker cache, dangling networks, leftover artifacts) | Each job runs on a fresh OSMO pod, destroyed within ~30 s of job completion. | | Fork PRs executing arbitrary code | Workflow's `if: github.event.pull_request.head.repo.full_name == github.repository` — fork PRs skipped. | -| Runner running as root | The runner runs as the unprivileged `ubuntu` user inside an instance whose only purpose is one job. | -| Docker socket gives root-equivalent access | Bounded to a single one-shot VM. The orchestrator host doesn't expose Docker at all. | +| Runner runs privileged (root) for docker-in-docker | The pod is privileged (needed to run `airstack up`/compose), but it is single-use, scoped to the dedicated CI pool, and only same-repo code ever reaches it. | +| Docker socket gives root-equivalent access | Bounded to a single one-shot pod. The orchestrator host doesn't expose Docker at all. | | Long-lived PAT on the runner host | The PAT lives only on the orchestrator. Workers receive a single-use **JIT runner config** — a base64 token bound to one runner registration. | -| Persistent OpenStack creds tied to a user password | Orchestrator authenticates with an **application credential** (revocable, scoped) instead of `openrc.sh`. | +| Persistent creds tied to a personal account | Orchestrator authenticates with a shared, non-personal **OSMO service-account token** (revocable, scoped to the CI pool), not an individual's login. | ### Setup -The orchestrator service code, cloud-init template, systemd unit, and full setup runbook live in [`.github/orchestrator/`](../../../../.github/orchestrator/). See [`.github/orchestrator/README.md`](ci-cd-orchestrator.md) for: +The orchestrator service code, OSMO runner-workflow template, runner image, systemd unit, and full setup runbook live in [`.github/orchestrator/`](../../../../.github/orchestrator/). See [`.github/orchestrator/README.md`](ci-cd-orchestrator.md) for: -- creating the OpenStack application credential and `clouds.yaml` -- staging the GitHub PAT -- running `setup.sh` on the orchestrator VM -- filling in flavor / network / keypair / security-group in `/etc/airstack-orchestrator/config.yaml` +- obtaining the OSMO service-account token and a dedicated CI GPU pool (with privileged mode enabled) +- building and pushing the runner image (`runner.Dockerfile`) +- staging the GitHub PAT and the OSMO token +- running `setup.sh` on the orchestrator host (installs the `osmo` CLI) +- filling in osmo_url / pool / platform / runner_image / resources in `/etc/airstack-orchestrator/config.yaml` - enabling and verifying the `airstack-orchestrator.service` systemd unit ### Runner labels diff --git a/tests/colcon_unit_test_packages.yaml b/tests/colcon_unit_test_packages.yaml index 65cdd38cc..6d96da1e5 100644 --- a/tests/colcon_unit_test_packages.yaml +++ b/tests/colcon_unit_test_packages.yaml @@ -11,5 +11,7 @@ robot: packages: - natnet_ros2 - lidar_point_cloud_filter - # Skips ament_copyright / flake8 / pep257 on Python packages; run linters separately. - pytest_args: "-m not linter" + # Linter skip lives in lidar_point_cloud_filter setup.cfg + test/conftest.py. + # ament pytest does not honor PYTEST_ADDOPTS -m. + # launch_testing is skipped via PYTEST_DISABLE_PLUGIN_AUTOLOAD in the test. + pytest_args: [] diff --git a/tests/conftest.py b/tests/conftest.py index 62644eb98..203e9d611 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -21,8 +21,9 @@ # ── pytest config / hooks ────────────────────────────────────────────────── def pytest_addoption(parser): - parser.addoption("--sim", default="msairsim,isaacsim", - help="Comma-separated sim targets: msairsim, isaacsim") + parser.addoption("--sim", default="isaacsim", + help="Comma-separated sim targets: isaacsim, msairsim. " + "Default isaacsim; pass --sim msairsim to opt in.") parser.addoption("--num-robots", default="1,3", help="Comma-separated robot counts, e.g. 1,3") parser.addoption("--stress-iterations", type=int, default=1, @@ -67,6 +68,9 @@ def pytest_addoption(parser): parser.addoption("--waypoint-timeout", default="120", help="Per-waypoint time budget (s, odometry clock) in " "test_waypoint_flight. Default: 120") + parser.addoption("--no-image-build", action="store_true", default=False, + help="CI flag: skip image-build in system-tests.yml. " + "Ignored by pytest itself.") def pytest_configure(config): diff --git a/tests/harness/__init__.py b/tests/harness/__init__.py index 05efe2144..d3fcf4e32 100644 --- a/tests/harness/__init__.py +++ b/tests/harness/__init__.py @@ -26,6 +26,7 @@ AIRSTACK_ROOT, COLCON_UNIT_TEST_PACKAGES_YAML, colcon_test_robot_command, + format_pytest_addopts, load_colcon_unit_test_config, repo_path, unit_test_dirs, @@ -44,7 +45,7 @@ __all__ = [ # discovery "AIRSTACK_ROOT", "COLCON_UNIT_TEST_PACKAGES_YAML", "repo_path", - "colcon_test_robot_command", "load_colcon_unit_test_config", + "colcon_test_robot_command", "format_pytest_addopts", "load_colcon_unit_test_config", "unit_test_dirs", "unit_test_files", # session "logger", diff --git a/tests/harness/commands.py b/tests/harness/commands.py index 9b6bc2fb9..7c593e4de 100644 --- a/tests/harness/commands.py +++ b/tests/harness/commands.py @@ -55,8 +55,12 @@ def _run_teed(cmd_list, timeout, log_name=None, env=None, cwd=None): return result -def docker_exec(container, cmd, timeout=60, log_name=None): - full_cmd = ["docker", "exec", container, "bash", "-c", cmd] +def docker_exec(container, cmd, timeout=60, log_name=None, env=None): + full_cmd = ["docker", "exec"] + if env: + for key, value in env.items(): + full_cmd.extend(["-e", f"{key}={value}"]) + full_cmd.extend([container, "bash", "-c", cmd]) return _run_teed(full_cmd, timeout=timeout, log_name=log_name) diff --git a/tests/harness/discovery.py b/tests/harness/discovery.py index c0178a92d..28e03737a 100644 --- a/tests/harness/discovery.py +++ b/tests/harness/discovery.py @@ -3,9 +3,11 @@ Driven by ``tests/colcon_unit_test_packages.yaml``. ``conftest.pytest_configure`` adds ``unit_test_files()`` to the pytest run, and ``pytest_itemcollected`` marks each of those items ``unit`` via ``_is_unit_item``. ament lint tests are excluded here — they run under -``colcon test`` (see ``colcon_test_robot_command``'s ``-m not linter``). +``colcon test`` (linter skip is in package pytest config; see +``colcon_test_robot_command``). """ import os +import shlex from pathlib import Path import yaml @@ -43,20 +45,42 @@ def load_colcon_unit_test_config(workspace="robot"): raise ValueError( f"'{workspace}.packages' is empty in {COLCON_UNIT_TEST_PACKAGES_YAML.name}" ) - return packages, cfg.get("pytest_args", "") + raw_args = cfg.get("pytest_args", []) + if isinstance(raw_args, str): + pytest_args = shlex.split(raw_args) if raw_args else [] + elif isinstance(raw_args, list): + pytest_args = [str(a) for a in raw_args] + else: + raise TypeError( + f"'{workspace}.pytest_args' must be a list or string in " + f"{COLCON_UNIT_TEST_PACKAGES_YAML.name}, got {type(raw_args).__name__}" + ) + return packages, pytest_args def colcon_test_robot_command(workspace="robot"): - """Shell command for colcon test over unit-test packages (robot workspace).""" - packages, pytest_args = load_colcon_unit_test_config(workspace) + """Shell command for colcon test over unit-test packages (robot workspace). + + Pytest flags from the YAML are *not* put on this command. colcon's + ``--pytest-args`` is a single nargs='*' option (last occurrence wins), + and nesting those tokens through ``bash -ic`` also breaks quoting. + Pass them as ``PYTEST_ADDOPTS`` via ``docker_exec(..., env=...)``. + """ + packages, _ = load_colcon_unit_test_config(workspace) pkg_list = " ".join(packages) - cmd = ( + return ( f"colcon test --packages-select {pkg_list} " "--event-handlers console_direct+ --return-code-on-test-failure" ) - if pytest_args: - cmd += f' --pytest-args "{pytest_args}"' - return cmd + + +def format_pytest_addopts(pytest_args): + """Build a PYTEST_ADDOPTS value that pytest will shlex-split back to tokens. + + Do not name this pytest_*: conftest functions with that prefix are treated + as pytest hooks and fail collection (exit code 3). + """ + return " ".join(shlex.quote(a) for a in pytest_args) # Each listed package resolves to its /test dir via these per-workspace globs. @@ -67,7 +91,7 @@ def colcon_test_robot_command(workspace="robot"): # ament lint tests ship in every ROS package's test/ dir and import ament_* at # module load (unavailable outside the built workspace). Skip them here — they run -# under `colcon test` instead (see colcon_test_robot_command's `-m not linter`). +# under `colcon test` instead (package pytest config skips ament linters). _LINTER_TEST_FILENAMES = { "test_copyright.py", "test_flake8.py", "test_pep257.py", "test_pep8.py", "test_xmllint.py", "test_lint_cmake.py", diff --git a/tests/system/test_build_packages.py b/tests/system/test_build_packages.py index 5c54cfee3..aa84b0464 100644 --- a/tests/system/test_build_packages.py +++ b/tests/system/test_build_packages.py @@ -1,10 +1,12 @@ +import shlex from pathlib import Path import pytest from conftest import (AIRSTACK_ROOT, airstack_cmd, colcon_test_robot_command, - docker_exec, load_colcon_unit_test_config, logger, - read_log_tail, wait_for_container) + docker_exec, format_pytest_addopts, + load_colcon_unit_test_config, logger, read_log_tail, + wait_for_container) def _warn_if_prebuilt(*ws_paths): @@ -52,7 +54,7 @@ def test_colcon_test_robot(self): Package list and pytest args come from tests/colcon_unit_test_packages.yaml. Workspace-wide ament linter tests are not gated here. """ - packages, _ = load_colcon_unit_test_config("robot") + packages, pytest_args = load_colcon_unit_test_config("robot") try: result = airstack_cmd("up", "robot-desktop", env_overrides={"AUTOLAUNCH": "false", "DISPLAY": ""}, @@ -69,10 +71,18 @@ def test_colcon_test_robot(self): ) assert build.returncode == 0, f"colcon build (with testing) failed:\n{read_log_tail()}" + # PYTEST_DISABLE_PLUGIN_AUTOLOAD=1 skips launch_testing before + # pytest 8.1+ validates its removed path= hook. Linter skip is in + # the package setup.cfg / test/conftest.py (ament pytest ignores + # PYTEST_ADDOPTS -m). + exec_env = {"PYTEST_DISABLE_PLUGIN_AUTOLOAD": "1"} + if pytest_args: + exec_env["PYTEST_ADDOPTS"] = format_pytest_addopts(pytest_args) test = docker_exec( container, - f"bash -ic '{colcon_test_robot_command('robot')}'", + f"bash -ic {shlex.quote(colcon_test_robot_command('robot'))}", timeout=300, + env=exec_env, ) assert test.returncode == 0, ( f"colcon test failed (packages: {', '.join(packages)}):\n{read_log_tail()}" From 0ae96fe8c40474dfb7b107be7a2a736d50976b11 Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Sat, 15 Aug 2026 15:44:19 -0400 Subject: [PATCH 15/21] OptiTrack (1/3): robot-side NatNet client + PX4 external-vision fusion (#374) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(perception): bring natnet_ros2 client up to the optitrack_emulation baseline Take the natnet_ros2 package from #367 onto the reworked base: the C++ NatNet client (natnet_ros2_node + client adapter + natnet_logic seam), the base mavros_gp_origin and vision_pose_converter nodes, per-robot natnet_config profiles, launch files, and the co-located C++/Python unit tests. natnet_ros2 is already listed in tests/colcon_unit_test_packages.yaml, so the base's YAML-driven collection picks up the updated unit tests directly — no proxy files. Real-robot PX4 external-vision fusion (px4_param_setter, geoid-corrected origin, EV-pose bounds) is layered on next. Co-Authored-By: Claude Opus 4.8 * feat(natnet): real-robot PX4 external-vision fusion (mocap → EKF2) Layer the Hummingbird real-robot fusion pipeline onto natnet_ros2 so an OptiTrack-only drone (no GNSS/mag/baro) fuses mocap pose into PX4 EKF2: - mavros_gp_origin_node: publishes a guarded synthetic GPS origin. On real HW, use_geoid_altitude feeds the egm96-5 geoid undulation (N ≈ 54 m at Lisbon) so mavros's ellipsoidal→AMSL conversion cancels and local z == OptiTrack z (fixes the ~36 m = 90 − 54 boot offset; see docs). Auto-skipped in sim. - vision_pose_converter_node: rate-limited mocap → MAVROS vision_pose bridge. - px4_params.yaml: the external-vision EKF2 param set. - natnet_ros2.launch.py wires the bridges when a robot's vision_pose block is on. px4_param_setter reworked into a **checker** (R3): auto_set=false by default — it reads and *flags* FCU params that differ from the desired set instead of writing them; on_mismatch=warn|halt (default warn). Set the params in QGroundControl; the node is the pre-flight safety net. auto_set=true restores the legacy enforce path. Excludes the duplicate vendored NatNet SDK (sensors/natnet_ros2) and deployment override .envs. Co-Authored-By: Claude Opus 4.8 * docs(natnet): PX4 external-vision setup guide + height-datum explainer Move the PX4 external-vision setup guide into docs/ (was a repo-root markdown) and wire it into the mkdocs nav under Perception. Adapt it to the reworked param checker (auto_set default off; check-and-flag, not enforce), and add a "height datum" section explaining the ~36 m local_z offset: AirStack's 90.0 ellipsoidal world datum minus the egm96-5 geoid undulation (N ≈ 54 m at Lisbon) = 36 m; fixed by publishing the geoid-corrected origin altitude so mavros's conversion cancels. Documents why it's invisible in sim and why the shared 90.0 datum must not be changed globally. Co-Authored-By: Claude Opus 4.8 * feat(perception): point natnet launch include at the natnet_config schema Refine the perception bringup comment on the LAUNCH_NATNET include so it points at the per-robot natnet_config.yaml schema parsed by natnet_ros2.launch.py. Co-Authored-By: Claude Opus 4.8 * chore: bump version to 0.19.0-alpha.14 * fix(natnet): make the NatNet client actually reachable + correct EV tuning Three defects that together meant the OptiTrack client could never connect to anything, in sim or on a real robot. 1. NATNET_SERVER_IP was unreachable config. natnet_config.yaml resolves it via $(env ...), but docker compose only injects variables named in a service's `environment:` block and no service declared it — not the compose files, not .env, not tests/system/test_optitrack_e2e.py. The client therefore always fell back to its hardcoded default (192.168.123.199), which is neither the in-sim emulator (172.31.0.200) nor any Motive host. Forwarded in robot-base-docker-compose.yaml, defaulting to the emulator so the sim path works unconfigured. 2. The tracked rigid body could never match. robot_1 pinned "Hummingbird" id 1146 while the emulator streams "Drone" id 1, and the NatNet client filters incoming frames by NUMERIC id — a mismatch yields a connected client that silently never publishes. Body name/id now accept $(env ...) (expanded in _build_node_params, with the id still coerced to int) and default to the emulator's body; sites override via NATNET_BODY_NAME / NATNET_BODY_ID. 3. EV tuning was not the deployment-validated set. EKF2_EV_DELAY 8.0 -> 7.0 and EKF2_EVP_NOISE 0.01 -> 0.05. EKF2_EVP_NOISE is not marker precision: it also sets the innovation gate at EKF2_EVP_GATE (default 5) sigma, so 0.01 gave a 5 cm gate that rejected legitimate mocap updates and refused to arm. 0.05 is a 25 cm gate, still far tighter than PX4's 0.1 default. px4_params.yaml keeps the evidence inline, including two results that are expensive to rediscover: raising EKF2_EV_DELAY to 50.0 measurably degrades tracking (the negative best-fit time shift shows the estimate running ahead of truth), and the drift-and-snap excursions were a 90 deg body-yaw offset in the Motive rigid-body definition, not a gate problem — so the fix belongs in Motive, never as yaw compensation in code. Adds two unit tests covering body-field env expansion and the emulator-matching defaults (natnet_ros2: 14 -> 16 passing). Co-Authored-By: Claude Opus 5 * add a real-robot OptiTrack deployment override Mocap counterpart to l4t-px4-realrobot.env: same Jetson stack, plus the NatNet server/body settings and LAUNCH_NATNET. Carries the two things that are easy to get wrong and produce no error. The body id must match Motive's streaming id, since the client filters frames numerically and a mismatch just never publishes. And nothing writes the EKF2 external-vision parameters to a real FCU — px4_param_setter only reads them back and warns — so they have to be set once in QGroundControl. Co-Authored-By: Claude Opus 5 * config bodies per robot profile; trim comments to the docs The rigid body a robot tracks is now set only in its natnet_config.yaml profile, keyed by ROBOT_NAME. NATNET_BODY_NAME / NATNET_BODY_ID are gone: a single global env var cannot express per-robot values, so it blocked the multi-robot case the profiles already handle. NATNET_SERVER_IP stays in the environment — one Motive host serves every robot. Comments across the package are cut back to what is not evident from the code. The EKF2 tuning results that were buried in px4_params.yaml move into docs/robot/px4_external_vision.md, which also had stale values (EV_DELAY 15.0, EVP_NOISE 0.01) contradicting the config: that raising EV_DELAY measurably hurts tracking, and that drift-and-snap was a Motive rigid-body yaw offset rather than a gate problem. Kept: the license header, and the note on why the SDK needs a reachability pre-check before Connect(). Co-Authored-By: Claude Opus 5 * put the mocap floor at the shared world datum desired_floor_amsl 0.0 -> 36.0, the world datum (90 m ellipsoidal) expressed in AMSL, so a mocap robot's reported global altitude agrees with sim and the GCS instead of sitting at sea level. The published ellipsoidal origin works out to ~90 m, the datum itself. local_position.z equals the OptiTrack height for any value of this parameter — it only moves the global altitude. Reasoning lives in the external-vision doc, which also now records that GeoPoint.altitude is ellipsoidal by contract, so AMSL must not be sent here. Not yet confirmed on hardware. Co-Authored-By: Claude Opus 5 * fail the build when the geoid dataset is missing MAVROS constructs the egm96-5 geoid in its UAS core, before any plugin loads, and throws std::invalid_argument if the dataset is absent — mavros_node terminates at startup, so there is no MAVROS at all, GPS or mocap. The image could ship without it. mavros' install_geographiclib_datasets.sh sends the downloader's output to /dev/null and, on failure, prints "Error while installing" and returns without a non-zero exit, so the RUN layer succeeded regardless. The tool it calls, geographiclib-get-geoids, was also only a transitive dependency of ros-mavros rather than something we pinned. Now pins geographiclib-tools and asserts the file landed, so a failed download fails the build. Verified against the shipped image: with the downloader broken the script still exits 0, and the new test -f returns non-zero. This is the dependency the OptiTrack external-vision path needs — mavros_gp_origin resolves the geoid undulation with the same egm96-5 model — hence landing it here. Co-Authored-By: Claude Opus 5 * abbreviated Dockerfile comment on geographic lib installation * fix repo-root doc links in the external-vision guide They resolved relative to docs/robot/, so mkdocs looked for docs/robot/robot/ros_ws/... and warned on every one. Prefixed with ../../; the file now builds warning-free. Co-Authored-By: Claude Opus 5 * comment trim * point the companion-link section at the PX4 docs Section 3 documented MAVLink serial setup at length — MAV_n_CONFIG / SER_TEL2_BAUD tables, wiring, USB-vs-TELEM2 comparison — all of which is standard PX4 setup that PX4 documents better and keeps current. Replaced with links to the companion computer, MAVLink peripherals, and serial configuration pages. Kept the part PX4 does not cover: the Cube Orange USB CDC-ACM stall, which starves EKF2 of vision updates and is why the companion link belongs on TELEM2. Four other sections and the troubleshooting table point here for that symptom. 65 lines -> 19. Co-Authored-By: Claude Opus 5 * frame section 4 around mavros_gp_origin, demote the 36 m note Section 4 now leads with what mavros_gp_origin does — inject a synthetic global position so PX4 will arm in modes that need one without GNSS — rather than presenting the height datum as a peer topic. The ~36 m offset becomes a note under it, scoped to real deployments and ending with why sim never sees it (the geoid path is skipped under use_sim_time, and sim's synthetic GPS is self-consistent with the spawn). Section 4b is gone; it had no inbound references. Dropped the "don't change the 90.0 globally" warning. Co-Authored-By: Claude Opus 5 * reject an unknown connection_type instead of defaulting to unicast validate_connection_type returned "unicast" for anything it did not recognise, so "mutlicast" or "Unicast" produced a client that connected on the wrong transport and then never received a frame — with only a warning to show for it. It now throws std::invalid_argument naming the offending value, and the node turns that into a fatal startup error rather than a warning it flies past. Case-sensitivity is deliberate: accepting "Unicast" would mean the config silently disagrees with itself. Tests updated from fallback to throw, plus one asserting the message names the bad value. 60 gtests pass. Co-Authored-By: Claude Opus 5 * px4 external vision docs trim * trim natnet node comments; note the latency figure is an estimate Comment trims in natnet_ros2_node.cpp (no code change). Records what cube_orange_latency_ms actually is: an estimate of the FCU hop, added to a logged total and never fused. Only the transport half of EKF2_EV_DELAY is measured, and that measurement starts at the NatNet server transmit, so Motive's own capture pipeline is not in it either. Also notes, for whoever retunes next, that the node stamps poses with its receive time — so delay after that stamp does not belong in EKF2_EV_DELAY, which points lower than 7.0 and matches the negative best-fit shift already recorded. Not chased down; 7.0 flies. CameraMidExposureTimestamp would replace the estimate with a measurement if it ever matters. Co-Authored-By: Claude Opus 5 * trim the external-vision tuning notes Replaces the two long tuning write-ups with a short troubleshooting tip (check the Motive rigid-body definition first — x forward, z up) and cuts the latency section back to what is measured versus estimated. Fixed a dangling "see below" in the EKF2_EV_DELAY table row, which pointed at the removed tuning result; the warning it carried is now stated inline. Co-Authored-By: Claude Opus 5 --- .env | 2 +- CHANGELOG.md | 8 + docs/robot/px4_external_vision.md | 216 +++++++++ mkdocs.yml | 1 + overrides/l4t-optitrack-realrobot.env | 35 ++ robot/docker/Dockerfile.robot | 7 +- robot/docker/robot-base-docker-compose.yaml | 2 + .../src/perception/natnet_ros2/CMakeLists.txt | 7 +- .../src/perception/natnet_ros2/README.md | 164 +++++-- .../natnet_ros2/config/mavros_gp_origin.yaml | 22 + .../natnet_ros2/config/natnet_config.yaml | 128 ++++-- .../natnet_ros2/config/px4_params.yaml | 43 ++ .../config/vision_pose_converter.yaml | 7 +- .../natnet_ros2/natnet_client_adapter.hpp | 11 + .../include/natnet_ros2/natnet_logic.hpp | 70 ++- .../launch/mavros_gp_origin.launch.xml | 34 ++ .../natnet_ros2/launch/natnet_ros2.launch.py | 197 +++++++- .../launch/px4_param_setter.launch.xml | 36 ++ .../launch/vision_pose_converter.launch.xml | 34 +- .../src/perception/natnet_ros2/package.xml | 2 + .../natnet_ros2/src/mavros_gp_origin_node.py | 198 ++++++++ .../natnet_ros2/src/natnet_client_adapter.cpp | 24 +- .../natnet_ros2/src/natnet_ros2_node.cpp | 429 ++++++++++++------ .../natnet_ros2/src/px4_param_setter_node.py | 296 ++++++++++++ .../src/vision_pose_converter_node.py | 66 ++- .../natnet_ros2/test/test_natnet_logic.cpp | 74 ++- .../natnet_ros2/test/test_natnet_ros2.py | 173 ++++++- .../launch/perception.launch.xml | 3 +- 28 files changed, 1956 insertions(+), 333 deletions(-) create mode 100644 docs/robot/px4_external_vision.md create mode 100644 overrides/l4t-optitrack-realrobot.env create mode 100644 robot/ros_ws/src/perception/natnet_ros2/config/mavros_gp_origin.yaml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/launch/mavros_gp_origin.launch.xml create mode 100644 robot/ros_ws/src/perception/natnet_ros2/launch/px4_param_setter.launch.xml create mode 100755 robot/ros_ws/src/perception/natnet_ros2/src/mavros_gp_origin_node.py create mode 100755 robot/ros_ws/src/perception/natnet_ros2/src/px4_param_setter_node.py diff --git a/.env b/.env index 4aa2502f1..70736e9ac 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.13" +VERSION="0.19.0-alpha.14" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/CHANGELOG.md b/CHANGELOG.md index 8c99c905e..0d12c4108 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `overrides/l4t-optitrack-realrobot.env` — deployment override for a real Jetson robot flying on OptiTrack mocap (PX4 EKF2 external vision instead of GPS): the NatNet server/body settings, plus the multi-NIC and FCU-parameter notes that path needs - Feature notebook workflow (`use-feature-notebook` skill): every agent-implemented feature gets a local, gitignored `notebook/NNN-feature-slug/` entry with a status-tracked `design_spec.md` (written before coding) and `results/` artifacts + self-contained `results_summary.md` that populate the feature's PR description - Battery and telemetry display in GCS RQT control panel (voltage and percentage per robot when MAVROS battery topic is bridged) - `TARGET_ARCH` build arg (default `x86_64`) in `Dockerfile.robot` to arch-parametrize `LD_LIBRARY_PATH`; `docker-compose.yaml` passes `TARGET_ARCH: aarch64` to the `voxl` and `l4t` real-robot image builds @@ -16,6 +17,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `overrides/l4t-px4-realrobot.env` — site-agnostic deployment override for a single real PX4 robot on a Jetson (aarch64/l4t) - `integration` test tier (`tests/integration/`, `integration` mark) with a shared `robot_autonomy_stack` fixture (robot container, no sim/GPU) - `waypoint_flight` system test (`tests/system/test_waypoint_flight.py`): takeoff → ordered waypoint route via `NavigateTask` (dispatched as a dense plan) → land, judged on the odometry track by the standalone stdlib-only `tests/waypoint_checker.py` (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`); validated end-to-end in Isaac Sim; serves as the standard acceptance check after integrating or swapping a planner module +- Real-robot PX4 external-vision fusion in `natnet_ros2` (OptiTrack mocap → EKF2): `mavros_gp_origin` (geoid-corrected synthetic GPS origin so `local_position.z` == OptiTrack z, fixing the ~36 m boot offset), `vision_pose_converter`, and a PX4 param **checker** (`px4_param_setter`, `auto_set` off by default; `on_mismatch` warn/halt) — setup guide at `docs/robot/px4_external_vision.md` ### Changed @@ -39,6 +41,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Dropped `ROBOT_NAME` / `ROS_DOMAIN_ID` from `overrides/l4t-px4-realrobot.env`: no compose service declares either, so an env file could never set them and the lines were inert - `bag_record/bag_recording_status` was bridged GCS -> robot in `domain_bridge.yaml`, the same direction as the command it answers, so recorder status never reached the GCS and every recording indicator stayed blank - `bag_record_node` passed `--exclude` to `ros2 bag record`, which Jazzy renamed to `--exclude-regex`. It is now an ambiguous prefix of four options, so argparse rejected the command and any section using `exclude:` (including `log.yaml`'s `airstack` section, i.e. everything but the cameras) recorded nothing — surfacing only as a usage dump in the node's stdout. Multiple `exclude:` entries are now alternated into one regex instead of repeating a single-valued flag, which had silently kept only the last +- `natnet_config.yaml`'s `$(env NATNET_SERVER_IP ...)` could never resolve: no compose service declared the variable, so the NatNet client always fell back to its hardcoded default and could reach neither the in-sim emulator nor a real Motive host. It is now forwarded in `robot-base-docker-compose.yaml`, defaulting to the in-sim emulator +- The NatNet rigid body tracked by `robot_1` defaulted to a site-specific body (id 1146) that no emulator streams; since the client filters frames by numeric id, that produced a connected client that never published. It now defaults to the emulator's body (`Drone`, id 1). Per-robot bodies are configured in each robot's profile in `natnet_config.yaml`, selected by `ROBOT_NAME` +- OptiTrack external-vision tuning corrected from real-flight bags: `EKF2_EV_DELAY` 8.0 → 7.0 and `EKF2_EVP_NOISE` 0.01 → 0.05. The old 0.01 gave a 5 cm innovation gate (`EKF2_EVP_GATE` × 5σ) that rejected valid mocap updates and blocked arming; `px4_params.yaml` now records the supporting measurements and the drift-and-snap misdiagnosis so neither is repeated +- The synthetic GPS origin now places the mocap floor at the shared world datum (`desired_floor_amsl: 36.0`, i.e. 90 m ellipsoidal in AMSL) rather than at sea level, so a mocap robot's reported global altitude agrees with sim and the GCS. `local_position.z` still equals the OptiTrack height either way +- The robot image could ship without the GeographicLib `egm96-5` geoid: mavros' `install_geographiclib_datasets.sh` swallows a failed download and still exits 0, so the `RUN` layer succeeded either way, and `geographiclib-tools` was only ever a transitive dependency. MAVROS builds that geoid in its UAS core before any plugin loads and throws if it is missing, so `mavros_node` died at startup on affected images. `Dockerfile.robot` now pins the tool and asserts the file exists, failing the build instead +- An unrecognised `connection_type` in `natnet_config.yaml` silently fell back to `unicast`, so a typo produced a client that connected on the wrong transport and never received frames. `validate_connection_type` now throws and `natnet_ros2_node` fails at startup naming the offending value ## [1.0.0] - 2024-12-19 diff --git a/docs/robot/px4_external_vision.md b/docs/robot/px4_external_vision.md new file mode 100644 index 000000000..4c7be444b --- /dev/null +++ b/docs/robot/px4_external_vision.md @@ -0,0 +1,216 @@ +# PX4 External-Vision (OptiTrack) Setup + +Runbook for flying a PX4 vehicle (Cube Orange) on **OptiTrack mocap as the sole +position source** — no GNSS, no magnetometer — with an onboard companion +computer (Jetson) running the AirStack robot stack. + +It covers three things that must all be right: + +1. **EKF2 parameters** — tell PX4 to fuse external vision instead of GPS/baro/mag. +2. **Companion MAVLink link** — how the Jetson talks to the Cube (see the PX4 docs). +3. **Vision pose pipeline** — how a mocap pose becomes a `VISION_POSITION_ESTIMATE`, + and how PX4 gets a global position without GNSS. + +> Scope: PX4 ≥ 1.14 (the `EKF2_EV_CTRL` / `EKF2_GPS_CTRL` era). For older +> firmware use `EKF2_AID_MASK: 24` and `EKF2_HGT_MODE: 3` instead of the bitmask +> params below. + +--- + +## 1. EKF2 parameters (external vision) + +These are enforced automatically at startup by the `px4_param_setter` node (see +below), sourced from +[`robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml`](../../robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml). +You can also set them by hand in QGroundControl — PX4 persists parameters, so +either way it's a one-time thing per airframe. + +| Parameter | Value | Meaning | +|---|---|---| +| `EKF2_EV_CTRL` | `11` | Fuse vision **horizontal pos (1) + vertical pos (2) + yaw (8)**. Add bit **4** (velocity) only if a vision *speed* source is also streamed. | +| `EKF2_HGT_REF` | `3` | Vision is the primary height reference (not baro / GPS). | +| `EKF2_GPS_CTRL` | `0` | No GPS fusion. | +| `EKF2_MAG_TYPE` | `5` | Magnetometer disabled — yaw comes from vision. | +| `EKF2_BARO_CTRL` | `0` | No baro fusion; height is pure vision. Set to `1` to keep baro as a backup height source. | +| `EKF2_EV_DELAY` | `7.0` | OptiTrack→EKF2 latency (ms): ~0.7 ms measured LAN transport + a ~5 ms *estimated* FCU hop. **Raising this does not compensate for apparent lag — it makes the estimate run ahead of truth.** | +| `EKF2_EV_NOISE_MD` | `1` | Use the `EKF2_EV*_NOISE` floors below instead of the message covariance (which is `1e-6` — too optimistic to fuse safely). | +| `EKF2_EVP_NOISE` | `0.05` | Vision **position** noise floor (m). Not marker precision — it also sets the innovation gate, `EKF2_EVP_GATE` (default 5) sigma wide, so this is a 25 cm gate. | +| `EKF2_EVA_NOISE` | `0.05` | Vision **angle** noise floor (rad). | +| `COM_ARM_WO_GPS` | `1` | Allow arming without GPS. | + +**Type matters.** Integers are written bare (`11`); floats need a decimal point +(`7.0`) so the MAVLink param type matches the FCU's declaration. Getting this +wrong makes the set silently reject. + +### Troubleshooting tips + +If you see drift-and-snap, **check the Motive PC rigid-body definition first** and ensure the x axis points forward. Then, make sure that Motive is streaming the position with z-axis up. + +### The latency figure is only partly measured + +`EKF2_EV_DELAY` is currently `7.0` ms: roughly `0.7` measured plus a `5.0` estimate +(`cube_orange_latency_ms` in `natnet_config.yaml`). Only the first part is empirically measured currently. + +- **Measured:** `natnet_ros2_node` derives transport latency from the NatNet + `TransmitTimestamp` — i.e. from *server transmit* to client receipt. It does not + include Motive's own capture→transmit pipeline (exposure, centroiding, solving), + which is typically several ms and happens before that clock starts. +- **Estimated:** `cube_orange_latency_ms` models the MAVROS → MAVLink → uORB → EKF2 hop. + It is **estimated only**. + +**Reboot after any change.** Fusion-source (`EKF2_*`) params are safest applied +from a clean estimator start — reboot the flight controller before flying. The +param setter prints a warning whenever it actually changes something. + +--- + +## 2. The param checker (`px4_param_setter`) + +Set the table above **once in QGroundControl**. To catch a mis-configured FCU +before flight, the stack runs a one-shot node at startup that **checks** the live +params against the desired set. **By default it only checks and flags — it does not +write to the FCU.** + +- **Node:** [`px4_param_setter_node.py`](../../robot/ros_ws/src/perception/natnet_ros2/src/px4_param_setter_node.py) +- **Config:** [`config/px4_params.yaml`](../../robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml) + (everything under `params.` is a desired FCU parameter) +- **Launch:** [`launch/px4_param_setter.launch.xml`](../../robot/ros_ws/src/perception/natnet_ros2/launch/px4_param_setter.launch.xml), + included from `natnet_ros2.launch.py` when the robot's `vision_pose` block is enabled. + +Two safety flags in `px4_params.yaml`: + +| Flag | Default | Behaviour | +|------|---------|-----------| +| `auto_set` | `false` | `false`: read + compare only, never write. `true`: also push mismatched params via `param/set` and verify (the legacy enforce path). | +| `on_mismatch` | `warn` | With `auto_set: false`, on a wrong param — `warn`: log the diffs, keep the stack up. `halt`: log fatal + exit non-zero so a `required` launch node tears the stack down before flight. | + +Per parameter it waits for an FCU connection + `settle_sec` (default 10 s), reads +the current value, and compares (float32 tolerance). A clean run logs +`10 already correct, 0 mismatched`. A mismatch under the default (`auto_set: false`, +`on_mismatch: warn`) logs, e.g., `EKF2_HGT_REF: FCU has 1, expected 3 (not set — +auto_set=false). Fix in QGroundControl.` + +Disable it entirely with `enabled: false`. + +> **The checker does NOT configure the companion link** (`MAV_*` / `SER_*` +> params in section 3) — those are set once in QGC. + +--- + +## 3. Companion MAVLink link (Jetson ↔ Cube) + +`mavros` reaches the FCU over the serial link named by `FCU_URL` in the deployment env. +Configuring that link is standard PX4 setup, not AirStack-specific — see the PX4 docs: + +- [Companion computer setup](https://docs.px4.io/main/en/companion_computer/) +- [MAVLink peripherals (`MAV_n_CONFIG`, `MAV_n_MODE`)](https://docs.px4.io/main/en/peripherals/mavlink_peripherals.html) +- [Serial port configuration](https://docs.px4.io/main/en/peripherals/serial_configuration.html) + +Use the **TELEM2 UART** for the companion link rather than USB. On Cube Orange the USB +CDC-ACM path intermittently stalls outbound transfers for 10–30 s at a time — visible as +`DROPPED Message-Id 102 … TX queue overflow` — which starves EKF2 of vision updates and +makes it dead-reckon between bursts. It is not a bandwidth problem and rate-limiting the +vision stream does not help. + +> In compose list-syntax `environment:`, values are literal — write `FCU_URL=/dev/ttyTHS1:115200` +> bare. Quoting it passes the quotes through and breaks MAVROS URL parsing. + + +## 4. Vision pose pipeline (mocap → PX4) + +``` +Motive (OptiTrack, 100 Hz) + → natnet_ros2_node publishes the rigid body as a ROS pose (ENU) + → vision_pose_converter rate-limit + quaternion canonicalize (passthrough) + → mavros vision_pose converts ENU→NED, sends VISION_POSITION_ESTIMATE (msg 102) + → PX4 EKF2 fuses per the params in section 1 +``` + +**Frame convention — the thing to get right.** MAVROS's `vision_pose` plugin +expects **ROS ENU** and converts to PX4 NED internally. The +[`vision_pose_converter_node.py`](../../robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py) +does **no coordinate transform** — it only rewrites `frame_id`, optionally +canonicalizes the quaternion sign (`qw ≥ 0`), and rate-limits. **So +`natnet_ros2_node` must already publish ENU.** If position/yaw come out rotated +or axis-swapped, fix it there, not in the converter. + +**Rate limiting.** `max_rate_hz` (default 50 in +[`vision_pose_converter.yaml`](../../robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml)) +caps the stream to MAVROS. EKF2 only needs 30–50 Hz. Note this is about not +saturating a healthy serial link — it does **not** fix the USB CDC stall in +section 3. + +### Injecting a global position — `mavros_gp_origin` + +Vision gives PX4 a valid *local* position, but with GNSS disabled it has no *global* +one, and modes that require a global position (e.g. `AUTO.LOITER`) refuse to arm. + +[`mavros_gp_origin_node.py`](../../robot/ros_ws/src/perception/natnet_ros2/src/mavros_gp_origin_node.py) +publishes a **synthetic GPS origin** once at startup, which lets PX4 derive a global +position from the fused vision estimate. It waits for MAVROS to connect, listens for an +existing origin, and only publishes if none is present — so a GNSS-equipped vehicle is +left untouched. Location and behaviour come from +[`mavros_gp_origin.yaml`](../../robot/ros_ws/src/perception/natnet_ros2/config/mavros_gp_origin.yaml); +the defaults match the AirStack shared world datum so sim, the GCS, and the robot agree +on where world origin sits on Earth. + +!!! note "Real deployments: the origin altitude needs a geoid correction" + + `geographic_msgs/GeoPoint.altitude` is a height above the **WGS-84 ellipsoid**, and + MAVROS converts it to AMSL with the **egm96-5 geoid** before handing it to PX4. Send + the shared datum's literal `90.0` and PX4 anchors its vertical frame at + `AMSL = 90 − N ≈ 36 m`, while OptiTrack says the floor is `z = 0` — the drone reads + ~36 m of altitude sitting on the floor. The gap is exactly the geoid undulation `N`. + + With `use_geoid_altitude: true` the node publishes `N + desired_floor_amsl` instead, + computing `N` at runtime via `GeoidEval` with the same egm96-5 model MAVROS uses, so + the conversion cancels out. + + `desired_floor_amsl` chooses what AMSL the mocap floor reports; `local_position.z` + equals the OptiTrack height either way. We use **36.0**, the shared datum in AMSL, so + the robot's global altitude agrees with sim and the GCS. + + **Not needed in sim.** The geoid path is skipped when `use_sim_time: true`: sim's + synthetic GPS is self-consistent with the spawn and uses the literal datum altitude + on both ends, so there is no ellipsoidal-vs-AMSL mismatch to correct. + + +## 5. Verify it's actually fusing + +**Live, in the QGC MAVLink _Console_** (not the Inspector — it can't see +companion→FCU messages): + +``` +listener vehicle_visual_odometry # should be steady ~50 Hz, not gappy +listener estimator_status +``` + +On the ROS side: `/{ROBOT_NAME}/interface/mavros/local_position/pose` should +publish and track the mocap. Hand-lift test: raise the vehicle, Z should go up +(Motive Z-up correct); translate it and check the sign/axis match. + +**Definitive, from the SD-card ulog** ([Flight Review](https://logs.px4.io) or +PlotJuggler): + +- `estimator_innovations` → **`ev_hpos` / `ev_vpos` / `ev_yaw`** and their + **test ratios**. Ratio > 1 ⇒ EKF2 is *rejecting* the measurement + (frame / timing / covariance). Near-zero with occasional gaps ⇒ fusing fine + but starved by dropped messages (section 3). +- `estimator_status_flags` → **`cs_ev_pos` / `cs_ev_yaw`** — confirms EV fusion + is actually active. If unset, EKF2 isn't fusing vision regardless of params. +- `vehicle_visual_odometry` rate in the log quantifies how many `102`s actually + arrived. + +--- + +## Troubleshooting quick reference + +| Symptom | Likely cause | Where to look | +|---|---|---| +| `DROPPED Message-Id 102 … TX queue overflow` | Cube USB CDC OUT stall | Section 3 → move to TELEM2 | +| mavros local pos drifts away from mocap over time | Dropped `102`s starving EKF2 | Fix link first, then recheck | +| Constant rotation between mocap and EKF2 pose | Yaw/frame misalignment | `natnet_ros2_node` frame (must be ENU); `EKF2_EV_CTRL` yaw bit | +| Axes swapped / uncorrelated | Wrong frame convention | `natnet_ros2_node`, not the converter | +| Param set "rejected or readback mismatch" | Wrong literal type (int vs float) | Section 1 — floats need a decimal point | +| Won't arm | GPS still required | `COM_ARM_WO_GPS: 1`, reboot | +| EV innovation test ratio > 1 | EKF2 rejecting vision | Retune `EKF2_EV_DELAY`, `EKF2_EVP_NOISE` / `EKF2_EVA_NOISE` | diff --git a/mkdocs.yml b/mkdocs.yml index 4b77be708..f45ef0963 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -118,6 +118,7 @@ nav: - Perception: - docs/robot/autonomy/perception/index.md - NatNet (OptiTrack): robot/ros_ws/src/perception/natnet_ros2/README.md + - PX4 External Vision (mocap): docs/robot/px4_external_vision.md - Local: - docs/robot/autonomy/local/index.md - World Model: diff --git a/overrides/l4t-optitrack-realrobot.env b/overrides/l4t-optitrack-realrobot.env new file mode 100644 index 000000000..c88253419 --- /dev/null +++ b/overrides/l4t-optitrack-realrobot.env @@ -0,0 +1,35 @@ +# Real-robot deployment on an NVIDIA Jetson (aarch64 / l4t) flying on OptiTrack mocap: +# PX4 EKF2 fuses the mocap pose as external vision instead of GPS. Use +# overrides/l4t-px4-realrobot.env instead if the vehicle flies on GPS. +# +# Build: airstack image-build --profile l4t robot-l4t +# Run: airstack up --env-file overrides/l4t-optitrack-realrobot.env robot-l4t +# +# Setup guide (PX4 parameters, frames, troubleshooting): +# docs/robot/px4_external_vision.md + +COMPOSE_PROFILES="l4t" +AUTOLAUNCH="true" +NUM_ROBOTS="1" +AUTONOMY_ROLE="full" + +# --- Robot identity ----------------------------------------------------------- +# Resolved from this device's hostname: name the Jetson robot-1 on the HOST +# hostnamectl set-hostname robot-1 -> robot_1 on domain 1 + +# --- OptiTrack / NatNet ------------------------------------------------------- +LAUNCH_NATNET="true" +# Motive host. No sensible default — set this before the first flight. +NATNET_SERVER_IP="192.168.1.100" + +# --- Flight controller (MAVROS) ---------------------------------------------- +# Jetson UART; some airframes wire the FCU through USB-serial instead +# (e.g. /dev/ttyUSB0:921600). +FCU_URL="/dev/ttyTHS4:115200" + +# --- Robot description -------------------------------------------------------- +URDF_FILE="robot_descriptions/iris/urdf/iris_with_sensors.pegasus.robot.urdf" + +# --- Flight-data recording ---------------------------------------------------- +BAG_STORAGE_PATH="/media/airlab/Storage/airstack_collection" +RECORD_BAGS="false" diff --git a/robot/docker/Dockerfile.robot b/robot/docker/Dockerfile.robot index ad73100eb..13ab03102 100644 --- a/robot/docker/Dockerfile.robot +++ b/robot/docker/Dockerfile.robot @@ -361,7 +361,12 @@ RUN apt-get update && apt-get install -y --no-install-recommends \ && fc-cache -f -v \ && rm -rf /var/lib/apt/lists/* -RUN /opt/ros/${ROS_DISTRO}/lib/mavros/install_geographiclib_datasets.sh +# MAVROS requires geographiclib-tools to be installed for any offboard control to work. +RUN apt-get update \ + && apt-get install -y --no-install-recommends geographiclib-tools \ + && /opt/ros/${ROS_DISTRO}/lib/mavros/install_geographiclib_datasets.sh \ + && test -f /usr/share/GeographicLib/geoids/egm96-5.pgm \ + && rm -rf /var/lib/apt/lists/* # Install DDS Router runtime library dependencies + OpenVDB RUN apt update && apt install -y --no-install-recommends \ diff --git a/robot/docker/robot-base-docker-compose.yaml b/robot/docker/robot-base-docker-compose.yaml index 8cb714e18..793cf6cab 100644 --- a/robot/docker/robot-base-docker-compose.yaml +++ b/robot/docker/robot-base-docker-compose.yaml @@ -21,6 +21,8 @@ services: - ONBOARD_BASE_PORT=${ONBOARD_BASE_PORT} - ROBOT_NAME_MAP_CONFIG_FILE=${ROBOT_NAME_MAP_CONFIG_FILE:-default_robot_name_map.yaml} - DEBUG_RVIZ=${DEBUG_RVIZ:-false} + # OptiTrack / NatNet + - NATNET_SERVER_IP=${NATNET_SERVER_IP:-172.31.0.200} volumes: # display stuff - $HOME/.Xauthority:/.Xauthority diff --git a/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt b/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt index ff47a00da..7d762f688 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt +++ b/robot/ros_ws/src/perception/natnet_ros2/CMakeLists.txt @@ -48,8 +48,7 @@ if(EXISTS "${_NATNET_LIB}" AND EXISTS "${_NATNET_INC}") # Install libNatNet.so alongside the node and register an environment hook so # that sourcing the workspace adds lib/natnet_ros2/ to LD_LIBRARY_PATH. - # Use PROGRAMS (not FILES) to preserve the execute bit — shared libraries - # must be executable for the dynamic linker to map them. + # Use PROGRAMS (not FILES) to preserve the execute bit install(PROGRAMS "${_NATNET_LIB}" DESTINATION lib/${PROJECT_NAME}) @@ -65,10 +64,12 @@ else() endif() # --------------------------------------------------------------------------- -# Python nodes (vision_pose_converter remains Python) +# Python nodes # --------------------------------------------------------------------------- install(PROGRAMS src/vision_pose_converter_node.py + src/mavros_gp_origin_node.py + src/px4_param_setter_node.py DESTINATION lib/${PROJECT_NAME}) # --------------------------------------------------------------------------- diff --git a/robot/ros_ws/src/perception/natnet_ros2/README.md b/robot/ros_ws/src/perception/natnet_ros2/README.md index eb44763b4..8a3970b4f 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/README.md +++ b/robot/ros_ws/src/perception/natnet_ros2/README.md @@ -15,8 +15,9 @@ This module provides a bridge between OptiTrack Motive motion capture systems an - Receives **NatNet UDP packets** from an external Motive PC (configurable IP/port) - **Decodes motion capture frames** containing rigid body positions and orientations - **Publishes pose data** to the AirStack perception layer in standard ROS 2 formats -- **Supports multi-robot** via ROBOT_NAME namespacing -- **Optionally bridges** to MAVROS for PX4 external pose feedback +- **Tracks multiple rigid bodies per robot** (e.g. a drone for state estimation plus a separate target), each mapped to its own topic +- **Supports multi-robot** via per-robot profiles selected by `ROBOT_NAME` +- **Optionally bridges** to MAVROS for PX4 external pose feedback (per-robot) - **Respects OptiTrack licensing** by keeping the NatNet SDK external (host-side download with explicit consent) ## Architecture @@ -25,13 +26,18 @@ This module provides a bridge between OptiTrack Motive motion capture systems an Motive (External PC) ↓ NatNet UDP (port 1511) ↓ -NatNet ROS 2 Node - ├→ /robot_1/perception/optitrack/{body_name} (PoseStamped, optional) - ├→ /robot_1/perception/optitrack/{body_name}/pose_cov (PoseWithCovarianceStamped, always) - └→ (Optional, publish_to_mavros: true) - vision_pose_converter_node - ├→ /robot_1/mavros/vision_pose/pose - └→ /robot_1/mavros/vision_pose/pose_cov +NatNet ROS 2 Node (loads the ROBOT_NAME profile from natnet_config.yaml) + │ per configured body (one or more): + ├→ /{ROBOT_NAME}/{topic} (PoseStamped, when pose: true) + ├→ /{ROBOT_NAME}/{topic}/pose_cov (PoseWithCovarianceStamped, when pose_cov: true) + └→ (Optional, vision_pose.enabled: true) + mavros_gp_origin_node + └→ /{ROBOT_NAME}/interface/mavros/global_position/set_gp_origin + px4_param_setter_node + └→ /{ROBOT_NAME}/interface/mavros/param/set (external-vision PX4 params) + vision_pose_converter_node (reads input/output topics from the profile) + ├→ /{ROBOT_NAME}/interface/mavros/vision_pose/pose + └→ /{ROBOT_NAME}/interface/mavros/vision_pose/pose_cov ``` ## Interfaces @@ -39,63 +45,122 @@ NatNet ROS 2 Node ### Inputs - **Network**: NatNet UDP stream from Motive PC (external network) -- **Configuration**: `natnet_config.yaml` with server IP, ports, `body_name`, and covariance +- **Configuration**: `natnet_config.yaml` — generic `server` settings plus a `robots` map of per-robot profiles (body list + optional MAVROS `vision_pose` block). The launch file selects the profile matching `ROBOT_NAME`. ### Outputs -For each tracked rigid body `{body_name}` from Motive: +For each rigid body in the robot's profile, `topic` is a **relative** leaf namespaced +under `/{ROBOT_NAME}/` (it defaults to `perception/optitrack/{rigid_body_name}` when +omitted): -#### Direct OptiTrack pose (optional) +#### Direct OptiTrack pose -- **Topic**: `/{ROBOT_NAME}/perception/optitrack/{body_name}` +- **Topic**: `/{ROBOT_NAME}/{topic}` - **Type**: `geometry_msgs/PoseStamped` - **Description**: Position and orientation only (no covariance) -- **Enabled by**: `publish_direct_optitrack: true` in config (default: `true`) +- **Enabled by**: `pose: true` on that body (per body) -#### Pose with covariance (always) +#### Pose with covariance -- **Topic**: `/{ROBOT_NAME}/perception/optitrack/{body_name}/pose_cov` +- **Topic**: `/{ROBOT_NAME}/{topic}/pose_cov` - **Type**: `geometry_msgs/PoseWithCovarianceStamped` -- **Description**: Same pose as above plus a 6×6 covariance matrix (`position_covariance` and `orientation_covariance` from config). Published whenever the rigid body is tracked — independent of `publish_direct_optitrack` and `publish_to_mavros`. - -#### MAVROS vision pose bridge (optional) - -When `publish_to_mavros: true`, `vision_pose_converter_node` subscribes to `pose_cov` and republishes for PX4: - -- **Topic**: `/{ROBOT_NAME}/mavros/vision_pose/pose` — `geometry_msgs/PoseStamped` (pose extracted from the covariance message) -- **Topic**: `/{ROBOT_NAME}/mavros/vision_pose/pose_cov` — `geometry_msgs/PoseWithCovarianceStamped` (full message, quaternion optionally canonicalized) -- **Enabled by**: `publish_to_mavros: true` in config +- **Description**: Same pose plus a 6×6 covariance matrix from that body's `position_covariance` / `orientation_covariance`. +- **Enabled by**: `pose_cov: true` on that body (per body) + +#### MAVROS vision pose bridge (optional, per robot) + +When the robot's `vision_pose.enabled: true`, `vision_pose_converter_node` subscribes to the configured `input_topic` (a body's `pose_cov`) and republishes for PX4 on the configured outputs: + +- **Topic** (`output_pose_topic`): `/{ROBOT_NAME}/interface/mavros/vision_pose/pose` — `geometry_msgs/PoseStamped` (pose extracted from the covariance message) +- **Topic** (`output_pose_cov_topic`): `/{ROBOT_NAME}/interface/mavros/vision_pose/pose_cov` — `geometry_msgs/PoseWithCovarianceStamped` (full message, quaternion optionally canonicalized) +- **Enabled by**: `vision_pose.enabled: true` in the robot's profile +- **Retargetable**: change `input_topic` / `output_pose_topic` / `output_pose_cov_topic` (relative, namespaced) to bridge to other middleware +- **PX4 side**: set `SITL_PARAM_PROFILE=px4-vision` in `.env` so Isaac SITL loads EKF2 external-vision params from `simulation/isaac-sim/docker/sitl-files/px4-vision.env` + +##### Synthetic GPS origin (mocap / no-GNSS arming) + +With GNSS disabled (`EKF2_GPS_CTRL=0`), PX4 fused EKF has **no global position**. This fails preflight checks and refuse to arm. When `vision_pose.enabled: true`, +`mavros_gp_origin_node` publishes a synthetic origin once at startup: + +- **Topic**: `/{ROBOT_NAME}/interface/mavros/global_position/set_gp_origin` — `geographic_msgs/GeoPointStamped` +- **Guarded**: waits for `mavros/state.connected`, then publishes only if no + origin already exists (it watches `…/global_position/gp_origin`), so a + GNSS-equipped vehicle is left untouched. +- **Params** (`config/mavros_gp_origin.yaml`): `enabled` (default `true`), + `latitude/longitude/altitude` (default Lisbon — the AirStack shared world + datum; **must match** the GCS origin in `gcs_visualizer/gcs_utils.py` and the + sim's `gps_utils.py` so Foxglove waypoints transform 1:1), `settle_sec`. + Set `enabled: false` to rely on real GNSS. + +##### PX4 parameter enforcement (external-vision EKF2 setup) + +When `vision_pose.enabled: true`, `px4_param_setter_node` pushes the PX4 +parameter set for OptiTrack-only flight through the MAVROS param plugin at +startup, so the FCU doesn't need manual QGroundControl configuration: + +- **Services used**: `/{ROBOT_NAME}/interface/mavros/param/get_parameters` + (read current), `…/param/set` (`mavros_msgs/ParamSetV2`, set + verify readback) +- **Idempotent**: waits for `mavros/state.connected` + `settle_sec` (initial + param-table pull), reads each param first, and skips ones already correct — + PX4 persists parameters, so subsequent boots are a verify-only pass. +- **Reboot warning**: if any parameter actually changed, it logs a warning to + reboot the FCU before flight so EKF2 restarts with a clean fusion config. +- **Params** (`config/px4_params.yaml`): `enabled`, `settle_sec`, + `retry_period_sec`, `max_attempts`, and the `params.*` map of desired FCU + values — external-vision fusion (`EKF2_EV_CTRL: 11`, `EKF2_HGT_REF: 3`), + GPS/mag/baro disabled (`EKF2_GPS_CTRL: 0`, `EKF2_MAG_TYPE: 5`, + `EKF2_BARO_CTRL: 0`), measured vision delay (`EKF2_EV_DELAY: 6.0` ms), and + EV noise floors (`EKF2_EV_NOISE_MD: 1`, `EKF2_EVP_NOISE`, `EKF2_EVA_NOISE`). + YAML type selects the MAVLink param type: write floats with a decimal point + (`6.0`), integers bare. Values assume PX4 ≥ 1.14; for older firmware use + `EKF2_AID_MASK: 24` / `EKF2_HGT_MODE: 3` instead. ## Configuration -Edit `config/natnet_config.yaml`: +`config/natnet_config.yaml` uses a custom `natnet:` schema (not a flat ROS 2 param +file): generic `server` settings shared by every agent, then a `robots` map of +per-robot profiles. The launch file parses it, selects the profile matching the +container's `ROBOT_NAME`, flattens the body list into node parameters, and brings up +the MAVROS bridge only when that robot's `vision_pose.enabled` is true. ```yaml -/**: - ros__parameters: - server_ip: "192.168.1.100" # IP of the Motive PC +natnet: + server: # generic across all agents + server_ip: "$(env NATNET_SERVER_IP 172.31.0.200)" client_ip: "0.0.0.0" command_port: 1510 data_port: 1511 - connection_type: "unicast" # or "multicast" - - body_name: "Drone" # rigid body name in Motive (case-sensitive) - body_id: -1 # -1 = publish all bodies in the frame - - publish_direct_optitrack: true # PoseStamped on …/optitrack/{body_name} - publish_to_mavros: false # include vision_pose_converter → MAVROS - + connection_type: "unicast" # or "multicast" + multicast_address: "239.255.42.99" frame_id: "world" - - position_covariance: [0.1, 0.0, 0.0, 0.0, 0.1, 0.0, 0.0, 0.0, 0.1] - orientation_covariance: [0.01, 0.0, 0.0, 0.0, 0.01, 0.0, 0.0, 0.0, 0.01] + debug: false + robots: + robot_1: + vision_pose: # per-robot MAVROS bridge (omit/false to skip) + enabled: true + input_topic: "perception/optitrack/drone/pose_cov" + output_pose_topic: "interface/mavros/vision_pose/pose" + output_pose_cov_topic: "interface/mavros/vision_pose/pose_cov" + bodies: # one or more tracked rigid bodies + - rigid_body_name: "Drone" # Motive name (case-sensitive) + id: 1 # Motive streaming id + topic: "perception/optitrack/drone" # relative → /{ROBOT_NAME}/ + pose: true # publish PoseStamped + pose_cov: true # publish PoseWithCovarianceStamped + position_covariance: [1.0e-6, 0, 0, 0, 1.0e-6, 0, 0, 0, 1.0e-6] + orientation_covariance: [3.0e-6, 0, 0, 0, 3.0e-6, 0, 0, 0, 3.0e-6] ``` +To track an additional body (e.g. a target) for a robot, add another entry under that +robot's `bodies`. To add a robot, add a new key under `robots`. The shipped file +includes commented scaffolding for a 3-drone fleet where `robot_1` and `robot_2` also +track a shared `Target` body and `robot_3` tracks only its drone. + ## Launch ### Basic launch -Parameters come from `config/natnet_config.yaml` (network, body, covariance). Optional overrides: +Parameters come from `config/natnet_config.yaml` (server + the `ROBOT_NAME` profile). Optional overrides: ```bash ros2 launch natnet_ros2 natnet_ros2.launch.py \ @@ -106,7 +171,7 @@ ros2 launch natnet_ros2 natnet_ros2.launch.py \ ### MAVROS bridge -Set `publish_to_mavros: true` in `natnet_config.yaml`. The launch file reads `publish_to_mavros` and `body_name` from that YAML to decide whether to include `vision_pose_converter.launch.xml`. +Set `vision_pose.enabled: true` in the robot's profile. The launch file includes `vision_pose_converter.launch.xml` (plus `mavros_gp_origin.launch.xml` and `px4_param_setter.launch.xml`) and forwards the profile's `input_topic` / `output_pose_topic` / `output_pose_cov_topic`. ### From perception bringup @@ -140,13 +205,18 @@ The SDK will be installed into `robot/ros_ws/src/perception/natnet_ros2/lib/` an ### Multi-Robot Support Each container instance gets its own `ROBOT_NAME` and `ROS_DOMAIN_ID`: -- Topics: `/{ROBOT_NAME}/perception/optitrack/{body_name}` and `/{ROBOT_NAME}/perception/optitrack/{body_name}/pose_cov` -- Supported via launch file argument forwarding +- The node loads the `robots[$ROBOT_NAME]` profile, so each robot tracks only the bodies (and runs the MAVROS bridge) configured for it. +- Topics are namespaced under `/{ROBOT_NAME}/` from each body's relative `topic`. +- Set `NUM_ROBOTS=N`; each replica resolves its own `ROBOT_NAME` (via `resolve_robot_name.py`) and auto-selects its profile — no per-robot env overrides. ### Error Handling - Invalid/malformed packets are skipped with debug logging - Lost connectivity logs warnings; gracefully recovers when stream resumes - Covariance in config allows tuning uncertainty per deployment +- **Connect retry:** the initial handshake is retried every 2 s until it + succeeds, so the node tolerates the NatNet server starting *after* the robot + (e.g. a Motive PC powered on later, or the Isaac Sim NatNet emulator which only + binds ~100 s into sim boot). The retry timer cancels itself on first success. ## Testing @@ -157,9 +227,9 @@ Each container instance gets its own `ROBOT_NAME` and `ROS_DOMAIN_ID`: ```bash ros2 launch natnet_ros2 natnet_ros2.launch.py ``` -4. Verify topics: +4. Verify topics (default profile maps the `Drone` body to `perception/optitrack/drone`): ```bash - ros2 topic echo /robot_1/perception/optitrack/Drone/pose_cov + ros2 topic echo /robot_1/perception/optitrack/drone/pose_cov ``` ### Without Real Hardware (Mock) @@ -167,7 +237,7 @@ TODO: Implement Motive simulator in Isaac Sim to generate fake NatNet packets ## Known Limitations -- When `body_id: -1`, all rigid bodies in the Motive frame get publishers; filter by subscribing to the `{body_name}` you care about +- The node publishes only bodies listed in the robot's profile (matched by `id`); bodies streamed by Motive but absent from the profile are ignored. - MAVROS bridge applies frame_id override and quaternion canonicalization; full PX4 frame alignment may still need tuning per airframe - No support for skeleton tracking or labeled markers yet (future enhancement) diff --git a/robot/ros_ws/src/perception/natnet_ros2/config/mavros_gp_origin.yaml b/robot/ros_ws/src/perception/natnet_ros2/config/mavros_gp_origin.yaml new file mode 100644 index 000000000..0035f017b --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/config/mavros_gp_origin.yaml @@ -0,0 +1,22 @@ +# Synthetic GPS origin for mocap / no-GNSS flight via MAVROS. +# Loaded by mavros_gp_origin.launch.xml when publish_to_mavros is enabled. +# See docs/robot/px4_external_vision.md for the height-datum explanation. + +/**: + ros__parameters: + # Skipped if an origin already exists (e.g. a GNSS-equipped vehicle). + enabled: true + # MUST match the GCS world origin (gcs_visualizer/gcs_utils.py) and the sim + # datum (launch_scripts/gps_utils.py), or the relay computes a huge ENU offset. + latitude: 38.736832 + longitude: -9.137977 + # Shared world datum; used directly when use_geoid_altitude is false or in sim. + altitude: 90.0 + # Real hardware: derive origin altitude from the geoid so local_position z + # equals OptiTrack height. Auto-skipped in sim. + use_geoid_altitude: true + # AMSL of the mocap floor. 36.0 = the shared world datum (90 m ellipsoidal) in AMSL, + # so the robot's global altitude agrees with sim and the GCS. + desired_floor_amsl: 36.0 + geoid_model: "egm96-5" + settle_sec: 5.0 diff --git a/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml b/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml index 69fc11d1c..6ad16e074 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml +++ b/robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml @@ -1,51 +1,101 @@ -# NatNet ROS 2 parameters — loaded by natnet_ros2.launch.py (NatNet node + MAVROS gate). -# publish_to_mavros / body_name are read by the launch file to decide vision_pose_converter include. -# -# Use /** so parameters apply regardless of namespace (e.g. /robot_1/perception/natnet_ros2_node). -# See: https://docs.ros.org/en/humble/Tutorials/Beginner-CLI-Tools/Understanding-ROS2-Parameters.html - -/**: - ros__parameters: - # IP address of the PC running Motive (OptiTrack server). - # Change this to match your local network before launching NatNet. - server_ip: "192.168.1.100" - # Motive learns unicast destination from outbound UDP source IP — bind explicitly when you have - # multiple NICs (e.g. Docker 172.17.* vs LAN). +# NatNet ROS 2 configuration — parsed by natnet_ros2.launch.py, which selects the +# profile matching the container's ROBOT_NAME. +# See docs/robot/px4_external_vision.md for the schema and setup guide. + +natnet: + + # --- Connection settings (generic across all agents) ----------------------- + server: + # Motive host; defaults to the in-sim emulator. Set NATNET_SERVER_IP per deployment. + server_ip: "$(env NATNET_SERVER_IP 172.31.0.200)" + # Bind explicitly when the client has multiple NICs. client_ip: "0.0.0.0" command_port: 1510 data_port: 1511 - # "unicast" — point-to-point; Motive streams directly to this machine's IP. - # Requires Motive unicast streaming enabled and client_ip set - # to the correct NIC when multiple interfaces are present. - # "multicast" — Motive broadcasts to a multicast group; any machine on the - # subnet that joins the group receives all body data. - # Use for multi-robot setups where every robot receives the - # full frame and filters by body_id. + # "unicast" (default) or "multicast"; multicast_address applies to the latter. connection_type: "unicast" - - # Only used when connection_type = "multicast". - # Must match Motive > Edit > Preferences > Data Streaming > Multicast Interface. - # OptiTrack default is 239.255.42.99. multicast_address: "239.255.42.99" - # Name of the rigid body as defined in Motive. Must match exactly (case-sensitive). - body_name: "Drone" - body_id: -1 - - publish_direct_optitrack: true - publish_to_mavros: true - frame_id: "world" debug: false - position_covariance: - [0.1, 0.0, 0.0, - 0.0, 0.1, 0.0, - 0.0, 0.0, 0.1] + # Per-message latency reporting. + latency_sampling_warmup_s: 5.0 + latency_sampling_window_s: 20.0 + cube_orange_latency_ms: 5.0 + + # --- Per-robot profiles (selected by ROBOT_NAME) --------------------------- + robots: + + robot_1: + # MAVROS vision_pose bridge; enabled=false skips the converter. + vision_pose: + enabled: true + input_topic: "perception/optitrack/drone/pose_cov" + output_pose_topic: "interface/mavros/vision_pose/pose" + output_pose_cov_topic: "interface/mavros/vision_pose/pose_cov" - orientation_covariance: - [0.01, 0.0, 0.0, - 0.0, 0.01, 0.0, - 0.0, 0.0, 0.01] + # Rigid bodies this robot tracks. name and id must match Motive exactly; the + # client filters frames by numeric id. Defaults match the in-sim emulator. + bodies: + - rigid_body_name: "Drone" + id: 1 + topic: "perception/optitrack/drone" + pose: true + pose_cov: true + position_covariance: + [1.0e-6, 0.0, 0.0, + 0.0, 1.0e-6, 0.0, + 0.0, 0.0, 1.0e-6] + orientation_covariance: + [3.0e-6, 0.0, 0.0, + 0.0, 3.0e-6, 0.0, + 0.0, 0.0, 3.0e-6] + robot_2: + vision_pose: + enabled: true + input_topic: "perception/optitrack/drone/pose_cov" + output_pose_topic: "interface/mavros/vision_pose/pose" + output_pose_cov_topic: "interface/mavros/vision_pose/pose_cov" + bodies: + - rigid_body_name: "Drone2" + id: 2 + topic: "perception/optitrack/drone" + pose: true + pose_cov: true + - rigid_body_name: "Target" # shared target — also tracked by robot_1 + id: 100 + topic: "perception/optitrack/target" + pose: true + pose_cov: false + position_covariance: + [1.0e-6, 0.0, 0.0, + 0.0, 1.0e-6, 0.0, + 0.0, 0.0, 1.0e-6] + orientation_covariance: + [3.0e-6, 0.0, 0.0, + 0.0, 3.0e-6, 0.0, + 0.0, 0.0, 3.0e-6] + + robot_3: + vision_pose: + enabled: true + input_topic: "perception/optitrack/drone/pose_cov" + output_pose_topic: "interface/mavros/vision_pose/pose" + output_pose_cov_topic: "interface/mavros/vision_pose/pose_cov" + bodies: + - rigid_body_name: "Drone3" + id: 3 + topic: "perception/optitrack/drone" + pose: true + pose_cov: true + position_covariance: + [1.0e-6, 0.0, 0.0, + 0.0, 1.0e-6, 0.0, + 0.0, 0.0, 1.0e-6] + orientation_covariance: + [3.0e-6, 0.0, 0.0, + 0.0, 3.0e-6, 0.0, + 0.0, 0.0, 3.0e-6] diff --git a/robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml b/robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml new file mode 100644 index 000000000..c2bd3b7be --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml @@ -0,0 +1,43 @@ +# PX4 parameters for OptiTrack-only (external vision) flight, checked at startup by +# px4_param_setter. See docs/robot/px4_external_vision.md for what each one does, the +# tuning rationale, and how to set them in QGroundControl. +# +# TYPE MATTERS: integers bare (11), floats with a decimal point (7.0), so the MAVLink +# param type matches the FCU's declaration. +# +# Values assume PX4 >= 1.14. For older firmware use EKF2_AID_MASK: 24 and +# EKF2_HGT_MODE: 3 instead. + +/**: + ros__parameters: + enabled: true + # Check-only by default: read the FCU's params and flag differences, never write. + auto_set: false + # On mismatch with auto_set:false — 'warn' (log diffs) or 'halt' (exit non-zero). + on_mismatch: "warn" + # Initial full param pull over serial (115200) is slow; give it time. + settle_sec: 10.0 + retry_period_sec: 2.0 + max_attempts: 30 + + params: + # Fuse vision horizontal position (1) + vertical position (2) + yaw (8). + EKF2_EV_CTRL: 11 + # Vision is the height reference. + EKF2_HGT_REF: 3 + EKF2_GPS_CTRL: 0 + # Magnetometer off; yaw comes from vision. + EKF2_MAG_TYPE: 5 + EKF2_BARO_CTRL: 0 + # Remove the baro at system level, not just from fusion, so the height datum is + # deterministic on every boot. WARNING: no baro backup — indoor mocap only. + SYS_HAS_BARO: 0 + EKF2_RNG_CTRL: 0 + # Do NOT raise to chase apparent lag; higher is measurably worse (see docs). + EKF2_EV_DELAY: 7.0 + # Use the NOISE floors below rather than the message covariance. + EKF2_EV_NOISE_MD: 1 + # Also sets the innovation gate (EKF2_EVP_GATE sigma wide): 0.05 -> ~25 cm. + EKF2_EVP_NOISE: 0.05 + EKF2_EVA_NOISE: 0.05 + COM_ARM_WO_GPS: 1 diff --git a/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml b/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml index a52181786..f14b4d2d7 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml +++ b/robot/ros_ws/src/perception/natnet_ros2/config/vision_pose_converter.yaml @@ -1,12 +1,13 @@ # Vision pose converter → MAVROS bridge parameters. # Loaded by vision_pose_converter.launch.xml via . -# $(env ROBOT_NAME ...) is expanded by launch substitution. /**: ros__parameters: frame_id: "world" child_frame_id: "$(env ROBOT_NAME robot_1)/base_link" # Normalise quaternion to canonical form (qw >= 0) before publishing. - # Recommended for ArduPilot EKF3 and any consumer sensitive to sign flips. - # PX4 EKF2 handles either sign internally, so this is optional for PX4. canonical_quaternion: true + # Cap the rate forwarded to MAVROS (0 = passthrough); EKF2 needs only 30-50 Hz. + max_rate_hz: 50.0 + # Which vision_pose topic(s) to forward: 'pose', 'pose_cov', or 'both'. + publish_mode: "pose_cov" diff --git a/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp index 04b3638ef..b64b254df 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp +++ b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_client_adapter.hpp @@ -52,9 +52,20 @@ class NatNetClientAdapter : public INatNetClient void set_frame_callback(std::function cb) override; void disconnect() override; + // Context handed to the SDK's C frame callback. Bundles the client (needed to + // convert TransmitTimestamp → latency via SecondsSinceHostTimestamp) with the + // user callback, since the SDK passes only a single void* through. Public so the + // file-scope trampoline in the .cpp can reinterpret the void* ctx. + struct FrameCallbackCtx + { + NatNetClient * client = nullptr; + std::function * cb = nullptr; + }; + private: std::unique_ptr client_; std::function user_cb_; + FrameCallbackCtx cb_ctx_{}; }; } // namespace natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp index f23216565..f2c93b838 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp +++ b/robot/ros_ws/src/perception/natnet_ros2/include/natnet_ros2/natnet_logic.hpp @@ -18,25 +18,17 @@ // OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE // SOFTWARE. -// natnet_logic.hpp — pure C++ helpers for natnet_ros2 (no ROS, no NatNet SDK). -// -// Five responsibility areas: -// -// 1. Covariance assembly -// 2. Topic names -// 3. Connection-configuration helpers (SDK-independent) -// 4. Rigid-body frame helpers (SDK-independent) -// 5. Abstraction seam: INatNetClient interface + negotiation logic -// -// NatNet SDK types (sNatNetClientConnectParams, sRigidBodyData, …) are only -// used inside natnet_ros2_node.cpp and natnet_client_adapter.cpp. -// All logic here uses plain C++ so test_natnet_logic.cpp compiles with only gtest. +// natnet_logic.hpp — pure C++ helpers for natnet_ros2 (no ROS, no NatNet SDK), so +// test_natnet_logic.cpp compiles with only gtest. SDK types stay in +// natnet_ros2_node.cpp / natnet_client_adapter.cpp. #pragma once +#include #include #include #include +#include #include #include @@ -91,16 +83,48 @@ inline std::string optitrack_pose_cov_topic( return optitrack_topic_base(robot_name, body_name) + "/pose_cov"; } +/// Namespace a relative topic leaf under /{robot_name}/. +/// +/// Leading slashes in \p relative are stripped so the result always has exactly +/// one. Used for the per-body ``topic`` overrides in natnet_config.yaml, which are +/// relative and namespaced by the node at runtime. +inline std::string namespaced_topic( + const std::string & robot_name, + const std::string & relative) +{ + const std::size_t start = relative.find_first_not_of('/'); + const std::string leaf = + (start == std::string::npos) ? std::string{} : relative.substr(start); + return "/" + robot_name + "/" + leaf; +} + +/// Topic base for one configured body: the per-body relative override when set, +/// otherwise the default /{robot_name}/perception/optitrack/{body_name}. +inline std::string body_topic_base( + const std::string & robot_name, + const std::string & body_name, + const std::string & relative_override) +{ + if (relative_override.empty()) { + return optitrack_topic_base(robot_name, body_name); + } + return namespaced_topic(robot_name, relative_override); +} + // =========================================================================== // 3. Connection-configuration helpers // =========================================================================== -/// Return ct if it is "unicast" or "multicast"; otherwise return "unicast". +/// Return ct if it is "unicast" or "multicast"; otherwise throw. +/// +/// Deliberately strict: silently falling back to "unicast" turns a typo into a +/// vehicle that connects to the wrong transport and never receives frames. inline std::string validate_connection_type(const std::string & ct) { if (ct == "unicast" || ct == "multicast") { return ct; } - return "unicast"; + throw std::invalid_argument( + "connection_type must be \"unicast\" or \"multicast\", got \"" + ct + "\""); } /// SDK-independent connection configuration aggregate. @@ -113,12 +137,12 @@ struct ConnectConfig std::string client_ip = "0.0.0.0"; uint16_t command_port = 1510u; uint16_t data_port = 1511u; - std::string connection_type = "unicast"; ///< validated + std::string connection_type = "unicast"; ///< "unicast" or "multicast" std::string multicast_address = "239.255.42.99"; }; /// Build a validated ConnectConfig from raw user-supplied strings. -/// connection_type is normalised via validate_connection_type(). +/// Throws std::invalid_argument when connection_type is not "unicast"/"multicast". inline ConnectConfig make_connect_config( const std::string & server_ip, const std::string & client_ip, @@ -177,6 +201,9 @@ struct FrameSample int32_t frame_num = 0; float timestamp = 0.f; int16_t params = 0; ///< NatNet frame.params bitmask + /// transit + client-processing latency the drone observes per message. + double transit_latency_s = 0.0; + bool has_latency = false; std::vector bodies; }; @@ -199,6 +226,15 @@ inline bool should_publish_body(int32_t filter_id, int32_t rb_id) return filter_id < 0 || rb_id == filter_id; } +/// Returns true when rb_id is one of the configured body ids. +/// +/// The node publishes only a fixed set of ids based on natnet_config.yaml. +inline bool body_is_configured(const std::vector & configured_ids, int32_t rb_id) +{ + return std::find(configured_ids.begin(), configured_ids.end(), rb_id) + != configured_ids.end(); +} + /// Double-precision pose extracted from a RigidBodySample. struct PoseData { diff --git a/robot/ros_ws/src/perception/natnet_ros2/launch/mavros_gp_origin.launch.xml b/robot/ros_ws/src/perception/natnet_ros2/launch/mavros_gp_origin.launch.xml new file mode 100644 index 000000000..4cb1f785b --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/launch/mavros_gp_origin.launch.xml @@ -0,0 +1,34 @@ + + + + + + + + + + + + + + + + + + + diff --git a/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py b/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py index cb7c178d8..440eaf8d6 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py +++ b/robot/ros_ws/src/perception/natnet_ros2/launch/natnet_ros2.launch.py @@ -1,5 +1,10 @@ #!/usr/bin/env python3 -"""Bring up NatNet node; optionally MAVROS bridge per natnet_config.yaml. +"""Bring up the NatNet node from natnet_config.yaml; optionally the MAVROS bridge. + +The config uses a custom ``natnet:`` schema (server settings + per-robot profiles), +so this launch file parses it, selects the profile matching ``ROBOT_NAME``, flattens +the body list into node parameters, and — when the robot's ``vision_pose`` block is +enabled — includes the MAVROS GP-origin + vision_pose_converter bridges. natnet_ros2_node is a C++ executable that requires the OptiTrack NatNet SDK. If the SDK was not installed (``airstack setup`` not run) and the workspace @@ -10,8 +15,9 @@ from __future__ import annotations import os +import re from pathlib import Path -from typing import cast +from typing import Any, cast import yaml from ament_index_python.packages import get_package_share_directory @@ -20,11 +26,28 @@ from launch.launch_description_sources import FrontendLaunchDescriptionSource from launch.substitutions import LaunchConfiguration from launch_ros.actions import Node -from launch_ros.parameter_descriptions import ParameterFile +# Per-body covariance fallback when a body omits its own (sub-0.1 mm / sub-0.1 deg). +_DEFAULT_POSITION_COVARIANCE = [1.0e-6, 0.0, 0.0, 0.0, 1.0e-6, 0.0, 0.0, 0.0, 1.0e-6] +_DEFAULT_ORIENTATION_COVARIANCE = [3.0e-6, 0.0, 0.0, 0.0, 3.0e-6, 0.0, 0.0, 0.0, 3.0e-6] + +_ENV_SUBST = re.compile(r"\$\(env\s+(\w+)(?:\s+([^)]*))?\)") + + +def _expand_env(value: Any) -> Any: + """Expand ``$(env VAR default)`` tokens in a string using os.environ.""" + if not isinstance(value, str): + return value + + def _replace(match: re.Match) -> str: + var, default = match.group(1), match.group(2) + return os.environ.get(var, default if default is not None else "") + + return _ENV_SUBST.sub(_replace, value) -def _ros_params_from_file(config_path: str) -> dict: - """Parse /** / ros__parameters block from a ROS 2 parameter YAML.""" + +def _load_natnet_config(config_path: str) -> dict: + """Parse the ``natnet:`` block from the config YAML.""" path = Path(config_path) if not path.is_file(): return {} @@ -32,34 +55,116 @@ def _ros_params_from_file(config_path: str) -> dict: data = yaml.safe_load(f) if not isinstance(data, dict): return {} - block = data.get('/**') - if not isinstance(block, dict): - return {} - params = block.get('ros__parameters', {}) - return cast(dict, params) if isinstance(params, dict) else {} + natnet = data.get('natnet', {}) + return cast(dict, natnet) if isinstance(natnet, dict) else {} + + +def _flatten_covariance(values: Any, fallback: list[float]) -> list[float]: + """Coerce a 9-element covariance block to floats, falling back when absent.""" + if not isinstance(values, (list, tuple)) or len(values) == 0: + return list(fallback) + return [float(v) for v in values] + + +def _build_node_params(server: dict, profile: dict) -> dict: + """Flatten the server block + a robot's body list into node parameters.""" + bodies = profile.get('bodies', []) or [] + + params: dict[str, Any] = { + 'server_ip': str(_expand_env(server.get('server_ip', '172.31.0.200'))), + 'client_ip': str(_expand_env(server.get('client_ip', '0.0.0.0'))), + 'command_port': int(server.get('command_port', 1510)), + 'data_port': int(server.get('data_port', 1511)), + 'connection_type': str(server.get('connection_type', 'unicast')), + 'multicast_address': str(server.get('multicast_address', '239.255.42.99')), + 'frame_id': str(server.get('frame_id', 'world')), + 'debug': bool(server.get('debug', False)), + 'latency_sampling_warmup_s': float(server.get('latency_sampling_warmup_s', 5.0)), + 'latency_sampling_window_s': float(server.get('latency_sampling_window_s', 20.0)), + 'cube_orange_latency_ms': float(server.get('cube_orange_latency_ms', 5.0)), + } + + body_names: list[str] = [] + body_ids: list[int] = [] + body_topics: list[str] = [] + body_pose: list[bool] = [] + body_pose_cov: list[bool] = [] + body_position_covariance: list[float] = [] + body_orientation_covariance: list[float] = [] + + for body in bodies: + body_names.append(str(body.get('rigid_body_name', ''))) + body_ids.append(int(body.get('id', -1))) + body_topics.append(str(body.get('topic', ''))) + body_pose.append(bool(body.get('pose', True))) + body_pose_cov.append(bool(body.get('pose_cov', True))) + body_position_covariance.extend( + _flatten_covariance(body.get('position_covariance'), _DEFAULT_POSITION_COVARIANCE) + ) + body_orientation_covariance.extend( + _flatten_covariance(body.get('orientation_covariance'), _DEFAULT_ORIENTATION_COVARIANCE) + ) + + params.update( + { + 'body_names': body_names, + 'body_ids': body_ids, + 'body_topics': body_topics, + 'body_pose': body_pose, + 'body_pose_cov': body_pose_cov, + 'body_position_covariance': body_position_covariance, + 'body_orientation_covariance': body_orientation_covariance, + } + ) + return params + + +def _namespaced(robot_name: str, relative: str) -> str: + """Namespace a relative topic under /{robot_name}/.""" + return '/' + robot_name + '/' + relative.lstrip('/') def generate_launch_description() -> LaunchDescription: pkg_share = get_package_share_directory('natnet_ros2') default_natnet_yaml = os.path.join(pkg_share, 'config', 'natnet_config.yaml') default_vp_yaml = os.path.join(pkg_share, 'config', 'vision_pose_converter.yaml') + default_gp_origin_yaml = os.path.join(pkg_share, 'config', 'mavros_gp_origin.yaml') + default_px4_params_yaml = os.path.join(pkg_share, 'config', 'px4_params.yaml') config_file = LaunchConfiguration('config_file') vision_pose_config_file = LaunchConfiguration('vision_pose_config_file') + gp_origin_config_file = LaunchConfiguration('gp_origin_config_file') + px4_params_config_file = LaunchConfiguration('px4_params_config_file') use_sim_time = LaunchConfiguration('use_sim_time') def launch_setup(context, *_args, **_kwargs): cfg_path = config_file.perform(context) vp_path = vision_pose_config_file.perform(context) + gp_path = gp_origin_config_file.perform(context) + px4_path = px4_params_config_file.perform(context) ust = use_sim_time.perform(context) - ros_params = _ros_params_from_file(cfg_path) - publish_mavros = bool(ros_params.get('publish_to_mavros', False)) - body_name = str(ros_params.get('body_name', 'robot_1')) + robot_name = os.environ.get('ROBOT_NAME', 'robot_1') + natnet = _load_natnet_config(cfg_path) + server = natnet.get('server', {}) if isinstance(natnet, dict) else {} + robots = natnet.get('robots', {}) if isinstance(natnet, dict) else {} + profile = robots.get(robot_name, {}) if isinstance(robots, dict) else {} + + if not profile: + print( + f"[natnet_ros2.launch] WARNING: no profile for ROBOT_NAME='{robot_name}' " + f"in {cfg_path}; node will start with no tracked bodies." + ) + + node_params = _build_node_params(server, profile) + # launch_ros / rclpy cannot infer the type of an empty-list parameter, so drop + # any empty arrays; the node declares matching empty defaults and tracks nothing. + node_params = { + k: v for k, v in node_params.items() if not (isinstance(v, list) and len(v) == 0) + } # pkg_share = /share/natnet_ros2 → go up two levels to reach , # then down into lib/natnet_ros2/ where colcon installs executables. - pkg_share = get_package_share_directory('natnet_ros2') node_path = Path(pkg_share).parent.parent / 'lib' / 'natnet_ros2' / 'natnet_ros2_node' if not node_path.exists(): raise RuntimeError( @@ -75,11 +180,49 @@ def launch_setup(context, *_args, **_kwargs): executable='natnet_ros2_node', name='natnet_ros2_node', output='screen', - parameters=[ParameterFile(config_file, allow_substs=True)], + parameters=[node_params], + # The closed-source NatNet SDK can assert (SIGABRT) on connect + # in odd network states; restart rather than losing mocap. + respawn=True, + respawn_delay=2.0, ), ] - if publish_mavros: + vision_pose = profile.get('vision_pose', {}) if isinstance(profile, dict) else {} + if vision_pose.get('enabled', False): + input_topic = _namespaced( + robot_name, str(vision_pose.get('input_topic', 'perception/optitrack/drone/pose_cov')) + ) + output_pose_topic = _namespaced( + robot_name, str(vision_pose.get('output_pose_topic', 'interface/mavros/vision_pose/pose')) + ) + output_pose_cov_topic = _namespaced( + robot_name, + str(vision_pose.get('output_pose_cov_topic', 'interface/mavros/vision_pose/pose_cov')), + ) + + actions.append( + IncludeLaunchDescription( + FrontendLaunchDescriptionSource( + os.path.join(pkg_share, 'launch', 'mavros_gp_origin.launch.xml'), + ), + launch_arguments=[ + ('config_file', gp_path), + ('use_sim_time', ust), + ], + ), + ) + actions.append( + IncludeLaunchDescription( + FrontendLaunchDescriptionSource( + os.path.join(pkg_share, 'launch', 'px4_param_setter.launch.xml'), + ), + launch_arguments=[ + ('config_file', px4_path), + ('use_sim_time', ust), + ], + ), + ) actions.append( IncludeLaunchDescription( FrontendLaunchDescriptionSource( @@ -87,7 +230,9 @@ def launch_setup(context, *_args, **_kwargs): ), launch_arguments=[ ('config_file', vp_path), - ('body_name', body_name), + ('input_topic', input_topic), + ('output_pose_topic', output_pose_topic), + ('output_pose_cov_topic', output_pose_cov_topic), ('use_sim_time', ust), ], ), @@ -99,18 +244,28 @@ def launch_setup(context, *_args, **_kwargs): DeclareLaunchArgument( 'config_file', default_value=default_natnet_yaml, - description='NatNet parameter YAML (/** ros__parameters). ' - 'publish_to_mavros and body_name control MAVROS include.', + description='NatNet config YAML (natnet: server + per-robot profiles). ' + 'The robot profile selected by ROBOT_NAME drives bodies + MAVROS include.', ), DeclareLaunchArgument( 'vision_pose_config_file', default_value=default_vp_yaml, - description='vision_pose_converter parameter YAML.', + description='vision_pose_converter parameter YAML (frame_id, canonical_quaternion).', + ), + DeclareLaunchArgument( + 'gp_origin_config_file', + default_value=default_gp_origin_yaml, + description='mavros_gp_origin parameter YAML.', + ), + DeclareLaunchArgument( + 'px4_params_config_file', + default_value=default_px4_params_yaml, + description='px4_param_setter parameter YAML (params.* = desired FCU parameters).', ), DeclareLaunchArgument( 'use_sim_time', default_value='false', - description='Forwarded to vision_pose_converter.launch.xml.', + description='Forwarded to MAVROS bridge launch files.', ), OpaqueFunction(function=launch_setup), ], diff --git a/robot/ros_ws/src/perception/natnet_ros2/launch/px4_param_setter.launch.xml b/robot/ros_ws/src/perception/natnet_ros2/launch/px4_param_setter.launch.xml new file mode 100644 index 000000000..a3852b0c8 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/launch/px4_param_setter.launch.xml @@ -0,0 +1,36 @@ + + + + + + + + + + + + + + + + + + + diff --git a/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml b/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml index aad9c4474..803cbdad7 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml +++ b/robot/ros_ws/src/perception/natnet_ros2/launch/vision_pose_converter.launch.xml @@ -3,26 +3,31 @@ - + + + - - - + + + diff --git a/robot/ros_ws/src/perception/natnet_ros2/package.xml b/robot/ros_ws/src/perception/natnet_ros2/package.xml index f9632b0f7..1f9e04f4d 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/package.xml +++ b/robot/ros_ws/src/perception/natnet_ros2/package.xml @@ -26,6 +26,8 @@ mavros_msgs + geographic_msgs + rcl_interfaces ament_index_python launch diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/mavros_gp_origin_node.py b/robot/ros_ws/src/perception/natnet_ros2/src/mavros_gp_origin_node.py new file mode 100755 index 000000000..b4151d835 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/src/mavros_gp_origin_node.py @@ -0,0 +1,198 @@ +#!/usr/bin/env python3 + +""" +MAVROS GPS Origin Node + +Publishes a synthetic GPS origin to MAVROS once at startup for mocap / no-GNSS +flight. With GNSS disabled, PX4 fuses vision into a valid local position but +has no global position, so modes that require one (e.g. AUTO.LOITER) refuse to +arm. Setting an origin lets PX4 derive global position from the fused estimate. + +The publish is guarded: it waits for MAVROS to connect, watches for an existing +origin, and only publishes if none is present — GNSS-equipped vehicles are left +untouched. +""" + +import shutil +import subprocess + +import rclpy +from rclpy.node import Node +from geographic_msgs.msg import GeoPointStamped +from mavros_msgs.msg import State + + +class MavrosGpOriginNode(Node): + """One-shot synthetic GPS origin publisher for MAVROS / PX4.""" + + def __init__(self): + super().__init__('mavros_gp_origin') + + self.declare_parameter('enabled', True) + # Defaults match the AirStack shared world datum (Lisbon) used by the GCS + # (gcs_utils.py) and sim (gps_utils.py). Normally overridden by + # config/mavros_gp_origin.yaml; kept in sync to avoid a stale fallback. + self.declare_parameter('latitude', 38.736832) + self.declare_parameter('longitude', -9.137977) + self.declare_parameter('altitude', 90.0) + # Real-hardware geoid handling. mavros/PX4 treat the origin altitude as a + # WGS-84 ELLIPSOIDAL height and internally apply the egm96-5 geoid model + # (mavros_uas::egm96_5) to convert to/from AMSL. With no GNSS/baro the + # vehicle height comes purely from vision (mocap floor ~ 0 AMSL), so to + # make local_position z equal the OptiTrack height the origin's ellipsoidal + # altitude must be: N(lat,lon) + desired_floor_amsl, where N is the geoid + # undulation. Because the SAME egm96-5 model computes N here and inside + # mavros, the undulation cancels EXACTLY (accuracy is independent of the + # model's absolute error). Skipped when use_sim_time=true: sim's synthetic + # GPS carries no geoid separation and uses the literal altitude. + self.declare_parameter('use_geoid_altitude', False) + # AMSL (m) assigned to the mocap floor / vision z = 0. Local z equals OptiTrack z + # for any value; this only sets what global altitude the floor reports. + self.declare_parameter('desired_floor_amsl', 36.0) + # Geoid model — MUST match mavros (egm96-5) for exact cancellation. + self.declare_parameter('geoid_model', 'egm96-5') + # Seconds to wait after MAVROS connects (listening for an existing + # origin) before publishing our synthetic one. + self.declare_parameter('settle_sec', 5.0) + + self._enabled = self.get_parameter('enabled').value + if not self._enabled: + self.get_logger().info('Synthetic GPS origin disabled (enabled=false).') + return + + self._lat = self.get_parameter('latitude').value + self._lon = self.get_parameter('longitude').value + self._settle_sec = self.get_parameter('settle_sec').value + self._alt = self._resolve_altitude() + + self._done = False + self._origin_exists = False + self._connected_since = None + self._publish_count = 0 + + self._set_origin_pub = self.create_publisher( + GeoPointStamped, 'set_gps_origin', 10 + ) + self._origin_sub = self.create_subscription( + GeoPointStamped, 'current_gps_origin', self._on_existing_origin, 10 + ) + self._state_sub = self.create_subscription( + State, 'mavros_state', self._on_mavros_state, 10 + ) + self._timer = self.create_timer(1.0, self._tick) + + self.get_logger().info( + f'MAVROS GPS origin node started ' + f'(lat={self._lat}, lon={self._lon}, alt={self._alt}, ' + f'settle_sec={self._settle_sec})' + ) + + def _geoid_undulation(self, lat, lon, model): + """ + Geoid undulation N (metres, height of the geoid above the WGS-84 + ellipsoid) at (lat, lon) via GeographicLib's GeoidEval — the same + egm96-5 dataset mavros loads (mavros_uas::egm96_5), so N cancels exactly + against mavros' internal ellipsoid<->AMSL conversion. Raises on failure. + """ + exe = shutil.which('GeoidEval') + if exe is None: + raise RuntimeError('GeoidEval not found on PATH (install GeographicLib tools)') + proc = subprocess.run( + [exe, '-n', model], + input=f'{lat:.9f} {lon:.9f}\n', + capture_output=True, text=True, timeout=10.0, + ) + if proc.returncode != 0: + raise RuntimeError( + f'GeoidEval rc={proc.returncode}: {proc.stderr.strip() or proc.stdout.strip()}' + ) + return float(proc.stdout.strip().split()[0]) + + def _resolve_altitude(self): + """ + Origin altitude to publish: the literal `altitude` param, unless + use_geoid_altitude is set on real hardware, in which case it is the + egm96-5 geoid undulation at (lat, lon) plus desired_floor_amsl. + """ + if not self.get_parameter('use_geoid_altitude').value: + return self.get_parameter('altitude').value + if self.get_parameter('use_sim_time').value: + self.get_logger().info( + 'use_sim_time=true: using literal altitude (sim datum), not geoid.' + ) + return self.get_parameter('altitude').value + floor = self.get_parameter('desired_floor_amsl').value + model = self.get_parameter('geoid_model').value + try: + n = self._geoid_undulation(self._lat, self._lon, model) + except Exception as e: + literal = self.get_parameter('altitude').value + self.get_logger().error( + f'use_geoid_altitude=true but geoid lookup failed ({e}); falling ' + f'back to literal altitude {literal} m. LOCAL Z WILL BE OFFSET BY ' + f'THE GEOID (tens of m) — fix GeographicLib/GeoidEval before flight.' + ) + return literal + alt = n + floor + self.get_logger().info( + f'Geoid origin altitude: N({model})={n:.4f} + floor_amsl={floor:.4f} ' + f'=> {alt:.4f} m ellipsoidal (local z will equal OptiTrack z).' + ) + return alt + + def _on_existing_origin(self, _msg: GeoPointStamped): + """An origin already exists (e.g. from GNSS) — never override it.""" + if not self._origin_exists and not self._done: + self.get_logger().info( + 'Existing GPS origin detected; skipping synthetic origin.' + ) + self._origin_exists = True + + def _on_mavros_state(self, msg: State): + if msg.connected and self._connected_since is None: + self._connected_since = self.get_clock().now() + + def _tick(self): + if self._done: + return + if self._origin_exists: + self._done = True + self._timer.cancel() + return + if self._connected_since is None: + return + elapsed = (self.get_clock().now() - self._connected_since).nanoseconds * 1e-9 + if elapsed < self._settle_sec: + return + + msg = GeoPointStamped() + msg.header.stamp = self.get_clock().now().to_msg() + msg.position.latitude = self._lat + msg.position.longitude = self._lon + msg.position.altitude = self._alt + self._set_origin_pub.publish(msg) + self._publish_count += 1 + self.get_logger().info( + f'Published synthetic GPS origin ' + f'(lat={self._lat}, lon={self._lon}, alt={self._alt}) ' + f'[{self._publish_count}/3]' + ) + # Publish a few times in case MAVROS subscribed late, then stop. + if self._publish_count >= 3: + self._done = True + self._timer.cancel() + + +def main(args=None): + rclpy.init(args=args) + try: + node = MavrosGpOriginNode() + rclpy.spin(node) + except KeyboardInterrupt: + pass + finally: + rclpy.shutdown() + + +if __name__ == '__main__': + main() diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp index 186f5572c..78caa940b 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp +++ b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp @@ -44,14 +44,22 @@ namespace void NATNET_CALLCONV sdk_frame_callback(sFrameOfMocapData * data, void * ctx) { - auto * frame_cb = static_cast *>(ctx); - if (!data || !frame_cb || !*frame_cb) { return; } + auto * cb_ctx = static_cast(ctx); + if (!data || !cb_ctx || !cb_ctx->cb || !*cb_ctx->cb) { return; } FrameSample fs; fs.frame_num = data->iFrame; fs.timestamp = data->fTimestamp; fs.params = static_cast(data->params); + // TransmitTimestamp is 0 on servers/streams that don't populate frame timing; + // only compute latency when it's present so downstream sampling can skip it. + if (data->TransmitTimestamp != 0 && cb_ctx->client) { + fs.transit_latency_s = + cb_ctx->client->SecondsSinceHostTimestamp(data->TransmitTimestamp); + fs.has_latency = true; + } + fs.bodies.reserve(static_cast(data->nRigidBodies)); for (int i = 0; i < data->nRigidBodies; ++i) { const sRigidBodyData & rb = data->RigidBodies[i]; @@ -63,7 +71,7 @@ void NATNET_CALLCONV sdk_frame_callback(sFrameOfMocapData * data, void * ctx) fs.bodies.push_back(s); } - (*frame_cb)(fs); + (*cb_ctx->cb)(fs); } /// Map NatNet SDK ErrorCode to our NatNetResult. @@ -159,8 +167,10 @@ std::vector NatNetClientAdapter::get_body_descriptors() void NatNetClientAdapter::set_frame_callback( std::function cb) { - user_cb_ = std::move(cb); - client_->SetFrameReceivedCallback(sdk_frame_callback, &user_cb_); + user_cb_ = std::move(cb); + cb_ctx_.client = client_.get(); + cb_ctx_.cb = &user_cb_; + client_->SetFrameReceivedCallback(sdk_frame_callback, &cb_ctx_); } // --------------------------------------------------------------------------- @@ -170,7 +180,9 @@ void NatNetClientAdapter::disconnect() client_->SetFrameReceivedCallback(sdk_frame_callback, nullptr); client_->Disconnect(); } - user_cb_ = nullptr; + user_cb_ = nullptr; + cb_ctx_.client = nullptr; + cb_ctx_.cb = nullptr; } } // namespace natnet_ros2 diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp index 65059f659..58b7c2fda 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp +++ b/robot/ros_ws/src/perception/natnet_ros2/src/natnet_ros2_node.cpp @@ -1,26 +1,22 @@ -// natnet_ros2_node.cpp -// -// ROS 2 NatNet SDK node for OptiTrack Motive integration. -// -// Published topics (per tracked rigid body): -// /{robot_name}/perception/optitrack/{body_name} → PoseStamped -// /{robot_name}/perception/optitrack/{body_name}/pose_cov → PoseWithCovarianceStamped -// -// Parameters (see config/natnet_config.yaml): -// server_ip, client_ip, command_port, data_port, -// body_name, body_id (-1 = all), publish_direct_optitrack, -// frame_id, debug, position_covariance, orientation_covariance -// -// ROBOT_NAME is read from the environment variable set by AirStack's -// robot_name_map resolver at container startup. +// natnet_ros2_node.cpp — ROS 2 NatNet SDK node for OptiTrack Motive. +// Parameters are flattened from config/natnet_config.yaml by natnet_ros2.launch.py. +// See docs/robot/px4_external_vision.md. +#include #include -#include +#include +#include #include #include -#include +#include #include #include +#include + +// POSIX sockets for the pre-connect reachability probe +#include +#include +#include // ROS 2 #include "rclcpp/rclcpp.hpp" @@ -32,6 +28,51 @@ #include "natnet_ros2/natnet_client_adapter.hpp" +// Ping the Motive command port before handing the server to the SDK: Connect() can +// assert deep in ClientCore::ValidateHostConnection (SIGABRT) rather than returning +// NetworkError when the host is unreachable. Do not remove this pre-check. +static bool natnet_server_reachable( + const std::string & server_ip, int command_port, int timeout_ms) +{ + const int fd = ::socket(AF_INET, SOCK_DGRAM, 0); + if (fd < 0) { return false; } + + timeval tv{}; + tv.tv_sec = timeout_ms / 1000; + tv.tv_usec = (timeout_ms % 1000) * 1000; + ::setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv)); + + sockaddr_in addr{}; + addr.sin_family = AF_INET; + addr.sin_port = htons(static_cast(command_port)); + if (::inet_pton(AF_INET, server_ip.c_str(), &addr.sin_addr) != 1) { + ::close(fd); + return false; + } + + // Official connect packet: header (msg_id=NAT_CONNECT(0), size=271), + // 270-byte payload starting with "Ping" + NatNet version at offset 265, + // then a trailing NUL (matches the SDK PythonClient's send_request). + std::array pkt{}; + pkt[2] = 271 & 0xFF; + pkt[3] = 271 >> 8; + pkt[4] = 'P'; pkt[5] = 'i'; pkt[6] = 'n'; pkt[7] = 'g'; + pkt[4 + 265] = 4; // requested NatNet version 4.2.0.0 + pkt[4 + 266] = 2; + + bool reachable = false; + if (::sendto(fd, pkt.data(), pkt.size(), 0, + reinterpret_cast(&addr), sizeof(addr)) == + static_cast(pkt.size())) + { + uint8_t reply[512]; + reachable = ::recv(fd, reply, sizeof(reply), 0) > 0; + } + ::close(fd); + return reachable; +} + + // --------------------------------------------------------------------------- // NatNetROS2Node // --------------------------------------------------------------------------- @@ -41,8 +82,14 @@ class NatNetROS2Node : public rclcpp::Node using PoseStamped = geometry_msgs::msg::PoseStamped; using PoseWithCovarianceStamped = geometry_msgs::msg::PoseWithCovarianceStamped; - struct BodyPublishers + struct BodyConfig { + int32_t id = -1; + std::string rigid_body_name; + std::string topic_base; + bool publish_pose = true; + bool publish_pose_cov = true; + std::array covariance{}; rclcpp::Publisher::SharedPtr pose_pub; rclcpp::Publisher::SharedPtr pose_cov_pub; }; @@ -52,55 +99,61 @@ class NatNetROS2Node : public rclcpp::Node : Node("natnet_ros2_node") { // ----- Parameters -------------------------------------------------- - this->declare_parameter("server_ip", "192.168.1.1"); - this->declare_parameter("client_ip", "0.0.0.0"); - this->declare_parameter("command_port", 1510); - this->declare_parameter("data_port", 1511); - this->declare_parameter("connection_type", std::string("unicast")); - this->declare_parameter("multicast_address", std::string("239.255.42.99")); - this->declare_parameter("body_name", "robot_1"); - this->declare_parameter("body_id", -1); - this->declare_parameter("publish_direct_optitrack", true); - this->declare_parameter("publish_to_mavros", false); - this->declare_parameter("frame_id", "world"); - this->declare_parameter("debug", false); - this->declare_parameter( - "position_covariance", - std::vector{0.1,0.,0., 0.,0.1,0., 0.,0.,0.1}); - this->declare_parameter( - "orientation_covariance", - std::vector{0.01,0.,0., 0.,0.01,0., 0.,0.,0.01}); + this->declare_parameter("server_ip", "192.168.1.1"); + this->declare_parameter("client_ip", "0.0.0.0"); + this->declare_parameter("command_port", 1510); + this->declare_parameter("data_port", 1511); + this->declare_parameter("connection_type", std::string("unicast")); + this->declare_parameter("multicast_address", std::string("239.255.42.99")); + this->declare_parameter("frame_id", "world"); + this->declare_parameter("debug", false); + + // Latency sampling: warmup + window for mean/stdev, then log a summary of measured latency. + this->declare_parameter("latency_sampling_warmup_s", 5.0); + this->declare_parameter("latency_sampling_window_s", 20.0); + // Suggested latency for the Cube Orange (PX4) from the OptiTrack Motive model. + // This is added to the measured transport latency to estimate total latency to PX4. + // NOTE: an estimate, not a measurement — see docs/robot/px4_external_vision.md. + // Diagnostic only; it is logged, never fused. + this->declare_parameter("cube_orange_latency_ms", 5.0); + + // Parallel per-body arrays (flattened from natnet_config.yaml by the launch file). + this->declare_parameter("body_names", std::vector{}); + this->declare_parameter("body_ids", std::vector{}); + this->declare_parameter("body_topics", std::vector{}); + this->declare_parameter("body_pose", std::vector{}); + this->declare_parameter("body_pose_cov", std::vector{}); + this->declare_parameter("body_position_covariance", std::vector{}); + this->declare_parameter("body_orientation_covariance", std::vector{}); // ----- Read parameters --------------------------------------------- - const auto connect_cfg = natnet_ros2::make_connect_config( - this->get_parameter("server_ip").as_string(), - this->get_parameter("client_ip").as_string(), - static_cast(this->get_parameter("command_port").as_int()), - static_cast(this->get_parameter("data_port").as_int()), - this->get_parameter("connection_type").as_string(), - this->get_parameter("multicast_address").as_string()); - - if (connect_cfg.connection_type != - this->get_parameter("connection_type").as_string()) - { - RCLCPP_WARN(get_logger(), - "Unknown connection_type '%s' — falling back to 'unicast'.", - this->get_parameter("connection_type").as_string().c_str()); + // Fatally fail if the config is invalid (e.g. unknown connection_type). + natnet_ros2::ConnectConfig connect_cfg; + try { + connect_cfg = natnet_ros2::make_connect_config( + this->get_parameter("server_ip").as_string(), + this->get_parameter("client_ip").as_string(), + static_cast(this->get_parameter("command_port").as_int()), + static_cast(this->get_parameter("data_port").as_int()), + this->get_parameter("connection_type").as_string(), + this->get_parameter("multicast_address").as_string()); + } catch (const std::invalid_argument & e) { + RCLCPP_FATAL(get_logger(), "Invalid natnet configuration: %s", e.what()); + throw; } - body_name_ = this->get_parameter("body_name").as_string(); - body_id_ = static_cast(this->get_parameter("body_id").as_int()); - publish_direct_ = this->get_parameter("publish_direct_optitrack").as_bool(); - frame_id_ = this->get_parameter("frame_id").as_string(); - debug_ = this->get_parameter("debug").as_bool(); + frame_id_ = this->get_parameter("frame_id").as_string(); + debug_ = this->get_parameter("debug").as_bool(); - covariance_6x6_ = natnet_ros2::build_covariance_6x6( - this->get_parameter("position_covariance").as_double_array(), - this->get_parameter("orientation_covariance").as_double_array()); + latency_warmup_s_ = this->get_parameter("latency_sampling_warmup_s").as_double(); + latency_window_s_ = this->get_parameter("latency_sampling_window_s").as_double(); + cube_orange_latency_ms_ = this->get_parameter("cube_orange_latency_ms").as_double(); const char * rn = std::getenv("ROBOT_NAME"); robot_name_ = rn ? rn : "robot_1"; + build_body_configs(); + RCLCPP_INFO(get_logger(), "========================================="); RCLCPP_INFO(get_logger(), "NatNet ROS 2 Node"); RCLCPP_INFO(get_logger(), " robot_name: %s", robot_name_.c_str()); @@ -110,18 +163,19 @@ class NatNetROS2Node : public rclcpp::Node if (natnet_ros2::is_multicast(connect_cfg)) { RCLCPP_INFO(get_logger(), " multicast_addr: %s", connect_cfg.multicast_address.c_str()); } - RCLCPP_INFO(get_logger(), " body_id: %d (%s)", - static_cast(body_id_), - (body_id_ < 0) ? "track all" : "single body"); + RCLCPP_INFO(get_logger(), " tracked bodies: %zu", bodies_.size()); RCLCPP_INFO(get_logger(), "========================================="); // Production client — NatNetClientAdapter wraps the SDK client_ = std::make_unique(); - connect_and_setup(connect_cfg); + connect_cfg_ = connect_cfg; - refresh_timer_ = this->create_wall_timer( - std::chrono::seconds(1), - std::bind(&NatNetROS2Node::refresh_descriptions_if_needed, this)); + // Try to connect now; keep retrying. + if (!connect_and_setup(connect_cfg_)) { + connect_timer_ = this->create_wall_timer( + std::chrono::seconds(2), + std::bind(&NatNetROS2Node::retry_connect, this)); + } } // ----------------------------------------------------------------------- @@ -132,14 +186,10 @@ class NatNetROS2Node : public rclcpp::Node // ----------------------------------------------------------------------- // Called from the NatNetClientAdapter's frame trampoline. - // publish() and Clock::now() are thread-safe; pub_mutex_ guards map access. + // publish() and Clock::now() are thread-safe; bodies_ is immutable after init. // ----------------------------------------------------------------------- void on_frame(const natnet_ros2::FrameSample & frame) { - if (natnet_ros2::model_list_changed(frame.params)) { - needs_description_refresh_.store(true, std::memory_order_relaxed); - } - if (debug_) { RCLCPP_DEBUG(get_logger(), "Frame %d: %zu rigid bodies, ts=%.4f s", frame.frame_num, frame.bodies.size(), static_cast(frame.timestamp)); @@ -147,6 +197,8 @@ class NatNetROS2Node : public rclcpp::Node const rclcpp::Time stamp = this->get_clock()->now(); + maybe_sample_latency(frame, stamp); + for (const auto & rb : frame.bodies) { if (!natnet_ros2::is_tracking_valid(rb.params)) { if (debug_) { @@ -154,20 +206,14 @@ class NatNetROS2Node : public rclcpp::Node } continue; } - if (!natnet_ros2::should_publish_body(body_id_, rb.id)) { continue; } - std::lock_guard lock(pub_mutex_); - - const auto pub_it = publishers_.find(rb.id); - if (pub_it == publishers_.end()) { - needs_description_refresh_.store(true, std::memory_order_relaxed); - continue; - } + const auto it = bodies_.find(rb.id); + if (it == bodies_.end()) { continue; } // not configured for this robot const natnet_ros2::PoseData pose = natnet_ros2::rb_to_pose(rb); - const BodyPublishers & bp = pub_it->second; + const BodyConfig & body = it->second; - if (publish_direct_ && bp.pose_pub) { + if (body.publish_pose && body.pose_pub) { PoseStamped msg; msg.header.frame_id = frame_id_; msg.header.stamp = stamp; @@ -178,10 +224,10 @@ class NatNetROS2Node : public rclcpp::Node msg.pose.orientation.y = pose.qy; msg.pose.orientation.z = pose.qz; msg.pose.orientation.w = pose.qw; - bp.pose_pub->publish(msg); + body.pose_pub->publish(msg); } - if (bp.pose_cov_pub) { + if (body.publish_pose_cov && body.pose_cov_pub) { PoseWithCovarianceStamped cov_msg; cov_msg.header.frame_id = frame_id_; cov_msg.header.stamp = stamp; @@ -192,22 +238,31 @@ class NatNetROS2Node : public rclcpp::Node cov_msg.pose.pose.orientation.y = pose.qy; cov_msg.pose.pose.orientation.z = pose.qz; cov_msg.pose.pose.orientation.w = pose.qw; - cov_msg.pose.covariance = covariance_6x6_; - bp.pose_cov_pub->publish(cov_msg); + cov_msg.pose.covariance = body.covariance; + body.pose_cov_pub->publish(cov_msg); } } } private: // ----------------------------------------------------------------------- - void connect_and_setup(const natnet_ros2::ConnectConfig & cfg) + // Returns true once the handshake succeeds. + bool connect_and_setup(const natnet_ros2::ConnectConfig & cfg) { + // Wire-level probe first + if (!natnet_server_reachable(cfg.server_ip, cfg.command_port, 500)) { + RCLCPP_WARN_THROTTLE(get_logger(), *get_clock(), 10000, + "Motive at %s:%d not answering NatNet ping — waiting to connect.", + cfg.server_ip.c_str(), cfg.command_port); + return false; + } + const natnet_ros2::NegotiationResult neg = natnet_ros2::negotiate(*client_, cfg); if (!neg.ok) { - RCLCPP_ERROR(get_logger(), "%s", neg.log_message.c_str()); - return; + RCLCPP_WARN(get_logger(), "%s", neg.log_message.c_str()); + return false; } if (neg.server_info.host_present) { @@ -216,103 +271,201 @@ class NatNetROS2Node : public rclcpp::Node RCLCPP_WARN(get_logger(), "%s", neg.log_message.c_str()); } - refresh_descriptions_locked(); - client_->set_frame_callback( [this](const natnet_ros2::FrameSample & f) { on_frame(f); }); RCLCPP_INFO(get_logger(), "Frame callback registered — receiving mocap data."); + connected_ = true; + return true; } // ----------------------------------------------------------------------- - void refresh_descriptions_if_needed() + // Timer-driven reconnect. + void retry_connect() { - if (!needs_description_refresh_.exchange(false, std::memory_order_relaxed)) { + if (connected_) { + if (connect_timer_) { connect_timer_->cancel(); } return; } - RCLCPP_INFO(get_logger(), "Model list change detected — refreshing data descriptions."); - std::lock_guard lock(pub_mutex_); - refresh_descriptions_locked(); + RCLCPP_INFO_THROTTLE(get_logger(), *get_clock(), 10000, + "NatNet not connected — retrying handshake to %s ...", + connect_cfg_.server_ip.c_str()); + if (connect_and_setup(connect_cfg_) && connect_timer_) { + connect_timer_->cancel(); + } } // ----------------------------------------------------------------------- - // Must be called with pub_mutex_ held (or from single-threaded init). + // Build the per-body config map + publishers from the parallel param arrays. + // Publishers are created up front (config-driven), so streaming begins as soon + // as frames arrive — no dependency on Motive's data-description handshake. // ----------------------------------------------------------------------- - void refresh_descriptions_locked() + void build_body_configs() { - if (!client_) { return; } - - // Always ensure the statically-configured body has a publisher - if (body_id_ >= 0) { - ensure_publisher_locked(body_id_, body_name_); + const auto names = this->get_parameter("body_names").as_string_array(); + const auto ids = this->get_parameter("body_ids").as_integer_array(); + const auto topics = this->get_parameter("body_topics").as_string_array(); + const auto pose = this->get_parameter("body_pose").as_bool_array(); + const auto pose_cov = this->get_parameter("body_pose_cov").as_bool_array(); + const auto pos_cov = this->get_parameter("body_position_covariance").as_double_array(); + const auto ori_cov = this->get_parameter("body_orientation_covariance").as_double_array(); + + const std::size_t n = std::min(names.size(), ids.size()); + if (names.size() != ids.size()) { + RCLCPP_WARN(get_logger(), + "body_names (%zu) and body_ids (%zu) length mismatch — using %zu.", + names.size(), ids.size(), n); } - const auto bodies = client_->get_body_descriptors(); - int newly_created = 0; - for (const auto & bd : bodies) { - // Store name for every body (including skeleton bones) - body_names_[bd.id] = bd.name; + for (std::size_t i = 0; i < n; ++i) { + BodyConfig body; + body.id = static_cast(ids[i]); + body.rigid_body_name = names[i]; + body.publish_pose = (i < pose.size()) ? pose[i] : true; + body.publish_pose_cov = (i < pose_cov.size()) ? pose_cov[i] : true; + + const std::string relative = (i < topics.size()) ? topics[i] : std::string{}; + body.topic_base = + natnet_ros2::body_topic_base(robot_name_, body.rigid_body_name, relative); - // Skip skeleton bones (parent_id >= 0) - if (bd.parent_id >= 0) { continue; } + body.covariance = natnet_ros2::build_covariance_6x6( + cov_slice(pos_cov, i, _DEFAULT_POSITION_COVARIANCE), + cov_slice(ori_cov, i, _DEFAULT_ORIENTATION_COVARIANCE)); - // When tracking a single body, skip others - if (!natnet_ros2::should_publish_body(body_id_, bd.id)) { continue; } + if (body.publish_pose) { + body.pose_pub = this->create_publisher(body.topic_base, 10); + } + if (body.publish_pose_cov) { + body.pose_cov_pub = this->create_publisher( + body.topic_base + "/pose_cov", 10); + } - if (ensure_publisher_locked(bd.id, bd.name)) { ++newly_created; } + RCLCPP_INFO(get_logger(), + "Tracking body id=%d name='%s' → %s (pose=%d pose_cov=%d)", + static_cast(body.id), body.rigid_body_name.c_str(), + body.topic_base.c_str(), + static_cast(body.publish_pose), + static_cast(body.publish_pose_cov)); + + bodies_.emplace(body.id, std::move(body)); } + } + + // ----------------------------------------------------------------------- + // Return the i-th 9-element covariance block from a flattened array, or the + // built-in default when the slice is missing. + static std::vector cov_slice( + const std::vector & flat, std::size_t i, const std::vector & fallback) + { + const std::size_t start = i * 9; + if (flat.size() < start + 9) { return fallback; } + return std::vector(flat.begin() + start, flat.begin() + start + 9); + } + + // ----------------------------------------------------------------------- + // Accumulate per-message transit latency over a fixed window and log a + // one-shot mean/stdev summary. Called once per frame from on_frame() (the SDK + // receive thread); all sampling state is touched only here, so no locking. + void maybe_sample_latency(const natnet_ros2::FrameSample & frame, + const rclcpp::Time & now) + { + if (latency_reported_ || !frame.has_latency) { return; } - if (newly_created > 0) { + if (!latency_first_seen_) { + latency_first_seen_ = true; + latency_first_time_ = now; RCLCPP_INFO(get_logger(), - "Data descriptions refreshed: %d new publisher(s) created.", newly_created); - } else { - RCLCPP_DEBUG(get_logger(), "Data descriptions refreshed: no new publishers."); + "Latency sampling armed: %.1fs warm-up, then %.1fs sampling window.", + latency_warmup_s_, latency_window_s_); + return; } + + const double elapsed = (now - latency_first_time_).seconds(); + if (elapsed < latency_warmup_s_) { return; } // still warming up + if (elapsed > latency_warmup_s_ + latency_window_s_) { // window closed + report_latency(); + return; + } + + const double lat = frame.transit_latency_s; + latency_count_ += 1; + latency_sum_s_ += lat; + latency_sum_sq_s_ += lat * lat; } // ----------------------------------------------------------------------- - bool ensure_publisher_locked(int32_t id, const std::string & name) + // Compute and log the latency summary once, then latch so it never repeats. + void report_latency() { - if (publishers_.count(id)) { return false; } + latency_reported_ = true; - const std::string topic_base = - natnet_ros2::optitrack_topic_base(robot_name_, name); + if (latency_count_ == 0) { + RCLCPP_WARN(get_logger(), + "Latency window elapsed but no timestamped frames were sampled " + "(server may not populate TransmitTimestamp)."); + return; + } - BodyPublishers bp; - if (publish_direct_) { - bp.pose_pub = this->create_publisher(topic_base, 10); + const double n = static_cast(latency_count_); + const double mean_s = latency_sum_s_ / n; + double var_s2 = 0.0; + if (latency_count_ > 1) { + // Sample variance (Bessel-corrected); clamp tiny negatives from round-off. + var_s2 = (latency_sum_sq_s_ - n * mean_s * mean_s) / (n - 1.0); + if (var_s2 < 0.0) { var_s2 = 0.0; } } - bp.pose_cov_pub = this->create_publisher( - natnet_ros2::optitrack_pose_cov_topic(robot_name_, name), 10); - publishers_.emplace(id, std::move(bp)); + const double mean_ms = mean_s * 1.0e3; + const double stdev_ms = std::sqrt(var_s2) * 1.0e3; + const double total_ms = mean_ms + cube_orange_latency_ms_; RCLCPP_INFO(get_logger(), - "Publisher registered: id=%d name='%s' → %s[/pose_cov]", - static_cast(id), name.c_str(), topic_base.c_str()); - return true; + "\n" + "========= OptiTrack -> drone message latency =========\n" + " sampling window : %.1f s (%llu frames)\n" + " transport mean : %.3f ms\n" + " transport std dev : %.3f ms\n" + " Cube Orange (model) : %.3f ms\n" + " estimated total : %.3f ms (to PX4 / EKF2 fusion)\n" + "======================================================", + latency_window_s_, + static_cast(latency_count_), + mean_ms, stdev_ms, cube_orange_latency_ms_, total_ms); } // ----------------------------------------------------------------------- // Parameters / state - std::string body_name_; - int32_t body_id_ = -1; - bool publish_direct_ = true; std::string frame_id_; - bool debug_ = false; + bool debug_ = false; std::string robot_name_; - std::array covariance_6x6_{}; + // Latency sampling parameters + running accumulators. + double latency_warmup_s_ = 5.0; + double latency_window_s_ = 20.0; + double cube_orange_latency_ms_ = 5.0; + bool latency_first_seen_ = false; + bool latency_reported_ = false; + rclcpp::Time latency_first_time_{0, 0, RCL_ROS_TIME}; + uint64_t latency_count_ = 0; + double latency_sum_s_ = 0.0; + double latency_sum_sq_s_ = 0.0; std::unique_ptr client_; + natnet_ros2::ConnectConfig connect_cfg_; + bool connected_ = false; + + std::unordered_map bodies_; - std::mutex pub_mutex_; - std::unordered_map body_names_; - std::unordered_map publishers_; + rclcpp::TimerBase::SharedPtr connect_timer_; - std::atomic needs_description_refresh_{false}; - rclcpp::TimerBase::SharedPtr refresh_timer_; + static const std::vector _DEFAULT_POSITION_COVARIANCE; + static const std::vector _DEFAULT_ORIENTATION_COVARIANCE; }; +const std::vector NatNetROS2Node::_DEFAULT_POSITION_COVARIANCE = + {1.0e-6, 0., 0., 0., 1.0e-6, 0., 0., 0., 1.0e-6}; +const std::vector NatNetROS2Node::_DEFAULT_ORIENTATION_COVARIANCE = + {3.0e-6, 0., 0., 0., 3.0e-6, 0., 0., 0., 3.0e-6}; + // --------------------------------------------------------------------------- int main(int argc, char ** argv) diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/px4_param_setter_node.py b/robot/ros_ws/src/perception/natnet_ros2/src/px4_param_setter_node.py new file mode 100755 index 000000000..1a08bef30 --- /dev/null +++ b/robot/ros_ws/src/perception/natnet_ros2/src/px4_param_setter_node.py @@ -0,0 +1,296 @@ +#!/usr/bin/env python3 + +""" +PX4 Parameter Checker Node + +Compares the FCU's PX4 parameters against a configured set (see +config/px4_params.yaml) for mocap-only flight (OptiTrack external vision, no GNSS, +no magnetometer). **By default it only checks and flags** — it does not write to the +FCU. The desired values are meant to be set once by a human in QGroundControl (see +docs/robot/…/px4_external_vision.md); this node is a safety net that catches a +mis-configured FCU before flight. + +Two safety flags control behaviour: + +- ``auto_set`` (default ``false``): when ``true``, the node also *writes* any + mismatched param via ``param/set`` (ParamSetV2) and verifies the readback — the + legacy enforce behaviour. When ``false`` (default) the node never writes. +- ``on_mismatch`` (``warn`` | ``halt``, default ``warn``): with ``auto_set=false``, + what to do when a param disagrees. ``warn`` logs the diffs and lets the stack come + up; ``halt`` logs fatal and exits non-zero so a ``required`` launch node tears the + stack down. + +For each entry under the ``params.`` prefix the node waits for an FCU connection + +settle, reads the current value via ``get_parameters``, and compares. Type mapping +follows the YAML literal: integers → MAVLink int params, floats → float params — +write ``6.0`` (not ``6``) for float params like EKF2_EV_DELAY so the type matches. +""" + +import math +import sys + +import rclpy +from rclpy.node import Node +from rclpy.parameter import Parameter +from rcl_interfaces.msg import ParameterValue, ParameterType +from rcl_interfaces.srv import GetParameters +from mavros_msgs.msg import State +from mavros_msgs.srv import ParamSetV2 + + +class Px4ParamSetterNode(Node): + """PX4 parameter checker (optionally setter) via the MAVROS param plugin.""" + + def __init__(self): + super().__init__( + 'px4_param_setter', + automatically_declare_parameters_from_overrides=True, + ) + + self._enabled = self._param_or('enabled', True) + if not self._enabled: + self.get_logger().info('PX4 param checker disabled (enabled=false).') + return + + # Safety flags: check-only by default; opt in to writing with auto_set. + self._auto_set = bool(self._param_or('auto_set', False)) + self._on_mismatch = str(self._param_or('on_mismatch', 'warn')).lower() + if self._on_mismatch not in ('warn', 'halt'): + self.get_logger().warn( + f"Invalid on_mismatch {self._on_mismatch!r}; falling back to 'warn'." + ) + self._on_mismatch = 'warn' + + # Seconds after MAVROS connects before the first attempt (initial + # param-table pull over serial takes a while at 115200 baud). + self._settle_sec = float(self._param_or('settle_sec', 10.0)) + self._retry_period_sec = float(self._param_or('retry_period_sec', 2.0)) + self._max_attempts = int(self._param_or('max_attempts', 30)) + + # Desired FCU params from the params.* prefix; YAML int → PX4 int32, + # YAML float → PX4 float. + self._desired = { + name: p.value + for name, p in self.get_parameters_by_prefix('params').items() + } + self._pending = dict(self._desired) + self._changed: list[str] = [] + self._skipped: list[str] = [] + # (param_id, current, desired) for params that disagree and were NOT set + # (auto_set=false). Drives the on_mismatch policy in _finish(). + self._mismatched: list[tuple] = [] + self._attempts = 0 + self._connected_since = None + self._done = False + self._inflight = False + + if not self._pending: + self.get_logger().warn('No params.* entries configured; nothing to do.') + self._done = True + return + + self._get_cli = self.create_client(GetParameters, 'param_get_parameters') + self._set_cli = self.create_client(ParamSetV2, 'param_set') + self._state_sub = self.create_subscription( + State, 'mavros_state', self._on_mavros_state, 10 + ) + self._timer = self.create_timer(self._retry_period_sec, self._tick) + + mode = 'auto-set' if self._auto_set else f'check-only (on_mismatch={self._on_mismatch})' + self.get_logger().info( + f'PX4 param checker started [{mode}]: {len(self._pending)} params ' + f'({", ".join(sorted(self._pending))}), settle_sec={self._settle_sec}' + ) + + def _param_or(self, name, default): + """Return a declared-from-overrides parameter value, or the default.""" + if self.has_parameter(name): + value = self.get_parameter(name).value + if value is not None: + return value + return default + + # --- MAVROS state ------------------------------------------------------ + + def _on_mavros_state(self, msg: State): + if msg.connected and self._connected_since is None: + self._connected_since = self.get_clock().now() + self.get_logger().info('FCU connected; waiting for param table to settle.') + + # --- Main retry loop --------------------------------------------------- + + def _tick(self): + if self._done or self._inflight: + return + if self._connected_since is None: + return + elapsed = (self.get_clock().now() - self._connected_since).nanoseconds * 1e-9 + if elapsed < self._settle_sec: + return + if not self._pending: + self._finish() + return + if self._attempts >= self._max_attempts: + self.get_logger().error( + f'Giving up after {self._attempts} attempts; ' + f'unset params: {", ".join(sorted(self._pending))}' + ) + self._finish() + return + + self._attempts += 1 + param_id = sorted(self._pending)[0] + if not self._get_cli.service_is_ready() or not self._set_cli.service_is_ready(): + self.get_logger().info('MAVROS param services not ready yet; retrying.') + return + + self._inflight = True + req = GetParameters.Request(names=[param_id]) + future = self._get_cli.call_async(req) + future.add_done_callback( + lambda f, pid=param_id: self._on_get_done(pid, f) + ) + + # --- Get → compare → set → verify chain -------------------------------- + + def _on_get_done(self, param_id: str, future): + try: + resp = future.result() + except Exception as e: # noqa: BLE001 — retry on any transport error + self.get_logger().warn(f'{param_id}: get_parameters failed ({e}); will retry.') + self._inflight = False + return + + current = resp.values[0] if resp.values else None + if current is not None and self._matches(current, self._desired[param_id]): + self.get_logger().info(f'{param_id}: already {self._desired[param_id]} — skipping.') + self._skipped.append(param_id) + del self._pending[param_id] + self._inflight = False + return + if current is None or current.type == ParameterType.PARAMETER_NOT_SET: + # Param table likely not pulled yet — retry rather than flag/force-set. + self.get_logger().info(f'{param_id}: not in MAVROS param table yet; will retry.') + self._inflight = False + return + + # Mismatch. Check-only mode (default): record and flag, never write. + if not self._auto_set: + self._mismatched.append( + (param_id, self._value_of(current), self._desired[param_id]) + ) + del self._pending[param_id] + self._inflight = False + return + + req = ParamSetV2.Request() + req.force_set = False + req.param_id = param_id + req.value = self._to_parameter_value(self._desired[param_id]) + set_future = self._set_cli.call_async(req) + set_future.add_done_callback( + lambda f, pid=param_id, old=self._value_of(current): self._on_set_done(pid, old, f) + ) + + def _on_set_done(self, param_id: str, old_value, future): + self._inflight = False + try: + resp = future.result() + except Exception as e: # noqa: BLE001 — retry on any transport error + self.get_logger().warn(f'{param_id}: set failed ({e}); will retry.') + return + + desired = self._desired[param_id] + if not resp.success or not self._matches(resp.value, desired): + self.get_logger().warn( + f'{param_id}: set rejected or readback mismatch ' + f'(wanted {desired}, got {self._value_of(resp.value)}); will retry.' + ) + return + + self.get_logger().info(f'{param_id}: {old_value} -> {desired}') + self._changed.append(param_id) + del self._pending[param_id] + + def _finish(self): + self._done = True + self._timer.cancel() + self.get_logger().info( + f'PX4 param check finished: {len(self._skipped)} already correct, ' + f'{len(self._changed)} set, {len(self._mismatched)} mismatched, ' + f'{len(self._pending)} unread.' + ) + if self._changed: + self.get_logger().warn( + f'FCU parameters changed ({", ".join(sorted(self._changed))}). ' + 'Reboot the flight controller before flying so EKF2 starts clean.' + ) + + # Check-only mismatches: report each, then apply the on_mismatch policy. + if self._mismatched: + for pid, current, desired in sorted(self._mismatched): + self.get_logger().warn( + f'{pid}: FCU has {current}, expected {desired} ' + '(not set — auto_set=false). Fix in QGroundControl.' + ) + names = ", ".join(sorted(p for p, _, _ in self._mismatched)) + if self._on_mismatch == 'halt': + self.get_logger().fatal( + f'{len(self._mismatched)} PX4 param(s) wrong for external-vision ' + f'flight ({names}); halting (on_mismatch=halt). Set them in ' + 'QGroundControl or enable auto_set.' + ) + # SystemExit propagates out of spin(); main()'s finally shuts down + # rclpy. Non-zero code lets a `required` launch node tear the stack down. + sys.exit(1) + self.get_logger().warn( + f'{len(self._mismatched)} PX4 param(s) differ from the external-vision ' + f'set ({names}); continuing (on_mismatch=warn).' + ) + + # --- Value helpers ------------------------------------------------------ + + @staticmethod + def _to_parameter_value(value) -> ParameterValue: + pv = ParameterValue() + if isinstance(value, bool) or isinstance(value, int): + pv.type = ParameterType.PARAMETER_INTEGER + pv.integer_value = int(value) + elif isinstance(value, float): + pv.type = ParameterType.PARAMETER_DOUBLE + pv.double_value = value + else: + raise TypeError(f'Unsupported PX4 param value type: {type(value)}') + return pv + + @staticmethod + def _value_of(pv: ParameterValue): + if pv.type == ParameterType.PARAMETER_INTEGER: + return pv.integer_value + if pv.type == ParameterType.PARAMETER_DOUBLE: + return pv.double_value + return None + + @classmethod + def _matches(cls, pv: ParameterValue, desired) -> bool: + current = cls._value_of(pv) + if current is None: + return False + # FCU floats are float32 — compare with a tolerance that absorbs the + # float64 → float32 round trip. + return math.isclose(float(current), float(desired), rel_tol=1e-5, abs_tol=1e-6) + + +def main(args=None): + rclpy.init(args=args) + try: + node = Px4ParamSetterNode() + rclpy.spin(node) + except (KeyboardInterrupt, rclpy.executors.ExternalShutdownException): + pass + finally: + rclpy.try_shutdown() + + +if __name__ == '__main__': + main() diff --git a/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py b/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py index 9a36f879d..88caede52 100755 --- a/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py +++ b/robot/ros_ws/src/perception/natnet_ros2/src/vision_pose_converter_node.py @@ -7,6 +7,12 @@ for PX4 external pose estimation and state fusion. Converts from NatNet coordinate frame to a frame suitable for MAVROS. + +Topics are configurable so the bridge can be retargeted to other middleware. +``input_topic`` / ``output_pose_topic`` / ``output_pose_cov_topic`` default to the +relative names ``input_pose`` / ``output_pose`` / ``output_pose_cov`` (remappable), +but natnet_ros2.launch.py overrides them with the absolute, ROBOT_NAME-namespaced +topics from the robot's ``vision_pose`` block in natnet_config.yaml. """ import rclpy @@ -28,15 +34,46 @@ def __init__(self): self.declare_parameter('frame_id', 'world') self.declare_parameter('child_frame_id', 'base_link') self.declare_parameter('canonical_quaternion', True) + # Max output rate to MAVROS (0 = passthrough). Each pose becomes a + # ~116-byte VISION_POSITION_ESTIMATE on the FCU serial link; at + # 115200 baud (~11.5 kB/s) a full-rate 100+ Hz mocap stream alone + # overflows the MAVROS TX queue. EKF2 only needs 30-50 Hz. + self.declare_parameter('max_rate_hz', 30.0) + # Which MAVROS vision_pose topic(s) to publish: 'pose', 'pose_cov', or + # 'both'. MAVROS turns EACH of vision_pose/pose and vision_pose/pose_cov + # into its own VISION_POSITION_ESTIMATE on the FCU link, so 'both' sends + # msg 102 at 2x the rate. Use a single topic to halve serial TX load. + self.declare_parameter('publish_mode', 'both') + # Topic names — overridden by the launch file from the per-robot + # vision_pose block; defaults are the historical remappable relative names. + self.declare_parameter('input_topic', 'input_pose') + self.declare_parameter('output_pose_topic', 'output_pose') + self.declare_parameter('output_pose_cov_topic', 'output_pose_cov') self.frame_id = self.get_parameter('frame_id').value self.child_frame_id = self.get_parameter('child_frame_id').value self.canonical_quaternion = self.get_parameter('canonical_quaternion').value + max_rate_hz = self.get_parameter('max_rate_hz').value + publish_mode = str(self.get_parameter('publish_mode').value).lower() + if publish_mode not in ('pose', 'pose_cov', 'both'): + self.get_logger().warn( + f"Invalid publish_mode {publish_mode!r}; falling back to 'both'" + ) + publish_mode = 'both' + self._publish_pose = publish_mode in ('pose', 'both') + self._publish_pose_cov = publish_mode in ('pose_cov', 'both') + # 0.95 factor so an input stream at exactly max_rate_hz doesn't beat + # against the period check and alias down to half rate. + self._min_period_ns = 0 if max_rate_hz <= 0.0 else int(0.95e9 / max_rate_hz) + self._last_pub_ns = 0 + input_topic = self.get_parameter('input_topic').value + output_pose_topic = self.get_parameter('output_pose_topic').value + output_pose_cov_topic = self.get_parameter('output_pose_cov_topic').value # Subscribers self.pose_sub = self.create_subscription( PoseWithCovarianceStamped, - 'input_pose', + input_topic, self._on_pose, 10 ) @@ -44,19 +81,22 @@ def __init__(self): # Publishers self.pose_pub = self.create_publisher( PoseStamped, - 'output_pose', + output_pose_topic, 10 ) self.pose_cov_pub = self.create_publisher( PoseWithCovarianceStamped, - 'output_pose_cov', + output_pose_cov_topic, 10 ) self.get_logger().info( f'Vision pose converter started ' f'(frame_id={self.frame_id!r}, child_frame_id={self.child_frame_id!r}, ' - f'canonical_quaternion={self.canonical_quaternion})' + f'canonical_quaternion={self.canonical_quaternion}, ' + f'max_rate_hz={max_rate_hz}, publish_mode={publish_mode!r}, ' + f'input_topic={input_topic!r}, output_pose_topic={output_pose_topic!r}, ' + f'output_pose_cov_topic={output_pose_cov_topic!r})' ) @staticmethod @@ -83,18 +123,26 @@ def _on_pose(self, msg: PoseWithCovarianceStamped): so that EKF consumers never see a sign-flip discontinuity. """ try: + if self._min_period_ns: + now_ns = self.get_clock().now().nanoseconds + if now_ns - self._last_pub_ns < self._min_period_ns: + return + self._last_pub_ns = now_ns + msg.header.frame_id = self.frame_id if self.canonical_quaternion: msg.pose.pose.orientation = self._canonical_quaternion( msg.pose.pose.orientation ) - self.pose_cov_pub.publish(msg) + if self._publish_pose_cov: + self.pose_cov_pub.publish(msg) - pose_msg = PoseStamped() - pose_msg.header = msg.header - pose_msg.pose = msg.pose.pose - self.pose_pub.publish(pose_msg) + if self._publish_pose: + pose_msg = PoseStamped() + pose_msg.header = msg.header + pose_msg.pose = msg.pose.pose + self.pose_pub.publish(pose_msg) except Exception as e: self.get_logger().error(f"Error converting pose: {e}") diff --git a/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp index 7a144ef9b..f469c2a29 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp +++ b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_logic.cpp @@ -136,6 +136,46 @@ TEST(TopicNames, LeadingSlashPresent) EXPECT_EQ(optitrack_topic_base("robot_1", "Body")[0], '/'); } +TEST(TopicNames, NamespacedTopicStripsLeadingSlashes) +{ + EXPECT_EQ(namespaced_topic("robot_1", "perception/optitrack/drone"), + "/robot_1/perception/optitrack/drone"); + EXPECT_EQ(namespaced_topic("robot_1", "/perception/optitrack/drone"), + "/robot_1/perception/optitrack/drone"); + EXPECT_EQ(namespaced_topic("robot_2", "///a/b"), "/robot_2/a/b"); +} + +TEST(TopicNames, BodyTopicBaseUsesOverrideWhenSet) +{ + // Empty override → default perception/optitrack/{name} + EXPECT_EQ(body_topic_base("robot_1", "Drone", ""), + "/robot_1/perception/optitrack/Drone"); + // Non-empty override → namespaced relative leaf (decoupled from body name) + EXPECT_EQ(body_topic_base("robot_1", "Drone", "perception/optitrack/drone"), + "/robot_1/perception/optitrack/drone"); + EXPECT_EQ(body_topic_base("robot_3", "Target", "perception/optitrack/target"), + "/robot_3/perception/optitrack/target"); +} + +// =========================================================================== +// Multi-body filtering — body_is_configured +// =========================================================================== + +TEST(BodyIsConfigured, MatchesConfiguredIds) +{ + const std::vector ids = {1, 100}; + EXPECT_TRUE(body_is_configured(ids, 1)); + EXPECT_TRUE(body_is_configured(ids, 100)); + EXPECT_FALSE(body_is_configured(ids, 2)); +} + +TEST(BodyIsConfigured, EmptySetMatchesNothing) +{ + const std::vector ids = {}; + EXPECT_FALSE(body_is_configured(ids, 0)); + EXPECT_FALSE(body_is_configured(ids, 1)); +} + // =========================================================================== // Server negotiation — validate_connection_type @@ -151,17 +191,28 @@ TEST(ValidateConnectionType, MulticastPassesThrough) EXPECT_EQ(validate_connection_type("multicast"), "multicast"); } -TEST(ValidateConnectionType, UnknownFallsBackToUnicast) +TEST(ValidateConnectionType, UnknownThrows) { - EXPECT_EQ(validate_connection_type("broadcast"), "unicast"); - EXPECT_EQ(validate_connection_type(""), "unicast"); - EXPECT_EQ(validate_connection_type("UDP"), "unicast"); + EXPECT_THROW(validate_connection_type("broadcast"), std::invalid_argument); + EXPECT_THROW(validate_connection_type(""), std::invalid_argument); + EXPECT_THROW(validate_connection_type("UDP"), std::invalid_argument); } -TEST(ValidateConnectionType, CaseSensitiveFallsBack) +TEST(ValidateConnectionType, CaseSensitiveThrows) { - EXPECT_EQ(validate_connection_type("Unicast"), "unicast"); - EXPECT_EQ(validate_connection_type("MULTICAST"), "unicast"); + // Accepting "Unicast" would mean the config silently disagrees with itself. + EXPECT_THROW(validate_connection_type("Unicast"), std::invalid_argument); + EXPECT_THROW(validate_connection_type("MULTICAST"), std::invalid_argument); +} + +TEST(ValidateConnectionType, MessageNamesTheOffendingValue) +{ + try { + validate_connection_type("broadcst"); + FAIL() << "expected std::invalid_argument"; + } catch (const std::invalid_argument & e) { + EXPECT_NE(std::string(e.what()).find("broadcst"), std::string::npos); + } } @@ -193,12 +244,11 @@ TEST(ConnectConfig, MulticastConfigIsMulticast) EXPECT_EQ(cfg.multicast_address, "239.255.42.99"); } -TEST(ConnectConfig, InvalidConnectionTypeFallsBackToUnicast) +TEST(ConnectConfig, InvalidConnectionTypeThrows) { - const auto cfg = make_connect_config( - "10.0.0.1", "0.0.0.0", 1510, 1511, "broadcast"); - EXPECT_EQ(cfg.connection_type, "unicast"); - EXPECT_FALSE(is_multicast(cfg)); + EXPECT_THROW( + make_connect_config("10.0.0.1", "0.0.0.0", 1510, 1511, "broadcast"), + std::invalid_argument); } TEST(ConnectConfig, PortsArePreserved) diff --git a/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py index cba5a3444..37526cd18 100644 --- a/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py +++ b/robot/ros_ws/src/perception/natnet_ros2/test/test_natnet_ros2.py @@ -1,20 +1,15 @@ # Copyright (c) 2024 Carnegie Mellon University # MIT License - see LICENSE in the repository root for full text. -"""Unit tests for natnet_ros2 Python source code. +"""Unit tests for natnet_ros2 Python helpers (no ROS install required). -These tests import the actual production source files and stub out ROS at the -import boundary so no ROS installation is required. +Stubs rclpy/launch at import time. Covers ``VisionPoseConverterNode`` quaternion +canonicalisation, configurable-topic wiring, and ``natnet_ros2.launch.py`` +profile-flattening helpers (server + per-body arrays, env expansion, namespacing). -Coverage here: - vision_pose_converter_node.py → VisionPoseConverterNode._canonical_quaternion() - → VisionPoseConverterNode._on_pose() frame_id assignment - -NOT covered here (C++ — requires colcon build + gtest): - natnet_ros2_node.cpp → build_covariance_6x6(), topic name construction, - connection_type validation, SDK frame callback logic. - These live in test_natnet_logic.cpp in the same test/ directory. +C++ logic (``natnet_logic.hpp``) is tested in ``test_natnet_logic.cpp`` via colcon. """ +import importlib.util import sys from pathlib import Path from types import SimpleNamespace @@ -28,22 +23,31 @@ # metaclass machinery returns a Mock for attribute access instead of running # __init_subclass__ / defining methods). We supply a real dummy base class # so the actual class body — including _canonical_quaternion — is defined. +# +# The fake also records declared params and created sub/pub topics so the +# configurable-topic wiring can be asserted without a ROS install. # --------------------------------------------------------------------------- class _FakeNode: + # Per-test parameter overrides keyed by name; consulted by declare_parameter so + # values survive the node's super().__init__ (which resets per-instance state). + _overrides: dict = {} + def __init__(self, name: str): - pass + self._params: dict = {} + self.created_subscriptions: list = [] + self.created_publishers: list = [] def get_logger(self): return MagicMock() - def declare_parameter(self, *args, **kwargs): - pass + def declare_parameter(self, name, default=None): + self._params[name] = self._overrides.get(name, default) def get_parameter(self, name): - m = MagicMock() - m.value = MagicMock() - return m - def create_subscription(self, *args, **kwargs): + return SimpleNamespace(value=self._params.get(name)) + def create_subscription(self, msg_type, topic, callback, qos): + self.created_subscriptions.append(topic) return MagicMock() - def create_publisher(self, *args, **kwargs): + def create_publisher(self, msg_type, topic, qos): + self.created_publishers.append(topic) return MagicMock() @@ -62,6 +66,36 @@ def create_publisher(self, *args, **kwargs): from vision_pose_converter_node import VisionPoseConverterNode # noqa: E402 +# --------------------------------------------------------------------------- +# Load natnet_ros2.launch.py with its heavy launch/ROS deps stubbed, so the +# pure flattening helpers can be unit-tested without a ROS install. +# --------------------------------------------------------------------------- + +for _mod in ( + "ament_index_python", + "ament_index_python.packages", + "launch", + "launch.actions", + "launch.launch_description_sources", + "launch.substitutions", + "launch_ros", + "launch_ros.actions", +): + sys.modules.setdefault(_mod, MagicMock()) + +# yaml is only needed by _load_natnet_config (not the flattening helpers); stub it +# if PyYAML is absent so the launch module still imports in a minimal unit env. +try: + import yaml # noqa: F401 +except ImportError: + sys.modules.setdefault("yaml", MagicMock()) + +_launch_path = Path(__file__).resolve().parent.parent / "launch" / "natnet_ros2.launch.py" +_spec = importlib.util.spec_from_file_location("natnet_ros2_launch_under_test", _launch_path) +natnet_launch = importlib.util.module_from_spec(_spec) +_spec.loader.exec_module(natnet_launch) + + # --------------------------------------------------------------------------- # Helpers # --------------------------------------------------------------------------- @@ -150,3 +184,104 @@ def test_canonical_quaternion_dual_sign_produces_same_result(): assert out_pos.x == pytest.approx(out_neg.x) assert out_pos.y == pytest.approx(out_neg.y) assert out_pos.z == pytest.approx(out_neg.z) + + +# --------------------------------------------------------------------------- +# VisionPoseConverterNode — configurable input/output topics +# --------------------------------------------------------------------------- + +@pytest.mark.unit +def test_vision_pose_converter_default_topics(): + """Defaults reproduce the historical relative (remappable) topic names.""" + node = VisionPoseConverterNode() + assert node.created_subscriptions == ["input_pose"] + assert node.created_publishers == ["output_pose", "output_pose_cov"] + + +@pytest.mark.unit +def test_vision_pose_converter_topic_overrides_applied(): + """When the topic params are set, sub/pub use those exact names.""" + _FakeNode._overrides = { + "input_topic": "/robot_2/perception/optitrack/drone/pose_cov", + "output_pose_topic": "/robot_2/custom/vision/pose", + "output_pose_cov_topic": "/robot_2/custom/vision/pose_cov", + } + try: + node = VisionPoseConverterNode() + finally: + _FakeNode._overrides = {} + assert node.created_subscriptions == ["/robot_2/perception/optitrack/drone/pose_cov"] + assert node.created_publishers == [ + "/robot_2/custom/vision/pose", + "/robot_2/custom/vision/pose_cov", + ] + + +# --------------------------------------------------------------------------- +# natnet_ros2.launch.py — pure config-flattening helpers +# --------------------------------------------------------------------------- + +@pytest.mark.unit +def test_expand_env_uses_default_when_unset(monkeypatch): + monkeypatch.delenv("NATNET_SERVER_IP", raising=False) + assert natnet_launch._expand_env("$(env NATNET_SERVER_IP 172.31.0.200)") == "172.31.0.200" + + +@pytest.mark.unit +def test_expand_env_uses_environment_value(monkeypatch): + monkeypatch.setenv("NATNET_SERVER_IP", "10.0.0.5") + assert natnet_launch._expand_env("$(env NATNET_SERVER_IP 172.31.0.200)") == "10.0.0.5" + + +@pytest.mark.unit +def test_namespaced_strips_and_prefixes(): + assert natnet_launch._namespaced("robot_1", "perception/optitrack/drone") == \ + "/robot_1/perception/optitrack/drone" + assert natnet_launch._namespaced("robot_2", "/already/abs") == "/robot_2/already/abs" + + +@pytest.mark.unit +def test_build_node_params_flattens_bodies(): + server = {"server_ip": "1.2.3.4", "command_port": 1510, "connection_type": "unicast"} + profile = { + "bodies": [ + { + "rigid_body_name": "Drone", + "id": 1, + "topic": "perception/optitrack/drone", + "pose": True, + "pose_cov": True, + "position_covariance": [9.0] * 9, + "orientation_covariance": [8.0] * 9, + }, + { + "rigid_body_name": "Target", + "id": 100, + "topic": "perception/optitrack/target", + "pose": True, + "pose_cov": False, + }, + ] + } + params = natnet_launch._build_node_params(server, profile) + + assert params["server_ip"] == "1.2.3.4" + assert params["body_names"] == ["Drone", "Target"] + assert params["body_ids"] == [1, 100] + assert params["body_topics"] == ["perception/optitrack/drone", "perception/optitrack/target"] + assert params["body_pose"] == [True, True] + assert params["body_pose_cov"] == [True, False] + # 9 floats per body, flattened in body order. + assert len(params["body_position_covariance"]) == 18 + assert params["body_position_covariance"][:9] == [9.0] * 9 + # Target omitted its covariance → built-in default fills its slice. + assert params["body_position_covariance"][9:] == natnet_launch._DEFAULT_POSITION_COVARIANCE + + +@pytest.mark.unit +def test_build_node_params_empty_profile(): + """A robot with no profile yields empty body arrays (node tracks nothing).""" + params = natnet_launch._build_node_params({}, {}) + assert params["body_names"] == [] + assert params["body_ids"] == [] + assert params["body_position_covariance"] == [] diff --git a/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml b/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml index a79ff85a1..1e9f8662e 100644 --- a/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml +++ b/robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml @@ -75,7 +75,8 @@ - + From 19402c2223812c9f454c1ffe0dd2d854b48d3e93 Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Sat, 15 Aug 2026 21:32:46 -0400 Subject: [PATCH 16/21] OptiTrack (2/3): NatNet server emulator + host integration tests (#375) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(sim): add NatNet server emulator (protocol core) + register unit tests The pure-Python NatNet server that emulates an OptiTrack Motive server so natnet_ros2 can be driven without hardware. USD/Isaac-free — this is the protocol + server core (unicast server, data/model/server types, serializers, default catalogs). The Isaac wrapper that maps a USD scene onto this server lands next. Registers the emulator package's co-located unit tests via a `sim:` entry in tests/colcon_unit_test_packages.yaml (base's simulation/**//test glob). The root conftest now puts each unit-test package's import root on sys.path so co-located tests import their package without a per-package conftest.py. Co-Authored-By: Claude Opus 4.8 * test(natnet): host integration tests — emulator server → natnet_ros2 Drive the real natnet_ros2 client from the host NatNet server emulator and check the drone pose reaches ROS at rate (single-body and multi-body profiles). No sim, no GPU — uses the base's `robot_autonomy_stack` fixture + `integration` mark. The Isaac-wrapper variant lands with the Isaac wrapper PR. Co-Authored-By: Claude Opus 4.8 * chore: bump version to 0.19.0-alpha.15 * pack frame sections through one helper * fix the labeled-marker struct format that raised on every pack sMarker.pack used ' --- .env | 2 +- CHANGELOG.md | 1 + .../optitrack.natnet.emulator/.gitignore | 11 + .../optitrack.natnet.emulator/README.md | 160 ++++++++ .../optitrack/__init__.py | 1 + .../optitrack/natnet/__init__.py | 1 + .../optitrack/natnet/emulator/__init__.py | 20 + .../optitrack/natnet/emulator/defaults.py | 25 ++ .../natnet/emulator/server/__init__.py | 11 + .../natnet/emulator/server/natnet_common.py | 27 ++ .../emulator/server/natnet_data_types.py | 224 +++++++++++ .../emulator/server/natnet_model_types.py | 134 +++++++ .../natnet/emulator/server/natnet_server.py | 353 +++++++++++++++++ .../emulator/server/natnet_server_types.py | 156 ++++++++ .../emulator/server/natnet_unicast_server.py | 172 ++++++++ .../optitrack.natnet.emulator/setup.py | 23 ++ .../test/natnet_test_helpers.py | 105 +++++ .../test/test_defaults.py | 26 ++ .../test/test_serializers.py | 367 ++++++++++++++++++ .../test/test_server_catalog.py | 78 ++++ .../test/test_unicast_protocol.py | 284 ++++++++++++++ tests/colcon_unit_test_packages.yaml | 6 + tests/conftest.py | 14 + tests/integration/natnet/README.md | 151 +++++++ .../natnet/test_natnet_integration.py | 302 ++++++++++++++ 25 files changed, 2653 insertions(+), 1 deletion(-) create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/.gitignore create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/__init__.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/__init__.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/__init__.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/defaults.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/__init__.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_common.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_data_types.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_model_types.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server_types.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_unicast_server.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/setup.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/natnet_test_helpers.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_defaults.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_serializers.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_server_catalog.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_unicast_protocol.py create mode 100644 tests/integration/natnet/README.md create mode 100644 tests/integration/natnet/test_natnet_integration.py diff --git a/.env b/.env index 70736e9ac..8cf4e408c 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.14" +VERSION="0.19.0-alpha.15" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d12c4108..1c65dc62e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `integration` test tier (`tests/integration/`, `integration` mark) with a shared `robot_autonomy_stack` fixture (robot container, no sim/GPU) - `waypoint_flight` system test (`tests/system/test_waypoint_flight.py`): takeoff → ordered waypoint route via `NavigateTask` (dispatched as a dense plan) → land, judged on the odometry track by the standalone stdlib-only `tests/waypoint_checker.py` (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`); validated end-to-end in Isaac Sim; serves as the standard acceptance check after integrating or swapping a planner module - Real-robot PX4 external-vision fusion in `natnet_ros2` (OptiTrack mocap → EKF2): `mavros_gp_origin` (geoid-corrected synthetic GPS origin so `local_position.z` == OptiTrack z, fixing the ~36 m boot offset), `vision_pose_converter`, and a PX4 param **checker** (`px4_param_setter`, `auto_set` off by default; `on_mismatch` warn/halt) — setup guide at `docs/robot/px4_external_vision.md` +- NatNet server emulator (`optitrack.natnet.emulator`, protocol core) — pure-Python OptiTrack Motive server emulation so `natnet_ros2` can be driven without hardware; host integration tests (`tests/integration/natnet/`) wire it to the robot client ### Changed diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/.gitignore b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/.gitignore new file mode 100644 index 000000000..adef4d964 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/.gitignore @@ -0,0 +1,11 @@ +# OptiTrack SDK archives and build artifacts (reference tree may exist locally) +**/*.obj +**/*.pdb +**/*.exe +**/*.iobj +**/*.ipdb +**/*.tlog/ +**/__pycache__/ +**/*.pyc +# Generated by the editable install (pip install -e) the Dockerfile and tests use. +**/*.egg-info/ diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md new file mode 100644 index 000000000..3bb05f0b6 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md @@ -0,0 +1,160 @@ +# OptiTrack NatNet Emulator (Isaac Sim Extension) + +Python NatNet **server** emulator for AirStack simulation and integration testing with [`natnet_ros2`](../../../../robot/ros_ws/src/perception/natnet_ros2/). + +The extension has two layers: + +1. **Transport + protocol** (`optitrack.natnet.emulator.server`) — UDP NatNet server, ctypes wire types, MODELDEF cache, frame streaming. Importable outside Isaac Sim (unit tests, host-side integration). +2. **Isaac integration** (`optitrack.natnet.emulator.isaac`) — stage-driven `/World/NatNetInterface` config prim, pose sampling on physics steps, Kit UI editor, and Pegasus launch-script helpers. + +## Layout + +``` +optitrack.natnet.emulator/ +├── config/extension.toml # Kit manifest (server module + UI entry point) +├── schema/schema.usda # Typed NatNet interface attribute definitions +├── setup.py +├── docs/ # (legacy design notes — see docs/simulation/isaac_sim/natnet_emulator.md) +├── test/ # Co-located unit tests (proxied by tests/sim/) +└── optitrack/natnet/emulator/ + ├── defaults.py # Reference Drone → prim bindings for tests + ├── server/ # NatNet UDP server (transport + protocol) + │ ├── natnet_server.py # Base server, queue, MODELDEF cache + │ ├── natnet_unicast_server.py + │ ├── natnet_data_types.py + │ ├── natnet_model_types.py + │ └── natnet_server_types.py + └── isaac/ # Isaac Sim wrapper (Kit + USD) + ├── config.py # Pure-Python NatNetInterfaceConfig model + ├── usd_bindings.py # Author/read interface prims on a stage + ├── catalog.py # Config → sDataDescriptions (MODELDEF) + ├── frames.py # Prim poses → sFrameOfMocapData + ├── manager.py # NatNetServerManager (lifecycle + sampling) + ├── scene_setup.py # Pegasus launch helpers (start_drone_natnet_server) + └── ui_extension.py # Docked editor panel (NatNetEmulatorExtension) +``` + +## Responsibilities + +| Layer | Role | +|-------|------| +| **Server** | UDP transport; `NAT_CONNECT` / `NAT_SERVERINFO`; `NAT_REQUEST_MODELDEF`; `NAT_KEEPALIVE`; `NAT_ECHOREQUEST` / `NAT_ECHORESPONSE`; `NAT_FRAMEOFDATA` on the **data port** (1511). MODELDEF stored as packed bytes via `set_model_def_payload()`. Frames enqueued with `enqueue_mocap_data()`. | +| **Isaac wrapper** | Authors and reads the NatNet interface config prim; builds MODELDEF from scene config; samples tracked prim world poses each physics step; calls `flush_mocap_data()` synchronously (background timer disabled — see below). | +| **`defaults.py`** | Hardcoded `Drone` → `/World/base_link` binding for legacy tests; production paths use the stage prim via `scene_setup.build_drone_config()`. | + +The server does **not** own prim-path bindings. The Isaac layer calls `set_model_def_payload(catalog.pack())` after building `sDataDescriptions` from the interface config. + +## Stage-driven config prim + +Configuration lives on a USD prim (conventionally `/World/NatNetInterface`) with `natnet:*` attributes: + +- Server: IP, unicast/multicast mode, command/data ports, publish rate, NatNet version, up-axis, optional pose noise. +- Bodies: multi-apply `natnet:body::*` fields mapping rigid-body name / streaming ID → target prim path. + +`NatNetServerManager` scans the stage, resyncs the catalog when the prim changes, and streams one rigid body per configured target. Missing prims emit **lost** bodies (NaN position, tracking-invalid bit clear) until the target appears — important for Pegasus drones spawned on first Play. + +**Up axis:** default `Z` passes Isaac/USD world poses through unchanged (matches `natnet_ros2`). Set `Y` to emulate a Y-up Motive room. + +## Streaming model (Isaac) + +Inside Kit, the server's background `_data_update_loop` is **disabled** (`auto_stream = False`) because the GIL-starved daemon thread does not reliably transmit frames. Instead, each physics step: + +1. `NatNetServerManager.sample_once()` reads prim poses and `enqueue_mocap_data(frame)`. +2. `NatNetUnicastServer.flush_mocap_data()` sends immediately on the physics-step thread. + +Outside Isaac (host unit tests), `auto_stream=True` uses the timer-driven loop. + +Default Docker sim IP: **`172.31.0.200`** (Isaac container on the AirStack bridge network). + +## Enabling in AirStack + +**Robot:** `LAUNCH_NATNET=true` in `.env` → `natnet_ros2` in perception bringup. Configure Motive/emulator IP in [`natnet_config.yaml`](../../../../robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml). + +**Isaac Sim:** set `ISAAC_SIM_SCRIPT_NAME` to a NatNet launch script (NatNet always starts — no `LAUNCH_NATNET` gate in the script): + +| Script | Use | +|--------|-----| +| `example_one_px4_pegasus_natnet_launch_script.py` | Single drone + static `Target` | +| `example_multi_px4_pegasus_natnet_launch_script.py` | `NUM_ROBOTS` drones + shared `Target` (system tests with NatNet use this even for `NUM_ROBOTS=1`) | + +Baseline Pegasus scripts (`example_one_px4_pegasus_launch_script.py`, `example_multi_px4_pegasus_launch_script.py`) have **no** NatNet integration. + +Convenience bundle for NatNet + external-vision PX4 SITL: + +```bash +airstack up --env-file overrides/isaac-natnet-vision.env +``` + +See [optitrack-development skill](../../../../.agents/skills/optitrack-development/SKILL.md) for wire-protocol details, libNatNet 4.4 unicast quirks, and debugging. + +## Usage + +### Server only (no Kit) + +```python +from optitrack.natnet.emulator import NatNetUnicastServer, make_default_drone_catalog +from optitrack.natnet.emulator.isaac.frames import BodySample, build_frame + +server = NatNetUnicastServer(local_interface="172.31.0.200") +server.set_model_def_payload(make_default_drone_catalog().pack()) +server.start() + +frame = build_frame(0, [BodySample(1, (0, 0, 1), (0, 0, 0, 1))]) +server.enqueue_mocap_data(frame) +server.flush_mocap_data() +``` + +### Isaac launch script + +```python +from optitrack.natnet.emulator.isaac import start_drone_natnet_server + +# Keep a reference to the manager for the sim lifetime. +manager = start_drone_natnet_server( + stage, + drones=[("Drone", 1, "/World/drone1/base_link")], + server_ip="172.31.0.200", +) +``` + +### Kit UI + +The extension registers **Window → NatNet Emulator** — a docked panel to create/edit the interface prim, start/stop the server, and view live body readouts. The same `NatNetServerManager` backs both the UI and launch-script paths. + +## Protocol notes (unicast, libNatNet 4.4) + +| Port | Traffic | +|------|---------| +| **1510** | Command: `NAT_CONNECT`, `NAT_REQUEST_MODELDEF`, keepalives, echo | +| **1511** | Data: `NAT_FRAMEOFDATA` — **must** be sent from a socket bound to the data port | + +Frames sent from the command socket are silently dropped by libNatNet. Every frame payload must include the 4-byte end-of-data tag expected by the C SDK unpacker. + +Full handshake layouts and sniffing workflow: [optitrack-development skill](../../../../.agents/skills/optitrack-development/SKILL.md). + +## Tests + +| Tier | Mark | What | +|------|------|------| +| Unit | `unit` | Serializers, protocol, config, USD authoring, catalog, pose sampling, server lifecycle, scene setup | +| Integration | `integration` | Host emulator → robot `natnet_ros2` pose Hz | + +Co-located tests live in `test/`. Pytest discovers them via thin proxies in [`tests/sim/optitrack_natnet_emulator/`](../../../../tests/sim/optitrack_natnet_emulator/). + +```bash +# Unit (no Docker / no SDK) +pytest tests/sim/optitrack_natnet_emulator/ -m unit -v + +# Integration (robot container + NatNet SDK) +pytest tests/integration/natnet/ -m integration -v +``` + +Representative unit modules: `test_unicast_protocol.py`, `test_pose_streaming.py`, `test_interface_authoring.py`, `test_server_lifecycle.py`, `test_scene_setup.py`. + +## Reference material + +- User guide: [`docs/simulation/isaac_sim/natnet_emulator.md`](../../../../docs/simulation/isaac_sim/natnet_emulator.md) +- Robot client: [`natnet_ros2/README.md`](../../../../robot/ros_ws/src/perception/natnet_ros2/README.md) +- Integration tier: [`tests/integration/natnet/README.md`](../../../../tests/integration/natnet/README.md) + +OptiTrack SDK sample headers may exist locally under `NatNetClientSDK/` for wire-format reference; they are **not** redistributed by AirStack (proprietary license). diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/__init__.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/__init__.py new file mode 100644 index 000000000..39ed38144 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/__init__.py @@ -0,0 +1 @@ +"""OptiTrack NatNet packages for AirStack Isaac Sim integration.""" diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/__init__.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/__init__.py new file mode 100644 index 000000000..b19da2cbc --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/__init__.py @@ -0,0 +1 @@ +"""NatNet simulation components.""" diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/__init__.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/__init__.py new file mode 100644 index 000000000..e819d8b1b --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/__init__.py @@ -0,0 +1,20 @@ +"""OptiTrack Motive NatNet emulator for Isaac Sim.""" + +from .defaults import ( + DEFAULT_DRONE_BINDING, + DEFAULT_TRACKED_BODY_BINDINGS, + TrackedBodyBinding, +) +from .server import Client, NatNetServer, NatNetUnicastServer, TransmissionType +from .server.natnet_model_types import make_default_drone_catalog + +__all__ = [ + "Client", + "DEFAULT_DRONE_BINDING", + "DEFAULT_TRACKED_BODY_BINDINGS", + "NatNetServer", + "NatNetUnicastServer", + "TrackedBodyBinding", + "TransmissionType", + "make_default_drone_catalog", +] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/defaults.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/defaults.py new file mode 100644 index 000000000..e888a6eff --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/defaults.py @@ -0,0 +1,25 @@ +"""Reference tracked-body defaults for tests and Isaac Sim wrapper.""" + +from __future__ import annotations + +from dataclasses import dataclass + + +@dataclass(frozen=True) +class TrackedBodyBinding: + """Maps a NatNet rigid body to a USD prim path (not sent on the NatNet wire).""" + + name: str + id: int + prim_path: str + parent_id: int = -1 + + +# Single-drone NatNet Pegasus scenes (example_one_px4_pegasus_natnet_launch_script.py). +DEFAULT_DRONE_BINDING = TrackedBodyBinding( + name="Drone", + id=1, + prim_path="/World/base_link", +) + +DEFAULT_TRACKED_BODY_BINDINGS: tuple[TrackedBodyBinding, ...] = (DEFAULT_DRONE_BINDING,) diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/__init__.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/__init__.py new file mode 100644 index 000000000..c84c1fe4b --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/__init__.py @@ -0,0 +1,11 @@ +"""NatNet UDP server implementation (unicast; multicast planned).""" + +from .natnet_server import Client, NatNetServer, TransmissionType +from .natnet_unicast_server import NatNetUnicastServer + +__all__ = [ + "Client", + "NatNetServer", + "NatNetUnicastServer", + "TransmissionType", +] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_common.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_common.py new file mode 100644 index 000000000..1eb32177c --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_common.py @@ -0,0 +1,27 @@ +from enum import IntEnum +import ctypes + +class ModelLimits(IntEnum): + MAX_MODELS = 2000 # maximum number of total models (data descriptions) + MAX_MARKERSETS = 1000 # maximum number of MarkerSets + MAX_RIGIDBODIES = 1000 # maximum number of RigidBodies + MAX_ASSETS = 1000 # Maximum number of Assets + MAX_NAMELENGTH = 256 # maximum length for strings + MAX_MARKERS = 200 # maximum number of markers per MarkerSet + MAX_RBMARKERS = 20 # maximum number of markers per RigidBody + MAX_SKELETONS = 100 # maximum number of skeletons + MAX_SKELRIGIDBODIES = 200 # maximum number of RididBodies per Skeleton + MAX_LABELED_MARKERS = 1000 # maximum number of labeled markers per frame + MAX_UNLABELED_MARKERS = 1000 # maximum number of unlabeled (other) markers per frame + + MAX_FORCEPLATES = 100 # maximum number of force plate 'bundles' + MAX_DEVICES = 100 # maximum number of peripheral device 'bundles' + MAX_ANALOG_CHANNELS = 32 # maximum number of data channels (signals) per analog/force plate device + MAX_ANALOG_SUBFRAMES = 30 # maximum number of analog/force plate frames per mocap frame + + MAX_PACKETSIZE = 65503 # max size of packet in bytes (actual packet size is dynamic) + # (65535 byte IP limit - 20 byte IP header - 8 byte UDP header - 4 byte sPacket header = 65503 bytes) + + + +MarkerData = ctypes.c_float * 3 diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_data_types.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_data_types.py new file mode 100644 index 000000000..1128cc351 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_data_types.py @@ -0,0 +1,224 @@ +import ctypes +import struct +from .natnet_common import ModelLimits, MarkerData + +class sMarker(ctypes.Structure): + _pack_ = 1 + _fields_ = [ + ("ID", ctypes.c_int32), + ("x", ctypes.c_float), + ("y", ctypes.c_float), + ("z", ctypes.c_float), + ("size", ctypes.c_float), + ("params", ctypes.c_int16), + ("residual", ctypes.c_float) + ] + + def pack(self) -> bytes: + return struct.pack(' bytes: + # szName is null-terminated on the wire. + name_bytes = self.szName.rstrip(b'\x00') + b'\x00' + payload = bytearray(name_bytes) + payload += struct.pack(' bytes: + return struct.pack(' bytes: + payload = bytearray(struct.pack(' bytes: + payload = bytearray(struct.pack(' bytes: + payload = bytearray(struct.pack(' bytes: + payload = bytearray(struct.pack(' bytes: + payload = bytearray(struct.pack(' bytes: + """NatNet 4.1+ prefixes each collection with a 4-byte byte count.""" + payload = bytearray(struct.pack(' 0) or natnet_major > 4: + payload += struct.pack(' bytes: + def pack_section(count: int, items, pack_item=lambda item: item.pack()) -> bytes: + """Count-prefixed section holding the first `count` entries of `items`.""" + data = bytearray() + for i in range(count): + data += pack_item(items[i]) + return self._pack_counted_section( + count, bytes(data), natnet_major=natnet_major, natnet_minor=natnet_minor + ) + + payload = bytearray() + + payload += struct.pack(' bytes: + # szName is null-terminated on the wire, not fixed MAX_NAMELENGTH. + name_bytes = self.szName.rstrip(b"\x00") + b"\x00" + payload = bytearray(name_bytes) + payload += struct.pack( + " bytes: + if self.type == int(DataDescriptors.Descriptor_RigidBody): + body = self.RigidBodyDescription.pack() + else: + raise ValueError(f"Unsupported data description type: {self.type}") + payload = bytearray(struct.pack(" bytes: + payload = bytearray(struct.pack(" sDataDescriptions: + """Build the default single-body catalog (Drone id=1) for natnet_ros2.""" + descriptions = sDataDescriptions() + descriptions.nDataDescriptions = 1 + desc = descriptions.arrDataDescriptions[0] + desc.type = int(DataDescriptors.Descriptor_RigidBody) + rb = desc.RigidBodyDescription + rb.szName = b"Drone" + rb.ID = 1 + rb.parentID = -1 + rb.offsetqw = 1.0 + rb.nMarkers = 0 + return descriptions diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server.py new file mode 100644 index 000000000..5ef19009e --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server.py @@ -0,0 +1,353 @@ +from . import natnet_data_types as DataMessages +from . import natnet_server_types as ServerMessages +from . import natnet_model_types as ModelTypes +from enum import Enum +import socket +import threading +import queue +import signal +import ctypes +import time +import typing + + +class TransmissionType(str, Enum): + UNICAST = "unicast" + MULTICAST = "multicast" + +class Client: + def __init__(self, ip: str, port: int, version: typing.Tuple[int, int, int, int] = (4, 4, 0, 0)): + self.ip = ip + self.port = port + self.version = version + self.subscribed_assets = set() + self.socket_lock = threading.Lock() + + def __hash__(self): + # Uniquely identify a client session by their IP and their unique command port. + return hash((self.ip, self.port)) + + def __eq__(self, other): + return (isinstance(other, Client) and + self.ip == other.ip and + self.port == other.port) + + +class NatNetServer: + def __init__(self, + local_interface : str = "172.31.0.200", + transmission_type: TransmissionType = TransmissionType.MULTICAST, + multicast_address : str = "239.255.42.99", + command_port: int = 1510, + data_port : int = 1511, + motive_app_version : typing.Tuple[int, int, int, int]=(3, 1, 0, 0), + natnet_version : typing.Tuple[int, int, int, int]=(4, 4, 0, 0), + high_res_clock_freq : int = 1_000_000_000, + publish_rate : int = 100 # Hz (default 100Hz) + ): + + self.local_interface = local_interface + self.transmission_type = transmission_type + self.multicast_address = multicast_address + self.command_port = command_port + self.data_port = data_port + self.motive_app_version = motive_app_version + self.natnet_version = natnet_version + self.high_res_clock_freq = high_res_clock_freq + self.publish_rate = publish_rate + + self._validate_init_params() + + self.server_description = self._build_server_description() + # Initialize synchronously safe data structures for server state and mocap data + + # Thread-safe queue for Mocp frames + self.mocap_data_queue = queue.Queue(maxsize=100) + self._last_mocap_frame: DataMessages.sFrameOfMocapData | None = None + self._last_mocap_lock = threading.Lock() + + # Thread list and shutdown event + self.threads = [] + self.shutdown_event = threading.Event() + + # Connected clients for unicast mode + self.connected_clients : typing.Set[Client] = set() + self.clients_lock : threading.Lock = threading.Lock() + + # MODELDEF wire cache (Isaac wrapper updates via set_model_def_payload) + self._model_def_lock = threading.Lock() + self._model_def_payload: bytes = ModelTypes.make_default_drone_catalog().pack() + + # Sockets + self.command_socket : socket.socket | None = None + self.data_socket : socket.socket | None = None + + self.running = False + + # When True (default), the background data loop streams frames on its own timer. + # Set False when an external driver (the Isaac wrapper's physics-step callback) + # sends frames synchronously via ``flush_mocap_data``. + self.auto_stream = True + + # start() launches two daemon threads: a command listener (handshake / MODELDEF / keepalive) + # and a data loop that streams mocap frames. The transmission-specific behavior lives in the unicast/multicast subclass. + + def _signal_handler(self, signum, frame): + print(f"\n[NatNetServer] Received interrupt signal {signum}. Initiating shutdown...") + self.shutdown() + + def enqueue_mocap_data(self, new_data: DataMessages.sFrameOfMocapData): + # Thread-safe method to push new physics frames (called by Isaac-Sim extension) + if self.mocap_data_queue.full(): + try: + # Drop oldest frame if falling behind + self.mocap_data_queue.get_nowait() + except queue.Empty: + pass + self.mocap_data_queue.put(new_data) + with self._last_mocap_lock: + self._last_mocap_frame = new_data + + def _get_last_known_mocap_frame(self) -> DataMessages.sFrameOfMocapData | None: + with self._last_mocap_lock: + return self._last_mocap_frame + + def set_model_def_payload(self, payload: bytes) -> None: + """Replace MODELDEF body served on NAT_REQUEST_MODELDEF (Isaac wrapper calls this).""" + with self._model_def_lock: + self._model_def_payload = payload + + def set_model_def_from_descriptions( + self, descriptions: ModelTypes.sDataDescriptions + ) -> None: + """Pack descriptions once and store as the MODELDEF wire cache.""" + self.set_model_def_payload(descriptions.pack()) + + def _get_model_def_payload(self) -> bytes: + """Return cached MODELDEF bytes (command thread only).""" + with self._model_def_lock: + return self._model_def_payload + + def start(self): + # Bind sockets and launch worker threads automatically on init + + # Register signal handlers for graceful shutdown (Catches Ctrl+C and kill) + try: + signal.signal(signal.SIGINT, self._signal_handler) + signal.signal(signal.SIGTERM, self._signal_handler) + except ValueError: + pass # Safe fallback if not called from the main thread + + # 1. Setup Command Socket (Receives connection/discovery requests) + self.command_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP) + self.command_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.command_socket.bind(('', self.command_port)) + + # 2. Setup Data Socket (Sends outward Mocap frames). + # Bind to the data port so frames leave with source port == data_port. + # libNatNet routes unicast NAT_FRAMEOFDATA by the server's data port + self.data_socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP) + self.data_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + self.data_socket.bind(('', self.data_port)) + if self.transmission_type == TransmissionType.MULTICAST: + self.data_socket.setsockopt(socket.IPPROTO_IP, socket.IP_MULTICAST_IF, socket.inet_aton(self.local_interface)) + + # 3. Launch Threads + cmd_thread = threading.Thread(target=self._command_listener_loop, daemon=True) + data_thread = threading.Thread(target=self._data_update_loop, daemon=True) + + self.threads.extend([cmd_thread, data_thread]) + + for t in self.threads: + t.start() + + self.running = True + + def shutdown(self): + # Cleanly shutdown threads and close sockets + self.running = False + self.shutdown_event.set() + + if self.command_socket: + self.command_socket.close() + + if self.data_socket: + self.data_socket.close() + + for t in self.threads: + if t.is_alive(): + t.join(timeout=1.0) + + def _validate_init_params(self): + + # Validate the local_interface is a valid IP address + if not self.local_interface or not isinstance(self.local_interface, str) or self.local_interface.count('.') != 3: + raise ValueError(f"Invalid local interface IP address: {self.local_interface}") + + # Validate between transmission types and address requirements + if self.transmission_type not in TransmissionType: + raise ValueError(f"Invalid transmission type: {self.transmission_type}. Must be 'unicast' or 'multicast'.") + + if self.transmission_type == TransmissionType.MULTICAST and not self.multicast_address: + raise ValueError("Multicast address must be provided for multicast transmission type.") + + if self.transmission_type == TransmissionType.UNICAST and self.multicast_address: + raise ValueError("Multicast address should not be provided for unicast transmission type.") + + if not (0 < self.command_port < 65536): + raise ValueError(f"Invalid command port: {self.command_port}. Must be between 1 and 65535.") + + if not (0 < self.data_port < 65536): + raise ValueError(f"Invalid data port: {self.data_port}. Must be between 1 and 65535.") + + if self.command_port == self.data_port: + raise ValueError("Command port and data port must be different.") + + if self.motive_app_version and (not isinstance(self.motive_app_version, tuple) or len(self.motive_app_version) != 4): + raise ValueError(f"Invalid Motive app version: {self.motive_app_version}. Must be a tuple of 4 integers (major, minor, build, revision).") + + if self.natnet_version and (not isinstance(self.natnet_version, tuple) or len(self.natnet_version) != 4): + raise ValueError(f"Invalid NatNet version: {self.natnet_version}. Must be a tuple of 4 integers (major, minor, build, revision).") + + if self.motive_app_version and not self.motive_app_version[0] == 3: + raise ValueError(f"Unsupported Motive app version: {self.motive_app_version}. Minimum supported version is 3.0.0.0. Recommended to use 3.1.0.0") + + if not self.natnet_version[0] == 4: + raise ValueError(f"Unsupported NatNet version: {self.natnet_version}. Minimum supported version is 4.0.0.0. Recommended to use 4.4.0.0") + + if self.high_res_clock_freq <= 0: + raise ValueError( + f"Invalid high resolution clock frequency: {self.high_res_clock_freq}. Must be a positive integer representing the frequency in Hz." + ) + + if self.publish_rate <= 0: + raise ValueError( + f"Invalid publish rate: {self.publish_rate}. Must be a positive number representing Hz." + ) + def _get_latest_mocap_packet(self) -> DataMessages.sFrameOfMocapData | None: + # Thread-safe method to retrieve the latest mocap data to be sent + try: + return self.mocap_data_queue.get_nowait() + except queue.Empty: + return None + + @staticmethod + def _pad_fixed_string(value: bytes) -> bytes: + """Null-pad a byte string to MAX_NAMELENGTH for fixed-size NatNet name fields.""" + truncated = value[: ServerMessages.MAX_NAMELENGTH - 1] + return truncated + b"\x00" * (ServerMessages.MAX_NAMELENGTH - len(truncated)) + + @staticmethod + def _assign_version_bytes(field: ctypes.Array, version: typing.Tuple[int, int, int, int]) -> None: + for index, component in enumerate(version): + field[index] = component + + @staticmethod + def _assign_ipv4_bytes(field: ctypes.Array, address: str | bytes) -> None: + octets = socket.inet_aton(address) if isinstance(address, str) else address + for index, octet in enumerate(octets): + field[index] = octet + + def _build_server_description(self) -> ServerMessages.sServerDescription: + # Helper to build the server description struct with current server info (e.g. on startup or in response to command request) + description = ServerMessages.sServerDescription() + description.HostPresent = True + description.szHostComputerName = self._pad_fixed_string( + socket.gethostname().encode("utf-8") + ) + self._assign_ipv4_bytes(description.HostComputerAddress, self.local_interface) + description.szHostApp = self._pad_fixed_string(b"Motive") + self._assign_version_bytes(description.HostAppVersion, self.motive_app_version) + self._assign_version_bytes(description.NatNetVersion, self.natnet_version) + description.HighResClockFrequency = self.high_res_clock_freq + description.bConnectionInfoValid = True + description.ConnectionDataPort = self.data_port + description.ConnectionMulticast = self.transmission_type == TransmissionType.MULTICAST + + if self.transmission_type == TransmissionType.MULTICAST: + self._assign_ipv4_bytes(description.ConnectionMulticastAddress, self.multicast_address) + else: + self._assign_ipv4_bytes(description.ConnectionMulticastAddress, b"\x00\x00\x00\x00") + + return description + + def _build_connect_response_payload(self) -> bytes: + """NAT_CONNECT reply: libNatNet parses NAT_SERVERINFO payload as sSender_Server.""" + sender = ServerMessages.sSender_Server() + sender.Common.szName = self._pad_fixed_string(b"Motive") + self._assign_version_bytes(sender.Common.Version, self.motive_app_version) + self._assign_version_bytes(sender.Common.NatNetVersion, self.natnet_version) + sender.HighResClockFrequency = self.high_res_clock_freq + sender.DataPort = self.data_port + sender.IsMulticast = self.transmission_type == TransmissionType.MULTICAST + if self.transmission_type == TransmissionType.MULTICAST: + self._assign_ipv4_bytes(sender.MulticastGroupAddress, self.multicast_address) + else: + self._assign_ipv4_bytes(sender.MulticastGroupAddress, b"\x00\x00\x00\x00") + return sender.pack() + + def _send_packet_to_client( + self, + client: Client, + message_id: ServerMessages.MessageId | int, + payload: bytes, + sock: socket.socket | None = None, + ) -> None: + """Send a NatNet packet to a unicast client (libNatNet 4.4). + + Command replies go out the command socket; mocap frames go out the data socket. + """ + if self.shutdown_event.is_set(): + return + sock = sock or self.command_socket + if not sock: + raise ValueError("[NatNetServer] Socket not initialized. Cannot send packet.") + + header = ServerMessages.sPacketHeader( + iMessage=int(message_id), + nDataBytes=len(payload), + ) + packet = header.pack() + payload + try: + with client.socket_lock: + sock.sendto(packet, (client.ip, client.port)) + except OSError as e: + raise ValueError( + f"[NatNetServer] Error sending message {int(message_id)} to " + f"client {client.ip}:{client.port}: {e}" + ) from e + + def _data_update_loop(self): # Stub: Different betweeen multicast and unicast server implementations, as they will need to handle client connections differently (multicast will just send to the multicast group address) + # Loop to update mocap data and send packets at regular intervals. + pass + + def _send_data_packet(self, client: Client, data_message: DataMessages.sFrameOfMocapData): + # Serialize frame payload and send via the data socket. + # + # Stamp the transmit time in the server's high-resolution clock domain so + # the client can recover per-message transit latency via + # NatNetClient::SecondsSinceHostTimestamp(TransmitTimestamp). This must match + # the clock used in the NAT_ECHORESPONSE handshake (time.time() nanoseconds) + # and the advertised HighResClockFrequency (defaults to 1e9 ticks/s), so the + # SDK's server-clock estimate and this timestamp share one timeline. + data_message.TransmitTimestamp = int(time.time() * 1_000_000_000) + try: + packet_bytes = data_message.pack() + except Exception as e: + raise ValueError(f"[NatNetServer] Error serializing data message: {e}") from e + + self._send_packet_to_client( + client, + ServerMessages.MessageId.NAT_FRAMEOFDATA, + packet_bytes, + sock=self.data_socket, + ) + + def _command_listener_loop(self): # Stub: Different betweeen multicast and unicast server implementations, as they will need to handle client connections differently (multicast will just send to the multicast group address) + # Loop to listen for and handle incoming command requests (e.g. from client apps) + pass + + def _handle_command_request(self, request_data: bytes): # Stub: Different betweeen multicast and unicast server implementations, as they will need to handle client connections differently (multicast will just send to the multicast group address) + # Parse incoming command request, perform requested action, and send response if needed + pass + diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server_types.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server_types.py new file mode 100644 index 000000000..3083f1a6e --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server_types.py @@ -0,0 +1,156 @@ +import ctypes +import struct +from enum import IntEnum + +MAX_NAMELENGTH = 256 +MAX_PACKETSIZE = 65503 + +# NatNet SDK sServerDescription uses default struct alignment (#pragma pack(pop)), not pack(1). +SERVER_DESCRIPTION_WIRE_SIZE = 552 +# NAT_CONNECT / NAT_SERVERINFO reply uses packed sSender_Server (#pragma pack(1) in NatNetTypes.h). +SENDER_SERVER_WIRE_SIZE = 256 + 4 + 4 + 8 + 2 + 1 + 4 # 279 + +# Client/server message ids +class MessageId(IntEnum): + NAT_CONNECT = 0 + NAT_SERVERINFO = 1 + NAT_REQUEST = 2 + NAT_RESPONSE = 3 + NAT_REQUEST_MODELDEF = 4 + NAT_MODELDEF = 5 + NAT_REQUEST_FRAMEOFDATA = 6 + NAT_FRAMEOFDATA = 7 + NAT_MESSAGESTRING = 8 + NAT_DISCONNECT = 9 + NAT_KEEPALIVE = 10 + NAT_DISCONNECTBYTIMEOUT = 11 + NAT_ECHOREQUEST = 12 + NAT_ECHORESPONSE = 13 + NAT_DISCOVERY = 14 + NAT_UNRECOGNIZED_REQUEST = 100 + +# Server/Sender configuration and info +def _fixed_name(field: ctypes.Array) -> bytes: + raw = bytes(field).split(b"\x00", 1)[0] + b"\x00" + if len(raw) > MAX_NAMELENGTH: + raw = raw[: MAX_NAMELENGTH - 1] + b"\x00" + return raw + b"\x00" * (MAX_NAMELENGTH - len(raw)) + + +class sSender(ctypes.Structure): + _pack_ = 1 + _fields_ = [ + ("szName", ctypes.c_char * MAX_NAMELENGTH), # host app's name + ("Version", ctypes.c_uint8 * 4), # host app's version [major.minor.build.revision] + ("NatNetVersion", ctypes.c_uint8 * 4) # host app's NatNet version + ] + + def pack(self) -> bytes: + payload = bytearray() + payload += _fixed_name(self.szName) + payload += bytes(self.Version) + payload += bytes(self.NatNetVersion) + return bytes(payload) + +class sSender_Server(ctypes.Structure): + _pack_ = 1 + _fields_ = [ + ("Common", sSender), + ("HighResClockFrequency", ctypes.c_uint64), + ("DataPort", ctypes.c_uint16), + ("IsMulticast", ctypes.c_bool), + ("MulticastGroupAddress", ctypes.c_uint8 * 4) + ] + + def pack(self) -> bytes: + payload = bytearray(self.Common.pack()) + payload += struct.pack(" bytes: + # Wire layout matches NatNet SDK on x86-64 (3 pad bytes before HighResClockFrequency). + payload = bytearray() + payload.append(1 if self.HostPresent else 0) + payload += _fixed_name(self.szHostComputerName) + payload += bytes(self.HostComputerAddress) + payload += _fixed_name(self.szHostApp) + payload += bytes(self.HostAppVersion) + payload += bytes(self.NatNetVersion) + while len(payload) % 8: + payload.append(0) + payload += struct.pack(" bytes: + return bytes(self) + +# Connection types enum matching NatNet SDK rules +class ConnectionType(IntEnum): + ConnectionType_Multicast = 0 + ConnectionType_Unicast = 1 + +class sNatNetClientConnectParams(ctypes.Structure): + """ + Python ctypes translation of the C++ sNatNetClientConnectParams struct. + Enforces a packed structure byte alignment matching the NatNet binary network protocol. + """ + _pack_ = 1 + _fields_ = [ + ("connectionType", ctypes.c_int32), # 4 bytes (mapping to standard ConnectionType enum) + ("serverCommandPort", ctypes.c_uint16), # 2 bytes + ("serverDataPort", ctypes.c_uint16), # 2 bytes + + # NOTE: Represented as void pointers (c_void_p) to safely match the host system's native bit size (e.g., 8 bytes on 64-bit) without string data unpacking overhead. + ("serverAddress", ctypes.c_void_p), + ("localAddress", ctypes.c_void_p), + ("multicastAddress", ctypes.c_void_p), + + ("subscribedDataOnly", ctypes.c_bool), # 1 byte + ("BitstreamVersion", ctypes.c_uint8 * 4) # 4 bytes: [Major, Minor, Build, Revision] + ] + + def pack(self) -> bytes: + return bytes(self) \ No newline at end of file diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_unicast_server.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_unicast_server.py new file mode 100644 index 000000000..d9c0a1e22 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_unicast_server.py @@ -0,0 +1,172 @@ +import ctypes +import time + +from . import natnet_server_types as ServerTypes +from .natnet_server import TransmissionType, Client, NatNetServer + + +class NatNetUnicastServer(NatNetServer): + def __init__(self, + local_interface="172.31.0.200", + transmission_type: TransmissionType = TransmissionType.UNICAST, + multicast_address=None, + command_port=1510, + data_port=1511 + ): + + if not transmission_type == TransmissionType.UNICAST: + raise ValueError("Transmission type 'MULTICAST' is not supported in NatNetUnicastServer. Please use NatNetMulticastServer instead.") + + super().__init__(local_interface, transmission_type, multicast_address, command_port, data_port) + + def _data_update_loop(self): + # Loop to update mocap data and send packets at regular intervals. + # When auto_stream is False the frames are pumped externally (Isaac physics step), + # so this thread only idles — but stays alive for clean shutdown. + while not self.shutdown_event.is_set(): + time.sleep(1 / self.publish_rate) + if not self.auto_stream: + continue + self.flush_mocap_data() + + def flush_mocap_data(self): + """Send the latest (or last) mocap frame to every connected client, once.""" + with self.clients_lock: + clients = list(self.connected_clients) + if not clients: + return + + data_messages = self._get_latest_mocap_packet() + + if data_messages is None: # If the server stops producing frames, use the last known frame. + data_messages = self._get_last_known_mocap_frame() + if data_messages is None: + return + + for client in clients: + try: + self._send_data_packet(client, data_messages) + except ValueError as e: + print(str(e)) + continue + + def _command_listener_loop(self): + # Listens on UDP command socket for incoming command requests from clients. + # Handles incoming client handshakes and teardown. + + print(f"[Command Listener] Command listener thread started. Listening for incoming client command requests on UDP address:port {self.local_interface}:{self.command_port}...") + + while not self.shutdown_event.is_set(): + try: + data, addr = self.command_socket.recvfrom(1024) # Buffer size of 1024 bytes should be sufficient for command requests + if not data: + continue + self._handle_command_request(data, addr) + except Exception as e: + if self.shutdown_event.is_set(): + break + print(f"[Command Listener] Error receiving command request: {e}") + time.sleep(0.1) # Sleep briefly to avoid tight loop on errors + + def _handle_command_request(self, request_data: bytes, client_address: tuple): + """ + Processes standard binary headers and registers unicast endpoints. + """ + header_size = ctypes.sizeof(ServerTypes.sPacketHeader) + if len(request_data) < header_size: + return + + # Parse the header via ctypes + header = ServerTypes.sPacketHeader.from_buffer_copy(request_data[:header_size]) + + # Handle Connection Handshake + if header.iMessage == int(ServerTypes.MessageId.NAT_CONNECT): + client_requested_version = self.natnet_version # Fallback to server's version. Version handshaking not supported in this extension. + + client_ip, client_port = client_address + + # Create and store a new client object + new_client = Client(client_ip, client_port, version=client_requested_version) + try: + with self.clients_lock: + self.connected_clients.discard(new_client) # Remove any existing client with the same IP and port + self.connected_clients.add(new_client) # Add the new client to the connected clients list + print(f"[Command Handler] Added client {new_client.ip}:{new_client.port} to connected clients list.") + except Exception as e: + print(f"[Command Handler] Error adding client {new_client.ip}:{new_client.port} to connected clients list: {e}") + return + + try: + self._send_packet_to_client( + new_client, + ServerTypes.MessageId.NAT_SERVERINFO, + self._build_connect_response_payload(), + ) + except ValueError as e: + raise ValueError( + f"[Command Handler] Error sending server description to client {client_address}: {e}" + ) from e + print( + f"[Command Handler] Sent server description to client address " + f"through its port {client_address}." + ) + return + + # Non-handshake commands require a prior NAT_CONNECT from this endpoint. + client_ip, client_port = client_address + client = self._find_client(client_ip, client_port) + if client is None: + print( + f"[Command Handler] Ignoring message {header.iMessage} from " + f"unregistered client {client_address}." + ) + return + + if header.iMessage == int(ServerTypes.MessageId.NAT_REQUEST_MODELDEF): + try: + self._send_packet_to_client( + client, + ServerTypes.MessageId.NAT_MODELDEF, + self._get_model_def_payload(), + ) + except ValueError as e: + print( + f"[Command Handler] Error sending MODELDEF to client " + f"{client_address}: {e}" + ) + return + + if header.iMessage == int(ServerTypes.MessageId.NAT_KEEPALIVE): + # Receiving a keepalive refreshes the client's liveness; nothing to send back. + return + + if header.iMessage == int(ServerTypes.MessageId.NAT_ECHOREQUEST): + echo_payload = request_data[header_size : header_size + header.nDataBytes] + # libNatNet expects clientRequestTimestamp + hostReceivedTimestamp (8 + 8 bytes). + host_ts = int(time.time() * 1_000_000_000).to_bytes(8, "little", signed=False) + response_payload = echo_payload[:8].ljust(8, b"\x00") + host_ts + try: + self._send_packet_to_client( + client, + ServerTypes.MessageId.NAT_ECHORESPONSE, + response_payload, + ) + except ValueError as e: + print( + f"[Command Handler] Error sending ECHORESPONSE to client " + f"{client_address}: {e}" + ) + return + + print( + f"[Command Handler] Unhandled message id {header.iMessage} from " + f"registered client {client_address}." + ) + + def _find_client(self, ip: str, port: int) -> Client | None: + target = Client(ip, port) + with self.clients_lock: + for client in self.connected_clients: + if client == target: + return client + return None diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/setup.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/setup.py new file mode 100644 index 000000000..1a1c153fc --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/setup.py @@ -0,0 +1,23 @@ +"""Isaac Sim extension install metadata for the OptiTrack NatNet emulator.""" + +import os + +from setuptools import find_packages, setup + +EXTENSION_PATH = os.path.dirname(os.path.realpath(__file__)) + +setup( + name="optitrack-natnet-emulator", + version="0.1.0", + description="NatNet UDP server emulator for Isaac Sim and natnet_ros2 integration", + license="MIT", + include_package_data=True, + python_requires=">=3.10", + install_requires=[ + "numpy", + "scipy", + ], + packages=find_packages(where="."), + package_dir={"": "."}, + zip_safe=False, +) diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/natnet_test_helpers.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/natnet_test_helpers.py new file mode 100644 index 000000000..07d6c8a5e --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/natnet_test_helpers.py @@ -0,0 +1,105 @@ +"""Shared helpers for optitrack.natnet.emulator unit tests.""" + +from __future__ import annotations + +import socket +import struct +import time +from contextlib import contextmanager + +from optitrack.natnet.emulator import NatNetUnicastServer, TransmissionType +from optitrack.natnet.emulator.server import natnet_server_types as st + + +def ephemeral_udp_port(host: str = "127.0.0.1") -> int: + """Return a free UDP port on *host* by binding and releasing a probe socket.""" + with socket.socket(socket.AF_INET, socket.SOCK_DGRAM) as probe: + probe.bind((host, 0)) + return probe.getsockname()[1] + + +class NatNetTestClient: + """Minimal UDP client for NatNet command-port protocol tests.""" + + def __init__(self, host: str = "127.0.0.1", timeout: float = 2.0) -> None: + self._host = host + self._sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) + self._sock.bind((host, 0)) + self._sock.settimeout(timeout) + + @property + def local_port(self) -> int: + return self._sock.getsockname()[1] + + def send_message( + self, + server_port: int, + message_id: st.MessageId | int, + payload: bytes = b"", + server_host: str | None = None, + ) -> None: + header = st.sPacketHeader( + iMessage=int(message_id), + nDataBytes=len(payload), + ) + self.send_raw(header.pack() + payload, server_port, server_host) + + def send_raw( + self, + data: bytes, + server_port: int, + server_host: str | None = None, + ) -> None: + """Send a raw UDP datagram (for malformed / malicious packet tests).""" + self._sock.sendto(data, (server_host or self._host, server_port)) + + def send_header_only( + self, + server_port: int, + message_id: st.MessageId | int, + declared_payload_len: int, + server_host: str | None = None, + ) -> None: + """Send a header whose nDataBytes does not match any trailing payload.""" + header = struct.pack(" tuple[int, bytes, tuple[str, int]]: + data, addr = self._sock.recvfrom(65535) + message_id, payload_len = struct.unpack(" None: + self._sock.close() + + +@contextmanager +def running_unicast_server( + command_port: int | None = None, + local_interface: str = "127.0.0.1", + publish_rate: int = 100, +): + """Start NatNetUnicastServer on ephemeral (or fixed) command + data ports. + + Both ports are ephemeral by default so concurrent/sequential tests never + collide on the well-known 1510/1511 pair. + """ + port = command_port if command_port is not None else ephemeral_udp_port(local_interface) + data_port = ephemeral_udp_port(local_interface) + while data_port == port: + data_port = ephemeral_udp_port(local_interface) + server = NatNetUnicastServer( + local_interface=local_interface, + transmission_type=TransmissionType.UNICAST, + multicast_address=None, + command_port=port, + data_port=data_port, + ) + server.publish_rate = publish_rate + server.start() + time.sleep(0.05) + try: + yield server, port + finally: + server.shutdown() diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_defaults.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_defaults.py new file mode 100644 index 000000000..7cc72b036 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_defaults.py @@ -0,0 +1,26 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Unit tests for hardcoded tracked-body defaults.""" + +from __future__ import annotations + +import pytest + +from optitrack.natnet.emulator.defaults import ( + DEFAULT_DRONE_BINDING, + DEFAULT_TRACKED_BODY_BINDINGS, +) + + +pytestmark = pytest.mark.unit + + +def test_default_drone_binding_matches_natnet_ros2_config(): + assert DEFAULT_DRONE_BINDING.name == "Drone" + assert DEFAULT_DRONE_BINDING.id == 1 + assert DEFAULT_DRONE_BINDING.parent_id == -1 + assert DEFAULT_DRONE_BINDING.prim_path == "/World/base_link" + + +def test_default_tracked_body_bindings_contains_drone_only(): + assert DEFAULT_TRACKED_BODY_BINDINGS == (DEFAULT_DRONE_BINDING,) diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_serializers.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_serializers.py new file mode 100644 index 000000000..47965b8f6 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_serializers.py @@ -0,0 +1,367 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Unit tests for NatNet wire serializers (no network).""" + +from __future__ import annotations + +import ctypes +import struct + +import pytest + +from optitrack.natnet.emulator import NatNetUnicastServer, TransmissionType +from optitrack.natnet.emulator.server import natnet_data_types as dt +from optitrack.natnet.emulator.server import natnet_model_types as mt +from optitrack.natnet.emulator.server import natnet_server_types as st +from optitrack.natnet.emulator.server.natnet_common import ModelLimits + + +pytestmark = pytest.mark.unit + + +# ============================================================================= +# natnet_server_types — transport / handshake +# ============================================================================= + + +def test_packet_header_pack_size_and_endianness(): + header = st.sPacketHeader( + iMessage=int(st.MessageId.NAT_FRAMEOFDATA), + nDataBytes=42, + ) + packed = header.pack() + + assert len(packed) == ctypes.sizeof(st.sPacketHeader) == 4 + message_id, payload_len = struct.unpack(" server only; real Motive sends no reply. An echo + # reply makes libNatNet log "Received unrecognized message Message=10". + with running_unicast_server() as (server, command_port): + client = NatNetTestClient(timeout=0.5) + try: + client.send_message(command_port, st.MessageId.NAT_CONNECT) + client.recv_message() + + client.send_message(command_port, st.MessageId.NAT_KEEPALIVE) + with pytest.raises(socket.timeout): + client.recv_message() + + # Client stays registered and keeps receiving frames. + assert len(server.connected_clients) == 1 + finally: + client.close() + + +# ============================================================================= +# Malformed datagrams — registered client & recovery +# ============================================================================= + + +def test_unknown_message_from_registered_client_gets_no_reply(): + with running_unicast_server() as (server, command_port): + client = NatNetTestClient(timeout=0.5) + try: + client.send_message(command_port, st.MessageId.NAT_CONNECT) + client.recv_message() + + client.send_message(command_port, 999) + with pytest.raises(socket.timeout): + client.recv_message() + + assert len(server.connected_clients) == 1 + finally: + client.close() + + +def test_server_survives_malformed_burst_then_valid_connect(): + with running_unicast_server() as (server, command_port): + client = NatNetTestClient(timeout=2.0) + try: + client.send_raw(b"", command_port) + client.send_raw(b"\xff", command_port) + client.send_header_only(command_port, 999, declared_payload_len=50000) + client.send_message(command_port, st.MessageId.NAT_REQUEST_MODELDEF) + + client.send_message(command_port, st.MessageId.NAT_CONNECT) + message_id, _payload, _addr = client.recv_message() + finally: + client.close() + + assert message_id == int(st.MessageId.NAT_SERVERINFO) + assert len(server.connected_clients) == 1 diff --git a/tests/colcon_unit_test_packages.yaml b/tests/colcon_unit_test_packages.yaml index 6d96da1e5..eb605498f 100644 --- a/tests/colcon_unit_test_packages.yaml +++ b/tests/colcon_unit_test_packages.yaml @@ -15,3 +15,9 @@ robot: # ament pytest does not honor PYTEST_ADDOPTS -m. # launch_testing is skipped via PYTEST_DISABLE_PLUGIN_AUTOLOAD in the test. pytest_args: [] + +# Simulation-side extensions (globbed under simulation/**//test). Collected by +# `pytest tests/` on the host runner; not part of the robot colcon workspace. +sim: + packages: + - optitrack.natnet.emulator diff --git a/tests/conftest.py b/tests/conftest.py index 203e9d611..29a7c7220 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -77,6 +77,20 @@ def pytest_configure(config): run_dir = session.init_run_dir(AIRSTACK_ROOT) config.option.xmlpath = str(run_dir / "results.xml") + # Co-located unit tests import their own package (e.g. `optitrack.natnet.emulator`, + # `lidar_point_cloud_filter.validation_core`). Put each package/extension import + # root (the parent of its test/ dir) on sys.path so they resolve without a + # per-package conftest.py — a second conftest.py collides with this root one as + # module `conftest` under --import-mode=importlib and breaks `from conftest import`. + # The test/ dir itself goes on too, for sibling helper modules, but only when it + # ships no conftest.py — otherwise that file wins the `conftest` name and the + # collision above is exactly what happens. + for d in unit_test_dirs(): + roots = [d.parent] if (d / "conftest.py").exists() else [d.parent, d] + for root in roots: + if str(root) not in sys.path: + sys.path.insert(0, str(root)) + # Collect co-located unit tests: their files live outside tests/, so add the # explicit non-linter test files to the collection args. Skip when an explicit # path was given on the CLI (args_source == ARGS) so `pytest tests/system/foo.py` diff --git a/tests/integration/natnet/README.md b/tests/integration/natnet/README.md new file mode 100644 index 000000000..4e9330c32 --- /dev/null +++ b/tests/integration/natnet/README.md @@ -0,0 +1,151 @@ +# NatNet ↔ robot autonomy integration + +Host-side NatNet wire-protocol tests that drive the Python emulator against +`natnet_ros2_node` in a real robot container. First resident of the +[`integration`](../README.md) tier (no sim, no GPU). + +Mark: `integration`. Filter this scenario with `tests/integration/natnet/`. + +For the **in-sim** end-to-end check (Isaac emulator + full stack), see +[Liveliness sentinel](#liveliness-sentinel-sim-end-to-end) below and +[`tests/system/test_liveliness.py`](../../system/test_liveliness.py). + +## What it verifies + +Three variants in [`test_natnet_integration.py`](test_natnet_integration.py). +All start a host-side `NatNetUnicastServer`, launch `natnet_ros2_node` in the +robot container pointed at the Docker bridge gateway, and assert a sustained +pose stream at **≥ 5 Hz** on the configured topic(s), e.g.: + +- `/{ROBOT_NAME}/perception/optitrack/drone/pose_cov` (wait for first message) +- `/{ROBOT_NAME}/perception/optitrack/drone` (Hz sample) + +| Test | Path | +|------|------| +| **`test_natnet_ros2_receives_drone_pose_hz`** | Hand-built `sFrameOfMocapData` frames enqueued on a raw `NatNetUnicastServer` (no USD). Minimal wire + SDK check. | +| **`test_natnet_ros2_receives_isaac_wrapper_pose_hz`** | Full Isaac data path: in-memory USD stage, `NatNetInterfaceConfig`, `author_interface`, `NatNetServerManager.sample_once()` on a moving prim — same sampling logic as the in-sim physics-step callback. Skips without `usd-core` (`pxr`). Pose-value fidelity is covered hermetically by the emulator's `test_pose_streaming.py` loopback. | +| **`test_natnet_ros2_multi_body_drone_and_target`** | Two bodies (drone id 1 + target id 100) with distinct relative topics; asserts both pose streams and that the target's `pose_cov` topic is **absent** (`body_pose_cov=false`). Exercises the multi-body profile + per-body `pose`/`pose_cov` toggles. | + +These tests **do not** start the full perception bringup or `LAUNCH_NATNET`; they +exec `natnet_ros2_node` directly with the flattened per-body params +(`body_names`/`body_ids`/`body_topics`/`body_pose`/`body_pose_cov`) and no MAVROS bridge. + +## Requirements + +- Docker daemon (robot-desktop container reachable from pytest). +- **`natnet_ros2_node` built** in the robot image (OptiTrack NatNet SDK is + license-gated — run `airstack setup --natnet`, then + `bws --packages-select natnet_ros2` in the container). Tests **skip** if the + node binary is missing. +- Host-side emulator package on `PYTHONPATH` (the test adds + `simulation/isaac-sim/extensions/optitrack.natnet.emulator` — not pip-installed + on the host). +- Ephemeral UDP ports on the host gateway IP (Docker default route as seen from + inside the container). + +The robot container comes from the shared **`robot_autonomy_stack`** fixture in +[`tests/conftest.py`](../../conftest.py) (see the [integration tier README](../README.md)). + +## Running + +```bash +# 1. One-time: NatNet SDK + build natnet_ros2 in the robot image +airstack setup --natnet # or NATNET_ACCEPT_LICENSE=1 airstack setup --natnet +docker exec airstack-robot-desktop-1 bash -lc 'bws --packages-select natnet_ros2' + +# 2a. Reuse an existing robot container (fast local iteration): +AUTOLAUNCH=false airstack up robot-desktop +pytest tests/integration/natnet/ -m integration -v + +# 2b. Let the harness bring the container up/down: +pytest tests/integration/natnet/ -m integration -v +``` +On CI / PR (write access): `/pytest -m integration` + +## Architecture + +``` +┌──────────────────────────────────────────────────────────────┐ +│ Host (pytest) │ +│ NatNetUnicastServer @ docker bridge gateway IP │ +│ • raw variant: hand-built frame queue │ +│ • Isaac variant: NatNetServerManager.sample_once(USD) │ +└────────────────────────────┬─────────────────────────────────┘ + │ UDP unicast (cmd + data ports) +┌────────────────────────────▼─────────────────────────────────┐ +│ Robot container (robot-desktop) │ +│ natnet_ros2_node (libNatNet 4.4 client) │ +│ → /{ROBOT_NAME}/{body topic}[/pose_cov] per configured body │ +└──────────────────────────────────────────────────────────────┘ +``` + +**In sim (liveliness tier):** the server runs inside the Isaac Sim container +(`172.31.0.200` by default). Use a NatNet Pegasus launch script +(`example_one_px4_pegasus_natnet_launch_script.py` or +`example_multi_px4_pegasus_natnet_launch_script.py`); `natnet_ros2` in the +robot stack connects via `natnet_config.yaml` (`server_ip` → emulator IP). + +**Catalog / MODELDEF:** The server holds a MODELDEF **wire cache** only +(`set_model_def_payload()`). Scene semantics (body names, streaming IDs, target +prim paths) come from the Isaac layer (`NatNetInterfaceConfig`, USD interface +prim, or launch-script `build_drone_config`). See the +[emulator README](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md). + +## Liveliness sentinel (sim end-to-end) + +The integration tier proves **robot client + host emulator** without Isaac. +The matching **system** check is +`TestLiveliness::test_natnet_pose_alive` in +[`test_liveliness.py`](../../system/test_liveliness.py): + +- **Gated on `LAUNCH_NATNET=true`** (skipped otherwise — normal liveliness runs + are unaffected). +- Asserts `/{robot_n}/{natnet pose topic}/pose_cov` ≥ 5 Hz per robot (the drone + body's configured topic — default `perception/optitrack/drone`). +- Override the checked topic with `NATNET_POSE_TOPIC` (default + `perception/optitrack/drone`). The sim body name (`NATNET_BODY_NAME`, default + `Drone`) is decoupled from the published topic, which the robot profile sets. + +Sim auto-start: set `ISAAC_SIM_SCRIPT_NAME` to a NatNet launch script and +`LAUNCH_NATNET=true` on the robot. Convenience bundle: +`airstack up --env-file overrides/isaac-natnet-vision.env` (NatNet script + +PX4 external-vision SITL profile). + +## libNatNet 4.4 unicast — verified wire contract + +The emulator is validated against the **real `libNatNet.so`** (not just the Python +`NatNetClient`) with a minimal C probe that registers `SetFrameReceivedCallback` +and `NatNet_SetLogCallback`. All of the following must hold for the SDK to deliver +frames to the callback: + +| Requirement | Why | +|-------------|-----| +| `NAT_CONNECT` → `sSender_Server` (279 B), name `Motive` | libNatNet reads `Motive 3.1 / NatNet 4.4` | +| `NAT_ECHOREQUEST` → `NAT_ECHORESPONSE` (16 B) | Prevents libNatNet assert | +| Frame ends with a **4-byte end-of-data tag** after `params` | libNatNet's frame unpacker reads it; without it the unpacked size mismatches `nDataBytes` and **every frame is silently dropped** | +| `NAT_FRAMEOFDATA` sent from the **data port** (source port == `data_port`) | libNatNet routes unicast frames by the server's data port. Frames sent from the **command** port are treated as command traffic and dropped — no error, no callback | +| `NAT_KEEPALIVE` gets **no reply** | An echo reply makes libNatNet log `Received unrecognized message Message=10` | + +With these in place the C probe reports `Server: Motive 3.1.0.0 NatNet 4.4.0.0`, +`data descriptions: 1`, and **~74 Hz** of frame callbacks. + +> The lenient Python `NatNetClient` accepts frames *without* the end-of-data tag +> and *on the command port*, which is why it appeared to work while libNatNet did +> not. Always validate against the C SDK. + +Full handshake notes and sniffing workflow: +[optitrack-development skill](../../../.agents/skills/optitrack-development/SKILL.md). + +## After changing natnet_ros2 or the emulator + +Rebuild in the robot container: + +```bash +docker exec airstack-robot-desktop-1 bash -lc 'bws --packages-select natnet_ros2' +``` + +Unit tests (protocol, serializers, Isaac wrapper loopback): + +```bash +pytest tests/sim/optitrack_natnet_emulator/ -m unit -v +``` diff --git a/tests/integration/natnet/test_natnet_integration.py b/tests/integration/natnet/test_natnet_integration.py new file mode 100644 index 000000000..e386fd1ee --- /dev/null +++ b/tests/integration/natnet/test_natnet_integration.py @@ -0,0 +1,302 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""NatNet - robot autonomy integration tests. + +Host-side variants stream frames to ``natnet_ros2_node`` in the robot container and +assert pose topics stay alive at >= 5 Hz: (1) raw ``NatNetUnicastServer`` hand-built +single-body frames; (2) ``NatNetServerManager`` sampling an in-memory USD stage +(Isaac wrapper path, no sim/GPU); (3) a multi-body profile (drone + target) that +exercises per-body topic overrides and the pose / pose_cov toggles. + +The node is parameterised with the flattened per-body arrays +(``body_names`` / ``body_ids`` / ``body_topics`` / ``body_pose`` / ``body_pose_cov``) +that natnet_ros2.launch.py derives from a robot's natnet_config.yaml profile. + +Multi-robot (NUM_ROBOTS=3, per-robot profiles) is exercised in-sim by +``tests/system/test_liveliness.py::test_natnet_pose_alive``. +""" + +from __future__ import annotations + +import subprocess +import sys +import threading +import time + +import pytest + +from conftest import ( # noqa: E402 — pytest adds tests/ to sys.path + docker_exec, + repo_path, + ros2_env, + sample_hz, + wait_for_first_message, +) + +# Emulator is not pip-installed on the host; add extension root + test helpers. +_EXT_ROOT = repo_path("simulation/isaac-sim/extensions/optitrack.natnet.emulator") +for _path in (_EXT_ROOT, _EXT_ROOT / "test"): + if str(_path) not in sys.path: + sys.path.insert(0, str(_path)) + +from optitrack.natnet.emulator import NatNetUnicastServer, TransmissionType # noqa: E402 +from optitrack.natnet.emulator.server import natnet_data_types as dt # noqa: E402 +from natnet_test_helpers import ephemeral_udp_port # noqa: E402 + +pytestmark = pytest.mark.integration + +_ROBOT_SETUP = "/root/AirStack/robot/ros_ws/install/setup.bash" +_NATNET_NODE = "/root/AirStack/robot/ros_ws/install/natnet_ros2/lib/natnet_ros2/natnet_ros2_node" +_WARMUP_S = 2.0 +_STREAM_HOLD_S = 12.0 +_MIN_HZ = 5.0 + +# Robot image has route/netstat but not `ip`; /proc/net/route is always present. +_DEFAULT_GATEWAY_CMD = ( + """awk '$2 == "00000000" { printf "%d.%d.%d.%d\\n", """ + """"0x" substr($3,7,2), "0x" substr($3,5,2), "0x" substr($3,3,2), "0x" substr($3,1,2); exit }' """ + """/proc/net/route""" +) + + +def _docker_default_gateway(container: str) -> str: + result = docker_exec(container, _DEFAULT_GATEWAY_CMD, timeout=10) + gateway = result.stdout.strip() + if not gateway: + pytest.skip(f"Could not resolve default gateway inside {container}") + return gateway + + +def _container_env(container: str, var: str, default: str) -> str: + # ROBOT_NAME / ROS_DOMAIN_ID are set in .bashrc (login shell), not container ENV. + # .bashrc may print "Sourcing ..." to stdout; take the last line as the value. + result = docker_exec(container, f"bash -lc 'echo ${var}'") + lines = [line.strip() for line in result.stdout.splitlines() if line.strip()] + value = lines[-1] if lines else "" + return value if value else default + + +def _natnet_node_available(container: str) -> bool: + result = docker_exec(container, f"test -x {_NATNET_NODE} && echo yes || echo no") + return "yes" in result.stdout + + +def _stop_stale_natnet_nodes(container: str) -> None: + docker_exec(container, "pkill -f natnet_ros2_node || true") + time.sleep(0.5) + + +# Each body: (streaming_id, rigid_body_name). The raw server frame carries ids only; +# the node maps ids → topics via its body_* params. +_DRONE_BODY = (1, "Drone") +_TARGET_BODY = (100, "Target") + + +def _make_frame(frame_num: int, body_ids) -> dt.sFrameOfMocapData: + frame = dt.sFrameOfMocapData() + frame.iFrame = frame_num + frame.nRigidBodies = len(body_ids) + for slot, body_id in enumerate(body_ids): + rb = frame.RigidBodies[slot] + rb.ID = body_id + rb.qw = 1.0 + # Bit 0 = tracking valid; natnet_ros2 skips bodies without it (natnet_logic.hpp). + rb.params = 1 + return frame + + +def _frame_publisher( + server: NatNetUnicastServer, stop_event: threading.Event, body_ids=(1,) +) -> None: + frame_num = 0 + interval = 1.0 / server.publish_rate + while not stop_event.is_set(): + server.enqueue_mocap_data(_make_frame(frame_num, body_ids)) + frame_num += 1 + time.sleep(interval) + + +def _launch_natnet_node(container, host_ip, command_port, domain_id, bodies=None): + """Start natnet_ros2_node in the container pointed at the host emulator. + + ``bodies`` is a list of (id, name, topic, pose, pose_cov); defaults to a single + Drone body on topic ``perception/optitrack/drone`` (the shipped config default). + """ + if bodies is None: + bodies = [(1, "Drone", "perception/optitrack/drone", "true", "true")] + ids = ",".join(str(b[0]) for b in bodies) + names = ",".join(b[1] for b in bodies) + topics = ",".join(b[2] for b in bodies) + pose = ",".join(b[3] for b in bodies) + pose_cov = ",".join(b[4] for b in bodies) + launch_cmd = ( + f"bash -lc '{ros2_env(_ROBOT_SETUP, domain_id)} && " + f"exec {_NATNET_NODE} --ros-args " + f"-p server_ip:={host_ip} " + f"-p command_port:={command_port} " + f"-p body_names:=[{names}] " + f"-p body_ids:=[{ids}] " + f"-p body_topics:=[{topics}] " + f"-p body_pose:=[{pose}] " + f"-p body_pose_cov:=[{pose_cov}]'" + ) + return subprocess.Popen( + ["docker", "exec", container, "bash", "-c", launch_cmd], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + ) + + +def _assert_pose_stream( + container, robot_name, domain_id, topic="perception/optitrack/drone", pose_cov=True +): + """Wait for the pose topic then assert a sustained rate >= _MIN_HZ. + + A body configured with ``body_pose_cov=false`` never publishes the ``/pose_cov`` + variant, so detect the first message on whichever topic the body actually emits. + """ + pose_topic = f"/{robot_name}/{topic}" + detect_topic = f"{pose_topic}/pose_cov" if pose_cov else pose_topic + + time.sleep(_WARMUP_S) + first_msg_s = wait_for_first_message( + container, detect_topic, domain_id, _ROBOT_SETUP, timeout=int(_STREAM_HOLD_S) + ) + assert first_msg_s is not None, ( + f"No messages on {detect_topic} within {_STREAM_HOLD_S}s " + "(NatNet connect or frame stream failed)" + ) + hz = sample_hz( + container, + pose_topic, + domain_id, + _ROBOT_SETUP, + duration=min(8, int(_STREAM_HOLD_S - first_msg_s)), + window=20, + ) + assert hz is not None, f"No sustained stream on {pose_topic}" + assert hz >= _MIN_HZ, f"Expected >= {_MIN_HZ} Hz on {pose_topic}, got {hz}" + + +def _terminate(proc) -> None: + if proc is None: + return + proc.terminate() + try: + proc.wait(timeout=5) + except subprocess.TimeoutExpired: + proc.kill() + + +def test_natnet_ros2_receives_drone_pose_hz(robot_autonomy_stack): + """Raw-server path: hand-built frames on NatNetUnicastServer.""" + container = robot_autonomy_stack["container"] + + if not _natnet_node_available(container): + pytest.skip( + "natnet_ros2_node not built — run airstack setup (NatNet SDK) and " + "bws --packages-select natnet_ros2 in the robot container" + ) + + _stop_stale_natnet_nodes(container) + + host_ip = _docker_default_gateway(container) + command_port = ephemeral_udp_port(host_ip) + robot_name = _container_env(container, "ROBOT_NAME", "robot_1") + domain_id = int(_container_env(container, "ROS_DOMAIN_ID", "0")) + + server = NatNetUnicastServer( + local_interface=host_ip, + transmission_type=TransmissionType.UNICAST, + multicast_address=None, + command_port=command_port, + ) + server.publish_rate = 50 + + stop_event = threading.Event() + publisher = threading.Thread( + target=_frame_publisher, args=(server, stop_event), daemon=True + ) + + node_proc: subprocess.Popen[str] | None = None + try: + # Seed dummy frames before the client connects; keep streaming the whole window. + publisher.start() + time.sleep(0.1) + server.start() + node_proc = _launch_natnet_node(container, host_ip, command_port, domain_id) + _assert_pose_stream(container, robot_name, domain_id) + finally: + stop_event.set() + publisher.join(timeout=2.0) + _terminate(node_proc) + server.shutdown() + + +def test_natnet_ros2_multi_body_drone_and_target(robot_autonomy_stack): + """Multi-body profile: one robot tracks a drone + a static target. + + Streams two bodies (drone id 1, target id 100) and configures the node like a + robot profile with two bodies and distinct relative topics. Asserts: the drone + pose streams >= 5 Hz on its custom topic; the target pose streams on its own + topic; and the target's pose_cov topic is absent (body_pose_cov=false). + """ + container = robot_autonomy_stack["container"] + + if not _natnet_node_available(container): + pytest.skip("natnet_ros2_node not built — run airstack setup (NatNet SDK)") + + _stop_stale_natnet_nodes(container) + + host_ip = _docker_default_gateway(container) + command_port = ephemeral_udp_port(host_ip) + robot_name = _container_env(container, "ROBOT_NAME", "robot_1") + domain_id = int(_container_env(container, "ROS_DOMAIN_ID", "0")) + + server = NatNetUnicastServer( + local_interface=host_ip, + transmission_type=TransmissionType.UNICAST, + multicast_address=None, + command_port=command_port, + ) + server.publish_rate = 50 + + bodies = [ + (_DRONE_BODY[0], _DRONE_BODY[1], "perception/optitrack/drone", "true", "true"), + (_TARGET_BODY[0], _TARGET_BODY[1], "perception/optitrack/target", "true", "false"), + ] + body_ids = (_DRONE_BODY[0], _TARGET_BODY[0]) + + stop_event = threading.Event() + publisher = threading.Thread( + target=_frame_publisher, args=(server, stop_event, body_ids), daemon=True + ) + + node_proc: subprocess.Popen[str] | None = None + try: + publisher.start() + time.sleep(0.1) + server.start() + node_proc = _launch_natnet_node(container, host_ip, command_port, domain_id, bodies) + # Drone (pose + pose_cov) and target (pose only) both stream. + _assert_pose_stream(container, robot_name, domain_id, "perception/optitrack/drone") + _assert_pose_stream( + container, robot_name, domain_id, "perception/optitrack/target", pose_cov=False + ) + + # body_pose_cov=false → the target pose_cov publisher must not exist. + target_cov = f"/{robot_name}/perception/optitrack/target/pose_cov" + topics = docker_exec( + container, + f"bash -lc '{ros2_env(_ROBOT_SETUP, domain_id)} && ros2 topic list'", + timeout=15, + ).stdout + assert target_cov not in topics.split(), ( + f"{target_cov} should not exist when body_pose_cov=false; topics:\n{topics}" + ) + finally: + stop_event.set() + publisher.join(timeout=2.0) + _terminate(node_proc) + server.shutdown() From 1c41f8c029a6b579fa3910e8c69c0dcc02e78c22 Mon Sep 17 00:00:00 2001 From: John Liu <63010779+JohnYanxinLiu@users.noreply.github.com> Date: Mon, 17 Aug 2026 01:15:49 -0400 Subject: [PATCH 17/21] OptiTrack (3/3): Isaac wrapper, mocap EV fusion in sim, and a Circle-trajectory e2e (#376) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(sim): Isaac wrapper for the NatNet emulator (USD scene → server) The Isaac integration layer that maps a live USD scene onto the NatNet server: catalog/config/frames/manager/scene_setup/ui_extension/usd_bindings, the extension manifest (config/), and the USD schema. Adds the natnet Pegasus launch scripts that spawn the emulator alongside PX4 in Isaac Sim, the isaac unit tests (incl. a float-tolerance loosen on the pose round-trip for float32/USD noise), and the Isaac-wrapper host integration test. scipy + usd-core added for the emulator's USD/pose-sampling tests. Co-Authored-By: Claude Opus 4.8 * test(natnet): dedicated OptiTrack sim e2e (optitrack mark) One dedicated Isaac bring-up (example_one_px4_pegasus_natnet_launch_script + LAUNCH_NATNET=true) that asserts the full NatNet chain: emulator → natnet_ros2 pose_cov >= 5 Hz, then PX4 local_position alive (EKF2 fusing the vision). Its own `optitrack` mark + _MODULE_ORDER slot — deliberately NOT a third parametrized sim, so the generic liveliness/sensors/flight suites aren't re-run under NatNet. Co-Authored-By: Claude Opus 4.8 * docs(natnet): emulator sim doc + optitrack-development skill Add the NatNet emulator Isaac Sim documentation (docs/simulation/isaac_sim/ natnet_emulator.md) and the optitrack-development agent skill covering the emulator, natnet_ros2, and the NatNet wire-protocol handshake. Co-Authored-By: Claude Opus 4.8 * chore: bump version to 0.19.0-alpha.16 * fix(sim): register the NatNet emulator via the Kit ext-folder The Isaac launch scripts import `optitrack.natnet.emulator`, but Kit was only pointed at the shared exts dir (`~/.local/share/ov/data/documents/Kit/shared/exts`), where Dockerfile.isaac-ros installs pegasus.simulator at image build. The emulator lives in the repo at simulation/isaac-sim/extensions/ and is never copied there, so it was not a registered extension and the import depended on ambient sys.path. Kit accepts repeated --ext-folder, so both standalone commands now pass the repo's extensions dir as a second search root. Chosen over copying the extension into the shared dir at build time because the repo tree is bind-mounted: emulator edits take effect on relaunch instead of requiring an image rebuild. Co-Authored-By: Claude Opus 5 * make the sim actually fuse the mocap stream EKF2_EV_CTRL defaults to 0, and the isaac compose set no PX4 params at all, so PX4 discarded the vision entirely and flew on sim GPS. The emulator could stream perfectly and change nothing. PX4 SITL's rcS applies any PX4_PARAM_ env var at boot and Pegasus passes the container env through, so no new mechanism is needed. Each entry defaults to PX4's own default, read out of the firmware in this image — unset is an explicit no-op and non-mocap sims are unaffected. They cannot be defined-but-empty: the rcS loop has no empty-value guard. Also hooks NATNET_BODY_ID in the single-drone launch script. The emulator hardcoded streaming id 1 while the client reads the env var, so a real Motive id would desync the two into a connected client that never publishes. Co-Authored-By: Claude Opus 5 * fly a circle on mocap fusion instead of asserting a topic exists test_px4_fuses_vision claimed to prove EKF2 fused the external vision but only waited for local_position/pose, which publishes off GPS regardless — it passed with vision disabled. The stack now comes up with GPS, baro and range aiding off, so mocap is the vehicle's only position source, and the module flies the Circle trajectory. Sustained lateral motion is where a wrong EV delay or a too-tight innovation gate shows up; a hover would not reveal either. Cross-track error is scored by the same helpers the autonomy benchmark uses, imported rather than reimplemented. test_px4_fuses_vision is kept as the pre-flight gate — it now establishes only that an estimate exists, and says so. Co-Authored-By: Claude Opus 5 * enforce only the mocap circle flight on PR open The pull_request branch passed no args, so opening a PR ran pytest's defaults: every mark, both sims, all four trajectory types. Now it runs the one end-to-end flight that covers the whole chain. Every other suite is unchanged and still reachable on demand — /pytest comments, workflow_dispatch inputs, and local airstack test. Co-Authored-By: Claude Opus 5 * add an isaac natnet mocap override Brings up the emulator plus PX4 on external-vision fusion in one command — the same configuration test_optitrack_e2e.py uses, so the test environment is reproducible by hand. Sets PLAY_SIM_ON_START explicitly because the root .env ships it false: the scene then loads paused, /clock never ticks, and every use_sim_time node sits frozen while the stack looks healthy. Co-Authored-By: Claude Opus 5 * install the natnet emulator as a real Kit extension The natnet launch scripts died with ModuleNotFoundError: No module named 'optitrack'. Pointing Kit's --ext-folder at the repo extensions dir was not enough — that only makes Kit aware of an extension, it does not put the package on sys.path. Handle it the same way pegasus.simulator already is: bake a copy into the Kit shared exts dir and pip-install it editable, then bind-mount the repo copy over it so edits stay live. The scripts now enable_extension() before importing, which registers the extension and its omni.isaac.core / omni.usd dependencies. The repo-extensions --ext-folder flag is dropped; the extension now lives in the dir the image already searches. Verified in a running container: extension starts, emulator serves on 172.31.0.200 :1510/:1511, and the robot sees /robot_1/perception/optitrack/drone/pose_cov at ~101 Hz feeding vision_pose and PX4 local_position at ~32 Hz. Co-Authored-By: Claude Opus 5 * set the streamed body in the script, not the environment The emulator read NATNET_BODY_NAME / NATNET_BODY_ID from the environment to stay in sync with the client. The client now takes its bodies from its per-robot profile in natnet_config.yaml, so the env hook was asymmetric and, being global, could not describe a multi-robot scene anyway. Both are now constants in the launch scripts, with the pairing spelled out inline, in the emulator sim doc, and in the optitrack-development skill — including that a mismatched id fails silently: the client connects and never publishes. Co-Authored-By: Claude Opus 5 * comment trim on isaac-sim docker compose * point the isaac-sim env blocks at their documentation * comment trim on editable installation of natnet emulator * keep the full default test run on PR open Narrowing the PR gate to `-m 'build_packages or optitrack'` also dropped the unit tier — 155 tests, including the emulator's own suite, which colcon test does not cover (it runs only the robot workspace packages). The optitrack e2e needs no gate of its own: with no -m filter it is collected like everything else, and it brings up its own mocap-EV stack via _E2E_ENV. Only the heavy-mark classification stays, so /pytest -m optitrack still builds sim images instead of taking the pull-only path. * wait for a converged estimate before arming in the optitrack e2e Gate on local_position/odom instead of /pose. odom goes live only once EKF2 has converged and home is set, which is what PX4's arming preflight requires; /pose fires earlier, and the takeoff dispatched in that window returned "failed to arm". Both autonomy suites already gate on odom for this reason (test_px4_ready). The gate alone is not sufficient under external vision: with GPS, baro and range aiding off, PX4's heading and horizontal-position stability checks settle after odom starts publishing — measured at ~26s past the gate. TakeoffTask does not retry its own ARM, so retry here first. * comment trim on optitrack e2e collection ordering * comment trim on the PR-open test args * rename the isaac natnet override to isaac-optitrack-simulation.env * select PX4 SITL parameters with a named env_file The isaac-sim service listed eleven PX4_PARAM_* entries, each defaulting to a hardcoded copy of PX4's own default so that an unset value stayed a no-op — rcS has no empty-value guard. Those copies can drift from firmware silently. Replaced with env_file: ./px4-params/${PX4_PARAM_SET:-default}.env. default.env is empty, so an unselected run injects nothing and PX4 keeps its firmware defaults; external-vision.env holds the mocap set. An unknown name fails the compose config rather than falling back. Also corrects the natnet_emulator doc table, which described three robots, the multi-drone script, and a SITL_PARAM_PROFILE variable that exists nowhere. * comment trim in compose file * trim verbose comments in the natnet sources Shorten multi-line inline comments that explained rationale or compared the chosen approach against alternatives. The longer explanations already live in docs/simulation/isaac_sim/natnet_emulator.md, so the comments now state what the code does and point there. Limited to files this PR adds: the natnet launch scripts and the emulator's isaac/ modules. The env files and the pre-existing launch script keep their original comments. Co-Authored-By: Claude Opus 5 * removed comment change * drop the GPS origin change from the baseline pegasus launch script example_one_px4_pegasus_launch_script.py is a pre-existing non-mocap script and does not need to change for the NatNet emulator work, so restore it to develop. The set_gps_origins call was also inert here: for a single drone spawned at the world origin it computes (38.736832, -9.137977, 90.07), which is the Lisbon default gps_utils already documents, and nothing in the Pegasus submodule reads the PX4_HOME_LAT_ vars it writes. Co-Authored-By: Claude Opus 5 * assert the external-vision params actually reached the FCU The rest of this module assumes PX4_PARAM_SET=external-vision took effect. If it silently does not, EKF2_EV_CTRL stays 0 and EKF2_GPS_CTRL stays 7, the vehicle flies the Circle on sim GPS, and every test still passes — the proof-by-elimination in test_px4_fuses_vision collapses because the elimination never happened. Read EKF2_EV_CTRL and EKF2_GPS_CTRL back off the FCU through the MAVROS param plugin, so the check covers the whole chain: compose env_file -> container env -> Pegasus -> PX4 rcS -> FCU. Runs before the flight tests so a param failure short-circuits in seconds instead of after two 2400s timeouts. Two params, not the full set: if these are right, PX4_PARAM_SET demonstrably applied and the rest came with it. Matching is on the printed value line, not the exit code — an unpulled param prints "Parameter not set." and still exits 0. Verified against a live sim: passes on the real config, fails with distinct messages for a wrong value and for a param that never appears. Co-Authored-By: Claude Opus 5 * docs(natnet): publish the emulator page and correct the setup examples Add the emulator doc to the nav as "MoCap Emulator" — it built and served but was orphaned, so it was only reachable by knowing the URL, and the "See docs/..." pointers in the code led somewhere unnavigable. Fix the launch-script examples in the doc and the extension README. Both omitted enable_extension(), which is the actual prerequisite: the package imports fine because Dockerfile.isaac-ros pip-installs it, but the emulator's modules pull omni.usd / omni.physx lazily, so Kit has to have the extension registered. The doc also carried a sys.path.insert pointing at ../utils (where scene_prep lives) that had nothing to do with the optitrack import. The README targeted /World/drone1/base_link rather than the /body child the launch scripts stream. Document that client registration does not survive a server restart: restart the robot container after Stop/Start Server. Stopping and starting the simulation is unaffected — frames are sampled on the physics step. Co-Authored-By: Claude Opus 5 * feat(natnet): the extension owns the server, tied to the sim timeline The Kit extension is the single owner of the NatNet server. It builds one from the /World/NatNetInterface prim on Play and shuts it down on Stop, so the server's lifetime matches the simulation and the panel reports its state rather than controlling it. Launch scripts author the interface prim before starting the timeline; author_drone_natnet_interface writes the prim and returns the authored config. Because the server is constructed on each Play, serverIp/ports/mode — bound into the socket at construction — pick up whatever is authored at that point. Bodies, up-axis and pose noise are re-read while running and need no rebuild. The panel opens on the interface authored on the stage, so Save writes back what is there; author_interface replaces the whole body set. A client registers with the server instance it connects to, and natnet_ros2 handshakes only until its first success, so a client from an earlier run is unknown to the server built by the next Play. Restart the robot container after each Stop -> Play cycle; documented in natnet_emulator.md. Not exercised against a live panel yet. Co-Authored-By: Claude Opus 5 * docs trim --------- Co-authored-by: Claude Opus 4.8 --- .agents/skills/optitrack-development/SKILL.md | 221 +++++++++ .env | 2 +- .github/workflows/system-tests.yml | 1 + CHANGELOG.md | 8 + docs/simulation/isaac_sim/natnet_emulator.md | 305 ++++++++++++ mkdocs.yml | 1 + overrides/isaac-optitrack-simulation.env | 36 ++ .../isaac-sim/docker/Dockerfile.isaac-ros | 5 + .../isaac-sim/docker/docker-compose.yaml | 5 + .../isaac-sim/docker/px4-params/default.env | 9 + .../docker/px4-params/external-vision.env | 18 + .../optitrack.natnet.emulator/README.md | 19 +- .../config/extension.toml | 23 + .../natnet/emulator/isaac/__init__.py | 62 +++ .../natnet/emulator/isaac/catalog.py | 53 +++ .../optitrack/natnet/emulator/isaac/config.py | 231 +++++++++ .../optitrack/natnet/emulator/isaac/frames.py | 129 +++++ .../natnet/emulator/isaac/manager.py | 444 ++++++++++++++++++ .../natnet/emulator/isaac/scene_setup.py | 128 +++++ .../natnet/emulator/isaac/ui_extension.py | 435 +++++++++++++++++ .../natnet/emulator/isaac/usd_bindings.py | 216 +++++++++ .../schema/schema.usda | 100 ++++ .../test/test_catalog.py | 111 +++++ .../test/test_discovery.py | 29 ++ .../test/test_frames.py | 129 +++++ .../test/test_interface_authoring.py | 126 +++++ .../test/test_interface_config.py | 186 ++++++++ .../test/test_pose_sampling.py | 238 ++++++++++ .../test/test_pose_streaming.py | 86 ++++ .../test/test_scene_setup.py | 100 ++++ .../test/test_server_from_config.py | 85 ++++ .../test/test_server_lifecycle.py | 155 ++++++ .../test/test_target_resolution.py | 84 ++++ ..._multi_px4_pegasus_natnet_launch_script.py | 261 ++++++++++ ...le_one_px4_pegasus_natnet_launch_script.py | 276 +++++++++++ tests/harness/collection.py | 1 + tests/integration/natnet/README.md | 6 +- .../natnet/test_natnet_integration.py | 74 +++ tests/pytest.ini | 1 + tests/requirements.txt | 2 + tests/system/test_optitrack_e2e.py | 299 ++++++++++++ 41 files changed, 4689 insertions(+), 11 deletions(-) create mode 100644 .agents/skills/optitrack-development/SKILL.md create mode 100644 docs/simulation/isaac_sim/natnet_emulator.md create mode 100644 overrides/isaac-optitrack-simulation.env create mode 100644 simulation/isaac-sim/docker/px4-params/default.env create mode 100644 simulation/isaac-sim/docker/px4-params/external-vision.env create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/config/extension.toml create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/__init__.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/catalog.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/config.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/frames.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/manager.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/ui_extension.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/usd_bindings.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/schema/schema.usda create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_catalog.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_discovery.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_frames.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_interface_authoring.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_interface_config.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_sampling.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_streaming.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_scene_setup.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_server_from_config.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_server_lifecycle.py create mode 100644 simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_target_resolution.py create mode 100644 simulation/isaac-sim/launch_scripts/example_multi_px4_pegasus_natnet_launch_script.py create mode 100644 simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_natnet_launch_script.py create mode 100644 tests/system/test_optitrack_e2e.py diff --git a/.agents/skills/optitrack-development/SKILL.md b/.agents/skills/optitrack-development/SKILL.md new file mode 100644 index 000000000..359afa3cd --- /dev/null +++ b/.agents/skills/optitrack-development/SKILL.md @@ -0,0 +1,221 @@ +--- +name: optitrack-development +description: Develop and integrate OptiTrack NatNet in AirStack — robot client (natnet_ros2), Isaac Sim Motive emulator, wire-protocol handshake, and libNatNet 4.4 unicast behavior. Use when working on natnet_ros2, optitrack.natnet.emulator, LAUNCH_NATNET, or NatNet UDP protocol compatibility. +license: Apache-2.0 +metadata: + author: AirLab CMU + repository: AirStack +--- + +# Skill: OptiTrack / NatNet Development + +## When to Use + +- Implementing or debugging the **Motive emulator** in Isaac Sim + (`simulation/isaac-sim/extensions/optitrack.natnet.emulator/`) +- Integrating or testing **`natnet_ros2`** on the robot stack +- Understanding **NatNet wire protocol** (connect, model def, frame streaming) +- Capturing what **`libNatNet.so`** actually sends on the network +- Enabling OptiTrack in sim: `LAUNCH_NATNET=true`, `natnet_config.yaml`, Docker IPs +- Sim testing with mocap: bring the stack up with `overrides/isaac-optitrack-simulation.env`, which starts Isaac + the emulator and switches PX4 EKF2 to external-vision fusion (GPS/baro/range aiding off, so mocap is the only position source) + +## Architecture in AirStack + +```mermaid +flowchart LR + subgraph sim ["Isaac Sim (172.31.0.200)"] + Emulator["optitrack.natnet.emulator\n(NatNet UDP server)"] + end + subgraph robot ["Robot container"] + Node["natnet_ros2_node"] + SDK["libNatNet.so client"] + Node --> SDK + end + SDK -->|"UDP 1510 (unicast: cmd + frames)"| Emulator + Node --> Topics["/{ROBOT_NAME}/perception/optitrack/..."] +``` + +| Component | Path | Role | +|-----------|------|------| +| Robot client | [`robot/ros_ws/src/perception/natnet_ros2/`](../../../robot/ros_ws/src/perception/natnet_ros2/) | ROS 2 node; uses **official NatNet SDK** (`NatNetClient::Connect`) | +| SDK install | `natnet_ros2/lib/libNatNet.so`, `include/natnet/` | Download via `airstack setup --natnet` (proprietary, not in git) | +| Emulator (WIP) | [`simulation/isaac-sim/extensions/optitrack.natnet.emulator/`](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/) | Python NatNet **server** for sim / integration tests | +| Integration tests | [`tests/integration/natnet/README.md`](../../../tests/integration/natnet/README.md) | End-to-end UDP tests against real SDK parser (mark: `integration`) | + +**Enable on robot:** `LAUNCH_NATNET=true` in `.env` → [`perception.launch.xml`](../../../robot/ros_ws/src/perception/perception_bringup/launch/perception.launch.xml) includes `natnet_ros2.launch.py`. + +**Enable in sim:** set ``ISAAC_SIM_SCRIPT_NAME`` to a NatNet Pegasus launch script (no env gate in the script — NatNet always starts): + +| Script | Use | +|--------|-----| +| [`example_one_px4_pegasus_natnet_launch_script.py`](../../../simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_natnet_launch_script.py) | Single drone + static ``Target`` | +| [`example_multi_px4_pegasus_natnet_launch_script.py`](../../../simulation/isaac-sim/launch_scripts/example_multi_px4_pegasus_natnet_launch_script.py) | ``NUM_ROBOTS`` drones + shared ``Target`` (pair with 3-profile ``natnet_config.yaml``) | + +Helpers: [`isaac/scene_setup.py`](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py) (`start_drone_natnet_server`, `author_static_target`). Drone body: single = ``Drone`` (id 1); multi = ``Drone`` (id ``i``); target = ``Target`` (id 100). These are **constants in the launch script**, not env vars — change them there AND in the matching ``natnet_config.yaml`` profile together. The client filters frames by numeric id, so a mismatch is silent: it connects and never publishes. Baseline Pegasus scripts (no NatNet) remain ``example_one_px4_pegasus_launch_script.py`` / ``example_multi_px4_pegasus_launch_script.py``. + +**Default client config:** unicast, `server_ip` → Motive/emulator (use `172.31.0.200` for Isaac container), ports 1510/1511. The config is per-robot: each `robots[$ROBOT_NAME]` profile lists the bodies it tracks (each a `rigid_body_name` + `id` mapped to a relative `topic`, with `pose`/`pose_cov` toggles and per-body covariance) and an optional `vision_pose` block that drives the MAVROS bridge. See [`natnet_config.yaml`](../../../robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml). + +## NatNet: Two UDP Channels + +| Port (server default) | Channel | Direction | +|----------------------|---------|-----------| +| **1510** | Command | Client → server: `NAT_CONNECT`, `NAT_REQUEST_MODELDEF`, keepalives. Server → client: `NAT_SERVERINFO`, `NAT_MODELDEF`, `NAT_RESPONSE` | +| **1511** | Data | Server → client: `NAT_FRAMEOFDATA` (mocap frames). Multicast group `239.255.42.99` when using multicast. **The server must send frames from a socket bound to the data port** (source port == `data_port`); see below. | + +**Critical rules (verified against the real `libNatNet.so` 4.4 unicast + `NatNet_SetLogCallback`):** + +- Command **responses** go to the client's endpoint from `recvfrom` on the server command listener (`1510`), sent via the **command** socket. +- **Frames must be sent from the server's DATA socket** (bound to `data_port`, e.g. `1511`) so the datagram **source port == `data_port`**. libNatNet routes inbound unicast datagrams by source port: frames from the **command** port are treated as command traffic and **silently dropped** (no error, no callback). This was the single biggest gotcha. +- **libNatNet 4.4 unicast uses one client UDP socket** (one ephemeral local port for command send/recv and frame recv). The client receives frames there regardless of the server's source port — but libNatNet only **dispatches** them to the frame callback when they came from the server's data port. Do **not** assume `data_port = cmd_port + 1`. +- **Every `NAT_FRAMEOFDATA` must end with a 4-byte end-of-data tag** (after the frame `params`). libNatNet's unpacker reads it; without it the unpacked length mismatches `nDataBytes` and the SDK drops the whole frame. (The lenient Python `NatNetClient` does not require it — always validate against the C SDK.) +- The **269-byte `NAT_CONNECT` payload does not include** the client port; the port is learned from the datagram **source address** on `NAT_CONNECT`. +- Do **not** trust `/proc`/`ss` alone for the client port — extra bound sockets may appear that do not match wire traffic. **`NAT_CONNECT` source `(ip, port)` is ground truth.** +- Do **not** parse connect payloads with in-memory `sNatNetClientConnectParams` (contains pointers). Use on-wire layouts below. + +## libNatNet 4.4 `NAT_CONNECT` (verified 2025-06) + +Observed against `127.0.0.1:1510` with the same unicast params as [`natnet_client_adapter.cpp`](../../../robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp). + +### What the client sends + +| Field | Observed value | +|-------|----------------| +| Message | `NAT_CONNECT` (0), `nDataBytes = 269`, total datagram 273 bytes | +| Payload layout | `sSender` (264 B) + `sConnectionOptions` (5 B) | +| `sSender.szName` | `"NatNetLib"` | +| `sSender.Version` | `[4, 4, 0, 0]` | +| `sSender.NatNetVersion` | `[4, 4, 0, 0]` | +| `subscribedDataOnly` | `0` | +| `BitstreamVersion` | `[0, 0, 0, 0]` → client defers to server version | +| Trailing port bytes | **None** (exactly 269 bytes; not PacketClient's optional +4) | +| UDP source port | Ephemeral (e.g. `41449`) — **client command + data port (same socket)** | + +Example hex (payload only, after 4-byte header): + +``` +NatNetLib\0 ... (256-byte name field) +04 04 00 00 (Version) +04 04 00 00 (NatNetVersion) +00 (subscribedDataOnly) +00 00 00 00 (BitstreamVersion) +``` + +## libNatNet 4.4 unicast: single client socket (verified 2025-06) + +Confirmed with wire capture on server `:1510`/`:1511`, `strace` on a minimal `NatNetClient::Connect()` binary, and `/proc//net/udp` cross-checks against the same `libNatNet.so` used by `natnet_ros2`. + +### What we observed + +| Signal | Result | +|--------|--------| +| Wire capture on server `:1510` | All client packets (`NAT_CONNECT`, `NAT_KEEPALIVE`, `NAT_REQUEST_MODELDEF`) from **one** source port | +| Wire capture on server `:1511` | **No** inbound packets from the client | +| strace on minimal client | **One** `bind()`, **one** fd for all `sendto` → server `:1510` and `recvfrom` ← server `:1510` | +| `NAT_CONNECT` payload | **No** trailing client port bytes (269 B total) | + +### Emulator rule (unicast + `natnet_ros2`) + +For libNatNet 4.4 unicast, treat the client as **single-endpoint**: + +```text +On NAT_CONNECT → store client_endpoint = (ip, port) from recvfrom +NAT_SERVERINFO → sendto(command_socket, client_endpoint) # source port = command_port +NAT_MODELDEF → sendto(command_socket, client_endpoint) # source port = command_port +NAT_FRAMEOFDATA → sendto(data_socket, client_endpoint) # source port = data_port (REQUIRED) +NAT_KEEPALIVE → no reply (client -> server only) +``` + +The client always learns its endpoint from the **`NAT_CONNECT` source address** (the +client uses a single socket), so the **destination** of frames is that endpoint. The +**source** of frames, however, must be the server's data port — bind a dedicated +`data_socket` to `('', data_port)` and `sendto` frames from it. + +`ConnectionDataPort = 1511` in `NAT_SERVERINFO` is required (the SDK uses it to +recognize the data channel — i.e. which source port valid frames arrive from). + +### When two client ports may still apply + +- **Multicast** clients (separate multicast data listener on `239.255.42.99:1511`) +- **PacketClient-style** samples that open explicit command + data sockets (optional +4 port bytes in connect) +- Other NatNet client implementations — always verify with protocol capture before assuming a two-socket model + +Do **not** assume `data_port = cmd_port + 1` for any client without capture. + +### What the server must reply (for `Connect()` + `GetServerDescription()`) + +1. **`NAT_SERVERINFO` (1)** on the **command port** to the connect datagram source. +2. Payload: packed **`sSender_Server`** (279 B), **not** `sServerDescription`. libNatNet + parses the `NAT_SERVERINFO` payload as `sSender_Server`; sending the larger + `sServerDescription` makes it misread the version/host. Fields: + - `Common.szName = "Motive"` (256-byte field) + - `Common.Version = {3, 1, 0, 0}` (Motive app), `Common.NatNetVersion = {4, 4, 0, 0}` + - `HighResClockFrequency`, `DataPort = 1511`, `IsMulticast = 0` (unicast) + +Pre-built in emulator: [`NatNetServer._build_connect_response_payload()`](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/server/natnet_server.py). + +### After connect (required for `natnet_ros2` topics) + +| SDK call | Server must handle | +|----------|-------------------| +| `GetDataDescriptionList()` | `NAT_REQUEST_MODELDEF` → `NAT_MODELDEF` with rigid body name/ID (e.g. `"Drone"`) | +| Frame callback | Stream `NAT_FRAMEOFDATA` to **`NAT_CONNECT` source `(ip, port)`** from the server **data socket** (source port = `data_port`); end each frame with the 4-byte EOD tag; set `rb.params & 0x01` (tracking valid) | +| Unicast keepalive | Accept `NAT_KEEPALIVE` on command port; **send no reply** | + +Verified end-to-end against the real `libNatNet.so` with a C probe that registers +`SetFrameReceivedCallback` + `NatNet_SetLogCallback`: with the data-port source, +EOD tag, `sSender_Server` reply, and no keepalive reply, the probe reports +`Server: Motive 3.1.0.0 NatNet 4.4.0.0`, `data descriptions: 1`, and ~74 Hz callbacks. + +## Wire format reference (do not confuse) + +| Client type | Connect payload | +|-------------|-----------------| +| **`libNatNet` / `natnet_ros2`** | `sSender` + `sConnectionOptions` (269 B observed) | +| **PacketClient sample** | Same + optional 4 trailing bytes (often zero in sample) | +| **Python NatNetClient sample** | Legacy 270-byte `"Ping"` blob — **not** used by `natnet_ros2` | + +API struct `sNatNetClientConnectParams` ([`NatNetTypes.h`](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/NatNetClientSDK/NatNetSDK/include/NatNetTypes.h)) is for `Connect()` in process memory only — **not** the on-wire layout. + +## Protocol capture (optional, for debugging) + +Not part of the repo. If you need to re-verify wire behavior or debug a new client/server pairing, build a **minimal out-of-band harness**: + +1. **Minimal C++ client** — tiny binary linking `libNatNet.so` from `natnet_ros2`; call `NatNetClient::Connect()` with the same params as [`natnet_client_adapter.cpp`](../../../robot/ros_ws/src/perception/natnet_ros2/src/natnet_client_adapter.cpp). Optional: `GetDataDescriptionList()`, frame callback, `--hold-seconds` sleep. +2. **Python UDP stub server** — bind `:1510` (and optionally `:1511`); reply to `NAT_CONNECT` with canned `NAT_SERVERINFO`, to `NAT_REQUEST_MODELDEF` with `NAT_MODELDEF`, to `NAT_KEEPALIVE` with ack; log every `(ip, port)` and message id. +3. **Connect capture** — run the client against the stub; hex-dump the first datagram; confirm 269-byte `sSender` + `sConnectionOptions` payload and ephemeral source port. +4. **Endpoint discovery** — during a full connect + model-def fetch: + - `tcpdump -i any udp and host ` or the stub's packet log + - `strace -e trace=bind,sendto,recvfrom` on the client binary + - `/proc//net/udp` or `ss -uapn` (treat **`NAT_CONNECT` source port** as ground truth if they disagree) +5. **Frame delivery check** — confirm the client's frame callback fires. Register both `SetFrameReceivedCallback` **and** `NatNet_SetLogCallback` (the log callback surfaces silent drops). Frames must be sent from the server **data socket** (source port = `data_port`) and end with the 4-byte EOD tag, or the SDK drops them with no callback. + +Use the SDK's `NatNetTypes.h` and `PacketClient.cpp` for on-wire layouts — not in-memory `sNatNetClientConnectParams`. + +## Emulator implementation checklist + +1. **Command listener** on `0.0.0.0:1510` +2. **`NAT_CONNECT`** → register `client_endpoint` from `recvfrom`; reply `NAT_SERVERINFO` +3. **`NAT_REQUEST_MODELDEF`** → reply `NAT_MODELDEF` (match `body_name` in config) +4. **Frame loop** → `NAT_FRAMEOFDATA` to `client_endpoint` **from the data socket** (source port = `data_port`); end each frame with the 4-byte EOD tag +5. **Isaac integration** → sample drone pose → `sFrameOfMocapData` → `enqueue_mocap_data()` +6. **Docker** → emulator on `172.31.0.200`; robot `server_ip` points there + +## Testing levels + +| Level | Approach | Validates | +|-------|----------|-----------| +| Unit (no network) | `test_natnet_logic.cpp`, `FakeNatNetClient` | Negotiation logic, topic names | +| Protocol capture | Minimal client + UDP stub (see above) | Wire-format `NAT_CONNECT`, client endpoint model | +| Integration | `tests/integration/natnet/` | Full SDK parser + `natnet_ros2_node` (mark: `integration`) | +| System (future) | `airstack test -m sensors` | Topic Hz on `/perception/optitrack/...` | + +```bash +# Unit tests (robot container) +docker exec airstack-robot-desktop-1 bash -c "sws && colcon test --packages-select natnet_ros2 --event-handlers console_direct+" +``` + +## References + +- OptiTrack NatNet docs: https://docs.optitrack.com/developer-tools/natnet-sdk/natnet-4.0 +- SDK samples (wire format): `NatNet_SDK_*/Samples/PacketClient/`, `PythonClient/` (legacy connect in Python only) +- Integration test: [`tests/integration/natnet/README.md`](../../../tests/integration/natnet/README.md) diff --git a/.env b/.env index 8cf4e408c..ea6070100 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.15" +VERSION="0.19.0-alpha.16" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/.github/workflows/system-tests.yml b/.github/workflows/system-tests.yml index 2818af58c..69a961a48 100644 --- a/.github/workflows/system-tests.yml +++ b/.github/workflows/system-tests.yml @@ -193,6 +193,7 @@ jobs: args_blob = ' '.join(args) heavy = any(m in marks_norm for m in ( 'liveliness', 'sensors', 'takeoff_hover_land', 'autonomy', 'build_docker', + 'optitrack', )) only_packages = marks_norm == 'build_packages' or ( not heavy and any(s in args_blob for s in ( diff --git a/CHANGELOG.md b/CHANGELOG.md index 1c65dc62e..15da258c6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- `overrides/isaac-optitrack-simulation.env` — brings up Isaac Sim with the NatNet emulator and PX4 flying on mocap EKF2 external vision (GPS/baro/range aiding off), i.e. the configuration `tests/system/test_optitrack_e2e.py` runs, reproducible by hand - `overrides/l4t-optitrack-realrobot.env` — deployment override for a real Jetson robot flying on OptiTrack mocap (PX4 EKF2 external vision instead of GPS): the NatNet server/body settings, plus the multi-NIC and FCU-parameter notes that path needs - Feature notebook workflow (`use-feature-notebook` skill): every agent-implemented feature gets a local, gitignored `notebook/NNN-feature-slug/` entry with a status-tracked `design_spec.md` (written before coding) and `results/` artifacts + self-contained `results_summary.md` that populate the feature's PR description - Battery and telemetry display in GCS RQT control panel (voltage and percentage per robot when MAVROS battery topic is bridged) @@ -19,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `waypoint_flight` system test (`tests/system/test_waypoint_flight.py`): takeoff → ordered waypoint route via `NavigateTask` (dispatched as a dense plan) → land, judged on the odometry track by the standalone stdlib-only `tests/waypoint_checker.py` (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`); validated end-to-end in Isaac Sim; serves as the standard acceptance check after integrating or swapping a planner module - Real-robot PX4 external-vision fusion in `natnet_ros2` (OptiTrack mocap → EKF2): `mavros_gp_origin` (geoid-corrected synthetic GPS origin so `local_position.z` == OptiTrack z, fixing the ~36 m boot offset), `vision_pose_converter`, and a PX4 param **checker** (`px4_param_setter`, `auto_set` off by default; `on_mismatch` warn/halt) — setup guide at `docs/robot/px4_external_vision.md` - NatNet server emulator (`optitrack.natnet.emulator`, protocol core) — pure-Python OptiTrack Motive server emulation so `natnet_ros2` can be driven without hardware; host integration tests (`tests/integration/natnet/`) wire it to the robot client +- Isaac wrapper for the NatNet emulator (USD scene → server) + natnet Pegasus launch scripts, and a dedicated OptiTrack sim e2e test (`optitrack` mark, `tests/system/test_optitrack_e2e.py`) that flies a **Circle trajectory on mocap EKF2 fusion** — GPS, baro and range aiding are disabled for the run, so the OptiTrack stream is the vehicle's only position source and cross-track error scores the whole chain ### Changed @@ -26,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Default system-test `--sim` is `isaacsim`; pass `--sim msairsim` to opt in to Microsoft AirSim - `-m build_packages` CI runs pull `cache_*` images instead of baking sim images - `docker-build.yml` retags unchanged images on VERSION bumps (content fingerprint) instead of always rebuilding; floating `cache_*` tags still seed PR layer cache +- The PR-open test run is unchanged (pytest's full defaults) and now also covers the OptiTrack Circle-trajectory e2e, which configures its own mocap-EV stack. `optitrack` joins the `heavy` mark list in `system-tests.yml`, so an optitrack run is never misclassified as colcon-only and sent down the pull-only image path - `robot-l4t` compose service knobs are now env-overridable (`AUTONOMY_ROLE`, `FCU_URL`, and the rosbag path via `BAG_STORAGE_PATH`); `FCU_URL` unquoted so the literal serial path reaches MAVROS - `zed-l4t` image: ZED SDK 4.2 → 5.2 with the coupled ROS deps (`zed_msgs` 5.2.1, `point_cloud_transport(_plugins)` 4.x, add `backward_ros`) - Unit tests are defined by `tests/colcon_unit_test_packages.yaml`: `conftest.py` collects each listed package's co-located `test/` dir under `--import-mode=importlib` and marks it `unit` (ament lint files are skipped and run under `colcon test`) @@ -48,6 +51,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - The synthetic GPS origin now places the mocap floor at the shared world datum (`desired_floor_amsl: 36.0`, i.e. 90 m ellipsoidal in AMSL) rather than at sea level, so a mocap robot's reported global altitude agrees with sim and the GCS. `local_position.z` still equals the OptiTrack height either way - The robot image could ship without the GeographicLib `egm96-5` geoid: mavros' `install_geographiclib_datasets.sh` swallows a failed download and still exits 0, so the `RUN` layer succeeded either way, and `geographiclib-tools` was only ever a transitive dependency. MAVROS builds that geoid in its UAS core before any plugin loads and throws if it is missing, so `mavros_node` died at startup on affected images. `Dockerfile.robot` now pins the tool and asserts the file exists, failing the build instead - An unrecognised `connection_type` in `natnet_config.yaml` silently fell back to `unicast`, so a typo produced a client that connected on the wrong transport and never received frames. `validate_connection_type` now throws and `natnet_ros2_node` fails at startup naming the offending value +- `natnet_ros2_node` on `robot_1` now compares the NatNet server's MODELDEF drone-body count (`Drone` / `Drone1`…`DroneN`, excluding `Target` and skeleton bones) against `NUM_ROBOTS` after the handshake and logs an error on mismatch, so a sim launch script and `natnet_config.yaml` that disagree about how many drones exist is caught at startup rather than as a robot that silently never receives frames. `NUM_ROBOTS` is forwarded into the robot container for it +- Isaac Sim PX4 never fused the mocap stream: `EKF2_EV_CTRL` defaults to 0 and the isaac compose set no PX4 parameters, so the emulator could stream perfectly while PX4 flew on sim GPS. The compose now passes the EKF2 external-vision set as `PX4_PARAM_*` (applied by PX4 SITL's `rcS` at boot), each defaulting to PX4's own default so non-mocap sims are unaffected; the mocap path opts in +- The NatNet emulator hardcoded the drone's streaming id to 1 while the client reads `NATNET_BODY_ID`, so a real Motive id desynced the two into a connected client that never published (`example_one_px4_pegasus_natnet_launch_script.py`) +- `test_optitrack_e2e.py::test_px4_fuses_vision` asserted only that `local_position/pose` publishes, which it does off GPS — the check passed with external vision disabled. It is now the pre-flight gate (an estimate exists) and the Circle flight is the actual proof of fusion +- The NatNet emulator is now installed as a Kit extension: `Dockerfile.isaac-ros` pip-installs it editable into the Isaac python and bind-mounts the repo copy over it (the same pattern as `pegasus.simulator`), and the natnet launch scripts `enable_extension` it before importing. Being on a Kit `--ext-folder` search path only makes Kit *aware* of an extension — it does not put the package on `sys.path` — so the scripts previously died with `ModuleNotFoundError: No module named 'optitrack'` ## [1.0.0] - 2024-12-19 diff --git a/docs/simulation/isaac_sim/natnet_emulator.md b/docs/simulation/isaac_sim/natnet_emulator.md new file mode 100644 index 000000000..4d6cb12da --- /dev/null +++ b/docs/simulation/isaac_sim/natnet_emulator.md @@ -0,0 +1,305 @@ +# NatNet Emulator (OptiTrack Simulation) + +The `optitrack.natnet.emulator` Isaac Sim extension lets you test the full +[`natnet_ros2`](../../../robot/ros_ws/src/perception/natnet_ros2/README.md) +perception stack in simulation without a physical OptiTrack system. It runs a +Motive-compatible NatNet UDP server inside Isaac Sim, streams rigid-body poses +sampled from USD prim world transforms, and presents the same wire protocol +that the real Motive software uses. + +## How it works + +``` +Isaac Sim (physics step) + ↓ sample prim world pose +NatNetServerManager (/World/NatNetInterface USD prim) + ↓ encode sFrameOfMocapData (NatNet 4.1 wire format) +NatNetUnicastServer ──UDP 1510/1511──► natnet_ros2_node (robot container) + ↓ + /robot_N/perception/optitrack/{body} + /robot_N/interface/mavros/vision_pose/pose +``` + +Configuration lives on a `/World/NatNetInterface` USD prim with `natnet:*` +attributes. Because it is USD, the config **persists when you save the stage** — +re-opening a `.usd` file restores the catalog and server settings without +re-running any script. + +Each physics step the extension: + +1. Reads the world transform of each tracked prim. +2. Packs a `sFrameOfMocapData` frame (one `sRigidBodyData` entry per body). +3. Flushes the frame immediately on the physics-step thread (no background timer). + +Bodies whose target prim is missing emit a **lost** frame (NaN position, +tracking-invalid bit clear) until the prim appears — this handles Pegasus drones +that are spawned on the first Play tick. + +Optional **sensor noise** (`pose_noise_std_meters`, `pose_noise_rotation_deg`) +adds Gaussian position and orientation perturbation to simulate real OptiTrack +measurement uncertainty. + +--- + +## Using the pre-built launch scripts + +The easiest way to start is with the provided Pegasus launch scripts. Set +`ISAAC_SIM_SCRIPT_NAME` in your environment or use the convenience override: + +```bash +# Single drone, NatNet emulator + PX4 flying on external vision +airstack up --env-file overrides/isaac-optitrack-simulation.env +``` + +`overrides/isaac-optitrack-simulation.env` sets: + +| Variable | Value | +|---|---| +| `NUM_ROBOTS` | `1` | +| `LAUNCH_NATNET` | `true` | +| `PX4_PARAM_SET` | `external-vision` | +| `ISAAC_SIM_SCRIPT_NAME` | `example_one_px4_pegasus_natnet_launch_script.py` | + +`PX4_PARAM_SET` selects `simulation/isaac-sim/docker/px4-params/.env`, whose +`PX4_PARAM_*` entries PX4's rcS applies at boot. It defaults to `default`, which is empty, +so every other Isaac Sim run keeps PX4's firmware defaults. Add a file there to save your +own parameter set. + +### Available NatNet launch scripts + +| Script | Use case | +|---|---| +| `example_one_px4_pegasus_natnet_launch_script.py` | Single drone + static `Target` body | +| `example_multi_px4_pegasus_natnet_launch_script.py` | `NUM_ROBOTS` drones + shared `Target` body | + +Both scripts set up GPS origins (via `gps_utils.py`) so the GCS datum matches +PX4, author the NatNet interface, and play the simulation automatically. + +!!! note "Baseline scripts have no NatNet" + `example_one_px4_pegasus_launch_script.py` and + `example_multi_px4_pegasus_launch_script.py` do **not** include NatNet. Use + the `*_natnet_*` variants above when you need mocap simulation. + +### Body naming + +| `NUM_ROBOTS` | Body names streamed | +|---|---| +| 1 | `Drone`, `Target` | +| N > 1 | `Drone1`, `Drone2`, …, `DroneN`, `Target` | + +### Changing which body is streamed + +The streamed body name and streaming id are **constants in the launch script** +(`NATNET_BODY_NAME` / `NATNET_BODY_ID` / `NATNET_TARGET_NAME`), not environment +variables. They must match a body entry in the robot's profile in +[`natnet_config.yaml`](../../../robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml), +which is the only place the client reads its bodies from — that is what lets each robot +in a multi-robot scene track a different body. + +To retarget, edit **both** together: + +| Where | What | +|---|---| +| `example_one_px4_pegasus_natnet_launch_script.py` | `NATNET_BODY_NAME`, `NATNET_BODY_ID` | +| `natnet_config.yaml` → `robots..bodies[]` | `rigid_body_name`, `id` | + +!!! warning "A mismatch fails silently" + The NatNet client filters incoming frames by **numeric id**. If the ids disagree, the + client connects, the emulator streams, and the pose topic never publishes — with no + error on either side. When debugging a silent stream, check the id first. + +--- + +## Adding NatNet to your own launch script + +Call `author_drone_natnet_interface` after your Pegasus drones are spawned and +before you start the timeline. The extension builds the server from the prim on +Play. + +```python +from isaacsim.core.utils.extensions import enable_extension + +# Enable the NatNet emulator extension through Kit extension manager. +enable_extension("optitrack.natnet.emulator") + +from optitrack.natnet.emulator.isaac import ( + author_drone_natnet_interface, + author_static_target, + DEFAULT_TARGET_PATH, + DEFAULT_TARGET_STREAMING_ID, +) + +stage = omni.usd.get_context().get_stage() + +# Optional: add a static target body the robot can navigate toward. +author_static_target(stage, DEFAULT_TARGET_PATH, position=(2.0, 0.0, 1.0)) + +# One entry per drone: (rigid_body_name, streaming_id, target_prim_path) +drones = [ + ("Drone", 1, "/World/drone1/base_link/body"), +] + +author_drone_natnet_interface( + stage, + drones=drones, + server_ip="172.31.0.200", # Isaac container IP on AirStack bridge network + pose_noise_enabled=True, + pose_noise_std_meters=0.0005, + pose_noise_rotation_deg=0.05, +) +``` + +For multiple drones, add one tuple per drone: + +```python +drones = [ + ("Drone1", 1, "/World/drone1/base_link/body"), + ("Drone2", 2, "/World/drone2/base_link/body"), + ("Drone3", 3, "/World/drone3/base_link/body"), +] +``` + +`author_drone_natnet_interface` also accepts the static target as a body — include +it explicitly if you want it: + +```python +from optitrack.natnet.emulator.isaac import DEFAULT_TARGET_STREAMING_ID, DEFAULT_TARGET_PATH + +drones = [ + ("Drone", 1, "/World/drone1/base_link/body"), + ("Target", DEFAULT_TARGET_STREAMING_ID, DEFAULT_TARGET_PATH), +] +``` + +--- + +## Using the Kit UI panel + +The extension registers a docked panel under **Window → NatNet Interface** in +the Isaac Sim menu bar (appears alongside the Pegasus panel). + +### Opening the panel + +Open Isaac Sim, load your scene, then go to **Window → NatNet Interface**. The +panel docks next to the Property panel in the bottom-right. + +### Panel controls + +| Button | Action | +|---|---| +| **Create Interface** | Author a fresh `/World/NatNetInterface` prim with current settings | +| **Save** | Push the form fields into the USD prim on the stage | +| **Load from Stage** | Pull the existing prim's values back into the form | +| **Print config** | Log the current config to the console | + +**The server's lifetime follows the simulation:** Play builds it from the prim, +Stop shuts it down. The panel's `Server:` label reports which state it is in. + +When edits take effect after **Save**: + +| Setting | Takes effect | +|---|---| +| Bodies — added, removed, renamed, retargeted | Next frame; the server re-reads the interface as it samples | +| `upAxis`, pose noise | Next frame | +| `serverIp`, ports, `mode` | Next **Play**; these are bound when the server is built | + +!!! warning "Restart the robot stack after each Play" + Clients register with the server instance they connect to, and `natnet_ros2` + handshakes only until its first success. A client connected during an earlier + run is unknown to the server built by the next Play and receives no frames; the + console shows `[Command Handler] Ignoring message N from unregistered client`. + + Assume one client connection per server lifetime: restart the robot container + after each Stop → Play cycle. + +### Server settings + +| Field | Default | Description | +|---|---|---| +| Server enabled | `true` | Uncheck to stop the server starting on Play | +| Server IP | `172.31.0.200` | IP the UDP socket binds to (Isaac container address) | +| Mode | `unicast` | `unicast` for direct; `multicast` for broadcast | +| Command port | `1510` | NatNet command channel | +| Data port | `1511` | NatNet data channel (frame stream) | +| Publish rate (Hz) | `120` | Target frame rate | +| Up axis | `Z` | `Z` passes poses through unchanged; `Y` re-axes for Y-up Motive | +| Pose noise enabled | `true` | Add Gaussian noise to simulate real sensor uncertainty | +| Pose noise std (m) | `0.0005` | Position noise std dev (0.5 mm, matching OptiTrack spec) | +| Pose noise rotation (deg) | `0.05` | Orientation noise std dev | + +### Adding tracked bodies + +1. In the Stage tree, **select the prim** you want to track (e.g. `/World/drone1/base_link/body`). +2. Click **Add body (from selection)** in the panel. +3. Fill in the **rigid body name** (must match the `rigid_body_name` in `natnet_config.yaml`) and **streaming ID**. +4. Click **Save**. The body starts streaming on the next frame. + +Each body row shows a live readout of the prim's current world position with a +colour-coded status indicator: + +- Green dot — server running, prim found, pose valid +- Grey dot — prim found but server not running +- Red — prim missing or NaN position + +### Persistence + +After configuring the panel, save your USD stage (**File → Save**). The +`natnet:*` attributes are written into the `.usd` file. Re-opening the stage +restores the full catalog automatically — no script or panel interaction needed +unless you want to change the config. + +--- + +## Configuration reference + +`author_drone_natnet_interface` and `build_drone_config` accept these keyword +arguments (all optional): + +| Parameter | Default | Description | +|---|---|---| +| `server_ip` | `"172.31.0.200"` | IP to bind the UDP server to | +| `mode` | `"unicast"` | `"unicast"` or `"multicast"` | +| `command_port` | `1510` | NatNet command port | +| `data_port` | `1511` | NatNet data port | +| `publish_rate` | `120.0` | Frame streaming rate (Hz) | +| `up_axis` | `"Z"` | Axis convention (`"Z"` or `"Y"`) | +| `pose_noise_enabled` | `True` | Enable sensor noise | +| `pose_noise_std_meters` | `0.0005` | Position noise std dev (m) | +| `pose_noise_rotation_deg` | `0.05` | Orientation noise std dev (degrees) | + +--- + +## Troubleshooting + +**`natnet_ros2` connects but no pose topics appear** + +- Check that `rigid_body_name` in `natnet_config.yaml` matches exactly what the + emulator is streaming (case-sensitive). Run `ros2 topic list` inside the robot + container and look for `/robot_N/perception/optitrack/...`. + +**Server starts but no data arrives in `natnet_ros2`** + +- Confirm the server IP matches the Isaac container's address (`172.31.0.200` + on the AirStack bridge). Check with `docker network inspect airstack_network`. +- The NatNet data port (1511) must be bound to the *data* socket — frames sent + from the command socket are silently dropped by libNatNet 4.4. + +**Data stopped after stopping and replaying the simulation** + +- Expected: Stop destroys the server, so the client's registration goes with it, + and `natnet_ros2` does not re-handshake after its first successful connect. The + console shows `Ignoring message N from unregistered client`. Restart the robot + container to force a fresh `NAT_CONNECT`. See the warning under + [Panel controls](#panel-controls). + +**Emulator streams but `vision_pose` is empty** + +- `vision_pose` forwarding requires `vision_pose.enabled: true` in the robot's + `natnet_config.yaml` profile and `SITL_PARAM_PROFILE=px4-vision` so PX4 + accepts external vision instead of GPS. + +**Body shows red / NaN in the UI panel** + +- The target prim doesn't exist yet. This is normal before pressing Play (Pegasus + spawns the drone `base_link` prim on the first physics tick). After Play the + indicator should turn green within one frame. diff --git a/mkdocs.yml b/mkdocs.yml index f45ef0963..d56789e58 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -93,6 +93,7 @@ nav: - Overhead Camera: docs/simulation/isaac_sim/overhead_camera.md - docs/simulation/isaac_sim/ascent_sitl_extension.md - docs/simulation/isaac_sim/export_stages_from_unreal.md + - MoCap Emulator: docs/simulation/isaac_sim/natnet_emulator.md - Microsoft AirSim (legacy): - docs/simulation/ms-airsim/index.md - Docker: docs/simulation/ms-airsim/docker.md diff --git a/overrides/isaac-optitrack-simulation.env b/overrides/isaac-optitrack-simulation.env new file mode 100644 index 000000000..32208f00c --- /dev/null +++ b/overrides/isaac-optitrack-simulation.env @@ -0,0 +1,36 @@ +# Isaac Sim + OptiTrack NatNet mocap, with PX4 flying on external-vision (EV) fusion. +# +# Usage: +# airstack up --env-file overrides/isaac-optitrack-simulation.env +# +# Data path: +# in-sim NatNet emulator -> natnet_ros2 -> vision_pose_converter +# -> /{robot}/interface/mavros/vision_pose/pose_cov -> PX4 EKF2 +# +# Real-robot counterpart: overrides/l4t-optitrack-realrobot.env + +COMPOSE_PROFILES="desktop,isaac-sim" +AUTOLAUNCH="true" +NUM_ROBOTS="1" + +# --- Isaac scene -------------------------------------------------------------- +ISAAC_SIM_USE_STANDALONE="true" +ISAAC_SIM_SCRIPT_NAME="example_one_px4_pegasus_natnet_launch_script.py" +# For multi-agent, use the following script instead. +# ISAAC_SIM_SCRIPT_NAME="example_multi_px4_pegasus_natnet_launch_script.py" +# Ensure each spawned robot has a corresponding rigid body name and streaming ID as in +# the natnet_config.yaml file. Also, increase NUM_ROBOTS to match. + +PLAY_SIM_ON_START="true" + +# --- OptiTrack / NatNet ------------------------------------------------------- +LAUNCH_NATNET="true" + +# The emulator runs inside the isaac-sim container, which holds this static IP on +# airstack_network. +NATNET_SERVER_IP="172.31.0.200" + +# --- PX4 EKF2 external-vision fusion ----------------------------------------- +# Selects simulation/isaac-sim/docker/px4-params/external-vision.env, which holds the +# EKF2 values. See docs/robot/px4_external_vision.md for what each one does. +PX4_PARAM_SET="external-vision" diff --git a/simulation/isaac-sim/docker/Dockerfile.isaac-ros b/simulation/isaac-sim/docker/Dockerfile.isaac-ros index 69bbd8117..b03816015 100644 --- a/simulation/isaac-sim/docker/Dockerfile.isaac-ros +++ b/simulation/isaac-sim/docker/Dockerfile.isaac-ros @@ -145,6 +145,11 @@ ENV ACCEPT_EULA="Y" # ENV ISAACSIM_PYTHON=/isaac-sim/python.sh RUN /isaac-sim/python.sh -m pip install --no-cache-dir -e /isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/pegasus.simulator +# Installing OptiTrack NatNet emulator extension into Kit for plugin discovery. +COPY extensions/optitrack.natnet.emulator \ + /isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/optitrack.natnet.emulator +RUN /isaac-sim/python.sh -m pip install --no-cache-dir -e /isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/optitrack.natnet.emulator + # Install PX4 things RUN git clone --branch ${PX4_VERSION} --recursive https://github.com/PX4/PX4-Autopilot.git diff --git a/simulation/isaac-sim/docker/docker-compose.yaml b/simulation/isaac-sim/docker/docker-compose.yaml index 50a9df16d..a78382ddc 100644 --- a/simulation/isaac-sim/docker/docker-compose.yaml +++ b/simulation/isaac-sim/docker/docker-compose.yaml @@ -38,6 +38,8 @@ services: ipv4_address: 172.31.0.200 # required to not conflict with other default docker networks on the host machine env_file: - ./omni_pass.env + # PX4 SITL parameter set, applied by rcS at boot. + - ./px4-params/${PX4_PARAM_SET:-default}.env environment: - AUTOLAUNCH=${AUTOLAUNCH:-'false'} - DISPLAY=${DISPLAY} @@ -73,6 +75,8 @@ services: - $HOME/docker/isaac-sim/pkg:/isaac-sim/.local/share/ov/pkg:rw \ # pegasus integration - ../extensions/PegasusSimulator/extensions/pegasus.simulator:/isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/pegasus.simulator/:rw + # natnet emulator integration + - ../extensions/optitrack.natnet.emulator:/isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/optitrack.natnet.emulator/:rw # omniverse - ./omniverse.toml:/isaac-sim/.nvidia-omniverse/config/omniverse.toml:rw - ./user.config.json:/isaac-sim/.local/share/ov/data/Kit/Isaac-Sim Full/5.1/user.config.json:rw # enables pegasus extension; IMPORTANT: set the version number without the trailing .0 @@ -148,6 +152,7 @@ services: - $HOME/docker/isaac-sim/data:/isaac-sim/.local/share/ov/data:rw - $HOME/docker/isaac-sim/pkg:/isaac-sim/.local/share/ov/pkg:rw - ../extensions/PegasusSimulator/extensions/pegasus.simulator:/isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/pegasus.simulator/:rw + - ../extensions/optitrack.natnet.emulator:/isaac-sim/.local/share/ov/data/documents/Kit/shared/exts/optitrack.natnet.emulator/:rw - ./omniverse.toml:/isaac-sim/.nvidia-omniverse/config/omniverse.toml:rw - ./user.config.json:/isaac-sim/.local/share/ov/data/Kit/Isaac-Sim Full/5.1/user.config.json:rw - .dev:/isaac-sim/.dev:rw diff --git a/simulation/isaac-sim/docker/px4-params/default.env b/simulation/isaac-sim/docker/px4-params/default.env new file mode 100644 index 000000000..2b2fb1893 --- /dev/null +++ b/simulation/isaac-sim/docker/px4-params/default.env @@ -0,0 +1,9 @@ +# Default PX4 SITL parameter set: no overrides. +# +# Selected when PX4_PARAM_SET is unset, so a plain `airstack up isaac-sim` leaves PX4 +# entirely on its firmware defaults. Keep this file empty — anything added here applies +# to every Isaac Sim run. +# +# To use a different set, put PX4_PARAM_SET= in .env or an overrides/*.env file; +# it selects px4-params/.env next to this file. Entries are PX4_PARAM_= +# and PX4's rcS applies them at boot. diff --git a/simulation/isaac-sim/docker/px4-params/external-vision.env b/simulation/isaac-sim/docker/px4-params/external-vision.env new file mode 100644 index 000000000..bfd95a2fd --- /dev/null +++ b/simulation/isaac-sim/docker/px4-params/external-vision.env @@ -0,0 +1,18 @@ +# PX4 SITL on external vision (OptiTrack mocap) instead of GPS. +# +# Select with PX4_PARAM_SET=external-vision. Mirrors the deployment-validated set in +# robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml, which is the source of +# truth — px4_param_setter checks the FCU against it on the real robot. +# See docs/robot/px4_external_vision.md for what each parameter does. + +PX4_PARAM_EKF2_EV_CTRL=11 # fuse vision horizontal position + vertical position + yaw +PX4_PARAM_EKF2_HGT_REF=3 # vision is the height reference +PX4_PARAM_EKF2_GPS_CTRL=0 # no GPS fusion +PX4_PARAM_EKF2_MAG_TYPE=5 # magnetometer off; yaw comes from vision +PX4_PARAM_EKF2_BARO_CTRL=0 # no barometer fusion +PX4_PARAM_SYS_HAS_BARO=0 # remove the baro at system level, so the height datum is vision +PX4_PARAM_EKF2_RNG_CTRL=0 # no range-finder aiding +PX4_PARAM_EKF2_EV_DELAY=7.0 # ms, measured end to end on the real deployment +PX4_PARAM_EKF2_EV_NOISE_MD=1 # use the NOISE floors below, not the message covariance +PX4_PARAM_EKF2_EVP_NOISE=0.05 # m; also sets the innovation gate (EKF2_EVP_GATE sigma wide) +PX4_PARAM_EKF2_EVA_NOISE=0.05 # rad diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md index 3bb05f0b6..3fedb5ecb 100644 --- a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md @@ -30,7 +30,7 @@ optitrack.natnet.emulator/ ├── catalog.py # Config → sDataDescriptions (MODELDEF) ├── frames.py # Prim poses → sFrameOfMocapData ├── manager.py # NatNetServerManager (lifecycle + sampling) - ├── scene_setup.py # Pegasus launch helpers (start_drone_natnet_server) + ├── scene_setup.py # Pegasus launch helpers (author_drone_natnet_interface) └── ui_extension.py # Docked editor panel (NatNetEmulatorExtension) ``` @@ -82,7 +82,7 @@ Baseline Pegasus scripts (`example_one_px4_pegasus_launch_script.py`, `example_m Convenience bundle for NatNet + external-vision PX4 SITL: ```bash -airstack up --env-file overrides/isaac-natnet-vision.env +airstack up --env-file overrides/isaac-optitrack-simulation.env ``` See [optitrack-development skill](../../../../.agents/skills/optitrack-development/SKILL.md) for wire-protocol details, libNatNet 4.4 unicast quirks, and debugging. @@ -107,19 +107,24 @@ server.flush_mocap_data() ### Isaac launch script ```python -from optitrack.natnet.emulator.isaac import start_drone_natnet_server +from isaacsim.core.utils.extensions import enable_extension -# Keep a reference to the manager for the sim lifetime. -manager = start_drone_natnet_server( +# Register the extension with Kit before importing from it. +enable_extension("optitrack.natnet.emulator") + +from optitrack.natnet.emulator.isaac import author_drone_natnet_interface + +# Author the interface prim; the extension starts the server on Play. +author_drone_natnet_interface( stage, - drones=[("Drone", 1, "/World/drone1/base_link")], + drones=[("Drone", 1, "/World/drone1/base_link/body")], server_ip="172.31.0.200", ) ``` ### Kit UI -The extension registers **Window → NatNet Emulator** — a docked panel to create/edit the interface prim, start/stop the server, and view live body readouts. The same `NatNetServerManager` backs both the UI and launch-script paths. +The extension registers **Window → NatNet Emulator** — a docked panel to create/edit the interface prim and view live body readouts. The extension owns the `NatNetServerManager`, building the server from the prim on Play and shutting it down on Stop. ## Protocol notes (unicast, libNatNet 4.4) diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/config/extension.toml b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/config/extension.toml new file mode 100644 index 000000000..a5df394da --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/config/extension.toml @@ -0,0 +1,23 @@ +[package] +version = "0.1.0" +title = "OptiTrack NatNet Emulator" +description = "NatNet UDP server emulator for Isaac Sim integration with natnet_ros2" +category = "Simulation" +keywords = ["optitrack", "natnet", "mocap", "simulation"] + +[dependencies] +"omni.isaac.core" = {} +"omni.usd" = {} +"omni.ui" = {} +"omni.kit.menu.utils" = {} + +# Pure transport/types package (no Kit UI; safe to import anywhere). +[[python.module]] +name = "optitrack.natnet.emulator" + +# Kit UI entry point: NatNetEmulatorExtension (menu + config-prim authoring window). +[[python.module]] +name = "optitrack.natnet.emulator.isaac.ui_extension" + +[python.build-system] +requires = ["setuptools"] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/__init__.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/__init__.py new file mode 100644 index 000000000..e28f9e412 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/__init__.py @@ -0,0 +1,62 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Isaac Sim integration for the NatNet emulator (stage-driven config prim). + +``config`` is pure Python. ``usd_bindings`` imports ``pxr`` lazily, so +importing this package is safe in non-Isaac environments. +""" + +from .config import ( + BodyBinding, + NatNetInterfaceConfig, + body_attr_name, + make_instance_key, +) +from .catalog import build_catalog, find_duplicate_targets +from .frames import BodySample, build_frame, make_rigid_body_data +from .manager import NatNetServerManager, default_server_factory, format_interface +from .scene_setup import ( + DEFAULT_INTERFACE_PATH, + DEFAULT_TARGET_PATH, + DEFAULT_TARGET_POSITION, + DEFAULT_TARGET_STREAMING_ID, + author_drone_natnet_interface, + author_static_target, + build_drone_config, +) +from .usd_bindings import ( + author_interface, + find_interfaces, + is_interface, + read_interface, + read_world_pose, + resolve_targets, +) + +__all__ = [ + "DEFAULT_INTERFACE_PATH", + "DEFAULT_TARGET_PATH", + "DEFAULT_TARGET_POSITION", + "DEFAULT_TARGET_STREAMING_ID", + "BodyBinding", + "BodySample", + "NatNetInterfaceConfig", + "NatNetServerManager", + "author_interface", + "author_drone_natnet_interface", + "author_static_target", + "body_attr_name", + "build_catalog", + "build_drone_config", + "build_frame", + "default_server_factory", + "find_duplicate_targets", + "find_interfaces", + "format_interface", + "is_interface", + "make_instance_key", + "make_rigid_body_data", + "read_interface", + "read_world_pose", + "resolve_targets", +] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/catalog.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/catalog.py new file mode 100644 index 000000000..9c3bfaa35 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/catalog.py @@ -0,0 +1,53 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +""" +Turn a :class:`NatNetInterfaceConfig` into the server's MODELDEF catalog +(``sDataDescriptions`` of rigid bodies). Pure Python + ctypes (the ``server`` +package is stdlib-only), so this is hermetically unit-testable — no USD, no Kit. +""" + +from __future__ import annotations + +from ..server.natnet_common import ModelLimits +from ..server.natnet_model_types import DataDescriptors, sDataDescriptions +from .config import NatNetInterfaceConfig + +# szName is null-terminated on the wire; reserve one byte for the terminator. +_MAX_NAME_BYTES = int(ModelLimits.MAX_NAMELENGTH) - 1 +_MAX_MODELS = int(ModelLimits.MAX_MODELS) + + +def build_catalog(config: NatNetInterfaceConfig) -> sDataDescriptions: + """Build an ``sDataDescriptions`` rigid-body catalog from the config bodies. + + No bodies -> an empty catalog (``nDataDescriptions == 0``). Names longer than + the NatNet name field are truncated. Raises ``ValueError`` if there are more + bodies than the protocol allows. + """ + bodies = config.bodies + if len(bodies) > _MAX_MODELS: + raise ValueError( + f"Too many bodies for one catalog: {len(bodies)} > {_MAX_MODELS} (MAX_MODELS)" + ) + + descriptions = sDataDescriptions() + descriptions.nDataDescriptions = len(bodies) + for i, body in enumerate(bodies): + desc = descriptions.arrDataDescriptions[i] + desc.type = int(DataDescriptors.Descriptor_RigidBody) + rb = desc.RigidBodyDescription + rb.szName = body.rigid_body_name.encode("utf-8")[:_MAX_NAME_BYTES] + rb.ID = int(body.streaming_id) + rb.parentID = int(body.parent_id) + rb.offsetqw = 1.0 # identity quaternion offset + rb.nMarkers = 0 + return descriptions + + +def find_duplicate_targets(config: NatNetInterfaceConfig) -> list[str]: + """Return target prim paths referenced by more than one body (empties ignored).""" + counts: dict[str, int] = {} + for body in config.bodies: + if body.target_prim: + counts[body.target_prim] = counts.get(body.target_prim, 0) + 1 + return [path for path, count in counts.items() if count > 1] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/config.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/config.py new file mode 100644 index 000000000..6b6943ded --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/config.py @@ -0,0 +1,231 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Pure-Python config model for the stage-driven NatNet interface. + +The USD binding layer (author/read against a ``Usd.Stage``) +lives in ``usd_bindings.py`` and depends on this model. + +Attribute names follow the multi-apply schema convention +(``natnet:body::``). +The custom-attribute backing is for a future typed applied schema. +""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any, Iterable, Mapping + +# --- attribute name constants (USD property names) ----------------------------- + +ATTR_NAMESPACE = "natnet" +MARKER_ATTR = "natnet:isInterface" + +ATTR_SERVER_ENABLED = "natnet:serverEnabled" +ATTR_SERVER_IP = "natnet:serverIp" +ATTR_MODE = "natnet:mode" +ATTR_MULTICAST_ADDR = "natnet:multicastAddr" +ATTR_COMMAND_PORT = "natnet:commandPort" +ATTR_DATA_PORT = "natnet:dataPort" +ATTR_PUBLISH_RATE = "natnet:publishRate" +ATTR_NATNET_VERSION = "natnet:natnetVersion" +ATTR_UP_AXIS = "natnet:upAxis" + +ATTR_POSE_NOISE_ENABLED = "natnet:poseNoiseEnabled" +ATTR_POSE_NOISE_STD_METERS = "natnet:poseNoiseStdMeters" +ATTR_POSE_NOISE_ROTATION_DEG = "natnet:poseNoiseRotationDeg" + +BODY_PREFIX = "natnet:body:" +BODY_FIELD_RIGID_BODY_NAME = "rigidBodyName" +BODY_FIELD_STREAMING_ID = "streamingId" +BODY_FIELD_PARENT_ID = "parentId" +BODY_FIELD_TARGET = "target" + +VALID_MODES = ("unicast", "multicast") + +# Streamed up-axis. "Z" (default) passes the USD pose through; "Y" emulates a Y-up +# Motive by rotating the streamed pose -90deg about X. +VALID_UP_AXES = ("Y", "Z") + +# defaults shared with NatNetUnicastServer +DEFAULT_SERVER_IP = "172.31.0.200" +DEFAULT_MULTICAST_ADDR = "239.255.42.99" +DEFAULT_COMMAND_PORT = 1510 +DEFAULT_DATA_PORT = 1511 +DEFAULT_PUBLISH_RATE = 100.0 +DEFAULT_NATNET_VERSION = "4.4.0.0" +DEFAULT_UP_AXIS = "Z" +DEFAULT_POSE_NOISE_ENABLED = True +DEFAULT_POSE_NOISE_STD_METERS = 0.0005 +DEFAULT_POSE_NOISE_ROTATION_DEG = 0.05 + + +def body_attr_name(key: str, field_name: str) -> str: + """USD property name for a body-binding field on the given instance key.""" + return f"{BODY_PREFIX}{key}:{field_name}" + + +def make_instance_key(name: str, used: set[str]) -> str: + """Derive a valid, unique multi-apply instance token from a rigid body name. + + USD property/instance tokens must be identifier-like; sanitize non-alnum chars + to underscores and disambiguate collisions with a numeric suffix. + """ + sanitized = "".join(c if c.isalnum() else "_" for c in name).strip("_") + if not sanitized: + sanitized = "body" + if sanitized[0].isdigit(): + sanitized = f"b_{sanitized}" + key = sanitized + i = 1 + while key in used: + key = f"{sanitized}_{i}" + i += 1 + used.add(key) + return key + + +@dataclass +class BodyBinding: + """One tracked rigid body: a Motive name/ID mapped to a USD prim path.""" + + rigid_body_name: str + target_prim: str + streaming_id: int = 1 + parent_id: int = -1 + + @classmethod + def from_dict(cls, data: Mapping[str, Any], *, target_prim: str | None = None) -> "BodyBinding": + d = dict(data) + resolved_target = target_prim if target_prim is not None else d.get("target_prim") + if not resolved_target: + raise ValueError("BodyBinding requires a target_prim (USD path of the tracked prim)") + if "rigid_body_name" not in d: + raise ValueError("BodyBinding requires a rigid_body_name") + return cls( + rigid_body_name=str(d["rigid_body_name"]), + target_prim=str(resolved_target), + streaming_id=int(d.get("streaming_id", 1)), + parent_id=int(d.get("parent_id", -1)), + ) + + def to_dict(self) -> dict[str, Any]: + return { + "rigid_body_name": self.rigid_body_name, + "target_prim": self.target_prim, + "streaming_id": self.streaming_id, + "parent_id": self.parent_id, + } + + +def _normalize_bodies(bodies: Any) -> list[BodyBinding]: + """Accept a list of dicts/BodyBindings, or a ``{prim_path: {...}}`` mapping.""" + if bodies is None: + return [] + out: list[BodyBinding] = [] + if isinstance(bodies, Mapping): + for prim_path, body in bodies.items(): + out.append(BodyBinding.from_dict(body, target_prim=prim_path)) + return out + if isinstance(bodies, Iterable): + for body in bodies: + if isinstance(body, BodyBinding): + out.append(body) + else: + out.append(BodyBinding.from_dict(body)) + return out + raise ValueError(f"`bodies` must be a list or a mapping, got {type(bodies).__name__}") + + +@dataclass +class NatNetInterfaceConfig: + """Server-level config plus the body catalog for one NatNet interface prim.""" + + server_enabled: bool = True + server_ip: str = DEFAULT_SERVER_IP + mode: str = "unicast" + multicast_addr: str = DEFAULT_MULTICAST_ADDR + command_port: int = DEFAULT_COMMAND_PORT + data_port: int = DEFAULT_DATA_PORT + publish_rate: float = DEFAULT_PUBLISH_RATE + natnet_version: str = DEFAULT_NATNET_VERSION + up_axis: str = DEFAULT_UP_AXIS + pose_noise_enabled: bool = DEFAULT_POSE_NOISE_ENABLED + pose_noise_std_meters: float = DEFAULT_POSE_NOISE_STD_METERS + pose_noise_rotation_deg: float = DEFAULT_POSE_NOISE_ROTATION_DEG + bodies: list[BodyBinding] = field(default_factory=list) + + @classmethod + def from_dict(cls, data: Mapping[str, Any]) -> "NatNetInterfaceConfig": + d = dict(data) + return cls( + server_enabled=bool(d.get("server_enabled", True)), + server_ip=str(d.get("server_ip", DEFAULT_SERVER_IP)), + mode=str(d.get("mode", "unicast")), + multicast_addr=str(d.get("multicast_addr", DEFAULT_MULTICAST_ADDR)), + command_port=int(d.get("command_port", DEFAULT_COMMAND_PORT)), + data_port=int(d.get("data_port", DEFAULT_DATA_PORT)), + publish_rate=float(d.get("publish_rate", DEFAULT_PUBLISH_RATE)), + natnet_version=str(d.get("natnet_version", DEFAULT_NATNET_VERSION)), + up_axis=str(d.get("up_axis", DEFAULT_UP_AXIS)).upper(), + pose_noise_enabled=bool(d.get("pose_noise_enabled", DEFAULT_POSE_NOISE_ENABLED)), + pose_noise_std_meters=float(d.get("pose_noise_std_meters", DEFAULT_POSE_NOISE_STD_METERS)), + pose_noise_rotation_deg=float(d.get("pose_noise_rotation_deg", DEFAULT_POSE_NOISE_ROTATION_DEG)), + bodies=_normalize_bodies(d.get("bodies")), + ) + + def to_dict(self) -> dict[str, Any]: + return { + "server_enabled": self.server_enabled, + "server_ip": self.server_ip, + "mode": self.mode, + "multicast_addr": self.multicast_addr, + "command_port": self.command_port, + "data_port": self.data_port, + "publish_rate": self.publish_rate, + "natnet_version": self.natnet_version, + "up_axis": self.up_axis, + "pose_noise_enabled": self.pose_noise_enabled, + "pose_noise_std_meters": self.pose_noise_std_meters, + "pose_noise_rotation_deg": self.pose_noise_rotation_deg, + "bodies": [b.to_dict() for b in self.bodies], + } + + def validate(self) -> "NatNetInterfaceConfig": + """Raise ``ValueError`` (aggregating all problems) if the config is invalid.""" + errors: list[str] = [] + if self.mode not in VALID_MODES: + errors.append(f"mode must be one of {VALID_MODES}, got {self.mode!r}") + if str(self.up_axis).upper() not in VALID_UP_AXES: + errors.append(f"up_axis must be one of {VALID_UP_AXES}, got {self.up_axis!r}") + for port_name, port in (("command_port", self.command_port), ("data_port", self.data_port)): + if not (0 < port < 65536): + errors.append(f"{port_name} must be in 1..65535, got {port}") + if self.command_port == self.data_port: + errors.append("command_port and data_port must differ") + if self.publish_rate <= 0: + errors.append(f"publish_rate must be > 0, got {self.publish_rate}") + if self.pose_noise_std_meters < 0: + errors.append( + f"pose_noise_std_meters must be >= 0, got {self.pose_noise_std_meters}" + ) + if self.pose_noise_rotation_deg < 0: + errors.append( + f"pose_noise_rotation_deg must be >= 0, got {self.pose_noise_rotation_deg}" + ) + for i, body in enumerate(self.bodies): + if not body.rigid_body_name: + errors.append(f"body[{i}] rigid_body_name must be non-empty") + names = [b.rigid_body_name for b in self.bodies] + if len(set(names)) != len(names): + errors.append("rigid_body_name values must be unique across bodies") + ids = [b.streaming_id for b in self.bodies] + if len(set(ids)) != len(ids): + errors.append("streaming_id values must be unique across bodies") + if errors: + raise ValueError("Invalid NatNetInterfaceConfig: " + "; ".join(errors)) + return self + + def assign_instance_keys(self) -> list[tuple[str, BodyBinding]]: + """Pair each body with a deterministic, unique multi-apply instance key.""" + used: set[str] = set() + return [(make_instance_key(b.rigid_body_name, used), b) for b in self.bodies] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/frames.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/frames.py new file mode 100644 index 000000000..2c92a99ce --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/frames.py @@ -0,0 +1,129 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Pose -> NatNet frame conversion (the data-enqueue path). + +Pure Python + ctypes. Sampled prim world poses become an +``sFrameOfMocapData`` of rigid bodies that the server +streams to the client. + +**Frame convention:** Motive exposes an "Up Axis" setting. AirStack's ``natnet_ros2`` +requires it set to **Z** and copy the rigid-body pose straight through +(``rb_to_pose`` is an identity copy). Isaac Sim coordinates are Z-up. +The default ``up_axis="Z"`` emits the prim's USD world pose as-is. +``up_axis="Y"`` emulates a default (Y-up) Motive by rotating the pose -90 deg about X. + +**params bits** (must match the client's ``is_tracking_valid`` / ``model_list_changed``): +- ``0x01`` on a rigid body marks tracking valid — the client *skips* bodies without it. +- ``0x02`` on the frame signals the model list changed so the client re-requests MODELDEF(set the frame after the catalog changes, e.g. a body added live). +""" + +from __future__ import annotations + +import math +import numpy as np +from dataclasses import dataclass +from scipy.spatial.transform import Rotation +from ..server.natnet_data_types import sFrameOfMocapData, sRigidBodyData + +TRACKING_VALID = 0x01 +MODEL_LIST_CHANGED = 0x02 + + +@dataclass +class BodySample: + """One sampled rigid body: streaming ID + world pose, or an invalid (lost) body.""" + + streaming_id: int + position: tuple[float, float, float] = (0.0, 0.0, 0.0) + orientation: tuple[float, float, float, float] = (0.0, 0.0, 0.0, 1.0) # qx,qy,qz,qw + valid: bool = True + + @classmethod + def lost(cls, streaming_id: int) -> "BodySample": + """An untracked body (missing prim): NaN position, tracking-invalid bit clear.""" + nan = float("nan") + return cls(streaming_id, (nan, nan, nan), (0.0, 0.0, 0.0, 1.0), valid=False) + + +def to_motive_pose(position: tuple[float, float, float], orientation: tuple[float, float, float, float], up_axis: str = "Z"): + """Re-express an Isaac (Z-up) world pose in Motive's streamed up-axis frame. + + Returns ``(position, orientation)`` re-axed for the given ``up_axis``: + + - ``"Z"`` (default) — identity pass-through. Isaac/USD is Z-up and the + reference Motive setup streams Z-up, so the pose flows through unchanged and + matches ``natnet_ros2`` (which does no axis conversion). + - ``"Y"`` — emulate a default Y-up Motive by rotating the pose -90 deg about X + (Isaac ``+Z`` -> Motive ``+Y``): ``(x, y, z) -> (x, z, -y)``. This is a + proper right-handed -> right-handed change of basis (det = +1), so the + quaternion's vector part takes the same swap and the scalar part is + unchanged: ``(qx, qy, qz, qw) -> (qx, qz, -qy, qw)``. + + Non-finite components (a lost body's NaN position) pass through unchanged. + """ + if str(up_axis).upper() != "Y": + return position, orientation + x, y, z = position + qx, qy, qz, qw = orientation + return (x, z, -y), (qx, qz, -qy, qw) + + +def make_rigid_body_data(sample: BodySample) -> sRigidBodyData: + """Build one ``sRigidBodyData`` from a sample (sets the tracking-valid bit).""" + rb = sRigidBodyData() + rb.ID = int(sample.streaming_id) + x, y, z = sample.position + qx, qy, qz, qw = sample.orientation + rb.x, rb.y, rb.z = float(x), float(y), float(z) + rb.qx, rb.qy, rb.qz, rb.qw = float(qx), float(qy), float(qz), float(qw) + rb.MeanError = 0.0 + rb.params = TRACKING_VALID if sample.valid else 0 + return rb + + +def build_frame( + frame_number: int, + samples, + *, + timestamp: float = 0.0, + model_list_changed: bool = False, +) -> sFrameOfMocapData: + """Assemble an ``sFrameOfMocapData`` of rigid bodies from samples.""" + frame = sFrameOfMocapData() + frame.iFrame = int(frame_number) + samples = list(samples) + frame.nRigidBodies = len(samples) + for i, sample in enumerate(samples): + frame.RigidBodies[i] = make_rigid_body_data(sample) + frame.fTimestamp = float(timestamp) + frame.params = MODEL_LIST_CHANGED if model_list_changed else 0 + return frame + + +def is_finite_pose(sample: BodySample) -> bool: + """True if all position/orientation components are finite (no NaN/inf).""" + return all(math.isfinite(v) for v in (*sample.position, *sample.orientation)) + + +def apply_pose_noise( + position: tuple[float, float, float], + orientation: tuple[float, float, float, float], + pose_noise_std_meters: float, + pose_noise_rotation_deg: float, +) -> tuple[tuple[float, float, float], tuple[float, float, float, float]]: + """Add independent Gaussian noise to position (m) and orientation (deg, XYZ euler).""" + + x, y, z = position + if pose_noise_std_meters > 0.0: + x += np.random.normal(0, pose_noise_std_meters) + y += np.random.normal(0, pose_noise_std_meters) + z += np.random.normal(0, pose_noise_std_meters) + + roll, pitch, yaw = Rotation.from_quat(orientation).as_euler("xyz", degrees=True) + if pose_noise_rotation_deg > 0.0: + roll += np.random.normal(0, pose_noise_rotation_deg) + pitch += np.random.normal(0, pose_noise_rotation_deg) + yaw += np.random.normal(0, pose_noise_rotation_deg) + + qx, qy, qz, qw = Rotation.from_euler("xyz", (roll, pitch, yaw), degrees=True).as_quat() + return (x, y, z), (float(qx), float(qy), float(qz), float(qw)) diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/manager.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/manager.py new file mode 100644 index 000000000..db64895a4 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/manager.py @@ -0,0 +1,444 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +""" +``NatNetServerManager`` detects interface prims, samples poses from the stage, +and owns a **single** server instance it can start and stop. +On each enable it builds a MODELDEF catalog from the config and constructs a +fresh server via an injectable factory. +""" + +from __future__ import annotations + +from .catalog import build_catalog, find_duplicate_targets +from .config import DEFAULT_UP_AXIS, NatNetInterfaceConfig +from .frames import BodySample, apply_pose_noise, build_frame, to_motive_pose +from .usd_bindings import find_interfaces, read_interface, read_world_pose, resolve_targets + + +def _catalog_signature(config: NatNetInterfaceConfig): + """Identity of the catalog (body id/name set) — changes trigger a MODELDEF refresh.""" + return tuple((b.streaming_id, b.rigid_body_name) for b in config.bodies) + + +def _parse_version(version_str: str) -> tuple[int, int, int, int]: + try: + parts = tuple(int(x) for x in str(version_str).split(".")) + except ValueError: + parts = () + return (parts + (0, 0, 0, 0))[:4] + + +def default_server_factory(config: NatNetInterfaceConfig): + """Construct (but do not start) a ``NatNetUnicastServer`` from a config.""" + from ..server import NatNetUnicastServer, TransmissionType + + if config.mode != "unicast": + raise NotImplementedError( + f"mode {config.mode!r} is not supported yet (unicast only)" + ) + server = NatNetUnicastServer( + local_interface=config.server_ip, + transmission_type=TransmissionType.UNICAST, + multicast_address=None, + command_port=config.command_port, + data_port=config.data_port, + ) + server.publish_rate = config.publish_rate + server.natnet_version = _parse_version(config.natnet_version) + return server + + +def format_interface(prim_path: str, cfg: NatNetInterfaceConfig) -> str: + """Render a human-readable multi-line summary of one interface config.""" + lines = [f"[natnet] Interface @ {prim_path}"] + lines.append(f" serverEnabled : {cfg.server_enabled}") + lines.append(f" serverIp : {cfg.server_ip}") + lines.append(f" mode : {cfg.mode}") + if cfg.mode == "multicast": + lines.append(f" multicastAddr : {cfg.multicast_addr}") + lines.append(f" commandPort : {cfg.command_port}") + lines.append(f" dataPort : {cfg.data_port}") + lines.append(f" publishRate : {cfg.publish_rate}") + lines.append(f" natnetVersion : {cfg.natnet_version}") + lines.append(f" upAxis : {cfg.up_axis}") + lines.append(f" poseNoise : enabled={cfg.pose_noise_enabled}") + lines.append( + f" std={cfg.pose_noise_std_meters} m, rot={cfg.pose_noise_rotation_deg} deg" + ) + if cfg.bodies: + lines.append(f" bodies ({len(cfg.bodies)}):") + for b in cfg.bodies: + target = b.target_prim or "" + lines.append( + f" - {b.rigid_body_name} (id={b.streaming_id}, parent={b.parent_id}) -> {target}" + ) + else: + lines.append(" bodies : (none)") + return "\n".join(lines) + + +class NatNetServerManager: + """Detects interface prims, prints config, and owns one server instance.""" + + def __init__(self, server_factory=None): + self._stage_event_sub = None + self._usd_listener = None + self._scan_tick_sub = None + self._scan_pending = False + self._timeline_sub = None + self._server = None + self._server_factory = server_factory or default_server_factory + # Sampling state. A NatNet prim edit sets ``_needs_resync``; the next physics + # sample re-reads the catalog/targets and clears it. + self._needs_resync = False + self._sample_cache: list = [] + self._frame_counter = 0 + self._catalog_signature = None + self._physx_sub = None + # Streamed up-axis, re-read on every resync. See frames.to_motive_pose. + self._up_axis = DEFAULT_UP_AXIS + # Pose noise. + self._pose_noise_enabled = False + self._pose_noise_std_meters = 0.0 + self._pose_noise_rotation_deg = 0.0 + + # --- lifecycle ------------------------------------------------------------- + + def on_startup(self): + import omni.usd + + usd_context = omni.usd.get_context() + self._stage_event_sub = usd_context.get_stage_event_stream().create_subscription_to_pop( + self._on_stage_event, name="natnet_manager_stage_events" + ) + self._register_usd_listener() + self._subscribe_physics() + self._subscribe_timeline() + print("[natnet] NatNetServerManager initialized") + self.scan_and_print() + + def on_shutdown(self): + self.stop_server() + self._physx_sub = None + self._timeline_sub = None + self._stage_event_sub = None + self._scan_tick_sub = None + self._scan_pending = False + self._revoke_usd_listener() + + def _subscribe_physics(self): + # Sample + enqueue poses on every physics step (only fires while playing). + try: + import omni.physx + + self._physx_sub = omni.physx.get_physx_interface().subscribe_physics_step_events( + self._on_physics_step + ) + except Exception as exc: # Kit/physx only + print(f"[natnet] Physics step subscription unavailable: {exc}") + self._physx_sub = None + + def _on_physics_step(self, _dt): + if self._server is not None: + self.sample_once() + + # --- timeline-driven lifecycle --------------------------------------------- + + def _subscribe_timeline(self): + """Bind the server's lifetime to the sim: Play starts it, Stop shuts it down. + + Play builds the server from the prim, so ``serverIp``/ports/``mode`` — bound + into the socket at construction — pick up whatever is authored at that point. + Body and noise edits need no rebuild; ``_resync`` re-reads them while running. + """ + try: + import omni.timeline + + self._timeline_sub = ( + omni.timeline.get_timeline_interface() + .get_timeline_event_stream() + .create_subscription_to_pop(self._on_timeline_event) + ) + except Exception as exc: # Kit only + print(f"[natnet] Timeline subscription unavailable: {exc}") + self._timeline_sub = None + + def _on_timeline_event(self, event): + import omni.timeline + + if event.type == int(omni.timeline.TimelineEventType.PLAY): + self._start_for_play() + elif event.type == int(omni.timeline.TimelineEventType.STOP): + self.stop_server() + + def _start_for_play(self): + """Start from the stage on Play, honouring the prim's ``serverEnabled``.""" + if self.is_running: + return + stage = self._get_stage() + if stage is None: + return + interfaces = find_interfaces(stage) + if not interfaces: + return + config = read_interface(interfaces[0]) + if not config.server_enabled: + print("[natnet] Play: serverEnabled is false — not starting.") + return + self.log_target_diagnostics(config) + self.start_server(config) + + # --- scanning -------------------------------------------------------------- + + def scan_and_print(self, *_): + """Find every interface prim and print its parsed config.""" + stage = self._get_stage() + if stage is None: + return + interfaces = find_interfaces(stage) + if not interfaces: + print("[natnet] Scan: no NatNetInterface prims on stage.") + return + print(f"[natnet] Scan: {len(interfaces)} interface(s) detected.") + for prim in interfaces: + cfg = read_interface(prim) + print(format_interface(prim.GetPath().pathString, cfg)) + + # --- server lifecycle (single instance; USD-free, factory-injectable) ------ + + @property + def is_running(self) -> bool: + return self._server is not None + + @property + def server(self): + return self._server + + def start_server(self, config: NatNetInterfaceConfig) -> bool: + """Build the catalog, construct a fresh server, and start it — once. + + Idempotent: if a server is already running this is a no-op returning False. + Returns True when a new server was created and started. + """ + if self._server is not None: + print("[natnet] start_server ignored: a server is already running.") + return False + catalog = build_catalog(config) + server = self._server_factory(config) + server.set_model_def_payload(catalog.pack()) + # Pump frames from the physics-step thread; the server's own background timer + # is starved by Kit's render/physics main loop. + if hasattr(server, "auto_stream"): + server.auto_stream = False + server.start() + self._server = server + # Build the prim->pose cache from the live stage on the first sampled frame. + self._needs_resync = True + self._frame_counter = 0 + # None so the first resync reports "changed" and the first frame flags + # model_list_changed, prompting the client to read MODELDEF. + self._catalog_signature = None + print( + f"[natnet] Server started on {config.server_ip} " + f"(cmd {config.command_port} / data {config.data_port}) " + f"with {len(config.bodies)} body(ies)." + ) + return True + + def stop_server(self) -> bool: + """Shut down the running server (fresh instance is built on next start). + + Idempotent: returns False if nothing was running. + """ + if self._server is None: + return False + try: + self._server.shutdown() + finally: + self._server = None + self._sample_cache = [] + self._needs_resync = False + print("[natnet] Server stopped.") + return True + + def toggle_server(self, config: NatNetInterfaceConfig) -> bool: + """Start if stopped, stop if running. Returns the resulting running state.""" + if self.is_running: + self.stop_server() + else: + self.start_server(config) + return self.is_running + + def apply_enabled(self, config: NatNetInterfaceConfig) -> None: + """Reconcile running state to ``config.server_enabled`` (start/stop).""" + if config.server_enabled and not self.is_running: + self.start_server(config) + elif not config.server_enabled and self.is_running: + self.stop_server() + + def log_target_diagnostics(self, config: NatNetInterfaceConfig) -> None: + """Warn about missing target prims and duplicate targets (best-effort).""" + stage = self._get_stage() + if stage is not None: + _existing, missing = resolve_targets(stage, config) + for body in missing: + print( + f"[natnet] WARNING: body '{body.rigid_body_name}' target prim " + f"missing or empty: {body.target_prim or ''}" + ) + for path in find_duplicate_targets(config): + print(f"[natnet] WARNING: multiple bodies target the same prim: {path}") + + # --- scripting entry point ------------------------------------------------- + + def start_from_stage(self) -> bool: + """Find the interface prim on the current stage, read it, and start. + + Convenience for scripts/Pegasus launchers: author the prim (see + ``author_interface``) then call this. Returns False if nothing to start. + """ + stage = self._get_stage() + if stage is None: + print("[natnet] start_from_stage: no active stage.") + return False + interfaces = find_interfaces(stage) + if not interfaces: + print("[natnet] start_from_stage: no NatNetInterface prim found.") + return False + config = read_interface(interfaces[0]) + self.log_target_diagnostics(config) + return self.start_server(config) + + # --- pose sampling + dynamic catalog (the data-enqueue path) --------------- + + def mark_dirty(self) -> None: + """Flag that the on-stage config changed; next sample re-reads the catalog.""" + self._needs_resync = True + + def _resync(self, stage) -> bool: + """Re-read the interface config, rebuild the catalog, and re-resolve targets. + + Returns True if the catalog (body id/name set) actually changed, so the next + frame can flag ``model_list_changed`` and the client re-requests MODELDEF. + """ + interfaces = find_interfaces(stage) + if not interfaces: + self._sample_cache = [] + return False + config = read_interface(interfaces[0]) + self._up_axis = config.up_axis + self._pose_noise_enabled = config.pose_noise_enabled + self._pose_noise_std_meters = config.pose_noise_std_meters + self._pose_noise_rotation_deg = config.pose_noise_rotation_deg + if self._server is not None: + self._server.set_model_def_payload(build_catalog(config).pack()) + # Cache target paths, not prim handles, so bodies whose target prim appears + # after the server starts begin streaming as soon as it exists. + self._sample_cache = [ + (body.streaming_id, body.rigid_body_name, body.target_prim) + for body in config.bodies + ] + signature = _catalog_signature(config) + changed = signature != self._catalog_signature + self._catalog_signature = signature + return changed + + def sample_once(self, stage=None): + """Sample every body's USD world pose and enqueue one frame to the server. + + Resyncs the catalog first if the config is dirty (so bodies added/removed + live are picked up). Returns the enqueued frame (or None if nothing to do). + """ + if self._server is None: + return None + if stage is None: + stage = self._get_stage() + if stage is None: + return None + + model_changed = False + if self._needs_resync: + model_changed = self._resync(stage) + self._needs_resync = False + + samples = [] + for streaming_id, _name, target_path in self._sample_cache: + prim = stage.GetPrimAtPath(target_path) if target_path else None + pose = read_world_pose(prim) if prim is not None else None + if pose is None: + samples.append(BodySample.lost(streaming_id)) + else: + position, orientation = to_motive_pose(*pose, up_axis=self._up_axis) + if self._pose_noise_enabled: + position, orientation = apply_pose_noise(position, orientation, self._pose_noise_std_meters, self._pose_noise_rotation_deg) + samples.append(BodySample(streaming_id, position, orientation, valid=True)) + + frame = build_frame( + self._frame_counter, samples, model_list_changed=model_changed + ) + self._frame_counter += 1 + self._server.enqueue_mocap_data(frame) + # Send synchronously from this (physics-step) thread. + flush_mocap_data = getattr(self._server, "flush_mocap_data", None) + if callable(flush_mocap_data): + flush_mocap_data() + return frame + + # --- stage / USD notifications -------------------------------------------- + + def _get_stage(self): + import omni.usd + + return omni.usd.get_context().get_stage() + + def _on_stage_event(self, event): + import omni.usd + + if event.type == int(omni.usd.StageEventType.OPENED): + self._register_usd_listener() + self.scan_and_print() + + def _register_usd_listener(self): + from pxr import Tf, Usd + + stage = self._get_stage() + if stage is None: + return + self._revoke_usd_listener() + self._usd_listener = Tf.Notice.Register( + Usd.Notice.ObjectsChanged, self._on_objects_changed, stage + ) + + def _revoke_usd_listener(self): + if self._usd_listener is not None: + self._usd_listener.Revoke() + self._usd_listener = None + + def _on_objects_changed(self, notice, sender): + # Only re-scan when something NatNet-related changed + try: + paths = list(notice.GetResyncedPaths()) + list(notice.GetChangedInfoOnlyPaths()) + except Exception: + paths = [] + if any(("NatNetInterface" in str(p)) or ("natnet:" in str(p)) for p in paths): + # A NatNet prim changed: mark the sampler dirty so the next physics step re-reads the catalog. + self._needs_resync = True + # Debounce author_interface() calls into one scan on the next update tick. + self._request_scan() + + def _request_scan(self): + if self._scan_pending: + return + self._scan_pending = True + import omni.kit.app + + self._scan_tick_sub = ( + omni.kit.app.get_app() + .get_update_event_stream() + .create_subscription_to_pop(self._on_scan_tick, name="natnet_manager_scan_tick") + ) + + def _on_scan_tick(self, _event): + self._scan_pending = False + self._scan_tick_sub = None + self.scan_and_print() diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py new file mode 100644 index 000000000..8c41053c3 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py @@ -0,0 +1,128 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Standalone-launch helpers: stand up a drone NatNet interface on scene load. + +Used by the Pegasus example launch scripts so a Motive-compatible NatNet server +comes up automatically with one rigid body per drone ``base_link`` — no UI clicks. + +Two layers, mirroring the rest of the package: + +- ``build_drone_config`` is **pure** (no USD / Kit), so it unit-tests hermetically. +- ``author_drone_natnet_interface`` writes the interface prim the extension builds + its server from. It imports ``pxr`` lazily (only when called). +""" + +from __future__ import annotations + +from typing import Iterable, Sequence, Tuple + +from .config import ( + DEFAULT_COMMAND_PORT, + DEFAULT_DATA_PORT, + DEFAULT_POSE_NOISE_ENABLED, + DEFAULT_POSE_NOISE_ROTATION_DEG, + DEFAULT_POSE_NOISE_STD_METERS, + DEFAULT_PUBLISH_RATE, + DEFAULT_SERVER_IP, + DEFAULT_UP_AXIS, + BodyBinding, + NatNetInterfaceConfig, +) + +# Where the example scripts author the single interface prim. +DEFAULT_INTERFACE_PATH = "/World/NatNetInterface" + +# Default world prim + position for the demo "target" body (a static placeholder +# the example scripts stream alongside the drones so a tracked target is available). +DEFAULT_TARGET_PATH = "/World/target" +DEFAULT_TARGET_POSITION = (2.0, 0.0, 1.0) +DEFAULT_TARGET_STREAMING_ID = 100 + +# (rigid_body_name, streaming_id, target_prim_path) +DroneSpec = Tuple[str, int, str] + + +def author_static_target( + stage, + prim_path: str = DEFAULT_TARGET_PATH, + position: Sequence[float] = DEFAULT_TARGET_POSITION, +): + """Author a static ``Xform`` prim to act as a NatNet-tracked target. + + Creates ``prim_path`` (a plain transform with a single translate op) at + ``position`` so the emulator can sample it like any other tracked body. The + prim is static — no physics, no animation — representing a fixed point of + interest that drones can be commanded toward. Imports ``pxr`` lazily so this + module stays importable outside Isaac. Returns ``prim_path``. + """ + from pxr import Gf, UsdGeom + + xform = UsdGeom.Xform.Define(stage, prim_path) + xform.AddTranslateOp().Set(Gf.Vec3d(float(position[0]), float(position[1]), float(position[2]))) + return prim_path + + +def build_drone_config( + drones: Iterable[DroneSpec], + *, + server_ip: str = DEFAULT_SERVER_IP, + mode: str = "unicast", + command_port: int = DEFAULT_COMMAND_PORT, + data_port: int = DEFAULT_DATA_PORT, + publish_rate: float = DEFAULT_PUBLISH_RATE, + server_enabled: bool = True, + up_axis: str = DEFAULT_UP_AXIS, + pose_noise_enabled: bool = DEFAULT_POSE_NOISE_ENABLED, + pose_noise_std_meters: float = DEFAULT_POSE_NOISE_STD_METERS, + pose_noise_rotation_deg: float = DEFAULT_POSE_NOISE_ROTATION_DEG, +) -> NatNetInterfaceConfig: + """Build a validated config with one rigid body per drone. + + ``drones`` is an iterable of ``(rigid_body_name, streaming_id, target_prim)`` + tuples — typically one per spawned drone, with ``target_prim`` pointing at the + drone's ``base_link``. Raises ``ValueError`` (via ``validate``) on duplicate + names/ids or bad ports. + """ + bodies = [ + BodyBinding( + rigid_body_name=str(name), + target_prim=str(target), + streaming_id=int(streaming_id), + ) + for name, streaming_id, target in drones + ] + cfg = NatNetInterfaceConfig( + server_enabled=server_enabled, + server_ip=server_ip, + mode=mode, + command_port=command_port, + data_port=data_port, + publish_rate=publish_rate, + up_axis=up_axis, + pose_noise_enabled=pose_noise_enabled, + pose_noise_std_meters=pose_noise_std_meters, + pose_noise_rotation_deg=pose_noise_rotation_deg, + bodies=bodies, + ) + cfg.validate() + return cfg + + +def author_drone_natnet_interface( + stage, + drones: Sequence[DroneSpec], + *, + prim_path: str = DEFAULT_INTERFACE_PATH, + **config_kwargs, +) -> NatNetInterfaceConfig: + """Author the NatNet interface prim from ``drones``. + + Writes ``prim_path`` (overwriting any existing interface) with one rigid body per + drone. Call this before starting the timeline: the extension builds the server + from this prim on Play. Returns the authored config. + """ + from .usd_bindings import author_interface + + cfg = build_drone_config(drones, **config_kwargs) + author_interface(stage, prim_path, cfg) + return cfg diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/ui_extension.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/ui_extension.py new file mode 100644 index 000000000..16a852d1f --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/ui_extension.py @@ -0,0 +1,435 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Kit extension entry: docked editor for the NatNet interface config prim. + +Create/manage the ``/World/NatNetInterface`` prim. The window docks to the +bottom-right (alongside the Property panel, like Pegasus) so it's easy to find. + +Sync model is explicit and user-driven via the top button row. +""" + +from __future__ import annotations + +import omni.ext + +from .config import VALID_MODES, VALID_UP_AXES, BodyBinding, NatNetInterfaceConfig +from .manager import NatNetServerManager +from .usd_bindings import author_interface, find_interfaces, read_interface, read_world_pose + +_DEFAULT_PRIM_PATH = "/World/NatNetInterface" +_LABEL_WIDTH = 140 +_POS_REFRESH_PERIOD = 1.0 / 6.0 # seconds between live USD position reads + +_COLOR_LIVE = 0xFF33CC33 # green: prim resolves and server is streaming +_COLOR_IDLE = 0xFFAAAAAA # grey: prim resolves but server not running +_COLOR_LOST = 0xFF3333FF # red: no prim / NaN + + +class NatNetEmulatorExtension(omni.ext.IExt): + """Registers the Window menu entry + the docked editor panel.""" + + def on_startup(self, ext_id): # noqa: D401 - Kit lifecycle hook + self._window = None + self._bodies_frame = None + self._cfg = NatNetInterfaceConfig() + self._row_readouts = {} + self._pos_refresh_sub = None + self._last_pos_refresh = 0.0 + self._manager = NatNetServerManager() + self._manager.on_startup() + self._add_menu() + self._subscribe_position_refresh() + + def on_shutdown(self): + self._remove_menu() + self._pos_refresh_sub = None + self._row_readouts = {} + if self._manager is not None: + self._manager.on_shutdown() + self._manager = None + if self._window is not None: + self._window.destroy() + self._window = None + + # --- live position readout ------------------------------------------------- + + def _subscribe_position_refresh(self): + try: + import omni.kit.app + except Exception: # pragma: no cover - Kit only + return + self._pos_refresh_sub = ( + omni.kit.app.get_app() + .get_update_event_stream() + .create_subscription_to_pop(self._on_pos_refresh, name="natnet_ui_pos_refresh") + ) + + def _on_pos_refresh(self, _event): + import time + + if self._window is None or not self._window.visible or not self._row_readouts: + return + now = time.monotonic() + if now - self._last_pos_refresh < _POS_REFRESH_PERIOD: + return + self._last_pos_refresh = now + stage = self._get_stage() + running = self._manager is not None and self._manager.is_running + for idx, (status_label, pos_label) in list(self._row_readouts.items()): + if not (0 <= idx < len(self._cfg.bodies)): + continue + target = self._cfg.bodies[idx].target_prim + symbol, color, text = self._row_readout(stage, target, running) + status_label.text = symbol + status_label.style = {"color": color} + pos_label.text = text + pos_label.style = {"color": color} + + def _row_readout(self, stage, target, running): + if not target: + return "\u25cb", _COLOR_IDLE, "no target prim" + prim = stage.GetPrimAtPath(target) if stage is not None else None + pose = read_world_pose(prim) if prim is not None else None + if pose is None: + return "\u2717", _COLOR_LOST, "NaN (prim missing)" + (x, y, z), _quat = pose + text = f"{x:+.3f}, {y:+.3f}, {z:+.3f}" + if running: + return "\u25cf", _COLOR_LIVE, text + return "\u25cf", _COLOR_IDLE, text + + # --- menu ------------------------------------------------------------------ + + def _add_menu(self): + try: + import omni.kit.menu.utils as menu_utils + from omni.kit.menu.utils import MenuItemDescription + except Exception: # pragma: no cover - Kit only + return + self._menu_entries = [ + MenuItemDescription(name="NatNet Interface", onclick_fn=self._toggle_window) + ] + menu_utils.add_menu_items(self._menu_entries, "Window") + + def _remove_menu(self): + try: + import omni.kit.menu.utils as menu_utils + except Exception: # pragma: no cover - Kit only + return + if getattr(self, "_menu_entries", None): + menu_utils.remove_menu_items(self._menu_entries, "Window") + self._menu_entries = None + + # --- window ---------------------------------------------------------------- + + def _toggle_window(self, *_): + import omni.ui as ui + + if self._window is None: + # Open on the interface authored on the stage, so Save writes back what is + # there — author_interface replaces the whole body set. + self._load_from_stage() + self._window = ui.Window("NatNet Interface", width=400, height=600) + self._window.frame.set_build_fn(self._build_window) + # Dock bottom-right next to the Property panel, like Pegasus. + self._window.deferred_dock_in("Property", ui.DockPolicy.CURRENT_WINDOW_IS_ACTIVE) + self._window.visible = True + return + self._window.visible = not self._window.visible + + def _refresh(self, *_): + if self._window is not None: + self._window.frame.rebuild() + + def _build_window(self): + import omni.ui as ui + + with ui.ScrollingFrame(): + with ui.VStack(spacing=6, height=0): + ui.Label("NatNet interface", height=0, style={"font_size": 16}) + + with ui.HStack(height=28, spacing=6): + ui.Button("Create Interface", clicked_fn=self._create_server) + ui.Button("Save", clicked_fn=self._save) + ui.Button("Load from Stage", clicked_fn=self._load_from_stage) + ui.Button("Print config", clicked_fn=self._print_config) + + running = self._manager is not None and self._manager.is_running + # Read-only: the server's lifetime follows the sim, so there is no + # control here. Play starts it from the prim, Stop shuts it down. + with ui.HStack(height=28, spacing=6): + ui.Label( + f"Server: {'RUNNING' if running else 'stopped (press Play)'}", + width=0, + style={"color": 0xFF33CC33 if running else 0xFF888888}, + ) + + ui.Label( + "\u26a0 Remember to save after each edit", + height=0, + word_wrap=True, + style={"color": 0xFF33CCFF, "font_size": 14}, + ) + + ui.Label(self._status_text(), height=0, word_wrap=True) + + ui.Separator(height=6) + self._bool_row(ui, "Server enabled", "server_enabled", self._cfg.server_enabled) + self._bool_row(ui, "Pose noise enabled", "pose_noise_enabled", self._cfg.pose_noise_enabled) + self._float_row(ui, "Pose noise std meters", "pose_noise_std_meters", self._cfg.pose_noise_std_meters) + self._float_row(ui, "Pose noise rotation deg", "pose_noise_rotation_deg", self._cfg.pose_noise_rotation_deg) + self._str_row(ui, "Server IP", "server_ip", self._cfg.server_ip) + self._combo_row(ui, "Mode", "mode", self._cfg.mode, VALID_MODES) + self._int_row(ui, "Command port", "command_port", self._cfg.command_port) + self._int_row(ui, "Data port", "data_port", self._cfg.data_port) + self._float_row(ui, "Publish rate (Hz)", "publish_rate", self._cfg.publish_rate) + self._combo_row(ui, "Up axis", "up_axis", self._cfg.up_axis, VALID_UP_AXES) + + ui.Separator(height=6) + ui.Label("Tracked bodies", height=0, style={"font_size": 14}) + self._bodies_frame = ui.Frame(height=0) + self._bodies_frame.set_build_fn(self._build_bodies) + with ui.HStack(height=0, spacing=6): + ui.Button("Add body (from selection)", clicked_fn=self._add_body) + + def _status_text(self): + prim = self._find_interface() + if prim is None: + return "No prim on stage yet — Save or Create Server to author one." + return f"Prim on stage: {prim.GetPath().pathString} (Save to push edits, Load to pull)" + + # --- server field rows (edit the working copy only) ------------------------ + + def _bool_row(self, ui, label, key, value): + with ui.HStack(height=0): + ui.Label(label, width=_LABEL_WIDTH) + cb = ui.CheckBox() + cb.model.set_value(bool(value)) + cb.model.add_value_changed_fn( + lambda m, k=key: self._set_cfg_field(k, m.get_value_as_bool()) + ) + + def _str_row(self, ui, label, key, value): + with ui.HStack(height=0): + ui.Label(label, width=_LABEL_WIDTH) + model = ui.StringField().model + model.set_value(str(value)) + model.add_value_changed_fn( + lambda m, k=key: self._set_cfg_field(k, m.get_value_as_string()) + ) + + def _int_row(self, ui, label, key, value): + with ui.HStack(height=0): + ui.Label(label, width=_LABEL_WIDTH) + model = ui.IntField().model + model.set_value(int(value)) + model.add_value_changed_fn( + lambda m, k=key: self._set_cfg_field(k, m.get_value_as_int()) + ) + + def _float_row(self, ui, label, key, value): + with ui.HStack(height=0): + ui.Label(label, width=_LABEL_WIDTH) + model = ui.FloatField().model + model.set_value(float(value)) + model.add_value_changed_fn( + lambda m, k=key: self._set_cfg_field(k, m.get_value_as_float()) + ) + + def _combo_row(self, ui, label, key, value, choices): + with ui.HStack(height=0): + ui.Label(label, width=_LABEL_WIDTH) + index = choices.index(value) if value in choices else 0 + combo = ui.ComboBox(index, *choices) + combo.model.get_item_value_model().add_value_changed_fn( + lambda m, k=key, c=choices: self._set_cfg_field(k, c[m.get_value_as_int()]) + ) + + def _set_cfg_field(self, attr, value): + setattr(self._cfg, attr, value) + + # --- bodies ---------------------------------------------------------------- + + def _rebuild_bodies(self, *_): + if self._bodies_frame is not None: + self._bodies_frame.rebuild() + + def _build_bodies(self): + import omni.ui as ui + + self._row_readouts = {} + with ui.VStack(spacing=6, height=0): + if not self._cfg.bodies: + ui.Label(" (no bodies — select a prim and click Add body)", height=0) + return + with ui.HStack(height=0, spacing=4): + ui.Label("Rigid body name", width=ui.Fraction(1)) + ui.Label("ID", width=40) + ui.Label("Parent", width=50) + ui.Label("Target prim", width=ui.Fraction(2)) + ui.Spacer(width=98) + for idx, body in enumerate(self._cfg.bodies): + self._build_body_row(ui, idx, body) + + def _build_body_row(self, ui, idx, body): + with ui.VStack(height=0, spacing=2): + with ui.HStack(height=0, spacing=4): + name = ui.StringField(width=ui.Fraction(1)).model + name.set_value(body.rigid_body_name) + name.add_value_changed_fn( + lambda m, i=idx: self._set_body_field(i, "rigid_body_name", m.get_value_as_string()) + ) + + sid = ui.IntField(width=40).model + sid.set_value(body.streaming_id) + sid.add_value_changed_fn( + lambda m, i=idx: self._set_body_field(i, "streaming_id", m.get_value_as_int()) + ) + + parent = ui.IntField(width=50).model + parent.set_value(body.parent_id) + parent.add_value_changed_fn( + lambda m, i=idx: self._set_body_field(i, "parent_id", m.get_value_as_int()) + ) + + target = ui.StringField(width=ui.Fraction(2), tooltip="USD path of the tracked prim").model + target.set_value(body.target_prim) + target.add_value_changed_fn( + lambda m, i=idx: self._set_body_field(i, "target_prim", m.get_value_as_string()) + ) + + ui.Button("set target", width=70, clicked_fn=lambda i=idx: self._retarget_body(i)) + ui.Button("x", width=24, clicked_fn=lambda i=idx: self._remove_body_at(i)) + + # Live readout: status dot + world position pulled from the USD stage. + stage = self._get_stage() + running = self._manager is not None and self._manager.is_running + symbol, color, text = self._row_readout(stage, body.target_prim, running) + with ui.HStack(height=0, spacing=6): + ui.Spacer(width=4) + status_label = ui.Label(symbol, width=14, style={"color": color}) + ui.Label("pos:", width=30, style={"color": _COLOR_IDLE}) + pos_label = ui.Label(text, width=ui.Fraction(1), style={"color": color}) + self._row_readouts[idx] = (status_label, pos_label) + + def _set_body_field(self, index, attr, value): + if 0 <= index < len(self._cfg.bodies): + setattr(self._cfg.bodies[index], attr, value) + + def _add_body(self): + next_id = max((b.streaming_id for b in self._cfg.bodies), default=0) + 1 + target = self._selected_target_path(self._find_interface()) + name = target.rsplit("/", 1)[-1] if target else f"Body{next_id}" + existing = {b.rigid_body_name for b in self._cfg.bodies} + while name in existing: + name = f"{name}_{next_id}" + self._cfg.bodies.append(BodyBinding(rigid_body_name=name, target_prim=target, streaming_id=next_id)) + self._rebuild_bodies() + + def _remove_body_at(self, index): + if 0 <= index < len(self._cfg.bodies): + self._cfg.bodies.pop(index) + self._rebuild_bodies() + + def _retarget_body(self, index): + import carb + + path = self._selected_target_path(self._find_interface()) + if not path: + carb.log_warn("[natnet] Select a prim in the viewport to retarget this body.") + return + if 0 <= index < len(self._cfg.bodies): + self._cfg.bodies[index].target_prim = path + self._rebuild_bodies() + + # --- stage helpers --------------------------------------------------------- + + def _get_stage(self): + import omni.usd + + return omni.usd.get_context().get_stage() + + def _find_interface(self): + stage = self._get_stage() + if stage is None: + return None + interfaces = find_interfaces(stage) + return interfaces[0] if interfaces else None + + def _interface_path(self): + prim = self._find_interface() + return prim.GetPath().pathString if prim is not None else _DEFAULT_PRIM_PATH + + def _select(self, prim_path): + import omni.usd + + omni.usd.get_context().get_selection().set_selected_prim_paths([prim_path], True) + + def _selected_target_path(self, interface_prim): + import omni.usd + + sel = omni.usd.get_context().get_selection().get_selected_prim_paths() + iface_path = interface_prim.GetPath().pathString if interface_prim else None + for path in sel: + if path != iface_path: + return path + return "" + + # --- explicit sync actions ------------------------------------------------- + + def _save(self): + import carb + + stage = self._get_stage() + if stage is None: + carb.log_error("[natnet] No active stage.") + return + try: + self._cfg.validate() + except ValueError as exc: + carb.log_error(f"[natnet] Not saved: {exc}") + return + path = self._interface_path() + author_interface(stage, path, self._cfg) + carb.log_info(f"[natnet] Saved interface to {path} ({len(self._cfg.bodies)} bodies).") + self._refresh() + + def _load_from_stage(self): + import carb + + prim = self._find_interface() + if prim is None: + self._cfg = NatNetInterfaceConfig() + carb.log_warn("[natnet] No interface on stage — reset to defaults.") + else: + self._cfg = read_interface(prim) + carb.log_info(f"[natnet] Loaded interface from {prim.GetPath().pathString}.") + self._refresh() + + def _print_config(self): + # Print whatever is authored on the stage (the source of truth). + if self._manager is not None: + self._manager.scan_and_print() + + def _create_server(self): + import carb + + stage = self._get_stage() + if stage is None: + carb.log_error("[natnet] No active stage.") + return + prim = self._find_interface() + if prim is None: + try: + self._cfg.validate() + except ValueError as exc: + carb.log_error(f"[natnet] Cannot create: {exc}") + return + author_interface(stage, _DEFAULT_PRIM_PATH, self._cfg) + path = _DEFAULT_PRIM_PATH + carb.log_info(f"[natnet] Created interface prim at {path}. (Server start: later commit.)") + else: + path = prim.GetPath().pathString + carb.log_info(f"[natnet] Interface already exists at {path}. (Server start: later commit.)") + self._select(path) + self._refresh() diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/usd_bindings.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/usd_bindings.py new file mode 100644 index 000000000..262216ee2 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/usd_bindings.py @@ -0,0 +1,216 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""USD binding layer: author / read / find NatNet interface prims on a stage. + +``pxr`` is imported lazily inside each function so importing this module doesn't +require USD. + +Backing today is plain namespaced custom attributes + relationships. Property names +follow the multi-apply schema convention (``natnet:body::``). +""" + +from __future__ import annotations + +from typing import Any + +from .config import ( + ATTR_COMMAND_PORT, + ATTR_DATA_PORT, + ATTR_MODE, + ATTR_MULTICAST_ADDR, + ATTR_NATNET_VERSION, + ATTR_POSE_NOISE_ENABLED, + ATTR_POSE_NOISE_ROTATION_DEG, + ATTR_POSE_NOISE_STD_METERS, + ATTR_PUBLISH_RATE, + ATTR_SERVER_ENABLED, + ATTR_SERVER_IP, + ATTR_UP_AXIS, + BODY_FIELD_PARENT_ID, + BODY_FIELD_RIGID_BODY_NAME, + BODY_FIELD_STREAMING_ID, + BODY_FIELD_TARGET, + BODY_PREFIX, + DEFAULT_COMMAND_PORT, + DEFAULT_DATA_PORT, + DEFAULT_MULTICAST_ADDR, + DEFAULT_NATNET_VERSION, + DEFAULT_POSE_NOISE_ENABLED, + DEFAULT_POSE_NOISE_ROTATION_DEG, + DEFAULT_POSE_NOISE_STD_METERS, + DEFAULT_PUBLISH_RATE, + DEFAULT_SERVER_IP, + DEFAULT_UP_AXIS, + MARKER_ATTR, + BodyBinding, + NatNetInterfaceConfig, + body_attr_name, +) + + +def author_interface(stage, prim_path: str, config: Any) -> Any: + """Create/overwrite a NatNet interface prim at ``prim_path`` from ``config``. + + ``config`` may be a :class:`NatNetInterfaceConfig` or a plain ``dict`` (passed + through ``from_dict``). Returns the ``Usd.Prim``. + """ + from pxr import Sdf + + cfg = config if isinstance(config, NatNetInterfaceConfig) else NatNetInterfaceConfig.from_dict(config) + cfg.validate() + + prim = stage.DefinePrim(prim_path, "Scope") + + # Overwrite semantics: drop any previously-authored body properties so removed + # bodies don't linger across re-authoring. + _clear_body_properties(prim) + + _set(prim, MARKER_ATTR, Sdf.ValueTypeNames.Bool, True) + _set(prim, ATTR_SERVER_ENABLED, Sdf.ValueTypeNames.Bool, cfg.server_enabled) + _set(prim, ATTR_SERVER_IP, Sdf.ValueTypeNames.String, cfg.server_ip) + _set(prim, ATTR_MODE, Sdf.ValueTypeNames.Token, cfg.mode) + _set(prim, ATTR_MULTICAST_ADDR, Sdf.ValueTypeNames.String, cfg.multicast_addr) + _set(prim, ATTR_COMMAND_PORT, Sdf.ValueTypeNames.Int, cfg.command_port) + _set(prim, ATTR_DATA_PORT, Sdf.ValueTypeNames.Int, cfg.data_port) + _set(prim, ATTR_PUBLISH_RATE, Sdf.ValueTypeNames.Float, cfg.publish_rate) + _set(prim, ATTR_NATNET_VERSION, Sdf.ValueTypeNames.String, cfg.natnet_version) + _set(prim, ATTR_UP_AXIS, Sdf.ValueTypeNames.Token, cfg.up_axis) + _set(prim, ATTR_POSE_NOISE_ENABLED, Sdf.ValueTypeNames.Bool, cfg.pose_noise_enabled) + _set(prim, ATTR_POSE_NOISE_STD_METERS, Sdf.ValueTypeNames.Float, cfg.pose_noise_std_meters) + _set(prim, ATTR_POSE_NOISE_ROTATION_DEG, Sdf.ValueTypeNames.Float, cfg.pose_noise_rotation_deg) + + for key, body in cfg.assign_instance_keys(): + _set(prim, body_attr_name(key, BODY_FIELD_RIGID_BODY_NAME), Sdf.ValueTypeNames.String, body.rigid_body_name) + _set(prim, body_attr_name(key, BODY_FIELD_STREAMING_ID), Sdf.ValueTypeNames.Int, body.streaming_id) + _set(prim, body_attr_name(key, BODY_FIELD_PARENT_ID), Sdf.ValueTypeNames.Int, body.parent_id) + rel = prim.CreateRelationship(body_attr_name(key, BODY_FIELD_TARGET), False) + rel.SetTargets([Sdf.Path(body.target_prim)] if body.target_prim else []) + + return prim + + +def read_interface(prim) -> NatNetInterfaceConfig: + """Reconstruct a :class:`NatNetInterfaceConfig` from an authored interface prim.""" + return NatNetInterfaceConfig( + server_enabled=bool(_get(prim, ATTR_SERVER_ENABLED, True)), + server_ip=str(_get(prim, ATTR_SERVER_IP, DEFAULT_SERVER_IP)), + mode=str(_get(prim, ATTR_MODE, "unicast")), + multicast_addr=str(_get(prim, ATTR_MULTICAST_ADDR, DEFAULT_MULTICAST_ADDR)), + command_port=int(_get(prim, ATTR_COMMAND_PORT, DEFAULT_COMMAND_PORT)), + data_port=int(_get(prim, ATTR_DATA_PORT, DEFAULT_DATA_PORT)), + publish_rate=float(_get(prim, ATTR_PUBLISH_RATE, DEFAULT_PUBLISH_RATE)), + natnet_version=str(_get(prim, ATTR_NATNET_VERSION, DEFAULT_NATNET_VERSION)), + up_axis=str(_get(prim, ATTR_UP_AXIS, DEFAULT_UP_AXIS)), + pose_noise_enabled=bool(_get(prim, ATTR_POSE_NOISE_ENABLED, DEFAULT_POSE_NOISE_ENABLED)), + pose_noise_std_meters=float(_get(prim, ATTR_POSE_NOISE_STD_METERS, DEFAULT_POSE_NOISE_STD_METERS)), + pose_noise_rotation_deg=float(_get(prim, ATTR_POSE_NOISE_ROTATION_DEG, DEFAULT_POSE_NOISE_ROTATION_DEG)), + bodies=_read_bodies(prim), + ) + + +def find_interfaces(stage) -> list: + """Return every prim on the stage marked as a NatNet interface.""" + interfaces = [] + for prim in stage.Traverse(): + attr = prim.GetAttribute(MARKER_ATTR) + if attr and attr.HasAuthoredValue() and bool(attr.Get()): + interfaces.append(prim) + return interfaces + + +def is_interface(prim) -> bool: + attr = prim.GetAttribute(MARKER_ATTR) + return bool(attr and attr.HasAuthoredValue() and bool(attr.Get())) + + +def read_world_pose(prim): + """Return ``((x, y, z), (qx, qy, qz, qw))`` from a prim's USD world transform. + + Reads the position/orientation **stored in the USD stage** (the local-to-world + transform), which is what the physics step writes back each frame. Returns + ``None`` for an invalid/non-xformable prim so callers can mark the body lost. + """ + from pxr import Usd, UsdGeom + + if prim is None or not prim.IsValid(): + return None + xformable = UsdGeom.Xformable(prim) + if not xformable: + return None + matrix = xformable.ComputeLocalToWorldTransform(Usd.TimeCode.Default()) + translation = matrix.ExtractTranslation() + quat = matrix.ExtractRotationQuat() # Gf.Quatd, normalized + imaginary = quat.GetImaginary() + position = (float(translation[0]), float(translation[1]), float(translation[2])) + orientation = ( + float(imaginary[0]), + float(imaginary[1]), + float(imaginary[2]), + float(quat.GetReal()), + ) + return position, orientation + + +def resolve_targets(stage, config): + """Split a config's bodies into (existing, missing) by target prim presence. + + A body whose ``target_prim`` is empty or points at a non-existent prim lands in + ``missing``. Returns two lists of :class:`BodyBinding`. + """ + existing = [] + missing = [] + for body in config.bodies: + prim = stage.GetPrimAtPath(body.target_prim) if body.target_prim else None + if prim is not None and prim.IsValid(): + existing.append(body) + else: + missing.append(body) + return existing, missing + + +# --- internal helpers ---------------------------------------------------------- + + +def _set(prim, name, type_name, value): + attr = prim.CreateAttribute(name, type_name) + attr.Set(value) + return attr + + +def _get(prim, name, default): + attr = prim.GetAttribute(name) + if attr and attr.HasAuthoredValue(): + return attr.Get() + return default + + +def _clear_body_properties(prim) -> None: + for name in list(prim.GetPropertyNames()): + if name.startswith(BODY_PREFIX): + prim.RemoveProperty(name) + + +def _read_bodies(prim) -> list[BodyBinding]: + suffix = f":{BODY_FIELD_RIGID_BODY_NAME}" + keys = [ + name[len(BODY_PREFIX): -len(suffix)] + for name in prim.GetPropertyNames() + if name.startswith(BODY_PREFIX) and name.endswith(suffix) + ] + + bodies: list[BodyBinding] = [] + for key in keys: + rel = prim.GetRelationship(body_attr_name(key, BODY_FIELD_TARGET)) + targets = rel.GetTargets() if rel else [] + bodies.append( + BodyBinding( + rigid_body_name=str(_get(prim, body_attr_name(key, BODY_FIELD_RIGID_BODY_NAME), "")), + target_prim=str(targets[0]) if targets else "", + streaming_id=int(_get(prim, body_attr_name(key, BODY_FIELD_STREAMING_ID), 1)), + parent_id=int(_get(prim, body_attr_name(key, BODY_FIELD_PARENT_ID), -1)), + ) + ) + + # Stable, deterministic order (independent of USD property iteration order). + bodies.sort(key=lambda b: (b.streaming_id, b.rigid_body_name)) + return bodies diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/schema/schema.usda b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/schema/schema.usda new file mode 100644 index 000000000..55defc731 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/schema/schema.usda @@ -0,0 +1,100 @@ +#usda 1.0 +( + """ + NatNet emulator applied API schemas (CODELESS). + + These schemas give the + interface prim typed, Property-panel-friendly attributes WITHOUT compiled + classes. They are codeless — `skipCodeGeneration = true` below — but still need + USD's plugin system to discover the generated registry. + + To produce the registry files (run once, in an env with USD tooling): + + usdGenSchema schema/schema.usda schema/ + + That emits `schema/generatedSchema.usda` and `schema/plugInfo.json`. The Kit + extension then registers the plugin dir on startup (Plug.Registry().RegisterPlugins). + + Until that registration is verified inside Kit, `optitrack.natnet.emulator.isaac` + authors the SAME attribute names as plain namespaced custom attributes (the + registration-free fallback), so nothing here is required for the facade to work. + """ + subLayers = [ + @usd/schema.usda@, + @usdGeom/schema.usda@ + ] +) +{ +} + +over "GLOBAL" ( + customData = { + bool skipCodeGeneration = true + string libraryName = "optitrackNatNet" + string libraryPath = "." + string libraryPrefix = "OptiTrackNatNet" + } +) +{ +} + +class "NatNetInterfaceAPI" ( + inherits = + customData = { + token apiSchemaType = "singleApply" + } + doc = "Marks a prim as a NatNet emulator interface and holds server-level config." +) +{ + bool natnet:isInterface = true ( + doc = "Discovery marker — find_interfaces() scans for prims with this set true." + ) + bool natnet:serverEnabled = true ( + doc = "When true the manager keeps a server running; toggling restarts it." + ) + string natnet:serverIp = "172.31.0.200" ( + doc = "Server interface IP (NatNetUnicastServer.local_interface)." + ) + token natnet:mode = "unicast" ( + allowedTokens = ["unicast", "multicast"] + doc = "Transmission mode." + ) + string natnet:multicastAddr = "239.255.42.99" ( + doc = "Multicast group (only used when mode = multicast)." + ) + int natnet:commandPort = 1510 ( + doc = "NatNet command port." + ) + int natnet:dataPort = 1511 ( + doc = "NatNet data port (frames stream from this source port)." + ) + float natnet:publishRate = 100 ( + doc = "Frame publish rate in Hz." + ) + string natnet:natnetVersion = "4.4.0.0" ( + doc = "Advertised NatNet protocol version." + ) +} + +class "NatNetBodyBindingAPI" ( + inherits = + customData = { + token apiSchemaType = "multipleApply" + token propertyNamespacePrefix = "natnet:body" + } + doc = "One tracked rigid body entry on a NatNet interface prim (apply once per body)." +) +{ + string natnet:body:__INSTANCE_NAME__:rigidBodyName = "" ( + doc = "Motive rigid body name (sRigidBodyDescription.szName)." + ) + int natnet:body:__INSTANCE_NAME__:streamingId = 1 ( + doc = "Streaming ID (sRigidBodyDescription.ID)." + ) + int natnet:body:__INSTANCE_NAME__:parentId = -1 ( + doc = "Parent rigid body ID (-1 if none)." + ) + rel natnet:body:__INSTANCE_NAME__:target ( + doc = "Tracked prim whose world pose is streamed for this body." + ) +} diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_catalog.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_catalog.py new file mode 100644 index 000000000..a3cc8cd34 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_catalog.py @@ -0,0 +1,111 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Catalog builder: body counts, wire fidelity, truncation, MAX_MODELS, duplicate targets.""" + +from __future__ import annotations + +import struct + +import pytest + +from optitrack.natnet.emulator.isaac.catalog import build_catalog, find_duplicate_targets +from optitrack.natnet.emulator.isaac.config import BodyBinding, NatNetInterfaceConfig +from optitrack.natnet.emulator.server import natnet_model_types as mt +from optitrack.natnet.emulator.server.natnet_common import ModelLimits + +pytestmark = pytest.mark.unit + + +def _unpack_bodies(payload: bytes): + """Decode a packed sDataDescriptions into [(name, id, parentID), ...].""" + (n,) = struct.unpack_from("frame builder tests (no USD, no Kit).""" + +from __future__ import annotations + +import math +import struct + +import pytest + +from optitrack.natnet.emulator.isaac.frames import ( + MODEL_LIST_CHANGED, + TRACKING_VALID, + BodySample, + apply_pose_noise, + build_frame, + make_rigid_body_data, + to_motive_pose, +) + +pytestmark = pytest.mark.unit + + +def test_to_motive_pose_z_is_identity(): + pos = (1.0, 2.0, 3.0) + quat = (0.1, 0.2, 0.3, 0.9) + assert to_motive_pose(pos, quat, up_axis="Z") == (pos, quat) + # Case-insensitive and default to Z. + assert to_motive_pose(pos, quat, up_axis="z") == (pos, quat) + assert to_motive_pose(pos, quat) == (pos, quat) + + +def test_to_motive_pose_y_swaps_axes_and_quat(): + # (x, y, z) -> (x, z, -y); quat vector part takes the same swap, scalar kept. + pos, quat = to_motive_pose((1.0, 2.0, 3.0), (0.1, 0.2, 0.3, 0.9), up_axis="Y") + assert pos == (1.0, 3.0, -2.0) + assert quat == (0.1, 0.3, -0.2, 0.9) + + +def test_to_motive_pose_y_maps_isaac_up_to_motive_up(): + # Isaac +Z (up) must become Motive +Y (up) under the Y-up emulation. + pos, _ = to_motive_pose((0.0, 0.0, 1.0), (0.0, 0.0, 0.0, 1.0), up_axis="y") + assert pos == (0.0, 1.0, 0.0) + + +def test_make_rigid_body_data_copies_pose_and_sets_valid_bit(): + rb = make_rigid_body_data( + BodySample(7, (1.0, 2.0, 3.0), (0.0, 0.0, 0.7071068, 0.7071068), valid=True) + ) + assert rb.ID == 7 + assert (rb.x, rb.y, rb.z) == (1.0, 2.0, 3.0) + assert rb.qw == pytest.approx(0.7071068) + assert rb.params & TRACKING_VALID # client requires this bit or it skips the body + + +def test_lost_sample_clears_valid_bit_and_is_nan(): + rb = make_rigid_body_data(BodySample.lost(3)) + assert rb.ID == 3 + assert rb.params & TRACKING_VALID == 0 + assert math.isnan(rb.x) and math.isnan(rb.y) and math.isnan(rb.z) + + +def test_build_frame_no_bodies(): + frame = build_frame(0, []) + assert frame.iFrame == 0 + assert frame.nRigidBodies == 0 + assert frame.params == 0 + + +def test_apply_pose_noise_zero_std_is_identity(): + position = (1.0, 2.0, 3.0) + orientation = (0.0, 0.0, 0.0, 1.0) + pos_out, quat_out = apply_pose_noise(position, orientation, 0.0, 0.0) + assert pos_out == position + assert quat_out == pytest.approx(orientation) + + +def test_apply_pose_noise_preserves_y_position(): + np = pytest.importorskip("numpy") + np.random.seed(0) + position = (0.0, 1.5, 0.0) + orientation = (0.0, 0.0, 0.0, 1.0) + pos_out, _ = apply_pose_noise(position, orientation, 0.001, 0.0) + # Before the euler-yaw shadowing bug, y collapsed to ~0 instead of staying near 1.5. + assert pos_out[1] == pytest.approx(1.5, abs=0.01) + + +def test_apply_pose_noise_adds_position_jitter(): + np = pytest.importorskip("numpy") + np.random.seed(1) + position = (0.0, 0.0, 0.0) + orientation = (0.0, 0.0, 0.0, 1.0) + pos_out, _ = apply_pose_noise(position, orientation, 0.001, 0.0) + assert pos_out != position + + +def test_build_frame_multiple_bodies_preserve_order(): + samples = [ + BodySample(1, (1.0, 0.0, 0.0)), + BodySample(2, (0.0, 2.0, 0.0)), + BodySample(5, (0.0, 0.0, 3.0)), + ] + frame = build_frame(42, samples) + assert frame.iFrame == 42 + assert frame.nRigidBodies == 3 + assert frame.RigidBodies[0].ID == 1 and frame.RigidBodies[0].x == 1.0 + assert frame.RigidBodies[1].ID == 2 and frame.RigidBodies[1].y == 2.0 + assert frame.RigidBodies[2].ID == 5 and frame.RigidBodies[2].z == 3.0 + + +def test_model_list_changed_sets_frame_param_bit(): + assert build_frame(0, [], model_list_changed=True).params & MODEL_LIST_CHANGED + assert build_frame(0, [], model_list_changed=False).params & MODEL_LIST_CHANGED == 0 + + +def test_frame_packs_and_rigid_body_section_decodes(): + frame = build_frame(9, [BodySample(4, (1.5, -2.5, 3.5), (0.0, 0.0, 0.0, 1.0))]) + payload = frame.pack(natnet_major=4, natnet_minor=4) + + # iFrame, then 4.4 counted sections (count+size each) for markersets & other markers. + (iframe,) = struct.unpack_from(" author -> read is stable + author_interface(stage, "/World/NatNetInterface", cfg) + assert read_interface(find_interfaces(stage)[0]) == cfg + + +def test_reauthoring_removes_stale_bodies(): + stage = _new_stage() + author_interface(stage, "/World/NatNetInterface", _CONFIG) + + single = NatNetInterfaceConfig.from_dict( + {"bodies": [{"rigid_body_name": "Drone", "target_prim": "/World/base_link", "streaming_id": 1}]} + ) + author_interface(stage, "/World/NatNetInterface", single) + + cfg = read_interface(find_interfaces(stage)[0]) + assert [b.rigid_body_name for b in cfg.bodies] == ["Drone"] + + +def test_up_axis_authors_and_reads_back(): + stage = _new_stage() + # Default (absent) -> Z. + author_interface(stage, "/World/NatNetInterface", _CONFIG) + assert read_interface(find_interfaces(stage)[0]).up_axis == "Z" + + # Explicit Y survives the USD round trip. + cfg = NatNetInterfaceConfig.from_dict({**_CONFIG, "up_axis": "Y"}) + author_interface(stage, "/World/NatNetInterface", cfg) + assert read_interface(find_interfaces(stage)[0]).up_axis == "Y" + + +def test_pose_noise_authors_and_reads_back(): + stage = _new_stage() + cfg = NatNetInterfaceConfig.from_dict( + { + **_CONFIG, + "pose_noise_enabled": False, + "pose_noise_std_meters": 0.001, + "pose_noise_rotation_deg": 0.1, + } + ) + author_interface(stage, "/World/NatNetInterface", cfg) + read = read_interface(find_interfaces(stage)[0]) + assert read.pose_noise_enabled is False + assert read.pose_noise_std_meters == pytest.approx(0.001) + assert read.pose_noise_rotation_deg == pytest.approx(0.1) + + +def test_empty_target_round_trips(): + # The UI's "Add body" can create a body with no target yet (set later in the + # Property panel); it must author and read back cleanly with an empty target. + stage = _new_stage() + cfg = NatNetInterfaceConfig(bodies=[BodyBinding("Drone", "", 1)]) + author_interface(stage, "/World/NatNetInterface", cfg) + read = read_interface(find_interfaces(stage)[0]) + assert read.bodies[0].rigid_body_name == "Drone" + assert read.bodies[0].target_prim == "" + + +def test_invalid_config_raises_before_authoring(): + stage = _new_stage() + with pytest.raises(ValueError): + author_interface(stage, "/World/NatNetInterface", {"mode": "bogus"}) + assert find_interfaces(stage) == [] diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_interface_config.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_interface_config.py new file mode 100644 index 000000000..9247c5f12 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_interface_config.py @@ -0,0 +1,186 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Hermetic unit tests for the pure-Python NatNet interface config model. + +No USD / Isaac imports — exercises dataclasses, dict normalization, the attribute +name builder, instance-key generation, and validation. +""" + +from __future__ import annotations + +import pytest + +from optitrack.natnet.emulator.isaac.config import ( + DEFAULT_POSE_NOISE_ENABLED, + DEFAULT_POSE_NOISE_ROTATION_DEG, + DEFAULT_POSE_NOISE_STD_METERS, + BodyBinding, + NatNetInterfaceConfig, + body_attr_name, + make_instance_key, +) + +pytestmark = pytest.mark.unit + + +def test_defaults_match_server_expectations(): + cfg = NatNetInterfaceConfig() + assert cfg.server_enabled is True + assert cfg.server_ip == "172.31.0.200" + assert cfg.mode == "unicast" + assert cfg.command_port == 1510 + assert cfg.data_port == 1511 + assert cfg.up_axis == "Z" # Isaac/USD native; matches the reference Motive setup + assert cfg.pose_noise_enabled is DEFAULT_POSE_NOISE_ENABLED + assert cfg.pose_noise_std_meters == DEFAULT_POSE_NOISE_STD_METERS + assert cfg.pose_noise_rotation_deg == DEFAULT_POSE_NOISE_ROTATION_DEG + assert cfg.bodies == [] + + +def test_up_axis_from_dict_normalizes_case(): + assert NatNetInterfaceConfig.from_dict({"up_axis": "y"}).up_axis == "Y" + assert NatNetInterfaceConfig.from_dict({"up_axis": "z"}).up_axis == "Z" + # Absent -> default Z. + assert NatNetInterfaceConfig.from_dict({}).up_axis == "Z" + + +def test_up_axis_survives_round_trip(): + cfg = NatNetInterfaceConfig.from_dict({"up_axis": "Y"}) + assert NatNetInterfaceConfig.from_dict(cfg.to_dict()).up_axis == "Y" + + +def test_pose_noise_survives_round_trip(): + cfg = NatNetInterfaceConfig.from_dict( + { + "pose_noise_enabled": False, + "pose_noise_std_meters": 0.001, + "pose_noise_rotation_deg": 0.1, + } + ) + restored = NatNetInterfaceConfig.from_dict(cfg.to_dict()) + assert restored.pose_noise_enabled is False + assert restored.pose_noise_std_meters == 0.001 + assert restored.pose_noise_rotation_deg == 0.1 + + +def test_from_dict_with_bodies_as_list(): + cfg = NatNetInterfaceConfig.from_dict( + { + "server_ip": "10.0.0.5", + "bodies": [ + {"rigid_body_name": "Drone", "target_prim": "/World/base_link", "streaming_id": 1}, + ], + } + ) + assert cfg.server_ip == "10.0.0.5" + assert len(cfg.bodies) == 1 + assert cfg.bodies[0] == BodyBinding("Drone", "/World/base_link", 1, -1) + + +def test_from_dict_with_bodies_as_prim_mapping(): + # The "dictionary of prims -> rigid body names and stuff" form. + cfg = NatNetInterfaceConfig.from_dict( + { + "bodies": { + "/World/base_link": {"rigid_body_name": "Drone", "streaming_id": 1}, + "/World/target": {"rigid_body_name": "Target", "streaming_id": 2}, + } + } + ) + by_name = {b.rigid_body_name: b for b in cfg.bodies} + assert by_name["Drone"].target_prim == "/World/base_link" + assert by_name["Target"].target_prim == "/World/target" + assert by_name["Target"].streaming_id == 2 + + +def test_to_dict_round_trip(): + cfg = NatNetInterfaceConfig.from_dict( + { + "mode": "multicast", + "publish_rate": 120, + "bodies": [{"rigid_body_name": "Drone", "target_prim": "/World/base_link"}], + } + ) + restored = NatNetInterfaceConfig.from_dict(cfg.to_dict()) + assert restored == cfg + + +def test_body_from_dict_requires_target_and_name(): + with pytest.raises(ValueError): + BodyBinding.from_dict({"rigid_body_name": "Drone"}) # no target_prim + with pytest.raises(ValueError): + BodyBinding.from_dict({"target_prim": "/World/base_link"}) # no name + + +def test_body_attr_name_builder(): + assert body_attr_name("Drone", "streamingId") == "natnet:body:Drone:streamingId" + + +def test_make_instance_key_sanitizes_and_dedupes(): + used: set[str] = set() + assert make_instance_key("Drone 1", used) == "Drone_1" + # collision after sanitization -> numeric suffix + assert make_instance_key("Drone-1", used) == "Drone_1_1" + # leading digit gets a safe prefix + assert make_instance_key("3PO", used).startswith("b_") + + +def test_assign_instance_keys_are_unique(): + cfg = NatNetInterfaceConfig( + bodies=[ + BodyBinding("Drone", "/World/a", 1), + BodyBinding("Drone", "/World/b", 2), # duplicate display name + ] + ) + keys = [k for k, _ in cfg.assign_instance_keys()] + assert len(set(keys)) == 2 + + +@pytest.mark.parametrize( + "overrides", + [ + {"mode": "bogus"}, + {"command_port": 0}, + {"data_port": 70000}, + {"command_port": 1510, "data_port": 1510}, + {"publish_rate": 0}, + {"up_axis": "X"}, + {"up_axis": "bogus"}, + {"pose_noise_std_meters": -0.001}, + {"pose_noise_rotation_deg": -0.1}, + ], +) +def test_validate_rejects_bad_server_config(overrides): + cfg = NatNetInterfaceConfig(**overrides) + with pytest.raises(ValueError): + cfg.validate() + + +def test_validate_rejects_duplicate_streaming_ids(): + cfg = NatNetInterfaceConfig( + bodies=[ + BodyBinding("A", "/World/a", 1), + BodyBinding("B", "/World/b", 1), + ] + ) + with pytest.raises(ValueError): + cfg.validate() + + +def test_validate_rejects_blank_rigid_body_name(): + cfg = NatNetInterfaceConfig(bodies=[BodyBinding("", "/World/a", 1)]) + with pytest.raises(ValueError): + cfg.validate() + + +def test_validate_allows_empty_target(): + # An empty target is valid: a freshly added body to be pointed in the UI/Property panel. + cfg = NatNetInterfaceConfig(bodies=[BodyBinding("Drone", "", 1)]) + assert cfg.validate() is cfg + + +def test_validate_accepts_good_config(): + cfg = NatNetInterfaceConfig( + bodies=[BodyBinding("Drone", "/World/base_link", 1)] + ) + assert cfg.validate() is cfg diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_sampling.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_sampling.py new file mode 100644 index 000000000..25db5ad96 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_sampling.py @@ -0,0 +1,238 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Pose sampling and catalog resync against an in-memory USD stage (fake server, no sockets).""" + +from __future__ import annotations + +import math + +import pytest + +pytest.importorskip("pxr") + +from pxr import Gf, Usd, UsdGeom # noqa: E402 + +from optitrack.natnet.emulator.isaac.config import BodyBinding, NatNetInterfaceConfig # noqa: E402 +from optitrack.natnet.emulator.isaac.frames import MODEL_LIST_CHANGED, TRACKING_VALID # noqa: E402 +from optitrack.natnet.emulator.isaac.manager import NatNetServerManager # noqa: E402 +from optitrack.natnet.emulator.isaac.usd_bindings import author_interface, read_world_pose # noqa: E402 + +pytestmark = pytest.mark.unit + + +class FakeServer: + def __init__(self): + self.frames = [] + self.payloads = [] + + def set_model_def_payload(self, payload): + self.payloads.append(payload) + + def start(self): + pass + + def shutdown(self): + pass + + def enqueue_mocap_data(self, frame): + self.frames.append(frame) + + +def _xform(stage, path, translate=(0.0, 0.0, 0.0)): + xform = UsdGeom.Xform.Define(stage, path) + xform.AddTranslateOp().Set(Gf.Vec3d(*translate)) + return xform + + +def _manager_with_fake(): + fake = FakeServer() + mgr = NatNetServerManager(server_factory=lambda cfg: fake) + return mgr, fake + + +# --- read_world_pose ------------------------------------------------------------- + + +def test_read_world_pose_returns_translation(): + stage = Usd.Stage.CreateInMemory() + _xform(stage, "/World/base_link", translate=(1.0, 2.0, 3.0)) + pose = read_world_pose(stage.GetPrimAtPath("/World/base_link")) + assert pose is not None + (x, y, z), (qx, qy, qz, qw) = pose + assert (round(x, 3), round(y, 3), round(z, 3)) == (1.0, 2.0, 3.0) + assert qw == pytest.approx(1.0) + + +def test_read_world_pose_invalid_prim_is_none(): + stage = Usd.Stage.CreateInMemory() + assert read_world_pose(stage.GetPrimAtPath("/World/nope")) is None + + +# --- sample_once ----------------------------------------------------------------- + + +def test_sample_once_no_bodies(): + stage = Usd.Stage.CreateInMemory() + author_interface(stage, "/World/NatNetInterface", NatNetInterfaceConfig()) + mgr, fake = _manager_with_fake() + mgr.start_server(NatNetInterfaceConfig(server_ip="127.0.0.1")) + frame = mgr.sample_once(stage) + assert frame is not None and frame.nRigidBodies == 0 + + +def test_sample_once_streams_world_pose(): + stage = Usd.Stage.CreateInMemory() + _xform(stage, "/World/base_link", translate=(4.0, 5.0, 6.0)) + cfg = NatNetInterfaceConfig( + server_ip="127.0.0.1", + pose_noise_enabled=False, + bodies=[BodyBinding("Drone", "/World/base_link", 1)], + ) + author_interface(stage, "/World/NatNetInterface", cfg) + mgr, fake = _manager_with_fake() + mgr.start_server(cfg) + + frame = mgr.sample_once(stage) + assert frame.nRigidBodies == 1 + rb = frame.RigidBodies[0] + assert rb.ID == 1 + assert (round(rb.x, 3), round(rb.y, 3), round(rb.z, 3)) == (4.0, 5.0, 6.0) + assert rb.params & TRACKING_VALID + # First frame after start resyncs -> client should be told the model list changed. + assert frame.params & MODEL_LIST_CHANGED + + +def test_sample_once_missing_prim_is_lost(): + stage = Usd.Stage.CreateInMemory() + cfg = NatNetInterfaceConfig( + server_ip="127.0.0.1", bodies=[BodyBinding("Ghost", "/World/missing", 9)] + ) + author_interface(stage, "/World/NatNetInterface", cfg) + mgr, fake = _manager_with_fake() + mgr.start_server(cfg) + + frame = mgr.sample_once(stage) + rb = frame.RigidBodies[0] + assert rb.ID == 9 + assert rb.params & TRACKING_VALID == 0 + assert math.isnan(rb.x) + + +def test_moving_prim_updates_streamed_position(): + stage = Usd.Stage.CreateInMemory() + xform = _xform(stage, "/World/base_link", translate=(0.0, 0.0, 0.0)) + cfg = NatNetInterfaceConfig( + server_ip="127.0.0.1", + pose_noise_enabled=False, + bodies=[BodyBinding("Drone", "/World/base_link", 1)], + ) + author_interface(stage, "/World/NatNetInterface", cfg) + mgr, fake = _manager_with_fake() + mgr.start_server(cfg) + + mgr.sample_once(stage) + xform.GetOrderedXformOps()[0].Set(Gf.Vec3d(10.0, 0.0, 0.0)) + frame = mgr.sample_once(stage) + assert round(frame.RigidBodies[0].x, 3) == 10.0 + + +def test_up_axis_z_streams_isaac_pose_as_is(): + stage = Usd.Stage.CreateInMemory() + _xform(stage, "/World/base_link", translate=(1.0, 2.0, 3.0)) + cfg = NatNetInterfaceConfig( + server_ip="127.0.0.1", + up_axis="Z", + pose_noise_enabled=False, + bodies=[BodyBinding("Drone", "/World/base_link", 1)], + ) + author_interface(stage, "/World/NatNetInterface", cfg) + mgr, _fake = _manager_with_fake() + mgr.start_server(cfg) + + rb = mgr.sample_once(stage).RigidBodies[0] + assert (round(rb.x, 3), round(rb.y, 3), round(rb.z, 3)) == (1.0, 2.0, 3.0) + + +def test_up_axis_y_reaxes_streamed_pose(): + # Y-up Motive emulation: Isaac (x, y, z) streams as (x, z, -y). + stage = Usd.Stage.CreateInMemory() + _xform(stage, "/World/base_link", translate=(1.0, 2.0, 3.0)) + cfg = NatNetInterfaceConfig( + server_ip="127.0.0.1", + up_axis="Y", + pose_noise_enabled=False, + bodies=[BodyBinding("Drone", "/World/base_link", 1)], + ) + author_interface(stage, "/World/NatNetInterface", cfg) + mgr, _fake = _manager_with_fake() + mgr.start_server(cfg) + + rb = mgr.sample_once(stage).RigidBodies[0] + assert (round(rb.x, 3), round(rb.y, 3), round(rb.z, 3)) == (1.0, 3.0, -2.0) + + +def test_body_added_while_live_is_picked_up_on_resync(): + stage = Usd.Stage.CreateInMemory() + _xform(stage, "/World/a", translate=(1.0, 0.0, 0.0)) + _xform(stage, "/World/b", translate=(0.0, 2.0, 0.0)) + cfg1 = NatNetInterfaceConfig( + server_ip="127.0.0.1", bodies=[BodyBinding("A", "/World/a", 1)] + ) + author_interface(stage, "/World/NatNetInterface", cfg1) + mgr, fake = _manager_with_fake() + mgr.start_server(cfg1) + + first = mgr.sample_once(stage) + assert first.nRigidBodies == 1 + + # Add a second body live: re-author the prim, then mark dirty (the UI/USD-notice + # path calls mark_dirty for us in Kit). + cfg2 = NatNetInterfaceConfig( + server_ip="127.0.0.1", + bodies=[BodyBinding("A", "/World/a", 1), BodyBinding("B", "/World/b", 2)], + ) + author_interface(stage, "/World/NatNetInterface", cfg2) + mgr.mark_dirty() + + second = mgr.sample_once(stage) + assert second.nRigidBodies == 2 + assert second.params & MODEL_LIST_CHANGED # catalog grew -> tell the client + ids = {second.RigidBodies[i].ID for i in range(second.nRigidBodies)} + assert ids == {1, 2} + # MODELDEF payload was refreshed on the server for the new catalog. + assert len(fake.payloads) >= 2 + + +def test_target_prim_created_after_start_becomes_valid(): + """A body whose target prim is spawned *after* the server starts (e.g. a Pegasus + drone base_link created on the first Play tick) must start streaming a valid pose + as soon as the prim appears — no mark_dirty/resync required, because the target + path is re-resolved every sample.""" + stage = Usd.Stage.CreateInMemory() + cfg = NatNetInterfaceConfig( + server_ip="127.0.0.1", + pose_noise_enabled=False, + bodies=[BodyBinding("Drone", "/World/drone1/base_link", 1)], + ) + author_interface(stage, "/World/NatNetInterface", cfg) + mgr, _fake = _manager_with_fake() + mgr.start_server(cfg) + + # Prim does not exist yet -> lost. + first = mgr.sample_once(stage) + assert first.RigidBodies[0].params & TRACKING_VALID == 0 + + # Spawn the target prim later (simulating the Play-tick drone creation). + _xform(stage, "/World/drone1/base_link", translate=(7.0, 8.0, 9.0)) + + # Next sample re-resolves the path -> valid pose, with no mark_dirty(). + second = mgr.sample_once(stage) + rb = second.RigidBodies[0] + assert rb.params & TRACKING_VALID + assert (round(rb.x, 3), round(rb.y, 3), round(rb.z, 3)) == (7.0, 8.0, 9.0) + + +def test_sample_once_noop_without_server(): + stage = Usd.Stage.CreateInMemory() + mgr, _fake = _manager_with_fake() + assert mgr.sample_once(stage) is None diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_streaming.py b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_streaming.py new file mode 100644 index 000000000..bc526c5b5 --- /dev/null +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_pose_streaming.py @@ -0,0 +1,86 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Loopback: sample_once on a USD prim → NAT_FRAMEOFDATA with sampled position. + +Real server + UDP sockets + in-memory stage. Requires pxr. +""" + +from __future__ import annotations + +import socket +import struct +import time + +import pytest + +pytest.importorskip("pxr") + +from pxr import Gf, Usd, UsdGeom # noqa: E402 + +from natnet_test_helpers import NatNetTestClient, ephemeral_udp_port # noqa: E402 + +from optitrack.natnet.emulator.isaac.config import BodyBinding, NatNetInterfaceConfig # noqa: E402 +from optitrack.natnet.emulator.isaac.manager import NatNetServerManager # noqa: E402 +from optitrack.natnet.emulator.isaac.usd_bindings import author_interface # noqa: E402 +from optitrack.natnet.emulator.server import natnet_server_types as st # noqa: E402 + +pytestmark = pytest.mark.unit + + +def _decode_first_rigid_body(payload: bytes): + # iFrame(4) + markersets(count4+size4) + othermarkers(count4+size4) = 20, then + # rigid bodies: count(4) + size(4) at 20, first body at 28: id + xyz. + rb_count, _rb_size = struct.unpack_from("1``: ``Drone1``, ``Drone2``, … (ids 1..N) + +Intended multi-robot profile pairing (see ``natnet_config.yaml`` commented scaffolding): + - ``robot_1``: tracks its drone + the shared ``Target`` + - ``robot_2``: tracks its drone + the shared ``Target`` + - ``robot_3``: tracks its drone only (no Target in profile) + +Set ``NUM_ROBOTS=3`` on both sim and robot stacks; each container picks its profile +via ``ROBOT_NAME``. + +Env: + - ``NUM_ROBOTS`` (default 1) + - ``ENABLE_LIDAR`` (default false) + - ``PLAY_SIM_ON_START`` (default true) +""" + +import asyncio +import os +import sys +import time + +import carb +from isaacsim import SimulationApp + +_headless = os.environ.get("ISAAC_SIM_HEADLESS", "false").lower() == "true" +simulation_app = SimulationApp({"headless": _headless}) + +import omni.kit.app +import omni.timeline +import omni.usd + +from omni.isaac.core.world import World + +from pegasus.simulator.params import SIMULATION_ENVIRONMENTS +from pegasus.simulator.logic.interface.pegasus_interface import PegasusInterface +from pegasus.simulator.ogn.api.spawn_multirotor import spawn_px4_multirotor_node +from pegasus.simulator.ogn.api.spawn_zed_camera import add_zed_stereo_camera_subgraph +from pegasus.simulator.ogn.api.spawn_rtx_lidar import add_rtx_lidar_subgraph + +sys.path.insert(0, os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "utils"))) +from scene_prep import scale_stage_prim, add_colliders, add_dome_light, save_scene_as_contained_usd + +# Register the emulator extension with Kit before importing from it. +# See docs/simulation/isaac_sim/natnet_emulator.md. +from isaacsim.core.utils.extensions import enable_extension # noqa: E402 + +enable_extension("optitrack.natnet.emulator") + +from optitrack.natnet.emulator.isaac import ( # noqa: E402 + DEFAULT_TARGET_PATH, + DEFAULT_TARGET_POSITION, + DEFAULT_TARGET_STREAMING_ID, + author_static_target, + author_drone_natnet_interface, +) + +# --------------------- CONFIGURATION --------------------- +ENV_URL = SIMULATION_ENVIRONMENTS["Default Environment"] +STAGE_SCALE = 1.0 +SAVE_SCENE_TO = None +DRONE_USD = "~/.local/share/ov/data/documents/Kit/shared/exts/pegasus.simulator/pegasus/simulator/assets/Robots/Iris/iris.usd" + +NUM_ROBOTS = int(os.environ.get("NUM_ROBOTS", "1")) +ENABLE_LIDAR = os.environ.get("ENABLE_LIDAR", "false").lower() == "true" +# Base name for the streamed drone bodies; drone i is streamed with id i. Must match +# the body entries in the per-robot profiles in natnet_config.yaml. +# See docs/simulation/isaac_sim/natnet_emulator.md. +NATNET_BODY_NAME = "Drone" +NATNET_TARGET_NAME = "Target" + +_NATNET_SERVER_KWARGS = { + "pose_noise_enabled": True, + "pose_noise_std_meters": 0.0005, + "pose_noise_rotation_deg": 0.05, +} +# --------------------------------------------------------- + + +ext_manager = omni.kit.app.get_app().get_extension_manager() +for ext in [ + "omni.graph.core", + "omni.graph.action", + "omni.graph.action_nodes", + "isaacsim.core.nodes", + "omni.graph.ui", + "omni.graph.visualization.nodes", + "omni.graph.scriptnode", + "omni.graph.window.action", + "omni.graph.window.generic", + "omni.graph.ui_nodes", + "pegasus.simulator", +]: + if not ext_manager.is_extension_enabled(ext): + ext_manager.set_extension_enabled_immediate(ext, True) + + +def wait_for_stage(stage, timeout_s: float = 10.0): + for _ in range(int(timeout_s / 0.1)): + omni.kit.app.get_app().update() + world_prim = stage.GetPrimAtPath("/World") + if world_prim.IsValid(): + non_physics = [c for c in world_prim.GetChildren() if c.GetName() != "PhysicsScene"] + if non_physics: + return True + time.sleep(0.1) + return False + + +def _drone_body_name(index: int) -> str: + """Single agent uses bare ``Drone``; multi uses ``Drone1``, ``Drone2``, …""" + return NATNET_BODY_NAME if NUM_ROBOTS == 1 else f"{NATNET_BODY_NAME}{index}" + + +def spawn_drone(index: int): + robot_name = f"robot_{index}" + drone_prim = f"/World/drone{index}/base_link" + init_x = 2.0 * (index - 1) - 2.0 * (NUM_ROBOTS - 1) / 2.0 + + graph_handle = spawn_px4_multirotor_node( + pegasus_node_name=f"PX4Multirotor_{index}", + drone_prim=drone_prim, + robot_name=robot_name, + vehicle_id=index, + domain_id=index, + usd_file=DRONE_USD, + init_pos=[init_x, 0.0, 0.07], + init_orient=[0.0, 0.0, 0.0, 1.0], + ) + + add_zed_stereo_camera_subgraph( + parent_graph_handle=graph_handle, + drone_prim=drone_prim, + robot_name=robot_name, + camera_name="ZEDCamera", + camera_offset=[0.2, 0.0, -0.05], + camera_rotation_offset=[0.0, 0.0, 0.0], + ) + + if ENABLE_LIDAR: + add_rtx_lidar_subgraph( + parent_graph_handle=graph_handle, + drone_prim=drone_prim, + robot_name=robot_name, + lidar_config="ouster_os1", + lidar_topic_name="point_cloud_raw", + lidar_offset=[0.0, 0.0, 0.025], + lidar_rotation_offset=[0.0, 0.0, 0.0], + min_range=0.75, + ) + + +class PegasusApp: + + def __init__(self): + self.timeline = omni.timeline.get_timeline_interface() + self.pg = PegasusInterface() + self.pg._world = World(**self.pg._world_settings) + self.world = self.pg.world + self.timeline.stop() + + self.pg.load_environment(ENV_URL) + + stage = omni.usd.get_context().get_stage() + if stage is None: + raise RuntimeError("Stage failed to load") + + if not wait_for_stage(stage): + carb.log_warn("Stage load timed out — continuing anyway.") + + stage_prim = stage.GetPrimAtPath("/World/stage") + if stage_prim.IsValid(): + scale_stage_prim(stage, "/World/stage", STAGE_SCALE) + add_colliders(stage_prim) + for _ in range(10): + omni.kit.app.get_app().update() + else: + carb.log_warn("/World/stage not found — skipping scale and collision.") + + add_dome_light(stage) + + if SAVE_SCENE_TO: + import tempfile + tmp_usd = os.path.join(tempfile.gettempdir(), "prepared_scene.usd") + success, error = asyncio.get_event_loop().run_until_complete( + omni.usd.get_context().export_as_stage_async(tmp_usd) + ) + if success: + os.makedirs(SAVE_SCENE_TO, exist_ok=True) + save_scene_as_contained_usd(tmp_usd, SAVE_SCENE_TO) + os.remove(tmp_usd) + else: + carb.log_error(f"Scene export failed: {error}") + + print(f"[example_multi_natnet] Spawning {NUM_ROBOTS} drone(s), lidar={'on' if ENABLE_LIDAR else 'off'}") + for i in range(1, NUM_ROBOTS + 1): + spawn_drone(i) + + self._setup_natnet(stage) + self.play_on_start = os.environ.get("PLAY_SIM_ON_START", "true").lower() == "true" + + def _setup_natnet(self, stage): + """Author NatNet bodies: one per drone plus one shared static target. + + Runs before the timeline starts; the emulator extension builds the server + from this prim on Play. + """ + try: + author_static_target(stage, DEFAULT_TARGET_PATH, DEFAULT_TARGET_POSITION) + bodies = [ + (_drone_body_name(i), i, f"/World/drone{i}/base_link/body") + for i in range(1, NUM_ROBOTS + 1) + ] + bodies.append((NATNET_TARGET_NAME, DEFAULT_TARGET_STREAMING_ID, DEFAULT_TARGET_PATH)) + + author_drone_natnet_interface(stage, bodies, **_NATNET_SERVER_KWARGS) + carb.log_warn( + f"[natnet] Interface authored with {NUM_ROBOTS} drone body(ies) " + f"and shared target '{NATNET_TARGET_NAME}' (robot_1/robot_2 subscribe via " + f"natnet_config; robot_3 omits Target)." + ) + except Exception as exc: # noqa: BLE001 - never let NatNet kill the sim + carb.log_error(f"[natnet] Failed to author interface: {exc}") + + def run(self): + if self.play_on_start: + self.timeline.play() + else: + self.timeline.stop() + + app = omni.kit.app.get_app() + while simulation_app.is_running(): + world = World.instance() + if world is not None and hasattr(world, '_scene'): + world.step(render=True) + if world is not self.world: + self.world = world + self.pg._world = world + else: + app.update() + + carb.log_warn("Closing simulation.") + self.timeline.stop() + simulation_app.close() + + +def main(): + PegasusApp().run() + + +if __name__ == "__main__": + main() diff --git a/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_natnet_launch_script.py b/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_natnet_launch_script.py new file mode 100644 index 000000000..8c0dde2c7 --- /dev/null +++ b/simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_natnet_launch_script.py @@ -0,0 +1,276 @@ +#!/usr/bin/env python +""" +Single-drone PX4 Pegasus launcher with OptiTrack NatNet mocap streaming. + +Same scene prep and sensor stack as ``example_one_px4_pegasus_launch_script.py``, plus +a Motive-compatible NatNet server that always streams: + + - ``Drone`` (id 1) from the Pegasus ``body`` prim under ``/World/base_link`` + - ``Target`` (id 100) from a static ``/World/target`` prim + +Pair with robot-side ``LAUNCH_NATNET=true`` and a matching ``natnet_config.yaml`` +profile. To consume the target on the robot, add a Target body to the profile +(see the commented scaffolding in ``natnet_config.yaml``). + +Override rigid-body names with ``NATNET_BODY_NAME`` / ``NATNET_TARGET_NAME``. +""" + +import os +import sys +import time +import asyncio + +import carb +from isaacsim import SimulationApp + +_LIVESTREAM = os.environ.get("ISAAC_SIM_LIVESTREAM", "").lower() == "true" + +if _LIVESTREAM: + _SIM_APP_CONFIG = { + "width": 1280, + "height": 720, + "window_width": 1920, + "window_height": 1080, + "headless": True, + "hide_ui": False, + "renderer": "RaytracedLighting", + "display_options": 3286, + } +else: + _SIM_APP_CONFIG = {"headless": False} + +simulation_app = SimulationApp(launch_config=_SIM_APP_CONFIG) + +if _LIVESTREAM: + from isaacsim.core.utils.extensions import enable_extension + simulation_app.set_setting("/app/window/drawMouse", True) + simulation_app.set_setting("/app/livestream/enabled", True) + LIVESTREAM_UDP_PORT = int(os.environ.get("ISAAC_SIM_LIVESTREAM_UDP_PORT", "49099")) + simulation_app.set_setting("/app/livestream/fixedHostPort", LIVESTREAM_UDP_PORT) + simulation_app.set_setting("/app/livestream/minHostPort", LIVESTREAM_UDP_PORT) + simulation_app.set_setting("/app/livestream/maxHostPort", LIVESTREAM_UDP_PORT) + enable_extension("omni.kit.livestream.webrtc") + +import omni.kit.app +import omni.timeline +import omni.usd + +from omni.isaac.core.world import World + +from pegasus.simulator.params import SIMULATION_ENVIRONMENTS +from pegasus.simulator.logic.interface.pegasus_interface import PegasusInterface +from pegasus.simulator.ogn.api.spawn_multirotor import spawn_px4_multirotor_node +from pegasus.simulator.ogn.api.spawn_zed_camera import add_zed_stereo_camera_subgraph +from pegasus.simulator.ogn.api.spawn_rtx_lidar import add_rtx_lidar_subgraph + +sys.path.insert(0, os.path.normpath(os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "utils"))) +from scene_prep import scale_stage_prim, add_colliders, add_dome_light, save_scene_as_contained_usd + +# gps_utils lives in this launch_scripts directory. +_LAUNCH_SCRIPTS_DIR = os.path.dirname(os.path.abspath(__file__)) +if _LAUNCH_SCRIPTS_DIR not in sys.path: + sys.path.insert(0, _LAUNCH_SCRIPTS_DIR) +from gps_utils import set_gps_origins, DEFAULT_WORLD_ORIGIN + +# Register the emulator extension with Kit before importing from it. +# See docs/simulation/isaac_sim/natnet_emulator.md. +from isaacsim.core.utils.extensions import enable_extension # noqa: E402 + +enable_extension("optitrack.natnet.emulator") + +from optitrack.natnet.emulator.isaac import ( # noqa: E402 + DEFAULT_TARGET_PATH, + DEFAULT_TARGET_POSITION, + DEFAULT_TARGET_STREAMING_ID, + author_static_target, + author_drone_natnet_interface, +) + +# --------------------- CONFIGURATION --------------------- +ENV_URL = SIMULATION_ENVIRONMENTS["Default Environment"] +STAGE_SCALE = 1.0 +SAVE_SCENE_TO = None +DRONE_USD = "~/.local/share/ov/data/documents/Kit/shared/exts/pegasus.simulator/pegasus/simulator/assets/Robots/Iris/iris.usd" + +# What world (0, 0, 0) maps to in GPS coordinates. Must match the GCS origin and +# the robot's natnet_ros2 mavros_gp_origin.yaml. +WORLD_GPS_ORIGIN = DEFAULT_WORLD_ORIGIN + +# Single drone spawned at the world origin. domain_id / spawn must match the +# spawn_px4_multirotor_node call below. +DRONE_CONFIGS = [ + {"domain_id": 1, "x_m": 0.0, "y_m": 0.0, "z_m": 0.07}, +] + +# Rigid body this scene streams. Must match a body entry in the robot's profile in +# natnet_ros2/config/natnet_config.yaml — a mismatch fails silently. +# See docs/simulation/isaac_sim/natnet_emulator.md. +NATNET_BODY_NAME = "Drone" +NATNET_BODY_ID = 1 +NATNET_TARGET_NAME = "Target" + +_NATNET_SERVER_KWARGS = { + "pose_noise_enabled": True, + "pose_noise_std_meters": 0.0005, + "pose_noise_rotation_deg": 0.05, +} +# --------------------------------------------------------- + + +ext_manager = omni.kit.app.get_app().get_extension_manager() +for ext in [ + "omni.graph.core", + "omni.graph.action", + "omni.graph.action_nodes", + "isaacsim.core.nodes", + "omni.graph.ui", + "omni.graph.visualization.nodes", + "omni.graph.scriptnode", + "omni.graph.window.action", + "omni.graph.window.generic", + "omni.graph.ui_nodes", + "pegasus.simulator", +]: + if not ext_manager.is_extension_enabled(ext): + ext_manager.set_extension_enabled_immediate(ext, True) + + +def wait_for_stage(stage, timeout_s: float = 10.0): + for _ in range(int(timeout_s / 0.1)): + omni.kit.app.get_app().update() + world_prim = stage.GetPrimAtPath("/World") + if world_prim.IsValid(): + non_physics = [c for c in world_prim.GetChildren() if c.GetName() != "PhysicsScene"] + if non_physics: + return True + time.sleep(0.1) + return False + + +class PegasusApp: + + def __init__(self): + # Must run before the PX4 SITL subprocess starts. + set_gps_origins(DRONE_CONFIGS, world_origin=WORLD_GPS_ORIGIN) + + self.timeline = omni.timeline.get_timeline_interface() + self.pg = PegasusInterface() + self.pg._world = World(**self.pg._world_settings) + self.world = self.pg.world + self.timeline.stop() + + self.pg.load_environment(ENV_URL) + + stage = omni.usd.get_context().get_stage() + if stage is None: + raise RuntimeError("Stage failed to load") + + if not wait_for_stage(stage): + carb.log_warn("Stage load timed out — continuing anyway.") + + stage_prim = stage.GetPrimAtPath("/World/stage") + if stage_prim.IsValid(): + scale_stage_prim(stage, "/World/stage", STAGE_SCALE) + add_colliders(stage_prim) + for _ in range(10): + omni.kit.app.get_app().update() + else: + carb.log_warn("/World/stage not found — skipping scale and collision.") + + add_dome_light(stage) + + if SAVE_SCENE_TO: + import tempfile + tmp_usd = os.path.join(tempfile.gettempdir(), "prepared_scene.usd") + success, error = asyncio.get_event_loop().run_until_complete( + omni.usd.get_context().export_as_stage_async(tmp_usd) + ) + if success: + os.makedirs(SAVE_SCENE_TO, exist_ok=True) + save_scene_as_contained_usd(tmp_usd, SAVE_SCENE_TO) + os.remove(tmp_usd) + else: + carb.log_error(f"Scene export failed: {error}") + + graph_handle = spawn_px4_multirotor_node( + pegasus_node_name="PX4Multirotor", + drone_prim="/World/base_link", + robot_name="robot_1", + vehicle_id=1, + domain_id=1, + usd_file=DRONE_USD, + init_pos=[0.0, 0.0, 0.07], + init_orient=[0.0, 0.0, 0.0, 1.0], + ) + + add_zed_stereo_camera_subgraph( + parent_graph_handle=graph_handle, + drone_prim="/World/base_link", + robot_name="robot_1", + camera_name="ZEDCamera", + camera_offset=[0.2, 0.0, -0.05], + camera_rotation_offset=[0.0, 0.0, 0.0], + ) + + add_rtx_lidar_subgraph( + parent_graph_handle=graph_handle, + drone_prim="/World/base_link", + robot_name="robot_1", + lidar_config="ouster_os1", + lidar_topic_name="point_cloud_raw", + lidar_offset=[0.0, 0.0, 0.025], + lidar_rotation_offset=[0.0, 0.0, 0.0], + min_range=0.75, + ) + + self._setup_natnet(stage) + self.play_on_start = os.environ.get("PLAY_SIM_ON_START", "true").lower() == "true" + + def _setup_natnet(self, stage): + """Author the NatNet interface prim (drone + static target). + + Runs before the timeline starts; the emulator extension builds the server + from this prim on Play. + """ + try: + author_static_target(stage, DEFAULT_TARGET_PATH, DEFAULT_TARGET_POSITION) + bodies = [ + (NATNET_BODY_NAME, NATNET_BODY_ID, "/World/base_link/body"), + (NATNET_TARGET_NAME, DEFAULT_TARGET_STREAMING_ID, DEFAULT_TARGET_PATH), + ] + author_drone_natnet_interface(stage, bodies, **_NATNET_SERVER_KWARGS) + carb.log_warn( + f"[natnet] Interface authored: '{NATNET_BODY_NAME}' (-> /World/base_link/body), " + f"'{NATNET_TARGET_NAME}' (-> {DEFAULT_TARGET_PATH})." + ) + except Exception as exc: # noqa: BLE001 - never let NatNet kill the sim + carb.log_error(f"[natnet] Failed to author interface: {exc}") + + def run(self): + if self.play_on_start: + self.timeline.play() + else: + self.timeline.stop() + + app = omni.kit.app.get_app() + while simulation_app.is_running(): + world = World.instance() + if world is not None and hasattr(world, '_scene'): + world.step(render=True) + if world is not self.world: + self.world = world + self.pg._world = world + else: + app.update() + + carb.log_warn("Closing simulation.") + self.timeline.stop() + simulation_app.close() + + +def main(): + PegasusApp().run() + + +if __name__ == "__main__": + main() diff --git a/tests/harness/collection.py b/tests/harness/collection.py index aa20c9f1e..ae78ed94c 100644 --- a/tests/harness/collection.py +++ b/tests/harness/collection.py @@ -24,6 +24,7 @@ "system.test_sensors", "system.test_takeoff_hover_land", "system.test_fixed_trajectory", + "system.test_optitrack_e2e", ] # Within test_takeoff_hover_land, each (env, velocity) runs phases in this chain order. diff --git a/tests/integration/natnet/README.md b/tests/integration/natnet/README.md index 4e9330c32..db4617312 100644 --- a/tests/integration/natnet/README.md +++ b/tests/integration/natnet/README.md @@ -103,12 +103,12 @@ The matching **system** check is - Asserts `/{robot_n}/{natnet pose topic}/pose_cov` ≥ 5 Hz per robot (the drone body's configured topic — default `perception/optitrack/drone`). - Override the checked topic with `NATNET_POSE_TOPIC` (default - `perception/optitrack/drone`). The sim body name (`NATNET_BODY_NAME`, default - `Drone`) is decoupled from the published topic, which the robot profile sets. + `perception/optitrack/drone`). The body name and the published topic are decoupled; + both are set in the robot's `natnet_config.yaml` profile. Sim auto-start: set `ISAAC_SIM_SCRIPT_NAME` to a NatNet launch script and `LAUNCH_NATNET=true` on the robot. Convenience bundle: -`airstack up --env-file overrides/isaac-natnet-vision.env` (NatNet script + +`airstack up --env-file overrides/isaac-optitrack-simulation.env` (NatNet script + PX4 external-vision SITL profile). ## libNatNet 4.4 unicast — verified wire contract diff --git a/tests/integration/natnet/test_natnet_integration.py b/tests/integration/natnet/test_natnet_integration.py index e386fd1ee..43fa4a7f3 100644 --- a/tests/integration/natnet/test_natnet_integration.py +++ b/tests/integration/natnet/test_natnet_integration.py @@ -234,6 +234,80 @@ def test_natnet_ros2_receives_drone_pose_hz(robot_autonomy_stack): server.shutdown() +def test_natnet_ros2_receives_isaac_wrapper_pose_hz(robot_autonomy_stack): + """Isaac-wrapper path: NatNetServerManager.sample_once on a moving USD prim. + + Tests that the wrapper feeds the real robot client end-to-end. Pose-value fidelity + is covered by test_pose_streaming.py loopback. + """ + pytest.importorskip("pxr") + import math + + from pxr import Gf, Usd, UsdGeom + + from optitrack.natnet.emulator.isaac import ( + BodyBinding, + NatNetInterfaceConfig, + NatNetServerManager, + author_interface, + ) + + container = robot_autonomy_stack["container"] + if not _natnet_node_available(container): + pytest.skip("natnet_ros2_node not built — run airstack setup (NatNet SDK)") + + _stop_stale_natnet_nodes(container) + + host_ip = _docker_default_gateway(container) + command_port = ephemeral_udp_port(host_ip) + data_port = ephemeral_udp_port(host_ip) + while data_port == command_port: + data_port = ephemeral_udp_port(host_ip) + robot_name = _container_env(container, "ROBOT_NAME", "robot_1") + domain_id = int(_container_env(container, "ROS_DOMAIN_ID", "0")) + + stage = Usd.Stage.CreateInMemory() + xform = UsdGeom.Xform.Define(stage, "/World/base_link") + translate_op = xform.AddTranslateOp() + translate_op.Set(Gf.Vec3d(0.0, 0.0, 1.0)) + cfg = NatNetInterfaceConfig( + server_ip=host_ip, + command_port=command_port, + data_port=data_port, + publish_rate=50.0, + bodies=[BodyBinding("Drone", "/World/base_link", streaming_id=1)], + ) + author_interface(stage, "/World/NatNetInterface", cfg) + + manager = NatNetServerManager(server_factory=None) # real server factory + stop_event = threading.Event() + + def _sampler(): + # Stand in for the in-sim physics-step callback: move the prim and sample. + interval = 1.0 / cfg.publish_rate + t = 0.0 + while not stop_event.is_set(): + translate_op.Set(Gf.Vec3d(math.sin(t), 0.0, 1.0)) + manager.sample_once(stage) + t += interval + time.sleep(interval) + + sampler = threading.Thread(target=_sampler, daemon=True) + + node_proc: subprocess.Popen[str] | None = None + try: + assert manager.start_server(cfg) is True + sampler.start() + time.sleep(0.1) + node_proc = _launch_natnet_node(container, host_ip, command_port, domain_id) + _assert_pose_stream(container, robot_name, domain_id) + finally: + stop_event.set() + sampler.join(timeout=2.0) + _terminate(node_proc) + manager.stop_server() + + def test_natnet_ros2_multi_body_drone_and_target(robot_autonomy_stack): """Multi-body profile: one robot tracks a drone + a static target. diff --git a/tests/pytest.ini b/tests/pytest.ini index 6f4547c0c..69538af34 100644 --- a/tests/pytest.ini +++ b/tests/pytest.ini @@ -9,6 +9,7 @@ markers = takeoff_hover_land: End-to-end takeoff / hover / land action tests autonomy: Fixed-pattern trajectory path-tracker benchmark (test_fixed_trajectory.py) waypoint_flight: Ordered-waypoint navigation judged on the odometry track (test_waypoint_flight.py) + optitrack: OptiTrack NatNet end-to-end (sim emulator → natnet_ros2 → PX4 EV fusion) testpaths = . addopts = -v --durations=0 --import-mode=importlib cache_dir = /tmp/.pytest_cache diff --git a/tests/requirements.txt b/tests/requirements.txt index a4b43e6bb..757d1ebdb 100644 --- a/tests/requirements.txt +++ b/tests/requirements.txt @@ -6,3 +6,5 @@ tabulate psutil pandas numpy +scipy +usd-core diff --git a/tests/system/test_optitrack_e2e.py b/tests/system/test_optitrack_e2e.py new file mode 100644 index 000000000..18530c7da --- /dev/null +++ b/tests/system/test_optitrack_e2e.py @@ -0,0 +1,299 @@ +"""OptiTrack NatNet end-to-end (sim). + +A single dedicated bring-up that exercises the whole OptiTrack path in Isaac Sim: +the in-sim NatNet **emulator** streams rigid-body poses → ``natnet_ros2`` publishes +the drone pose → the ``vision_pose`` bridge feeds MAVROS → PX4 EKF2 fuses it. + +This brings the NatNet stack up **once** and asserts only one NatNet-specific test. +The cheap, GPU-free half of this (host emulator → ``natnet_ros2`` Hz) lives in ``tests/integration/natnet/``. + +Mark: ``optitrack``. Needs Docker + GPU + Isaac Sim license; skips cleanly when the +isaac-sim image isn't built locally. +""" +import os +import re +import time + +import pytest + +from conftest import ( # noqa: E402 — pytest adds tests/ to sys.path + airstack_cmd, + container_running, + find_container, + get_metrics, + get_robot_containers, + logger, + missing_images, + read_log_tail, + ros2_exec, + sample_hz, + wait_for_container, + wait_for_first_message, +) +from system.test_fixed_trajectory import ( + TARGET_ALTITUDE_M, + _landing_one_robot, + _run_parallel, + _takeoff_one_robot, + _trajectory_one_robot, +) + +pytestmark = pytest.mark.optitrack + +# Single-drone NatNet Isaac stack: the natnet Pegasus script spawns the emulator +# alongside PX4, and LAUNCH_NATNET=true brings up natnet_ros2 + the vision_pose / +# gp_origin / param bridges on the robot. +# +# PX4_PARAM_SET selects simulation/isaac-sim/docker/px4-params/external-vision.env, which +# switches PX4 SITL's EKF2 to mocap external vision and turns GPS, baro and range aiding +# OFF, so the OptiTrack stream is the vehicle's ONLY position source. PX4's rcS applies +# those PX4_PARAM_* entries at boot; they mirror the deployment-validated set in +# robot/ros_ws/src/perception/natnet_ros2/config/px4_params.yaml. +# +# Without it EKF2_EV_CTRL is 0, PX4 silently discards the vision and flies on sim GPS — +# which is what made the previous version of this module's fusion check vacuous. +_E2E_ENV = { + "NUM_ROBOTS": "1", + "COMPOSE_PROFILES": "desktop,isaac-sim", + "AUTOLAUNCH": "true", + "ISAAC_SIM_USE_STANDALONE": "true", + "ISAAC_SIM_SCRIPT_NAME": "example_one_px4_pegasus_natnet_launch_script.py", + "PLAY_SIM_ON_START": "true", + "LAUNCH_NATNET": "true", + # EKF2 external-vision (mocap) configuration — see comment above. Selects + # simulation/isaac-sim/docker/px4-params/external-vision.env, which turns GPS, baro + # and range aiding off, leaving mocap as the vehicle's only position source. + "PX4_PARAM_SET": "external-vision", + # Headless: no X on the CI runner. + "QT_QPA_PLATFORM": "offscreen", +} + +# The trajectory flown to prove fusion. Circle is the enforced PR gate: sustained lateral +# motion is where a wrong EV delay or a too-tight innovation gate actually shows up, which +# a stationary hover would never reveal. +_E2E_TRAJECTORY = "Circle" + +_ROBOT_PATTERN = "robot.*desktop" +_ROBOT_SETUP_BASH = "/root/AirStack/robot/ros_ws/install/setup.bash" +_ROBOT_DOMAIN = 1 +# Drone body's relative pose topic from natnet_config.yaml, namespaced per robot. +_NATNET_POSE_TOPIC = os.environ.get("NATNET_POSE_TOPIC", "perception/optitrack/drone") +_NATNET_MIN_HZ = 5.0 +# PX4 fused local position (proves EKF2 accepted the external vision). odom, not pose: +# it goes live only once EKF2 has converged and home is set, which is what PX4's arming +# preflight requires. See test_px4_ready in test_takeoff_hover_land.py — pose-era signals +# fire ~25s earlier, and arming in that window returns "failed to arm". +_PX4_LOCAL_POSE_TOPIC = "interface/mavros/local_position/odom" +# MAVROS param plugin node; hosts the FCU param table as ROS 2 parameters. Same +# service px4_param_setter reads through. +_EV_PARAM_NODE = "interface/mavros/param" +# One proves vision fusion is on, the other proves GPS aiding is off — together they +# are the precondition every test below assumes but none of them check. +_EV_EXPECTED = {"EKF2_EV_CTRL": 11, "EKF2_GPS_CTRL": 0} +# MAVROS pulls the param table lazily after FCU connect; px4_param_setter budgets +# settle_sec 10 + 30 retries for the same reason. +_EV_PARAM_TIMEOUT = 90 +# `ros2 param get` prints "Integer value is: 11" / "Double value is: 7.0". +_PARAM_VALUE_RE = re.compile(r"value is:\s*(-?[\d.]+)") +# Cold Isaac boot: Pegasus load + Play + emulator UDP connect. +_FIRST_MSG_TIMEOUT = 180 + +# Belt-and-braces behind the odom gate: EV-only convergence is slower and less predictable +# than the GPS case the autonomy suites were tuned against, and TakeoffTask does not retry +# its own ARM (takeoff_landing_task.cpp send_robot_command). +_ARM_ATTEMPTS = 5 +_ARM_RETRY_S = 2.0 +_ARM_SERVICE = "interface/robot_command" +_ARM_COMMAND = 1 # airstack_msgs/srv/RobotCommand.Request.ARM + +# The flight helpers imported from test_fixed_trajectory take an `airstack_env`-style cfg; +# `robot_setup_bash` is the only key any of them reads. +_TRAJ_CFG = {"robot_setup_bash": _ROBOT_SETUP_BASH} + + +def _arm_with_retries(container: str) -> None: + """Arm the vehicle, retrying while PX4's preflight is still rejecting it. + + Uses the same robot_command service TakeoffTask arms through, so a successful + call here leaves TakeoffTask's `is_armed_` set and it skips its own arming. + """ + service = f"/robot_{_ROBOT_DOMAIN}/{_ARM_SERVICE}" + last = "" + started = time.time() + for attempt in range(1, _ARM_ATTEMPTS + 1): + result = ros2_exec( + container, + f'timeout 10 ros2 service call {service} ' + f'airstack_msgs/srv/RobotCommand "{{command: {_ARM_COMMAND}}}"', + domain_id=_ROBOT_DOMAIN, setup_bash=_ROBOT_SETUP_BASH, timeout=20, + ) + last = (result.stdout or "") + (result.stderr or "") + if "success=True" in last.replace(" ", ""): + logger.info("armed on attempt %d/%d", attempt, _ARM_ATTEMPTS) + return + logger.info("arm attempt %d/%d refused (PX4 preflight not ready yet)", + attempt, _ARM_ATTEMPTS) + if attempt < _ARM_ATTEMPTS: + time.sleep(_ARM_RETRY_S) + + pytest.fail( + f"could not arm after {_ARM_ATTEMPTS} attempts over " + f"{time.time() - started:.0f}s — PX4 preflight still rejecting. " + "With GPS/baro/range aiding off, this means EKF2 has not converged on the " + f"vision estimate. Last response:\n{last.strip()[-400:]}" + ) + + +@pytest.fixture(scope="module") +def optitrack_sim_stack(request): + """Bring the NatNet Isaac stack up once for the module; tear it down after. + + Reuses an already-running robot-desktop container (fast local iteration); + otherwise brings the stack up. Skips when the isaac-sim image isn't built. + """ + existing = find_container(_ROBOT_PATTERN) + if existing and container_running(existing): + yield {"container": existing, "brought_up": False} + return + + missing = missing_images(env=_E2E_ENV) + if missing: + pytest.skip("isaac-sim / robot image not built locally: " + ", ".join(missing)) + + airstack_cmd("down", timeout=120, log_name="optitrack_e2e") + result = airstack_cmd("up", env_overrides=_E2E_ENV, timeout=300, log_name="optitrack_e2e") + if result.returncode != 0: + pytest.fail(f"`airstack up` (natnet isaac) failed:\n{read_log_tail('optitrack_e2e')}") + + container = wait_for_container(_ROBOT_PATTERN, timeout=180) + assert container, "robot-desktop container not Running after 180s" + try: + yield {"container": container, "brought_up": True} + finally: + airstack_cmd("down", timeout=120, log_name="optitrack_e2e") + + +def _robot_container(stack): + # robot_1 lives on the first (index-1) replica. + return get_robot_containers(_ROBOT_PATTERN)[0] if not stack["brought_up"] \ + else wait_for_container(_ROBOT_PATTERN, timeout=60) + + +class TestOptitrackE2E: + + @pytest.mark.dependency(name="natnet_pose") + def test_natnet_pose_alive(self, optitrack_sim_stack): + """Emulator → natnet_ros2 → vision_pose: the drone pose_cov streams >= 5 Hz.""" + container = _robot_container(optitrack_sim_stack) + topic = f"/robot_{_ROBOT_DOMAIN}/{_NATNET_POSE_TOPIC}/pose_cov" + + first = wait_for_first_message( + container, topic, domain_id=_ROBOT_DOMAIN, + setup_bash=_ROBOT_SETUP_BASH, timeout=_FIRST_MSG_TIMEOUT, + ) + assert first is not None, ( + f"no NatNet pose on {topic} within {_FIRST_MSG_TIMEOUT}s " + "(emulator → natnet_ros2 path down)" + ) + hz = sample_hz(container, topic, domain_id=_ROBOT_DOMAIN, + setup_bash=_ROBOT_SETUP_BASH, duration=5, window=20) + get_metrics().record("test_optitrack_e2e.natnet_pose_hz", + "natnet_pose_hz", hz if hz is not None else "none", unit="Hz") + assert hz is not None and hz >= _NATNET_MIN_HZ, \ + f"{topic} at {hz} Hz (< {_NATNET_MIN_HZ})" + + @pytest.mark.dependency(name="ev_params", depends=["natnet_pose"]) + def test_ev_params_applied(self, optitrack_sim_stack): + """The external-vision param set actually reached the FCU. + + Everything below assumes PX4_PARAM_SET=external-vision took effect. If it + silently did not, EKF2_EV_CTRL stays 0 and EKF2_GPS_CTRL stays 7, the vehicle + flies the Circle on sim GPS, and every other test here still passes. This reads + the live values back off the FCU, so it covers the whole chain: compose env_file + -> container env -> Pegasus -> PX4 rcS -> FCU. + """ + container = _robot_container(optitrack_sim_stack) + node = f"/robot_{_ROBOT_DOMAIN}/{_EV_PARAM_NODE}" + + unread = dict(_EV_EXPECTED) + actual = {} + deadline = time.time() + _EV_PARAM_TIMEOUT + while unread and time.time() < deadline: + for name in list(unread): + result = ros2_exec( + container, f"ros2 param get {node} {name}", + domain_id=_ROBOT_DOMAIN, setup_bash=_ROBOT_SETUP_BASH, timeout=20, + ) + # An unpulled param prints "Parameter not set." and still exits 0, so + # match on the value line rather than the return code. + match = _PARAM_VALUE_RE.search(result.stdout or "") + if match: + actual[name] = float(match.group(1)) + del unread[name] + if unread: + time.sleep(2.0) + + assert not unread, ( + f"{', '.join(sorted(unread))} never appeared in the MAVROS param table " + f"within {_EV_PARAM_TIMEOUT}s — the MAVROS/FCU link is down, which is a " + "different failure from a wrong parameter." + ) + wrong = {k: v for k, v in sorted(actual.items()) if v != _EV_EXPECTED[k]} + assert not wrong, ( + f"PX4 is not configured for external vision: {wrong} (expected " + f"{ {k: _EV_EXPECTED[k] for k in wrong} }). PX4_PARAM_SET=external-vision " + "did not reach the FCU, so the flight below would fly on GPS and pass anyway." + ) + logger.info("EV params confirmed on FCU: %s", actual) + + @pytest.mark.dependency(name="ev_ready", depends=["ev_params"]) + def test_px4_fuses_vision(self, optitrack_sim_stack): + """PX4 publishes local_position/odom, so EKF2 has converged and home is set. + + This only establishes that a converged estimate EXISTS — it is deliberately not + the proof that vision is being fused, because odom publishes off any aiding + source. The flight below is the proof: with GPS, baro and range aiding disabled in + _E2E_ENV, mocap is the only thing that can produce this estimate at all. + """ + container = _robot_container(optitrack_sim_stack) + topic = f"/robot_{_ROBOT_DOMAIN}/{_PX4_LOCAL_POSE_TOPIC}" + + first = wait_for_first_message( + container, topic, domain_id=_ROBOT_DOMAIN, + setup_bash=_ROBOT_SETUP_BASH, timeout=_FIRST_MSG_TIMEOUT, + ) + assert first is not None, ( + f"no PX4 local_position/odom on {topic} within {_FIRST_MSG_TIMEOUT}s — " + "EKF2 never converged or never set a home position. With GPS/baro/range " + "aiding off, that means the external-vision path never reached it." + ) + + @pytest.mark.dependency(name="ev_takeoff", depends=["ev_ready"]) + @pytest.mark.timeout(2400) + def test_takeoff(self, optitrack_sim_stack): + """Take off to TARGET_ALTITUDE_M flying on the mocap-fused estimate.""" + container = _robot_container(optitrack_sim_stack) + _arm_with_retries(container) + _run_parallel(1, lambda n: _takeoff_one_robot( + n, container, _TRAJ_CFG, TARGET_ALTITUDE_M)) + + @pytest.mark.dependency(name="ev_circle", depends=["ev_takeoff"]) + @pytest.mark.timeout(2400) + def test_circle_trajectory(self, optitrack_sim_stack): + """Fly a Circle with mocap as the only position source. + + This is the end-to-end proof: emulator → natnet_ros2 → vision_pose → MAVROS → + EKF2 → controller → airframe. Cross-track error is scored by the same code the + autonomy benchmark uses, so a mocap regression shows up as path deviation rather + than as a topic that merely exists. + """ + container = _robot_container(optitrack_sim_stack) + _run_parallel(1, lambda n: _trajectory_one_robot( + n, container, _TRAJ_CFG, _E2E_TRAJECTORY)) + + @pytest.mark.dependency(name="ev_land", depends=["ev_takeoff"]) + @pytest.mark.timeout(2400) + def test_landing(self, optitrack_sim_stack): + """Land the drone; runs even when the trajectory phase fails.""" + container = _robot_container(optitrack_sim_stack) + _run_parallel(1, lambda n: _landing_one_robot(n, container, _TRAJ_CFG)) From f4697265e40d65a39ba0127593adf172ee51f840 Mon Sep 17 00:00:00 2001 From: pvkumara <99618405+pvkumara@users.noreply.github.com> Date: Tue, 18 Aug 2026 18:34:09 -0400 Subject: [PATCH 18/21] CI/CD Tuning PR - pytest collection bug fix (#384) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * docs(tests): align unit-test docs with the co-located layout Unit test source moved into /test/ and is collected from colcon_unit_test_packages.yaml, but the surrounding documentation still described the mirror-directory-and-proxy scheme that replaced. Six per-layer stubs under tests/robot/ told authors to add tests in directories tests no longer live in, and tests/sim/motive_emulator/README.md proposed a NatNet emulator that was built at simulation/isaac-sim/extensions/optitrack.natnet.emulator/ instead. Remove them and rewrite the two tree READMEs as signposts. Correct the add-unit-tests and run-system-tests skills, which future agents read to work in this area, on four points they had wrong: - Running them. `pytest tests/` does not collect co-located unit tests — the injection in conftest.pytest_configure is skipped whenever a path is given on the command line. It reports "no tests collected" and exits 5, which reads as a failure but means nothing ran. `airstack test -m unit` and `cd tests && pytest -m unit` are the working forms; verified 155 passed vs exit 5. - CI. No workflow runs unit tests. system-tests.yml invokes `pytest tests/`, and fires only on PR-open, /pytest, or workflow_dispatch. - The mark. pytest_itemcollected applies @pytest.mark.unit by file location, so test sources should not declare it. The skill previously said "always decorate", which is where the redundant declarations came from. - colcon. It runs only what a package's CMakeLists registers. natnet_ros2 has ament_add_gtest but no ament_add_pytest_test, so its Python tests run only under the root harness. Also fixes a pytest_args example that would silently do nothing (`-m not linter`; ament's pytest runner ignores -m via PYTEST_ADDOPTS, and the real value is []), and the same stale layout claim in the testing docs and the emulator README. Co-Authored-By: Claude Opus 5 * docs(tests): record how C++ and Python unit tests reach CI C++ gtests run under colcon test, which CI executes inside the robot container via the build_packages mark (test_build_packages.py::test_colcon_test_robot). Python unit tests run under the root harness, which no workflow invokes. Whether colcon test also picks up a package's Python tests depends on its build type: lidar_point_cloud_filter is ament_python and exposes them via setup.cfg (testpaths = test), so they run in both places; natnet_ros2 is ament_cmake and registers only ament_add_gtest, so its Python tests run nowhere in CI. Co-Authored-By: Claude Opus 5 * fix(tests): collect co-located unit tests when the run is not narrowed Unit-test source lives outside tests/, so pytest_configure appends it to the collection args. That injection was gated on args_source != ARGS, which pytest sets for any positional path — including `tests/`. The intent was that `pytest tests/system/foo.py` should not drag in 155 unrelated tests, but the guard could not tell narrowing from naming the whole suite, so CI's `pytest tests/` collected 97 of 252 items and the Python unit tests ran nowhere. Decide on the paths instead: a positional is broad when it names tests/ itself or an ancestor, narrow otherwise. `pytest tests/` and `pytest .` inject; `pytest tests/system`, a single file, and a node id do not. Node ids are split on `::` first, since only the part before it addresses the filesystem. `any` rather than `all` is deliberate — pytest_configure appends the co-located files (narrow, absolute) to config.args, so `all` would flip the answer for anything re-deriving it after that mutation. The decision is also stashed on config for the contract test to read. tests/meta/test_collection_contract.py pins the behaviour: a table over broad/narrow invocations, a check that the command in system-tests.yml is classified broad (the test that would have caught this), and a check that every discovered file produced collected items. It lives under tests/ on purpose — co-located, it would stop being collected at the same moment it stopped guarding anything. Verified: `pytest tests/ -m unit` 0 -> 170 passed; `cd tests && pytest -m unit` unchanged at 170; `pytest tests/system/test_liveliness.py` still collects 16. Unit tests now run with every system-tests.yml invocation. That workflow's triggers are unchanged and intentional — PR open, /pytest, workflow_dispatch — since the same run drives the GPU system tests. Co-Authored-By: Claude Opus 5 * docs(tests): explain why C++ and Python unit tests use different runners The split was documented as a fact without its reason. A gtest is a binary compiled against the package's headers and rclcpp, so it can only run where the ROS toolchain is — colcon test inside the robot container, which build_packages reaches after building with -DBUILD_TESTING=ON. Python unit tests stub ROS at the import boundary and touch no ROS runtime, so they need neither a build nor a container, which is what keeps the suite under a second. State the invariant that follows: a Python test needing a live ROS node belongs in tests/integration/ or tests/system/, not in a package test/ dir. Co-Authored-By: Claude Opus 5 * test: run the collection contract tests with the fast tier They are hermetic and they guard the collection of everything above them, so running them after the GPU sim suites is backwards — a hung flight test would mean they never execute. Rank them in _MODULE_ORDER right after the co-located unit tests, ahead of system.test_build_docker. Also drop the `from conftest import repo_path` in favour of harness.discovery, which the module already imports from — one less thing between the test and the function it needs. Co-Authored-By: Claude Opus 5 * fix(ci): make PR validation and metrics trustworthy Run fast unit checks automatically, constrain host collection, and distinguish infrastructure failures from comparable simulation results. --------- Co-authored-by: John Co-authored-by: Claude Opus 5 Co-authored-by: Pranav Kumara --- .agents/skills/add-unit-tests/SKILL.md | 90 +++-- .../skills/bump-version-and-release/SKILL.md | 2 +- .agents/skills/run-system-tests/SKILL.md | 31 +- .env | 2 +- .github/workflows/system-tests.yml | 163 ++++++--- .github/workflows/unit-tests.yml | 46 +++ AGENTS.md | 7 +- CHANGELOG.md | 10 +- .../development/intermediate/testing/ci_cd.md | 75 +++-- .../development/intermediate/testing/index.md | 13 +- .../intermediate/testing/unit_testing.md | 48 ++- osmo/README.md | 12 +- .../optitrack.natnet.emulator/README.md | 6 +- tests/README.md | 31 +- tests/conftest.py | 48 ++- tests/harness/__init__.py | 8 +- tests/harness/collection.py | 4 + tests/harness/discovery.py | 47 ++- tests/harness/run_meta.py | 310 ++++++++++++++++++ tests/harness/test_ids.py | 15 + tests/integration/natnet/README.md | 2 +- tests/meta/test_collection_contract.py | 150 +++++++++ tests/meta/test_metrics_reporting_contract.py | 263 +++++++++++++++ tests/parse_metrics.py | 141 ++++++-- tests/robot/README.md | 10 +- tests/robot/behavior/README.md | 3 - tests/robot/global/README.md | 3 - tests/robot/interface/README.md | 3 - tests/robot/local/README.md | 3 - tests/robot/perception/README.md | 3 - tests/robot/sensors/README.md | 4 - tests/run_summary.py | 42 +-- tests/sim/README.md | 24 +- tests/sim/motive_emulator/README.md | 61 ---- 34 files changed, 1377 insertions(+), 303 deletions(-) create mode 100644 .github/workflows/unit-tests.yml create mode 100644 tests/harness/run_meta.py create mode 100644 tests/harness/test_ids.py create mode 100644 tests/meta/test_collection_contract.py create mode 100644 tests/meta/test_metrics_reporting_contract.py delete mode 100644 tests/robot/behavior/README.md delete mode 100644 tests/robot/global/README.md delete mode 100644 tests/robot/interface/README.md delete mode 100644 tests/robot/local/README.md delete mode 100644 tests/robot/perception/README.md delete mode 100644 tests/robot/sensors/README.md delete mode 100644 tests/sim/motive_emulator/README.md diff --git a/.agents/skills/add-unit-tests/SKILL.md b/.agents/skills/add-unit-tests/SKILL.md index d49a36d73..27aeb3a87 100644 --- a/.agents/skills/add-unit-tests/SKILL.md +++ b/.agents/skills/add-unit-tests/SKILL.md @@ -1,6 +1,6 @@ --- name: add-unit-tests -description: Add Python or C++ unit tests to an AirStack ROS 2 package. Covers the co-location pattern (test source in package/test/), registering the package in colcon_unit_test_packages.yaml so pytest tests/ and airstack test -m unit collect it, and how to extend to sim and GCS modules. +description: Add Python or C++ unit tests to an AirStack ROS 2 package. Covers the co-location pattern (test source in package/test/), registering the package in colcon_unit_test_packages.yaml so pytest tests/ and airstack test -m unit collect it, and how to extend to sim components. license: MIT metadata: author: AirLab CMU @@ -15,8 +15,8 @@ Use this skill when: - Adding Python unit tests for a ROS 2 package (perception, sensors, local, global, behavior, interface) - Adding C++ unit tests (`gtest`) to a package already using `ament_cmake` -- Extending unit tests to sim-side Python (`tests/sim/`) or GCS modules (`tests/gcs/`) -- Verifying that `airstack test -m unit` and `pytest tests/` (CI) pick up your new tests +- Extending unit tests to sim-side Python (`simulation/**//test/`) +- Verifying that `airstack test -m unit` picks up your new tests For system tests (full Docker stack, sim, sensors, takeoff/hover/land) see the `run-system-tests` skill instead. @@ -24,8 +24,8 @@ For system tests (full Docker stack, sim, sensors, takeoff/hover/land) see the ## Architecture Overview Unit test **source lives co-located with its package** (ROS 2 / colcon convention). -`tests/colcon_unit_test_packages.yaml` lists which packages have unit tests, and -`pytest tests/` collects them from there — you only edit files under the package itself. +`tests/colcon_unit_test_packages.yaml` lists which packages have unit tests, and the root +harness collects them from there — you only edit files under the package itself. ``` robot/ros_ws/src/// @@ -47,10 +47,37 @@ auto-tagged `@pytest.mark.unit` by path, so `-m unit` selects it. ament lint fil | Invocation | What runs | |---|---| -| `pytest tests/ -m unit` | Package `test/test_*.py`, collected directly from source | -| `airstack test -m unit` | Same path | -| CI `system-tests.yml` (PR open / approved) | Same path via `pytest tests/` | -| `colcon test --packages-select ` | Real test in `package/test/` (incl. linters + C++) | +| `airstack test -m unit` | Package `test/test_*.py`, collected directly from source | +| `cd tests && pytest -m unit` | Same path — the containerless equivalent | +| `pytest tests/ -m unit` | Same path — what CI runs | +| `colcon test --packages-select ` | C++ gtests and linters; Python only for `ament_python` packages (see below) | + +**Two runners, split by language — because C++ needs a build and Python does not.** A +gtest is a binary: it must be compiled against the package's headers and rclcpp, so it can +only run where the ROS toolchain is. That is `colcon test` inside the robot container, +which CI reaches via the **`build_packages`** mark +(`tests/system/test_build_packages.py::test_colcon_test_robot`, which builds with +`-DBUILD_TESTING=ON` first). Python unit tests are deliberately hermetic — they stub ROS +at the import boundary and touch no ROS runtime — so they need no build and no container, +which is what lets the root harness run all of them in about a second. + +Preserve that property when adding tests: a Python test that needs a live ROS node belongs +in `tests/integration/` or `tests/system/`, not here. + +Whether `colcon test` *also* picks up a package's Python tests depends on its build type: + +| Package | Build type | Python tests under `colcon test` | +|---|---|---| +| `natnet_ros2` | `ament_cmake` | **No** — `CMakeLists.txt` registers `ament_add_gtest` but no `ament_add_pytest_test` | +| `lidar_point_cloud_filter` | `ament_python` | **Yes** — `setup.cfg` sets `testpaths = test`, so colcon's pytest runner finds them | + +So a Python test in an `ament_cmake` package runs *only* via the root harness — which is +fine, since that is what CI invokes. + +Naming a path *below* `tests/` narrows the run and skips the injection, so +`pytest tests/system/test_x.py` stays fast and does not drag in unit tests. The rule lives +in `harness.discovery.collection_is_broad` and is pinned by +`tests/meta/test_collection_contract.py`. ## Step-by-Step: Adding a Python Unit Test @@ -87,13 +114,17 @@ if str(_src) not in sys.path: from my_module import my_function # noqa: E402 -@pytest.mark.unit def test_my_function_basic(): assert my_function(1, 2) == 3 ``` **Key points:** -- Always decorate with `@pytest.mark.unit` — this is the filter for fast runs. +- **Do not write `@pytest.mark.unit`.** `pytest_itemcollected` in `tests/conftest.py` + applies it by file location to everything under a registered package's `test/` dir. + Writing it by hand is redundant, and it warns (`PytestUnknownMarkWarning`) under any + invocation where `tests/pytest.ini` is not the configfile — e.g. `colcon test`. +- Import `pytest` only if you need its API (`approx`, `raises`, `parametrize`, + `importorskip`). - Compute paths relative to `__file__` (`parent.parent / "src"`) — never hardcode absolute paths. - For packages with a Python module directory (`//`), add the package @@ -127,35 +158,40 @@ robot: - natnet_ros2 - lidar_point_cloud_filter - # ← add here - pytest_args: "-m not linter" + pytest_args: [] ``` +Leave `pytest_args` empty. It is forwarded to `colcon test` via `PYTEST_ADDOPTS`, and +ament's pytest runner ignores `-m` there — a marker expression in this field silently +does nothing. + That's the whole registration. `conftest.py` globs `robot/ros_ws/src/**//test`, collects its non-linter `test_*.py`, and marks them `unit`. The test file must be self-contained: if it imports package code, set up `sys.path` at the top of the test file (see `test_validation_core.py`, which inserts its package root). Same YAML, different workspace key (`sim:`), for Isaac-extension unit tests. -### 5. Run locally to verify +### 4. Run locally to verify ```bash -# From repo root — no container needed -cd tests -pytest -m unit -v -# or airstack test -m unit -v +# or, containerless: +AIRSTACK_ROOT=$(pwd) pytest tests/ -m unit -v ``` -All 14+ existing tests plus your new ones should pass. Collected items point straight +All 155 existing tests plus your new ones should pass. Collected items point straight at the co-located source: ``` ../robot/ros_ws/src///test/test_.py::test_my_function_basic PASSED ``` -### 6. CI picks it up automatically +### 5. Running in CI -Unit tests are discovered by `pytest tests/` and run as part of `system-tests.yml` -(triggered on PR open) — no changes to CI needed. +`unit-tests.yml` invokes `pytest tests/ -m unit` on GitHub-hosted `ubuntu-latest` +whenever a PR targeting `main` or `develop` is opened, synchronized, or reopened. +It does not consume an OSMO GPU. +C++ gtests still run through the OSMO `build_packages` mark because they require the +ROS workspace and toolchain inside the robot container. --- @@ -242,11 +278,11 @@ sim: | Where does test source live? | `/…//test/` (co-located with the package) | | Where does pytest discover tests? | From the package `test/` dir listed in `colcon_unit_test_packages.yaml` | | How are duplicate basenames handled? | `--import-mode=importlib` (set in `pytest.ini`) | -| What mark do all unit tests use? | `@pytest.mark.unit` (auto-applied by path in `conftest.py`) | -| What CI workflow runs them? | `system-tests.yml` — runs `pytest tests/` which includes unit tests | -| When does that workflow trigger? | PR opened, `/pytest` comment, `workflow_dispatch` | +| What mark do all unit tests use? | `@pytest.mark.unit` — auto-applied by path in `conftest.py`; do not write it yourself | +| How do I run them? | `airstack test -m unit`, `cd tests && pytest -m unit`, or `pytest tests/ -m unit` | +| What CI workflow runs them? | Python: `unit-tests.yml`; C++: the `build_packages` path in `system-tests.yml` — see §5 | | Do system tests (`liveliness`, etc.) run too? | No — `-m unit` filters to hermetic tests only | -| Does `colcon test` also run these? | Yes — Python tests in `package/test/` are discovered by colcon's pytest runner | +| Does `colcon test` also run these? | Only if the package registers them. `ament_add_gtest` covers C++; a Python test needs `ament_add_pytest_test`, which `natnet_ros2` does **not** have — its Python tests run only under the root harness | | Can I add pure C++ gtests? | Yes — `ament_add_gtest` in CMakeLists.txt | ## Reference Implementations @@ -261,8 +297,8 @@ Both are collected from their package `test/` dir. ## Files to Know -- `.github/workflows/system-tests.yml` — CI workflow (runs `pytest tests/` including unit tests) -- `tests/pytest.ini` — mark registration + `--import-mode=importlib` +- `.airstack/modules/dev.sh` — what `airstack test` runs (bare `pytest` with `working_dir` `tests/`) +- `tests/pytest.ini` — mark registration + `--import-mode=importlib` + `testpaths` - `tests/colcon_unit_test_packages.yaml` — the package list driving unit-test collection - `tests/conftest.py` — `unit_test_files()` / `pytest_configure` inject package tests; `pytest_itemcollected` auto-marks `unit` - `tests/README.md` — full test harness reference diff --git a/.agents/skills/bump-version-and-release/SKILL.md b/.agents/skills/bump-version-and-release/SKILL.md index 18792a965..a4a74c77e 100644 --- a/.agents/skills/bump-version-and-release/SKILL.md +++ b/.agents/skills/bump-version-and-release/SKILL.md @@ -260,5 +260,5 @@ For a true release (dropping the pre-release suffix): ## Related Skills -- [`run-system-tests`](../run-system-tests) — what fires on every PR alongside the version check +- [`run-system-tests`](../run-system-tests) — automatic unit/package gates and how to request simulation campaigns - [`update-documentation`](../update-documentation) — for docs-only PRs that may still need a VERSION bump to clear the gate diff --git a/.agents/skills/run-system-tests/SKILL.md b/.agents/skills/run-system-tests/SKILL.md index 22ce20520..c2260d29e 100644 --- a/.agents/skills/run-system-tests/SKILL.md +++ b/.agents/skills/run-system-tests/SKILL.md @@ -1,6 +1,6 @@ --- name: run-system-tests -description: Run, interpret, and extend AirStack's pytest system test suite (build_packages, build_docker, liveliness, sensors, takeoff_hover_land, autonomy), trigger runs via /pytest PR comments, and read metrics.json regression reports. Use for invoking tests, debugging failures from results.xml/metrics.json, or adding a new system test. +description: Run, interpret, and extend AirStack's pytest system test suite (build_packages, build_docker, liveliness, sensors, takeoff_hover_land, autonomy), trigger runs via /pytest PR comments, and read run_meta.json/metrics.json reports. Use for invoking tests, distinguishing infrastructure failures from policy regressions, or adding a new system test. license: Apache-2.0 metadata: author: AirLab CMU @@ -14,7 +14,7 @@ metadata: Use this skill when you need to: - Invoke the pytest system tests locally (via `airstack test`) or on CI (via `/pytest` PR comment or `workflow_dispatch`) -- Diagnose a failing system test — interpret `summary.txt`, `results.xml`, and `metrics.json` from `tests/results//` +- Diagnose a failing system test — interpret `summary.txt`, `results.xml`, `run_meta.json`, and `metrics.json` from `tests/results//` - Compare metrics against a baseline run (`parse_metrics.py --baseline`) to confirm a regression or improvement - Add a new system test to `tests/`: pick the right mark, wire up `airstack_env` parametrization, and record metrics with `MetricsRecorder` @@ -33,8 +33,8 @@ The suite lives at `tests/` (repo root) and is fully pytest-based. Configuration | Concern | Unit (`-m unit`) | System (`-m liveliness` etc.) | |---|---|---| | Hardware required | None — pure Python | Docker daemon, NVIDIA GPU, sim license | -| CI workflow | `system-tests.yml` (included in `pytest tests/`) | `system-tests.yml` (GPU OpenStack VM) | -| Trigger | Every push + PR (automatic) | PR opened, `/pytest` comment, `workflow_dispatch` | +| CI workflow | `unit-tests.yml` (`ubuntu-latest`) | `system-tests.yml` (ephemeral OSMO GPU pod) | +| Trigger | PR opened, synchronized, or reopened | Automatic `build_packages` on PR open/update/reopen; simulation via `/pytest` or `workflow_dispatch` | | Source location | `/test/test_*.py` (collected directly, listed in `colcon_unit_test_packages.yaml`) | `tests/system/` | | How to add | See `add-unit-tests` skill | See *Adding a New System Test* below | @@ -42,8 +42,8 @@ Run unit tests without any Docker stack: ```bash airstack test -m unit -v -# or -pytest tests/ -m unit -v # AIRSTACK_ROOT=$(pwd) for direct pytest +# or directly: +AIRSTACK_ROOT=$(pwd) pytest tests/ -m unit -v ``` For details on the co-located layout and adding new unit tests, see the @@ -180,7 +180,10 @@ Total parametrize cardinality for sim tests = `len(sims) × len(num_robots) × s The `system-tests.yml` workflow accepts three trigger paths: -1. **PR opened** (same-repo only) — auto-runs pytest with conftest defaults. Fork PRs are skipped to keep arbitrary code off the self-hosted runner. +1. **PR opened, synchronized, or reopened** (same-repo only) — auto-runs the + `build_packages` mark. Fork PRs are skipped to keep arbitrary code off the + privileged self-hosted runner. Python unit tests run separately in + `unit-tests.yml`, including for fork PRs. 2. **`/pytest` issue comment** on a PR — only honored from users with `OWNER`, `MEMBER`, or `COLLABORATOR` author association. Fork PRs are explicitly rejected by the `Resolve PR head` step (the PR's head repo must equal `${context.repo.owner}/${context.repo.repo}`). 3. **`workflow_dispatch`** — manual run from the Actions tab with form inputs (`marks`, `sim`, `num_robots`, `stress_iterations`, `stable_duration`, `baseline_run_id`). @@ -205,14 +208,14 @@ notes: testing the new altitude controller The workflow: 1. Posts an acknowledgment PR comment showing the resolved `pytest tests/ ` command and a link to the run 2. Opens an in-progress GitHub Check Run on the PR's head SHA so the run shows up in the **Checks** tab (issue_comment events otherwise associate runs with the default branch) -3. Runs pytest on a freshly-spawned ephemeral OpenStack runner (`runs-on: [self-hosted, airstack-ephemeral]`) +3. Runs pytest on a freshly-spawned ephemeral OSMO GPU pod (`runs-on: [self-hosted, airstack-ephemeral]`) 4. Uploads `tests/results/` as artifact `test-results--` (90-day retention) -5. The downstream `report` job runs `parse_metrics.py` against the latest baseline artifact from the PR's base branch and posts a markdown table back as a PR comment + job summary +5. The downstream `report` job runs `parse_metrics.py`, compares only a matching complete simulation baseline, posts the result, and finalizes the PR-head Check Run 6. Closes the Check Run with the final conclusion ### Why fork PRs are blocked -The runner is GPU-equipped, has Docker root access, and is reused (briefly) across the lifetime of one job. Running arbitrary fork code on it would let a contributor exfiltrate registry creds, mine crypto, or pivot into the OpenStack tenant. The same-repo guard is the only line of defense and **must not be removed**. If you need to test a fork PR, mirror the branch into the upstream repo first. +The runner is GPU-equipped, has Docker root access, and is reused (briefly) across the lifetime of one job. Running arbitrary fork code on it would let a contributor exfiltrate registry creds, mine crypto, or abuse the privileged OSMO CI pool. The same-repo guard is the only line of defense and **must not be removed**. If you need to test a fork PR, mirror the branch into the upstream repo first. ## Interpreting Results and Metrics @@ -273,7 +276,7 @@ The report has three sections per test module: - **Sim publishing rates** — pivoted Hz aggregates per topic (`mean`, `start_mean`, `end_mean`, `min`, `max`) from the `sensors` mark (sim + robot streams) - **Compute usage** — pivoted CPU/mem/GPU per container -Regressions exceeding `--threshold` (default 20%) are flagged `:red_circle:`; improvements beyond threshold get `:green_circle:`. CI fails the job on any regression. +Regressions exceeding `--threshold` (default 20%) are flagged `:red_circle:`; improvements beyond threshold get `:green_circle:`. CI fails only when both artifacts are complete and have the same simulation campaign fingerprint. When local-debugging a CI regression, download both artifacts (`test-results--` from the PR run and from the base branch's most recent run), unzip them under `tests/results/`, and run `parse_metrics.py` locally to see the same table the bot posted. @@ -360,7 +363,7 @@ If multiple tests need the same setup, add a fixture in `conftest.py` (not in yo - **Letting parametrize cardinality explode**. Default `--num-robots 1,3` (and `--sim msairsim` if you opt in) multiplies stack bring-ups for each selected mark (`liveliness`, `sensors`, `takeoff_hover_land`, …) — expensive. Override locally to a single tuple while iterating. `--sim` defaults to `isaacsim` only. - **Hardcoded container names**. Always use `find_container`, `get_robot_containers`, or `wait_for_container` — replica suffixes (`-1`, `-2`, `-3`) and compose project prefixes change. - **Asserting on stdout instead of using `read_log_tail`**. The conftest captures each subprocess's combined stdout/stderr in memory; assertions should reference it via `read_log_tail()` (`f"airstack up failed:\n{read_log_tail()}"`) so failures attach the relevant context to the JUnit XML. -- **Trying to SSH into a CI runner mid-job**. Workers are ephemeral OpenStack VMs destroyed within ~30s of job completion. Re-running the job creates a fresh VM. For genuine debugging on the runner, see `.github/orchestrator/README.md` (also exposed at `tests/ci-cd-orchestrator.md`) — but in 99% of cases, reproduce locally with `airstack test`. +- **Trying to SSH into a CI runner mid-job**. Workers are ephemeral OSMO pods destroyed after job completion. Re-running creates a fresh pod. For genuine runner debugging, see `.github/orchestrator/README.md` (also exposed at `tests/ci-cd-orchestrator.md`) — but in most cases, reproduce locally with `airstack test`. - **Forgetting to register a new mark**. Adding `@pytest.mark.my_new_mark` without updating `tests/pytest.ini` produces "PytestUnknownMarkWarning" and makes `-m my_new_mark` fail to filter as expected. ## Quick Reference @@ -424,12 +427,12 @@ python tests/parse_metrics.py \ ### Files to know - `tests/conftest.py` — pytest hooks + the `airstack_env` / `robot_autonomy_stack` fixtures (re-exports the harness API) -- `tests/harness/` — helpers split by concern: `session`, `discovery`, `commands`, `containers`, `metrics` (`MetricsRecorder`), `sim`, `collection` (ordering) +- `tests/harness/` — helpers split by concern: `session`, `discovery`, `commands`, `containers`, `metrics` (`MetricsRecorder`), `run_meta`, `test_ids`, `sim`, `collection` (ordering) - `tests/pytest.ini` — mark registration, log format - `tests/parse_metrics.py` — markdown reporter, regression diff - `tests/README.md` — user-facing docs (CLI options, output layout, CI/CD orchestrator) - `.github/workflows/system-tests.yml` — CI workflow with `/pytest` comment trigger -- `.github/orchestrator/README.md` — ephemeral OpenStack runner setup and SSH-debug procedure +- `.github/orchestrator/README.md` — ephemeral OSMO runner setup and worker-debug procedure ## References diff --git a/.env b/.env index ea6070100..c97a01e5b 100644 --- a/.env +++ b/.env @@ -12,7 +12,7 @@ PROJECT_NAME="airstack" # If you've run ./airstack.sh setup, then this will auto-generate from the git commit hash every time a change is made # to a Dockerfile or docker-compose.yaml file. Otherwise this can also be set explicitly to make a release version. # auto-generated from git commit hash -VERSION="0.19.0-alpha.16" +VERSION="0.19.0-alpha.17" # Choose "dev" or "prebuilt". "dev" is for mounted code that must be built live. "prebuilt" is for built ros_ws baked into the image DOCKER_IMAGE_BUILD_MODE="dev" # Where to push and pull images from. Can replace with your docker hub username if using docker hub. diff --git a/.github/workflows/system-tests.yml b/.github/workflows/system-tests.yml index 69a961a48..48ab4f639 100644 --- a/.github/workflows/system-tests.yml +++ b/.github/workflows/system-tests.yml @@ -2,7 +2,7 @@ name: System Tests on: pull_request: - types: [opened] + types: [opened, synchronize, reopened] issue_comment: types: [created] workflow_dispatch: @@ -41,7 +41,7 @@ jobs: runs-on: [self-hosted, airstack-ephemeral] # Triggers: # - workflow_dispatch (manual) - # - PR opened from the same repo (not a fork) — same-repo guard + # - PR opened, synchronized, or reopened from the same repo — same-repo guard # prevents arbitrary code execution on the self-hosted runner from # untrusted contributors. # - PR comment starting with `/pytest` from a user with write access @@ -57,6 +57,9 @@ jobs: github.event.issue.pull_request != null && startsWith(github.event.comment.body, '/pytest') && contains(fromJSON('["OWNER","MEMBER","COLLABORATOR"]'), github.event.comment.author_association)) + concurrency: + group: system-tests-${{ github.event_name == 'pull_request' && github.event.pull_request.number || github.run_id }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} timeout-minutes: 120 # Adding any `permissions:` entry disables GITHUB_TOKEN's defaults, so # every scope used here has to be re-granted explicitly: @@ -76,6 +79,10 @@ jobs: # is not addressable from `if:` expressions. env: DOCKER_REGISTRY_PASSWORD: ${{ secrets.DOCKER_REGISTRY_PASSWORD }} + outputs: + tested_sha: ${{ steps.identity.outputs.tested_sha }} + pr_number: ${{ steps.identity.outputs.pr_number }} + check_run_id: ${{ steps.check_create.outputs.id }} steps: # Uses actions/github-script (Node, bundled with the runner) instead # of `gh` so we don't depend on system tools — the ephemeral @@ -103,6 +110,24 @@ jobs: core.setOutput('head_sha', pr.data.head.sha); core.setOutput('base_ref', pr.data.base.ref); + - name: Resolve tested revision identity + if: always() + id: identity + env: + EVENT_NAME: ${{ github.event_name }} + COMMENT_HEAD_SHA: ${{ steps.pr.outputs.head_sha }} + EVENT_SHA: ${{ github.sha }} + COMMENT_PR_NUMBER: ${{ github.event.issue.number }} + EVENT_PR_NUMBER: ${{ github.event.pull_request.number }} + run: | + if [[ "$EVENT_NAME" == "issue_comment" ]]; then + echo "tested_sha=${COMMENT_HEAD_SHA:-$EVENT_SHA}" >> "$GITHUB_OUTPUT" + echo "pr_number=$COMMENT_PR_NUMBER" >> "$GITHUB_OUTPUT" + else + echo "tested_sha=$EVENT_SHA" >> "$GITHUB_OUTPUT" + echo "pr_number=$EVENT_PR_NUMBER" >> "$GITHUB_OUTPUT" + fi + # Parsed up-front (before checkout) so the acknowledgment comment below # can echo the resolved args. This step only reads env vars, so it # doesn't need the working tree. @@ -134,9 +159,10 @@ jobs: if (st := os.environ.get('INPUT_STABLE', '').strip()): args.extend(['--stable-duration', st]) elif event == 'pull_request': - # PR-opened auto-run uses pytest's conftest defaults — same as - # /pytest with no args. - args = [] + # Automatic PR validation is deliberately build-scoped. Fast + # Python unit tests run in unit-tests.yml; GPU simulation remains + # selectable via /pytest or workflow_dispatch. + args = ['-m', 'build_packages'] else: body = os.environ.get('COMMENT_BODY', '') # Only the first line is parsed — everything below it is @@ -223,7 +249,7 @@ jobs: # Reply on the PR thread so the commenter sees their /pytest was # picked up and can confirm we parsed the args correctly. The # workflow_dispatch path skips this (no PR to comment on); the - # pull_request-opened path skips it too (the PR Checks tab is + # pull_request path skips it too (the PR Checks tab is # already showing the native run). - name: Post acknowledgment comment if: github.event_name == 'issue_comment' @@ -403,57 +429,75 @@ jobs: run: | # Re-split the shell-quoted args from the parse step so we forward # them to pytest as a proper argv list (preserving values like - # `-m 'a or b'`). Empty PYTEST_ARGS yields an empty array, so - # pytest falls back to its conftest defaults. - mapfile -t ARGS < <(python3 -c "import os, shlex; print('\n'.join(shlex.split(os.environ['PYTEST_ARGS'])))") + # `-m 'a or b'`). sys.stdout.write is intentional: print('') emits + # one blank line, which mapfile turns into an empty positional path + # and makes pytest recurse from the repository root. + mapfile -t ARGS < <(python3 -c "import os, shlex, sys; sys.stdout.write(''.join(f'{arg}\\n' for arg in shlex.split(os.environ['PYTEST_ARGS'])))") + for arg in "${ARGS[@]}"; do + if [[ -z "$arg" ]]; then + echo "::error::Refusing an empty pytest argument because it expands collection to the repository root." + exit 2 + fi + done + set +e pytest tests/ \ "${ARGS[@]}" \ -v -s \ --log-cli-level=INFO \ --log-cli-format='%(asctime)s [%(levelname)s] %(name)s: %(message)s' \ --log-cli-date-format='%H:%M:%S' + pytest_status=$? + set -e + if (( pytest_status != 0 )); then + exit "$pytest_status" + fi + + # A successful collect-only/non-executed campaign is not a passing + # system test. Make that distinction visible in the job conclusion. + python3 <<'PYEOF' + import json + from pathlib import Path + + candidates = list(Path("tests/results").glob("*/run_meta.json")) + if not candidates: + raise SystemExit("::error::pytest succeeded without run_meta.json") + latest = max(candidates, key=lambda path: path.stat().st_mtime) + outcome = json.loads(latest.read_text()).get("outcome") + if outcome not in {"simulation", "non_simulation"}: + raise SystemExit( + f"::error::pytest did not execute a complete campaign ({outcome})" + ) + PYEOF - name: Upload test results uses: actions/upload-artifact@v4 if: always() with: - name: test-results-${{ github.sha }}-${{ github.run_id }} + name: test-results-${{ steps.identity.outputs.tested_sha }}-${{ github.run_id }} path: tests/results/ retention-days: 90 - # Close out the Check Run with the job's final conclusion. The - # `steps.check_create.outputs.id` guard skips this when the open - # step didn't run (workflow_dispatch) or failed before producing - # an id. - - name: Finalize check on PR head - if: always() && github.event_name == 'issue_comment' && steps.check_create.outputs.id - uses: actions/github-script@v7 - with: - script: | - await github.rest.checks.update({ - owner: context.repo.owner, - repo: context.repo.repo, - check_run_id: ${{ steps.check_create.outputs.id }}, - status: 'completed', - conclusion: '${{ job.status }}', - details_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`, - }); - report: name: Metrics Report runs-on: ubuntu-latest needs: run-tests # Skip when run-tests was skipped (e.g., comment didn't match `/pytest`) # so we don't post empty-report comments on every PR comment. - if: always() && needs.run-tests.result != 'skipped' + if: > + always() && + needs.run-tests.result != 'skipped' && + (needs.run-tests.result != 'cancelled' || github.event_name != 'pull_request') permissions: + actions: read + checks: write + contents: read pull-requests: write steps: - name: Checkout uses: actions/checkout@v4 with: - submodules: recursive + ref: ${{ needs.run-tests.outputs.tested_sha }} - name: Set up Python uses: actions/setup-python@v5 @@ -478,8 +522,9 @@ jobs: - name: Download current test results uses: actions/download-artifact@v4 + continue-on-error: true with: - name: test-results-${{ github.sha }}-${{ github.run_id }} + name: test-results-${{ needs.run-tests.outputs.tested_sha }}-${{ github.run_id }} path: current-results/ # PR mode (opened or comment-triggered): fetch latest artifact from @@ -504,9 +549,10 @@ jobs: uses: actions/download-artifact@v4 continue-on-error: true with: + github-token: ${{ github.token }} run-id: ${{ inputs.baseline_run_id }} - name_is_regexp: true - name: "test-results-.*" + pattern: "test-results-*" + merge-multiple: true path: baseline-results/ # Manual dispatch without explicit baseline: fetch latest from main @@ -548,6 +594,19 @@ jobs: CURRENT="${{ steps.dirs.outputs.current }}" BASELINE="${{ steps.dirs.outputs.baseline }}" + if [ -z "$CURRENT" ]; then + cat > report.md <<'EOF' + ## Run status + + **Simulation metrics are not comparable.** The test job produced no finalized `results.xml` artifact. The runner may have timed out, been cancelled, or failed before pytest started. + + Pass-rate and regression tables are suppressed because no completed test campaign is available. + EOF + echo "parser_exit=0" >> "$GITHUB_OUTPUT" + exit 0 + fi + + set +e if [ -n "$BASELINE" ]; then python tests/parse_metrics.py \ --current "$CURRENT" \ @@ -558,6 +617,10 @@ jobs: --current "$CURRENT" \ --output report.md fi + parser_exit=$? + set -e + echo "parser_exit=$parser_exit" >> "$GITHUB_OUTPUT" + exit "$parser_exit" - name: Post PR comment if: github.event_name == 'issue_comment' || github.event_name == 'pull_request' @@ -571,9 +634,9 @@ jobs: } catch { body = '_No metrics report generated._'; } - const header = `## Test Metrics — \`${{ github.sha }}\`\n\n`; + const header = `## Test Metrics — \`${{ needs.run-tests.outputs.tested_sha }}\`\n\n`; await github.rest.issues.createComment({ - issue_number: context.issue.number, + issue_number: Number('${{ needs.run-tests.outputs.pr_number }}'), owner: context.repo.owner, repo: context.repo.repo, body: header + body, @@ -583,7 +646,7 @@ jobs: if: always() run: | if [ -f report.md ]; then - echo "## Test Metrics — \`${{ github.sha }}\`" >> "$GITHUB_STEP_SUMMARY" + echo "## Test Metrics — \`${{ needs.run-tests.outputs.tested_sha }}\`" >> "$GITHUB_STEP_SUMMARY" echo "" >> "$GITHUB_STEP_SUMMARY" cat report.md >> "$GITHUB_STEP_SUMMARY" else @@ -593,5 +656,31 @@ jobs: - name: Fail on regression if: steps.report.outcome == 'failure' run: | - echo "::error::Metric regression detected — see the report above for details." + if [ "${{ steps.report.outputs.parser_exit }}" = "1" ]; then + echo "::error::Metric regression detected — see the report above for details." + else + echo "::error::Metrics report generation failed — see the report step log." + fi exit 1 + + - name: Finalize check on PR head + if: always() && github.event_name == 'issue_comment' && needs.run-tests.outputs.check_run_id + uses: actions/github-script@v7 + with: + script: | + const tests = '${{ needs.run-tests.result }}'; + const report = '${{ steps.report.outcome }}'; + let conclusion = 'failure'; + if (tests === 'success' && report === 'success') { + conclusion = 'success'; + } else if (tests === 'cancelled') { + conclusion = 'cancelled'; + } + await github.rest.checks.update({ + owner: context.repo.owner, + repo: context.repo.repo, + check_run_id: Number('${{ needs.run-tests.outputs.check_run_id }}'), + status: 'completed', + conclusion, + details_url: `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`, + }); diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml new file mode 100644 index 000000000..a2bf33022 --- /dev/null +++ b/.github/workflows/unit-tests.yml @@ -0,0 +1,46 @@ +name: Unit Tests + +on: + pull_request: + branches: [main, develop] + types: [opened, synchronize, reopened] + +permissions: + contents: read + +concurrency: + group: unit-tests-${{ github.event.pull_request.number }} + cancel-in-progress: true + +jobs: + unit: + name: Python unit and harness contracts + runs-on: ubuntu-latest + timeout-minutes: 15 + + steps: + - name: Checkout + uses: actions/checkout@v4 + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.12" + cache: pip + cache-dependency-path: tests/requirements.txt + + - name: Install test dependencies + run: python -m pip install -r tests/requirements.txt + + - name: Run unit tests + env: + AIRSTACK_ROOT: ${{ github.workspace }} + run: pytest tests/ -m unit + + - name: Upload unit-test results + if: always() + uses: actions/upload-artifact@v4 + with: + name: unit-test-results-${{ github.sha }}-${{ github.run_id }} + path: tests/results/ + retention-days: 30 diff --git a/AGENTS.md b/AGENTS.md index 76987f8eb..8c113f219 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -222,7 +222,7 @@ docker exec airstack-robot-desktop-1 bash -c "ros2 topic echo --onc docker exec airstack-robot-desktop-1 bash -c "sws && colcon test --packages-select natnet_ros2 --event-handlers console_direct+" ``` -2. **Unit tests (`pytest`, `unit` mark):** Fast, hermetic checks. Test **source** lives co-located with each ROS 2 package in `/test/` (standard colcon convention). [`tests/colcon_unit_test_packages.yaml`](tests/colcon_unit_test_packages.yaml) lists which packages have unit tests, and `tests/conftest.py` collects them from there under `--import-mode=importlib`. To add a package's unit tests, list it in that YAML. Unit tests run as part of the `system-tests.yml` suite. Example: `airstack test -m unit -v`. See `add-unit-tests` skill. +2. **Unit tests (`pytest`, `unit` mark):** Fast, hermetic checks. Test **source** lives co-located with each ROS 2 package in `/test/` (standard colcon convention). [`tests/colcon_unit_test_packages.yaml`](tests/colcon_unit_test_packages.yaml) lists which packages have unit tests, and `tests/conftest.py` collects them from there under `--import-mode=importlib`. To add a package's unit tests, list it in that YAML. Python unit tests run automatically in `unit-tests.yml` on `ubuntu-latest`; C++ gtests run through the `system-tests.yml` `build_packages` path. Example: `airstack test -m unit -v`. See `add-unit-tests` skill. 3. **System Level (`tests/system/`):** Full simulation tests (Isaac Sim or Microsoft AirSim legacy) - End-to-end autonomy stack testing @@ -244,7 +244,7 @@ Pytest-based system tests live under [`tests/system/`](tests/system/). They brin | [`tests/system/test_fixed_trajectory.py`](tests/system/test_fixed_trajectory.py) | `autonomy` | 4-phase flight chain (PX4 ready → takeoff → execute Circle/Figure8/Racetrack/Line trajectory → land) per (sim, num_robots, iter, trajectory_type); records cross-track error and path RMSE | Docker, GPU, sim license | | [`tests/system/test_waypoint_flight.py`](tests/system/test_waypoint_flight.py) | `waypoint_flight` | 4-phase flight chain (PX4 ready → takeoff → NavigateTask waypoint route → land) per (sim, num_robots, iter); pass/fail judged on the odometry track by the standalone [`tests/waypoint_checker.py`](tests/waypoint_checker.py) (in-order corridor arrival within `--waypoint-tolerance`, final goal within `--goal-tolerance`, per-waypoint `--waypoint-timeout`) | Docker, GPU, sim license | -The pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures live in [`tests/conftest.py`](tests/conftest.py); the shared helpers are split by concern into the [`tests/harness/`](tests/harness/) package (`session`, `discovery`, `commands`, `containers`, `metrics` (with `MetricsRecorder`), `sim`, `collection`) and re-exported through `conftest`, so `from conftest import ` still resolves. Each run produces a timestamped directory under `tests/results//` with `summary.txt`, `results.xml`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) generates a markdown report (single-run or diff-vs-baseline; exits 1 on regression). +The pytest hooks and the `airstack_env` / `robot_autonomy_stack` fixtures live in [`tests/conftest.py`](tests/conftest.py); the shared helpers are split by concern into the [`tests/harness/`](tests/harness/) package (`session`, `discovery`, `commands`, `containers`, `metrics` (with `MetricsRecorder`), `run_meta`, `test_ids`, `sim`, `collection`) and re-exported through `conftest`, so `from conftest import ` still resolves. Each run produces a timestamped directory under `tests/results//` with `summary.txt`, `results.xml`, `run_meta.json`, and `metrics.json` (no per-test log files — live output streams to the terminal via `log_cli`). [`tests/parse_metrics.py`](tests/parse_metrics.py) compares only matching, complete simulation campaigns and exits 1 on a genuine metric regression. **Run via the CLI** (containerized runner — no local Python needed): @@ -280,7 +280,8 @@ GitHub Actions workflows live in [`.github/workflows/`](.github/workflows/): | Workflow | Trigger | Purpose | |----------|---------|---------| -| [`system-tests.yml`](.github/workflows/system-tests.yml) | PR opened, `/pytest` PR comment (write-access only), or `workflow_dispatch` | Runs the `tests/` suite on an ephemeral GPU runner; posts metrics report (with regression diff vs base branch / `main`) as a PR comment and to the job summary | +| [`unit-tests.yml`](.github/workflows/unit-tests.yml) | PR to `main`/`develop` opened, synchronized, or reopened | Runs Python unit tests and harness contracts on `ubuntu-latest` | +| [`system-tests.yml`](.github/workflows/system-tests.yml) | PR opened/synchronized/reopened, `/pytest` PR comment (write-access only), or `workflow_dispatch` | Runs automatic package builds or selected simulation marks on an ephemeral GPU runner; only complete simulation campaigns are compared in metrics reports | | [`docker-build.yml`](.github/workflows/docker-build.yml) | Push to `main`/`develop` that changes `.env` (`VERSION=`), or manual dispatch | Builds, pushes, and cosign-signs all compose images on the ephemeral runner | | [`check-version-increment.yml`](.github/workflows/check-version-increment.yml) | Pull request | Validates `.env` `VERSION=` is valid semver and strictly greater than the base branch | | `deploy_docs_from_{main,develop,release}.yaml` | Push to the matching branch (`docs/**`, `mkdocs.yml`, `*.md`) | Publishes versioned MkDocs site via `mike` | diff --git a/CHANGELOG.md b/CHANGELOG.md index 15da258c6..d1409cd61 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Automatic `unit-tests.yml` PR gate on `ubuntu-latest`, plus `run_meta.json` outcome metadata so reports distinguish completed simulation campaigns from collection errors, empty selections, timeouts, and cancellations - `overrides/isaac-optitrack-simulation.env` — brings up Isaac Sim with the NatNet emulator and PX4 flying on mocap EKF2 external vision (GPS/baro/range aiding off), i.e. the configuration `tests/system/test_optitrack_e2e.py` runs, reproducible by hand - `overrides/l4t-optitrack-realrobot.env` — deployment override for a real Jetson robot flying on OptiTrack mocap (PX4 EKF2 external vision instead of GPS): the NatNet server/body settings, plus the multi-NIC and FCU-parameter notes that path needs - Feature notebook workflow (`use-feature-notebook` skill): every agent-implemented feature gets a local, gitignored `notebook/NNN-feature-slug/` entry with a status-tracked `design_spec.md` (written before coding) and `results/` artifacts + self-contained `results_summary.md` that populate the feature's PR description @@ -24,17 +25,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Unit-test documentation now matches the co-located layout: the `add-unit-tests` and `run-system-tests` skills and the testing docs record which runner each language uses (C++ gtests via `colcon test` under the `build_packages` mark; Python via the root harness, plus `colcon test` for `ament_python` packages), and stop instructing authors to write `@pytest.mark.unit` by hand — `conftest.py` applies it by file location - Ephemeral CI GPU runners spawn via NVIDIA OSMO (not OpenStack); `system-tests.yml` / `docker-build.yml` still use `airstack-ephemeral` - Default system-test `--sim` is `isaacsim`; pass `--sim msairsim` to opt in to Microsoft AirSim - `-m build_packages` CI runs pull `cache_*` images instead of baking sim images - `docker-build.yml` retags unchanged images on VERSION bumps (content fingerprint) instead of always rebuilding; floating `cache_*` tags still seed PR layer cache -- The PR-open test run is unchanged (pytest's full defaults) and now also covers the OptiTrack Circle-trajectory e2e, which configures its own mocap-EV stack. `optitrack` joins the `heavy` mark list in `system-tests.yml`, so an optitrack run is never misclassified as colcon-only and sent down the pull-only image path +- Automatic OSMO validation runs the pull-only `build_packages` gate whenever a PR is opened, updated, or reopened; GPU-intensive simulation campaigns (including OptiTrack) are selected through `/pytest` or `workflow_dispatch` - `robot-l4t` compose service knobs are now env-overridable (`AUTONOMY_ROLE`, `FCU_URL`, and the rosbag path via `BAG_STORAGE_PATH`); `FCU_URL` unquoted so the literal serial path reaches MAVROS - `zed-l4t` image: ZED SDK 4.2 → 5.2 with the coupled ROS deps (`zed_msgs` 5.2.1, `point_cloud_transport(_plugins)` 4.x, add `backward_ros`) - Unit tests are defined by `tests/colcon_unit_test_packages.yaml`: `conftest.py` collects each listed package's co-located `test/` dir under `--import-mode=importlib` and marks it `unit` (ament lint files are skipped and run under `colcon test`) +### Removed + +- Pre-co-location unit-test scaffolding: the six per-layer stub READMEs under `tests/robot/` (which instructed authors to add tests in directories tests no longer live in) and `tests/sim/motive_emulator/README.md` (superseded by `simulation/isaac-sim/extensions/optitrack.natnet.emulator/` and `tests/integration/natnet/`) + ### Fixed +- `pytest tests/` now collects the co-located unit tests before mark filtering. The old guard skipped injection whenever any path was on the command line, and `tests/` is a path — CI collected 97 of 252 items and the Python unit tests ran nowhere. Narrowing (`pytest tests/system/test_x.py`) still skips injection; repository-root and empty-path collection are rejected +- Empty CI pytest arguments no longer become `pytest tests/ ""` and recurse through the repository; collection/import, setup/teardown, partial, and interrupted artifacts are reported as non-comparable instead of false 0% simulation-policy results, and metric regression runs only for an identical simulation campaign fingerprint - Isaac Sim image: PX4 `ubuntu.sh` no longer fails dpkg configure on the NVIDIA base (`ca-certificates` / `software-properties-common`); use `--no-nuttx --no-sim-tools` like ms-airsim - Robot image: pin `pytest<8.1` and disable `launch_testing` for colcon unit tests so ROS Jazzy's outdated pytest hook does not abort `colcon test` - Robot name resolution now honors a pre-set `ROBOT_NAME` (e.g. injected via docker compose) instead of always overriding it from the container/hostname mapping (`robot/docker/.bashrc`) diff --git a/docs/development/intermediate/testing/ci_cd.md b/docs/development/intermediate/testing/ci_cd.md index a53f5926d..3727642bd 100644 --- a/docs/development/intermediate/testing/ci_cd.md +++ b/docs/development/intermediate/testing/ci_cd.md @@ -1,11 +1,12 @@ # CI/CD Pipeline on OSMO -AirStack's continuous integration runs the **full drone stack** — simulator, -robot autonomy, and GCS — on a GPU for every change. Because that needs a -GPU, a Docker daemon, and a clean filesystem, jobs cannot run on GitHub's -hosted runners and should not run on a shared always-on machine. Instead, a -small orchestrator service watches the GitHub Actions queue and submits one -**ephemeral [NVIDIA OSMO](https://nvidia.github.io/OSMO/) pod per job**. The pod +AirStack continuous integration has two tiers. Fast Python unit and harness +contract tests run on GitHub-hosted runners for updates to PRs targeting +`main` or `develop`. Container, +ROS workspace, and selectable **full drone stack** campaigns — simulator, +robot autonomy, and GCS — run on an ephemeral GPU worker. A small orchestrator +service watches the GitHub Actions queue and submits one **ephemeral +[NVIDIA OSMO](https://nvidia.github.io/OSMO/) pod per system-test job**. The pod registers as a single-use GitHub Actions runner, executes exactly one job, and is destroyed. @@ -24,11 +25,11 @@ to fit CI into your day-to-day development loop. | Question | Answer | |---|---| -| Where do CI jobs run? | A fresh GPU pod on the OSMO `airstack` pool, one per job, destroyed after. | -| What triggers a run? | A PR being **opened**, a `/pytest` comment from a maintainer, or manual `workflow_dispatch`. | -| What gets tested? | Docker image builds, `colcon` builds, unit tests, stack liveliness, sensor rates, takeoff/hover/land, fixed-trajectory tracking. | -| How do I see results? | A metrics report comment on the PR, plus the `test-results-*` artifact (`summary.txt`, `results.xml`, `metrics.json`). | -| What fails the build? | Any failed test, **or** a metric regressing more than 20 % against the base branch's last run. | +| Where do CI jobs run? | Python unit tests: `ubuntu-latest`. Build and simulation tests: a fresh OSMO GPU pod, destroyed afterward. | +| What triggers a run? | PR open/update/reopen runs unit + package-build gates; maintainers select simulations with `/pytest`; `workflow_dispatch` is also available. | +| What gets tested? | Automatically: Python units/contracts and ROS package builds/tests. Selectably: Docker builds, liveliness, sensors, flight policies, and OptiTrack. | +| How do I see results? | Checks plus a report comment and `test-results-*` artifact (`summary.txt`, `results.xml`, `run_meta.json`, `metrics.json`). | +| What fails the build? | Any failed test, or a comparable simulation metric regressing more than 20%. Invalid/incomplete campaigns are labeled, not scored as policy failures. | | Who holds the secrets? | Only the orchestrator host. Workers get a single-use JIT token valid for one registration. | --- @@ -181,12 +182,14 @@ it to Harbor. | Trigger | When it fires | What it runs | |---|---|---| -| `pull_request` (`types: [opened]`) | Only when the PR is first opened, and only for same-repo branches | pytest's `conftest` defaults — the full mark set | +| `unit-tests.yml` pull request | PR to `main`/`develop` opened, synchronized, or reopened (including forks) | `pytest tests/ -m unit` on `ubuntu-latest` | +| `system-tests.yml` pull request | PR opened, synchronized, or reopened, same-repo branches only | `-m build_packages` on an OSMO worker | | `/pytest` PR comment | Any time, from a user with `OWNER`/`MEMBER`/`COLLABORATOR` association | Whatever args you put on the first line of the comment | | `workflow_dispatch` | Manual, from the Actions tab | The form inputs: `marks`, `sim`, `num_robots`, `stress_iterations`, `stable_duration`, `baseline_run_id` | -Pushes to an open PR deliberately do **not** re-trigger. GPU pods are a shared -resource, so re-runs are opt-in via `/pytest`. +PR pushes re-run the fast unit gate and the pull-only `build_packages` gate. +GPU-intensive simulations do **not** run automatically; select the campaign +whose policy or integration changed with `/pytest`. ### Comment syntax @@ -329,7 +332,7 @@ flowchart LR | Mark | Module | What it verifies | Bugs it is good at catching | |---|---|---|---| -| `unit` | `tests/robot/`, `tests/sim/` proxies | Hermetic Python/numpy logic co-located with each ROS 2 package | Off-by-one and boundary errors in filters, converters, validators; regressions in pure algorithm code | +| `unit` | `/test/` (co-located) | Hermetic Python/numpy logic co-located with each ROS 2 package | Off-by-one and boundary errors in filters, converters, validators; regressions in pure algorithm code | | `build_docker` | `system/test_build_docker.py` | Every image builds; records image sizes | Broken Dockerfiles, deleted apt packages, upstream base-image drift, accidental image bloat | | `build_packages` | `system/test_build_packages.py` | `colcon build` inside robot, GCS, and ms-airsim workspaces | Missing `package.xml` dependencies, uninstalled launch/config files, C++ breakage on a clean tree | | `liveliness` | `system/test_liveliness.py` | Containers reach Running, `/clock` publishes, tmux panes alive, sentinel ROS 2 nodes present, compute snapshot, stability poll | Launch files that crash on start, nodes that die after 30 s, `ROBOT_NAME`/domain-ID misconfiguration, runaway CPU or memory | @@ -382,7 +385,8 @@ Run one mark at a time unless you genuinely need both. After `run-tests` finishes — pass or fail — a `report` job on `ubuntu-latest` downloads the current artifact plus a **baseline** artifact and runs [`parse_metrics.py`](https://github.com/castacks/AirStack/blob/main/tests/parse_metrics.py) -in diff mode. +in diff mode only when both artifacts have the same complete simulation +campaign fingerprint (selected tests and parameters). | Run type | Baseline used | |---|---| @@ -390,13 +394,19 @@ in diff mode. | `workflow_dispatch` with `baseline_run_id` | That specific run | | `workflow_dispatch` without it | Latest artifact on `main` | -The comment has three sections per test module: a flat **Metrics** table, a -**Sim publishing rates** pivot (topic Hz aggregates from the `sensors` mark), -and a **Compute usage** pivot (CPU / memory / GPU per container). Regressions -are marked with a red circle, improvements with a green one, and the job -**fails** if any metric moves more than the 20 % threshold in the wrong -direction. That is the mechanism that catches slow degradation — the kind of -change where nothing throws but the tracker is quietly 30 % worse. +For a complete simulation campaign, the comment has pass rates plus a flat +**Metrics** table, a **Sim publishing rates** pivot (topic Hz aggregates from +the `sensors` mark), and a **Compute usage** pivot (CPU / memory / GPU per +container). Regressions are marked with a red circle, improvements with a +green one, and the job **fails** if any comparable metric moves more than the +20% threshold in the wrong direction. + +`run_meta.json` separates those policy results from CI failures. A collection +error, zero-test selection, internal pytest error, cancellation, or timeout is +reported as **simulation metrics are not comparable**. Pass-rate and regression +tables are suppressed in that case; the infrastructure problem cannot appear as +a false 0% policy score. A policy assertion that runs and fails remains a real +simulation result and keeps its recorded error metrics. ### The artifact @@ -406,6 +416,7 @@ change where nothing throws but the tracker is quietly 30 % worse. tests/results/2026-08-06_14-30-00/ ├── summary.txt # human-readable per-chain summary — open this first ├── results.xml # JUnit XML: durations, pass/fail per test +├── run_meta.json # completion state, pytest exit, selected/executed sim counts └── metrics.json # every recorded metric, including time series ``` @@ -442,9 +453,12 @@ flowchart TD l --> pr s --> pr a --> pr - pr --> ci["Full suite runs on the ephemeral GPU pod"] - ci --> rep["Read the metrics comment"] - rep --> iter["/pytest with a narrowed mark to confirm a fix"] + pr --> fast["unit-tests.yml on ubuntu-latest"] + pr --> ci["build_packages on an ephemeral OSMO pod"] + fast --> rep["Read automatic check results"] + ci --> rep + rep --> iter["/pytest with the relevant simulation mark"] + iter --> metrics["Read like-for-like policy metrics"] ``` Practical rules that follow from how the system is built: @@ -453,7 +467,7 @@ Practical rules that follow from how the system is built: - **Narrow before you re-run.** A `/pytest` with no args re-runs everything. `/pytest -m autonomy --sim msairsim --trajectory-types Circle` re-runs the one chain you are fixing, in a fraction of the time. - **Never trust a green launch test against a stale build.** This is why `build_packages` is auto-prepended; keep it that way when writing your own `/pytest` line. - **Read `summary.txt` before the raw log.** It groups each flight chain with per-phase wall times and status, so the failing phase is obvious without scrolling a 40-minute log. -- **Treat the metrics diff as a review artifact.** A PR that turns a metric red needs an explanation in the thread, even when every test passed. +- **Treat a like-for-like metrics diff as a review artifact.** The reporter compares only identical selected simulation campaigns; a PR that turns a metric red needs an explanation even when every assertion passed. - **Bump `VERSION` in `.env` when image content changes.** [`check-version-increment.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/check-version-increment.yml) gates the PR on a strictly-greater semver, and merging that bump is what triggers the release build below. --- @@ -486,7 +500,8 @@ covers that digest; the job re-signs the same digest under the new tags’ refs) | Workflow | Runner | Purpose | |---|---|---| -| `system-tests.yml` | Ephemeral OSMO GPU pod | Full test suite + metrics report | +| `unit-tests.yml` | `ubuntu-latest` | Python unit tests and harness contracts on updates to PRs targeting `main`/`develop` | +| `system-tests.yml` | Ephemeral OSMO GPU pod | Automatic package-build gate and selectable simulation campaigns + metrics report | | `docker-build.yml` | Ephemeral OSMO GPU pod | Retag or rebuild, push, and sign compose images | | `check-version-increment.yml` | `ubuntu-latest` | Semver gate on `.env` `VERSION=` | | `deploy_docs_from_*.yaml` | `ubuntu-latest` | Versioned MkDocs publish via `mike` | @@ -525,7 +540,8 @@ down the list. | `Cannot connect to the Docker daemon` mid-test | Pod | Inner dockerd crashed — `osmo workflow exec "$WF" runner`, then read `/var/log/dockerd.log` | | `No space left on device` | Pod | Bump `storage` in `config.yaml`; Isaac assets plus all images are large | | Runner registered, then pytest failed | Tests | A real test failure — the GitHub Actions log and `summary.txt` are canonical | -| Metrics report job failed with no test failures | Report | A metric regressed past the 20 % threshold; read the diff table | +| Report says “simulation metrics are not comparable” | Collection/infrastructure | Read the run outcome and pytest exit status in `run_meta.json`; no policy regression was scored | +| Metrics report job failed with no test failures | Report | A like-for-like metric regressed past the 20% threshold, or report generation itself failed; read the report step log | To map a GitHub job to its pod: @@ -548,6 +564,7 @@ Full runbook, including credential rotation and worker-side diagnostics: | Path | Role | |---|---| +| [`.github/workflows/unit-tests.yml`](../../../../.github/workflows/unit-tests.yml) | Fast Python unit/harness gate on GitHub-hosted runners | | [`.github/workflows/system-tests.yml`](https://github.com/castacks/AirStack/blob/main/.github/workflows/system-tests.yml) | The test workflow: triggers, arg parsing, image prep, pytest, artifact, metrics report | | [`.github/orchestrator/orchestrator.py`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/orchestrator.py) | The spawn and reap loops, GitHub polling, JIT minting, OSMO CLI plumbing | | [`.github/orchestrator/runner-workflow.yaml.j2`](https://github.com/castacks/AirStack/blob/main/.github/orchestrator/runner-workflow.yaml.j2) | Per-job OSMO workflow template | diff --git a/docs/development/intermediate/testing/index.md b/docs/development/intermediate/testing/index.md index 07c5fb887..8d35551b5 100644 --- a/docs/development/intermediate/testing/index.md +++ b/docs/development/intermediate/testing/index.md @@ -5,15 +5,15 @@ hardware requirement: | Layer | Where | Mark / Tool | Hardware | |---|---|---|---| -| **Unit tests** | `tests/robot/`, `tests/sim/` | `pytest -m unit` | None — pure Python | +| **Unit tests** | `/test/` (co-located) | `airstack test -m unit` | None — pure Python | | **Package tests** | `/test/` | `colcon test` | Robot container | | **System tests** | `tests/system/` | `pytest -m liveliness` etc. | Docker, GPU, sim license | -## Unit tests (`pytest -m unit`) +## Unit tests (`airstack test -m unit`) Fast, hermetic Python tests that run in seconds with no Docker or GPU. Test source lives **co-located with its ROS 2 package** (`/test/`); the packages with unit -tests are listed in `tests/colcon_unit_test_packages.yaml`, and `pytest tests/` collects +tests are listed in `tests/colcon_unit_test_packages.yaml`, and the root harness collects them from there. ```bash @@ -22,8 +22,9 @@ airstack test -m unit -v pytest tests/ -m unit -v ``` -Unit tests run as part of `system-tests.yml` via `pytest tests/` and can also be -run locally with no Docker or GPU needed. +Unit tests run automatically on every update to PRs targeting `main` or +`develop` through `unit-tests.yml` on `ubuntu-latest`, and can also be run +locally with no Docker or GPU needed. → **[Unit Testing Guide](unit_testing.md)** — patterns, CI workflow, how to add tests for new packages (Python and C++ gtest). @@ -87,4 +88,4 @@ airstack test -m "build_packages or autonomy" \ - [Unit Testing](unit_testing.md) — `@pytest.mark.unit`, co-located tests, CI workflow - [Testing frameworks](testing_frameworks.md) — `colcon test`, rostest patterns - [Integration testing](integration_testing.md) -- [CI/CD Pipeline on OSMO](ci_cd.md) — how CI runs the full stack on ephemeral GPU pods: architecture, triggers, what each mark catches, and the metrics regression gate +- [CI/CD Pipeline on OSMO](ci_cd.md) — automatic unit/build gates, selectable full-stack GPU campaigns, triggers, and like-for-like metrics reporting diff --git a/docs/development/intermediate/testing/unit_testing.md b/docs/development/intermediate/testing/unit_testing.md index dd2f72cf3..84423ceb0 100644 --- a/docs/development/intermediate/testing/unit_testing.md +++ b/docs/development/intermediate/testing/unit_testing.md @@ -1,12 +1,12 @@ # Unit Testing -AirStack unit tests are **fast, hermetic, and purely Python** — no Docker stack, no GPU, no running containers. They run locally in seconds and gate every pull request via a dedicated GitHub Actions workflow on a standard `ubuntu-latest` runner. +AirStack unit tests are **fast, hermetic, and purely Python** — no Docker stack, no GPU, no running containers. They run locally in seconds via `airstack test -m unit` and automatically on every update to PRs targeting `main` or `develop` through `unit-tests.yml`. ## Design principles - **Co-located with source.** Test files live in `/test/` alongside the code they test. This is the standard ROS 2 / colcon convention and ensures tests are discovered by both `colcon test` and `pytest`. - **Listed in one place.** `tests/colcon_unit_test_packages.yaml` lists which packages have unit tests. `tests/conftest.py` resolves each to its `test/` dir and collects the non-linter `test_*.py` files under `--import-mode=importlib`. To add a package's unit tests, list it in that YAML. -- **`@pytest.mark.unit` on every test.** Auto-applied by path in `conftest.py` (source files may also declare it). The `unit` mark keeps unit tests isolated from system tests that need Docker, GPUs, and sim licenses. +- **`@pytest.mark.unit` on every test, applied for you.** `conftest.py` marks items by file location, so test sources should not declare it themselves. The `unit` mark keeps unit tests isolated from system tests that need Docker, GPUs, and sim licenses. ## Repository layout @@ -35,19 +35,40 @@ Collected items point straight at the co-located source: # Locally — no container or Docker stack required airstack test -m unit -v -# Or directly with pytest (AIRSTACK_ROOT must point to the repo root) +# Or directly with pytest export AIRSTACK_ROOT=$(pwd) -pip install pytest numpy +pip install -r tests/requirements.txt pytest tests/ -m unit -v ``` -Unit tests complete in under one second for the current suite. +The current suite completes in about 20 seconds on a developer workstation. ## CI -Unit tests are collected and run as part of `system-tests.yml` via `pytest tests/` -(no marks specified on PR open = all tests including `unit`). Run them locally at -any time with no infrastructure required: +**The two languages take different runners because C++ needs a build and Python does +not.** A gtest is a binary compiled against the package's headers and rclcpp, so it only +runs where the ROS toolchain is — `colcon test` inside the robot container. Python unit +tests stub ROS at the import boundary and touch no ROS runtime, so they need neither a +build nor a container, which keeps the whole suite in the fast feedback tier. Both are +gated in CI: + +| Test | Runner | In CI via | +|---|---|---| +| C++ gtest | `colcon test` inside the robot container | the `build_packages` mark (`tests/system/test_build_packages.py::test_colcon_test_robot`) | +| Python, `ament_python` package | root harness **and** `colcon test` | `unit-tests.yml` **and** `build_packages` | +| Python, `ament_cmake` package | root harness only | `unit-tests.yml` | + +`colcon test` picks up Python tests only when the package's build type makes it: an +`ament_python` package like `lidar_point_cloud_filter` exposes them through +`setup.cfg` (`testpaths = test`), while an `ament_cmake` package like `natnet_ros2` +would need an explicit `ament_add_pytest_test` — it has none, so its Python tests reach +CI only through the root harness. + +Python unit tests are collected by `unit-tests.yml`'s `pytest tests/ -m unit` +invocation on PR open, synchronize, and reopen. That job uses GitHub-hosted +`ubuntu-latest`; it does not queue for an OSMO GPU. The OSMO `system-tests.yml` +invocation uses the same safe `tests/` collection boundary, but mark filtering may +deselect unit tests for targeted build/simulation runs. Run the same gate locally with: ```bash airstack test -m unit -v @@ -73,7 +94,6 @@ AIRSTACK_ROOT=$(pwd) pytest tests/ -m unit -v # robot/ros_ws/src///test/test_my_module.py import sys from pathlib import Path -import pytest # Make the package importable without a colcon install _src = Path(__file__).resolve().parent.parent / "src" @@ -83,11 +103,13 @@ if str(_src) not in sys.path: from my_module import my_function # noqa: E402 -@pytest.mark.unit def test_basic(): assert my_function(1, 2) == 3 ``` +No `@pytest.mark.unit` — `conftest.py` applies it by file location. Import `pytest` +only if you need its API (`approx`, `raises`, `parametrize`, `importorskip`). + If the production code inherits from `rclpy.node.Node`, stub ROS at the import boundary: @@ -117,7 +139,7 @@ sys.modules["rclpy.node"] = _rclpy_node_mod robot: packages: - # ← add here; conftest.py collects /test/test_*.py - pytest_args: "-m not linter" + pytest_args: [] # forwarded to colcon via PYTEST_ADDOPTS; `-m` is ignored there ``` That's the whole registration. If the test imports package code, set up `sys.path` at the @@ -128,7 +150,7 @@ across packages don't collide. **3. Verify:** ```bash -pytest tests/ -m unit -v +airstack test -m unit -v ``` ### C++ (gtest) @@ -180,7 +202,7 @@ sim: ``` `pytest tests/ -m unit` discovers them automatically — no changes to `pytest.ini` -or CI changes needed. +or CI needed. ## See also diff --git a/osmo/README.md b/osmo/README.md index 91b41dbd5..e3f6041bd 100644 --- a/osmo/README.md +++ b/osmo/README.md @@ -21,9 +21,9 @@ README is the **lab admin / operator** reference: pool requirements, workspace image build & push, validation stages, plus a credential summary for context. -> **Scope:** developer workflow only. CI/CD on OSMO is **not** part of this -> integration — the existing `system-tests.yml` + OpenStack orchestrator path -> is unchanged. +> **Scope:** this directory documents the interactive developer workflow. +> AirStack CI also uses OSMO, but through the separate ephemeral-runner +> orchestrator in [`.github/orchestrator/`](../.github/orchestrator/). ## Architecture in one minute @@ -296,6 +296,6 @@ If you see Isaac Sim's "Login Required" popup at startup: layout leaves room for additional workflow files when this is done. - **Persistent workspace** — mount `/root/AirStack` to a PVC so uncommitted edits survive `osmo workflow cancel`. Pool-policy dependent. -- **CI/CD on OSMO** — the existing `.github/workflows/system-tests.yml` + - OpenStack ephemeral runner path is unchanged. Migrating CI to OSMO is a - separate effort. +- **Shared interactive/CI worker design** — CI already runs on one-shot OSMO + pods through `.github/orchestrator/`; this interactive workspace intentionally + remains a separate image and lifecycle. diff --git a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md index 3fedb5ecb..cd822ac5f 100644 --- a/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md +++ b/simulation/isaac-sim/extensions/optitrack.natnet.emulator/README.md @@ -15,7 +15,7 @@ optitrack.natnet.emulator/ ├── schema/schema.usda # Typed NatNet interface attribute definitions ├── setup.py ├── docs/ # (legacy design notes — see docs/simulation/isaac_sim/natnet_emulator.md) -├── test/ # Co-located unit tests (proxied by tests/sim/) +├── test/ # Co-located unit tests (listed in colcon_unit_test_packages.yaml) └── optitrack/natnet/emulator/ ├── defaults.py # Reference Drone → prim bindings for tests ├── server/ # NatNet UDP server (transport + protocol) @@ -144,11 +144,11 @@ Full handshake layouts and sniffing workflow: [optitrack-development skill](../. | Unit | `unit` | Serializers, protocol, config, USD authoring, catalog, pose sampling, server lifecycle, scene setup | | Integration | `integration` | Host emulator → robot `natnet_ros2` pose Hz | -Co-located tests live in `test/`. Pytest discovers them via thin proxies in [`tests/sim/optitrack_natnet_emulator/`](../../../../tests/sim/optitrack_natnet_emulator/). +Co-located tests live in `test/`. The root harness collects them via the `sim:` key in [`colcon_unit_test_packages.yaml`](../../../../tests/colcon_unit_test_packages.yaml). ```bash # Unit (no Docker / no SDK) -pytest tests/sim/optitrack_natnet_emulator/ -m unit -v +airstack test -m unit -v # Integration (robot container + NatNet SDK) pytest tests/integration/natnet/ -m integration -v diff --git a/tests/README.md b/tests/README.md index 558c3fe15..306567a1a 100644 --- a/tests/README.md +++ b/tests/README.md @@ -117,7 +117,7 @@ Writes custom metrics to `tests/results//metrics.json` after each `re ### Output files Every test run produces a timestamped directory containing only `summary.txt`, -`results.xml`, and `metrics.json` — there is **no** `logs/` subdirectory and no +`results.xml`, `run_meta.json`, and `metrics.json` — there is **no** `logs/` subdirectory and no per-test log files are written under the run directory. ``` @@ -125,6 +125,7 @@ tests/results/ └── 2025-04-21_14-30-00/ ├── summary.txt # Human-readable key metrics — open this first ├── results.xml # JUnit XML — test durations and pass/fail status + ├── run_meta.json # Completion/outcome and campaign fingerprint └── metrics.json # Custom metrics (image sizes, Hz, compute, timing) ``` @@ -517,13 +518,17 @@ python tests/parse_metrics.py \ Prints a side-by-side comparison. Exits **1** if any metric regresses beyond the threshold; exits 0 otherwise. -The report has three sections per test module: +For a completed test campaign, the report has three sections per test module: - **Metrics** — flat table of scalar metrics (test name, metric key, value/baseline, change%) - **Sim publishing rates** — pivot table of topic Hz aggregates from the `sensors` mark (`mean`, `start_mean`, `end_mean`, `min`, `max`; sim + robot topics) - **Compute usage** — pivot table of CPU/memory/GPU metrics per container Regressions are flagged with :red_circle:, improvements with :green_circle:. +Collection errors, command/internal errors, zero-test runs, and jobs that stop before +pytest finalizes are labeled **not comparable**. Their pass-rate and regression tables +are suppressed so an infrastructure failure cannot appear as 0% policy performance. +`run_meta.json` records the pytest exit status and simulation tests selected/completed. --- @@ -534,19 +539,25 @@ Regressions are flagged with :red_circle:, improvements with :green_circle:. reference, what each mark catches, and how to fold CI into your development loop — see **[CI/CD Pipeline on OSMO](../docs/development/intermediate/testing/ci_cd.md)**. -### Workflow: `system-tests.yml` +### CI workflows -[`.github/workflows/system-tests.yml`](../../../../.github/workflows/system-tests.yml) runs on: +[`.github/workflows/unit-tests.yml`](../.github/workflows/unit-tests.yml) +runs all Python unit and harness-contract tests on `ubuntu-latest` whenever a PR is +opened, updated, or reopened against `main` or `develop`. -- **Pull requests** to `main` or `develop` — automatically runs `build_docker or build_packages` tests (no GPU-intensive liveliness run on every PR) +[`.github/workflows/system-tests.yml`](../.github/workflows/system-tests.yml) runs on: + +- **Same-repository pull requests** when opened, updated, or reopened — automatically + runs `build_packages` on OSMO (no GPU-intensive simulation campaign on every push) +- **`/pytest` PR comments** from maintainers — runs the requested registered marks - **Manual dispatch** (`workflow_dispatch`) — fully configurable for liveliness runs and metric comparisons #### Manual dispatch inputs | Input | Default | Description | |-------|---------|-------------| -| `marks` | `liveliness` | pytest marks expression | -| `sim` | `msairsim` | Sim targets | +| `marks` | `liveliness or takeoff_hover_land` | pytest marks expression | +| `sim` | `isaacsim` | Sim targets | | `num_robots` | `1` | Robot counts | | `stress_iterations` | `1` | Iterations per config | | `stable_duration` | `120` | Stability polling seconds | @@ -560,9 +571,9 @@ Regressions are flagged with :red_circle:, improvements with :green_circle:. 1. Downloads the current artifact 2. Downloads a baseline artifact (from the base branch for PRs, from `main` for manual runs, or from the specified `baseline_run_id`) -3. Runs `parse_metrics.py` in diff mode if a baseline is found, otherwise in single-run mode +3. Runs `parse_metrics.py` in diff mode only when both artifacts have the same complete simulation campaign fingerprint; otherwise reports the current run without comparison 4. Posts the markdown report as a PR comment (PR runs) or to the job summary (all runs) -5. Fails with `::error::` if `parse_metrics.py` exits 1 (regression detected) +5. Fails with `::error::` only for a comparable metric regression; invalid/incomplete campaigns are reported as infrastructure outcomes #### Required third-party action @@ -610,7 +621,7 @@ AirStack's tests require a GPU, Docker, and a clean filesystem per run, so they ### Setup -The orchestrator service code, OSMO runner-workflow template, runner image, systemd unit, and full setup runbook live in [`.github/orchestrator/`](../../../../.github/orchestrator/). See [`.github/orchestrator/README.md`](ci-cd-orchestrator.md) for: +The orchestrator service code, OSMO runner-workflow template, runner image, systemd unit, and full setup runbook live in [`.github/orchestrator/`](../.github/orchestrator/). See [`.github/orchestrator/README.md`](ci-cd-orchestrator.md) for: - obtaining the OSMO service-account token and a dedicated CI GPU pool (with privileged mode enabled) - building and pushing the runner image (`runner.Dockerfile`) diff --git a/tests/conftest.py b/tests/conftest.py index 29a7c7220..38aec5c5f 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -11,12 +11,13 @@ if _TESTS_DIR not in sys.path: sys.path.insert(0, _TESTS_DIR) -from harness import collection, session +from harness import collection, session as harness_session # Re-export the harness helper API so existing `from conftest import ` in the # system tests + sensor_probes keeps working unchanged. from harness import * # noqa: F401,F403 from harness.commands import _nodeid_dotted from harness.discovery import _is_unit_item +from harness.run_meta import write_run_meta # ── pytest config / hooks ────────────────────────────────────────────────── @@ -74,7 +75,7 @@ def pytest_addoption(parser): def pytest_configure(config): - run_dir = session.init_run_dir(AIRSTACK_ROOT) + run_dir = harness_session.init_run_dir(AIRSTACK_ROOT) config.option.xmlpath = str(run_dir / "results.xml") # Co-located unit tests import their own package (e.g. `optitrack.natnet.emulator`, @@ -91,12 +92,15 @@ def pytest_configure(config): if str(root) not in sys.path: sys.path.insert(0, str(root)) - # Collect co-located unit tests: their files live outside tests/, so add the - # explicit non-linter test files to the collection args. Skip when an explicit - # path was given on the CLI (args_source == ARGS) so `pytest tests/system/foo.py` - # still narrows as expected. + # Collect co-located unit tests: their files live outside tests/, so pytest never + # reaches them by recursion — append the non-linter test files explicitly. Only for + # a run that means "everything": `pytest tests/system/foo.py` must still narrow. + # See harness.discovery.collection_is_broad. src_name = getattr(getattr(config, "args_source", None), "name", "TESTPATHS") - if src_name != "ARGS": + config.airstack_unit_tests_injected = src_name != "ARGS" or collection_is_broad( + config.args, config.invocation_params.dir + ) + if config.airstack_unit_tests_injected: for f in unit_test_files(): entry = str(f) if entry not in config.args: @@ -113,18 +117,36 @@ def pytest_itemcollected(item): def pytest_runtest_setup(item): - session.set_current_item(item) + harness_session.set_current_item(item) def pytest_runtest_teardown(item): - session.set_current_item(None) + harness_session.set_current_item(None) -def pytest_sessionfinish(exitstatus): - """Write summary.txt with key metrics so users don't need to dig through logs.""" - run_dir = session.run_dir() +@pytest.hookimpl(trylast=True) +def pytest_sessionfinish(session, exitstatus): + """Persist run outcome metadata and a human-readable summary.""" + run_dir = harness_session.run_dir() if run_dir is None: return + try: + terminal = session.config.pluginmanager.getplugin("terminalreporter") + reports = [ + report + for entries in getattr(terminal, "stats", {}).values() + for report in entries + ] + meta_path = write_run_meta( + run_dir, + session.items, + exitstatus, + session.config.option.markexpr, + reports, + ) + logger.info("Wrote run metadata to %s", meta_path) + except Exception as exc: + logger.warning("Failed to write run metadata: %s", exc) try: from run_summary import write_summary summary_path = write_summary(run_dir) @@ -183,7 +205,7 @@ def airstack_env(request): # test id (see pytest_collection_modifyitems), so airstack up/down output # lands next to the triggering test's own log instead of under pytest's # stale callspec.id. - log = f"airstack_env.{_nodeid_dotted(session.current_item().nodeid, with_path_sep=True)}" + log = f"airstack_env.{_nodeid_dotted(harness_session.current_item().nodeid, with_path_sep=True)}" headless = not request.config.getoption("--gui") env_overrides = { diff --git a/tests/harness/__init__.py b/tests/harness/__init__.py index d3fcf4e32..7b1210a7c 100644 --- a/tests/harness/__init__.py +++ b/tests/harness/__init__.py @@ -25,7 +25,9 @@ from harness.discovery import ( AIRSTACK_ROOT, COLCON_UNIT_TEST_PACKAGES_YAML, + TESTS_DIR, colcon_test_robot_command, + collection_is_broad, format_pytest_addopts, load_colcon_unit_test_config, repo_path, @@ -44,9 +46,9 @@ __all__ = [ # discovery - "AIRSTACK_ROOT", "COLCON_UNIT_TEST_PACKAGES_YAML", "repo_path", - "colcon_test_robot_command", "format_pytest_addopts", "load_colcon_unit_test_config", - "unit_test_dirs", "unit_test_files", + "AIRSTACK_ROOT", "COLCON_UNIT_TEST_PACKAGES_YAML", "TESTS_DIR", "repo_path", + "colcon_test_robot_command", "collection_is_broad", "format_pytest_addopts", + "load_colcon_unit_test_config", "unit_test_dirs", "unit_test_files", # session "logger", # commands diff --git a/tests/harness/collection.py b/tests/harness/collection.py index ae78ed94c..f0eada6ba 100644 --- a/tests/harness/collection.py +++ b/tests/harness/collection.py @@ -14,6 +14,10 @@ # Unit tests first — fast, hermetic, no Docker. Co-located package unit tests # (see unit_test_files) sort into this leading slot via the path check below. "__unit__", + # Harness contract tests: hermetic, and they guard the collection of everything + # above, so they belong with the fast tier rather than after the sim suites. + "test_collection_contract", + "test_metrics_reporting_contract", # System tests follow in dependency order. "system.test_build_docker", "system.test_build_packages", diff --git a/tests/harness/discovery.py b/tests/harness/discovery.py index 28e03737a..b54fb5644 100644 --- a/tests/harness/discovery.py +++ b/tests/harness/discovery.py @@ -1,8 +1,9 @@ """Unit-test discovery: which packages have unit tests and where their files live. Driven by ``tests/colcon_unit_test_packages.yaml``. ``conftest.pytest_configure`` adds -``unit_test_files()`` to the pytest run, and ``pytest_itemcollected`` marks each of those -items ``unit`` via ``_is_unit_item``. ament lint tests are excluded here — they run under +``unit_test_files()`` to the pytest run whenever ``collection_is_broad`` says the command +line did not narrow the run, and ``pytest_itemcollected`` marks each of those items +``unit`` via ``_is_unit_item``. ament lint tests are excluded here — they run under ``colcon test`` (linter skip is in package pytest config; see ``colcon_test_robot_command``). """ @@ -17,6 +18,10 @@ Path(AIRSTACK_ROOT) / "tests" / "colcon_unit_test_packages.yaml" ) +# The tests/ tree, derived from this file rather than AIRSTACK_ROOT so the guard always +# agrees with the conftest that is actually running. +TESTS_DIR = Path(__file__).resolve().parents[1] + def repo_path(*parts: str) -> Path: """Resolve a path relative to the repo root (``AIRSTACK_ROOT``). @@ -147,3 +152,41 @@ def unit_test_files(): if f.name not in _LINTER_TEST_FILENAMES: files.append(f) return files + + +def _arg_path(arg, invocation_dir): + """Absolute path addressed by one pytest positional. + + Positionals are raw CLI strings and may be node ids + (``system/test_x.py::TestY::test_z``); only the part before ``::`` addresses the + filesystem. The path need not exist — pytest reports bad paths itself. + """ + return Path(invocation_dir, str(arg).split("::", 1)[0]).resolve() + + +def collection_is_broad(args, invocation_dir, tests_root=None) -> bool: + """True only when the positional names the complete ``tests/`` harness. + + Co-located unit tests live outside ``tests/``, so ``pytest_configure`` appends them + to ``config.args`` by hand. It must do that only for a run that already means + "everything", or ``pytest tests/system/test_x.py`` would drag in every unit test. + + Repository-root collection is intentionally *not* broad: importing every + ``test_*.py`` under ROS, Isaac Sim, and vendored submodules on the host is invalid. + + pytest (testpaths ``.``, cwd tests/) -> broad + pytest tests/ (CI, and the documented commands) -> broad + pytest . (cwd repo root) -> invalid/narrow + pytest tests/system -> narrow + pytest tests/system/test_x.py::TestY::test_z -> narrow + pytest ../simulation/.../test/test_frames.py -> narrow + + Exactly one non-empty positional is required so an accidental empty argument + cannot silently add the repository root to pytest's recursion. + """ + root = Path(tests_root or TESTS_DIR).resolve() + invocation_dir = Path(invocation_dir).resolve() + positionals = [str(arg) for arg in args if not str(arg).startswith("-")] + if len(positionals) != 1 or not positionals[0]: + return False + return _arg_path(positionals[0], invocation_dir) == root diff --git a/tests/harness/run_meta.py b/tests/harness/run_meta.py new file mode 100644 index 000000000..8cd0821a9 --- /dev/null +++ b/tests/harness/run_meta.py @@ -0,0 +1,310 @@ +"""Run-level outcome metadata for honest CI and metrics reporting.""" + +from __future__ import annotations + +import hashlib +import json +import xml.etree.ElementTree as ET +from pathlib import Path + +from harness.test_ids import canonical_test_id + + +RUN_META_FILENAME = "run_meta.json" + +SIMULATION_MODULES = ( + "system.test_liveliness.", + "system.test_sensors.", + "system.test_takeoff_hover_land.", + "system.test_fixed_trajectory.", + "system.test_waypoint_flight.", + "system.test_optitrack_e2e.", +) + + +def is_simulation_test_id(test_id: str) -> bool: + """Whether a test belongs to a GPU/simulation campaign.""" + canonical = canonical_test_id(test_id) + return canonical.startswith(SIMULATION_MODULES) + + +def campaign_fingerprint(test_ids) -> str: + """Stable identity for the exact selected simulation campaign.""" + canonical_ids = sorted( + canonical_test_id(str(test_id).replace("::", ".")).replace(".py.", ".") + for test_id in test_ids + ) + if not canonical_ids: + return "" + payload = "\n".join(canonical_ids).encode() + return hashlib.sha256(payload).hexdigest() + + +def _item_outcome(item) -> str | None: + """Return the final outcome recorded on a pytest item.""" + reports = [ + getattr(item, "_rep_setup", None), + getattr(item, "_rep_call", None), + getattr(item, "_rep_teardown", None), + ] + if any(rep is not None and rep.failed for rep in reports): + return "failed" + if any(rep is not None and rep.skipped for rep in reports): + return "skipped" + call = getattr(item, "_rep_call", None) + if call is not None and call.passed: + return "passed" + return None + + +def _report_details(reports) -> tuple[dict[str, str], set[str], set[str]]: + """Collapse phase reports and identify items that reached call phase.""" + priority = {"passed": 0, "skipped": 1, "failed": 2} + outcomes = {} + call_nodeids = set() + infrastructure_error_nodeids = set() + for report in reports or []: + nodeid = getattr(report, "nodeid", None) + when = getattr(report, "when", None) + if not nodeid or when not in ("setup", "call", "teardown"): + continue + if report.failed: + outcome = "failed" + if when != "call": + infrastructure_error_nodeids.add(nodeid) + elif report.skipped: + outcome = "skipped" + elif when == "call" and report.passed: + outcome = "passed" + else: + continue + if when == "call": + call_nodeids.add(nodeid) + previous = outcomes.get(nodeid, "passed") + outcomes[nodeid] = max((previous, outcome), key=priority.get) + return outcomes, call_nodeids, infrastructure_error_nodeids + + +def build_run_meta(items, exitstatus: int, mark_expression: str = "", + reports=None) -> dict: + """Build serializable run metadata from a completed pytest session.""" + report_outcomes, call_nodeids, infrastructure_error_nodeids = _report_details( + reports + ) + if report_outcomes: + completed_by_id = report_outcomes + else: + completed_by_id = { + str(item.nodeid): outcome + for item in items + if (outcome := _item_outcome(item)) is not None + } + call_nodeids = { + str(item.nodeid) + for item in items + if getattr(item, "_rep_call", None) is not None + } + infrastructure_error_nodeids = { + str(item.nodeid) + for item in items + if any( + report is not None and report.failed + for report in ( + getattr(item, "_rep_setup", None), + getattr(item, "_rep_teardown", None), + ) + ) + } + completed = list(completed_by_id.values()) + simulation_items = [ + item for item in items if is_simulation_test_id(str(item.nodeid)) + ] + simulation_completed = [ + item for item in simulation_items if str(item.nodeid) in call_nodeids + ] + simulation_infrastructure_errors = [ + item + for item in simulation_items + if str(item.nodeid) in infrastructure_error_nodeids + ] + + if exitstatus == 2: + # Pytest uses exit 2 for both collection aborts and user/runner + # interruption. Reports prove that execution had already begun. + outcome = "incomplete" if completed else "collection_error" + elif exitstatus in (3, 4): + outcome = "internal_error" + elif exitstatus == 5 or not items: + outcome = "no_tests" + elif not call_nodeids: + outcome = ( + "simulation_not_executed" if simulation_items + else "tests_not_executed" + ) + elif simulation_items and not simulation_completed: + outcome = "simulation_not_executed" + elif simulation_infrastructure_errors: + outcome = "incomplete" + elif len(simulation_completed) != len(simulation_items): + outcome = "incomplete" + elif simulation_items: + outcome = "simulation" + else: + outcome = "non_simulation" + + return { + "schema_version": 1, + "complete": outcome != "incomplete", + "outcome": outcome, + "pytest_exitstatus": int(exitstatus), + "mark_expression": mark_expression, + "selected_tests": len(items), + "completed_tests": len(completed), + "passed": completed.count("passed"), + "failed": completed.count("failed"), + "skipped": completed.count("skipped"), + "simulation_selected": len(simulation_items), + "simulation_completed": len(simulation_completed), + "campaign_fingerprint": campaign_fingerprint( + item.nodeid for item in simulation_items + ), + } + + +def write_run_meta(run_dir: Path, items, exitstatus: int, + mark_expression: str = "", reports=None) -> Path: + """Write ``run_meta.json`` for a normally completed pytest session.""" + path = Path(run_dir) / RUN_META_FILENAME + path.write_text(json.dumps( + build_run_meta(items, exitstatus, mark_expression, reports), + indent=2, + sort_keys=True, + ) + "\n") + return path + + +def _classify_junit(results_xml: Path) -> dict: + """Infer legacy run state when ``run_meta.json`` is unavailable.""" + cases = list(ET.parse(results_xml).iter("testcase")) + errors = sum(tc.find("error") is not None for tc in cases) + failures = sum(tc.find("failure") is not None for tc in cases) + skipped = sum(tc.find("skipped") is not None for tc in cases) + simulation = sum( + is_simulation_test_id(f"{tc.get('classname')}.{tc.get('name')}") + for tc in cases + ) + simulation_ids = [ + f"{tc.get('classname')}.{tc.get('name')}" + for tc in cases + if is_simulation_test_id(f"{tc.get('classname')}.{tc.get('name')}") + ] + simulation_errors = sum( + tc.find("error") is not None + for tc in cases + if is_simulation_test_id(f"{tc.get('classname')}.{tc.get('name')}") + ) + + if errors: + outcome = "incomplete" if simulation else "collection_error" + elif not cases: + outcome = "no_tests" + elif simulation: + outcome = "simulation" + else: + outcome = "non_simulation" + + return { + "schema_version": 1, + "complete": outcome != "incomplete", + "outcome": outcome, + "pytest_exitstatus": None, + "mark_expression": "", + "selected_tests": len(cases), + "completed_tests": len(cases), + "passed": len(cases) - errors - failures - skipped, + "failed": errors + failures, + "skipped": skipped, + "simulation_selected": simulation, + "simulation_completed": simulation - simulation_errors, + "campaign_fingerprint": campaign_fingerprint(simulation_ids), + "inferred": True, + } + + +def classify_run(run_dir: Path) -> dict: + """Read run metadata or infer whether an artifact is comparable.""" + run_dir = Path(run_dir) + meta_path = run_dir / RUN_META_FILENAME + if meta_path.exists(): + try: + meta = json.loads(meta_path.read_text()) + except (OSError, json.JSONDecodeError) as exc: + return { + "schema_version": 1, + "complete": False, + "outcome": "incomplete", + "reason": f"Run metadata could not be read: {exc}", + } + if meta.get("outcome") in ("simulation", "non_simulation"): + results_xml = run_dir / "results.xml" + if not results_xml.exists(): + return { + "schema_version": 1, + "complete": False, + "outcome": "incomplete", + "reason": "Run metadata exists but JUnit results are missing.", + } + try: + ET.parse(results_xml) + except (OSError, ET.ParseError) as exc: + return { + "schema_version": 1, + "complete": False, + "outcome": "incomplete", + "reason": f"JUnit results were not finalized: {exc}", + } + return meta + + results_xml = run_dir / "results.xml" + if results_xml.exists(): + try: + return _classify_junit(results_xml) + except (OSError, ET.ParseError) as exc: + return { + "schema_version": 1, + "complete": False, + "outcome": "incomplete", + "reason": f"JUnit results were not finalized: {exc}", + } + + if (run_dir / "metrics.json").exists(): + return { + "schema_version": 1, + "complete": False, + "outcome": "incomplete", + "reason": "Metrics exist but pytest did not finalize JUnit/run metadata.", + } + + return { + "schema_version": 1, + "complete": False, + "outcome": "missing_results", + "reason": "No pytest result artifact was produced.", + } + + +def simulation_metrics_comparable(meta: dict, baseline: dict | None = None) -> bool: + """Whether a complete simulation may be compared with a like campaign.""" + valid = bool( + meta + and meta.get("complete") + and meta.get("outcome") == "simulation" + and meta.get("campaign_fingerprint") + ) + if not valid or baseline is None: + return valid + return bool( + baseline.get("complete") + and baseline.get("outcome") == "simulation" + and baseline.get("campaign_fingerprint") == meta["campaign_fingerprint"] + ) diff --git a/tests/harness/test_ids.py b/tests/harness/test_ids.py new file mode 100644 index 000000000..5b0fd825c --- /dev/null +++ b/tests/harness/test_ids.py @@ -0,0 +1,15 @@ +"""Canonical test identifiers shared by metrics and summary reporting.""" + + +def canonical_test_id(name: str) -> str: + """Unify pytest node-id path slashes with JUnit classname dots. + + ``metrics.json`` keys start with paths such as + ``system/test_liveliness.Class.test`` while JUnit uses + ``system.test_liveliness.Class.test``. + """ + head, dot, rest = name.partition(".") + if "/" in head: + head = head.replace("/", ".") + return head + dot + rest if dot else head + return name diff --git a/tests/integration/natnet/README.md b/tests/integration/natnet/README.md index db4617312..f373e7d85 100644 --- a/tests/integration/natnet/README.md +++ b/tests/integration/natnet/README.md @@ -147,5 +147,5 @@ docker exec airstack-robot-desktop-1 bash -lc 'bws --packages-select natnet_ros2 Unit tests (protocol, serializers, Isaac wrapper loopback): ```bash -pytest tests/sim/optitrack_natnet_emulator/ -m unit -v +airstack test -m unit -v ``` diff --git a/tests/meta/test_collection_contract.py b/tests/meta/test_collection_contract.py new file mode 100644 index 000000000..998de89a4 --- /dev/null +++ b/tests/meta/test_collection_contract.py @@ -0,0 +1,150 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Contract tests for co-located unit-test collection. + +Unit-test source lives outside ``tests/``, so ``conftest.pytest_configure`` appends it to +the collection args when ``collection_is_broad`` says the command line did not narrow the +run. Get that wrong in the permissive direction and a narrowed run drags in every unit +test; get it wrong the other way and CI silently runs none of them. + +These live under ``tests/`` on purpose. Co-located, they would stop being collected at +the same moment they stopped guarding anything — here plain recursion finds them, so a +broken guard makes them run and fail. +""" +import re +from pathlib import Path + +import pytest + +from harness.discovery import ( # noqa: E402 — pytest adds tests/ to sys.path + TESTS_DIR, + collection_is_broad, + repo_path, + unit_test_files, +) + +# Not co-located, so `_is_unit_item` will not mark it — the one place the mark is +# written by hand. +pytestmark = pytest.mark.unit + +_REPO = TESTS_DIR.parent + + +@pytest.mark.parametrize( + "cwd, args", + [ + (_REPO, ["tests/"]), # CI, and the documented commands + (_REPO, ["tests"]), + (_REPO, ["./tests/"]), + (_REPO, [str(TESTS_DIR)]), + (TESTS_DIR, ["."]), # testpaths, i.e. `airstack test` + (TESTS_DIR, [str(TESTS_DIR)]), + ], +) +def test_broad_invocations_collect_unit_tests(cwd, args): + assert collection_is_broad(args, cwd) is True + + +@pytest.mark.parametrize( + "cwd, args", + [ + (TESTS_DIR, ["system"]), + (TESTS_DIR, ["system/test_liveliness.py"]), + (TESTS_DIR, ["system/test_liveliness.py::TestLiveliness::test_x"]), + (_REPO, ["tests/system/test_sensors.py"]), + (_REPO, ["tests/integration/natnet"]), + (_REPO, ["simulation/isaac-sim/extensions/optitrack.natnet.emulator/test/test_frames.py"]), + (_REPO, ["."]), # never recurse over the repo on host + (_REPO, [""]), + (_REPO, ["tests/", ""]), + (_REPO, ["tests/", "tests/system"]), + (_REPO, []), + ], +) +def test_narrowed_invocations_do_not(cwd, args): + assert collection_is_broad(args, cwd) is False + + +def test_ci_invocation_is_broad(): + """The shared system harness path must permit co-located injection. + + This catches the original bug: CI ran `pytest tests/`, which the guard classified + as narrowed. The CPU workflow executes the injected tests with ``-m unit``; + mark-scoped system runs may intentionally deselect them after safe collection. + """ + workflow = repo_path(".github", "workflows", "system-tests.yml").read_text() + match = re.search(r"^\s*pytest\s+(\S+)", workflow, re.M) + assert match, "no `pytest ` invocation found in system-tests.yml" + assert collection_is_broad([match.group(1)], _REPO), ( + f"system-tests.yml runs `pytest {match.group(1)}`, which does not collect " + "co-located unit tests" + ) + + +def test_ci_empty_args_do_not_emit_an_empty_positional(): + """Guard the mapfile bug that changed bare `/pytest` into `pytest tests/ ""`.""" + workflow = repo_path(".github", "workflows", "system-tests.yml").read_text() + assert "sys.stdout.write" in workflow + assert "print('\\\\n'.join(shlex.split" not in workflow + assert "Refusing an empty pytest argument" in workflow + + +def test_cpu_unit_workflow_uses_the_broad_harness_path(): + workflow = repo_path(".github", "workflows", "unit-tests.yml").read_text() + match = re.search(r"^\s*run:\s+pytest\s+(\S+)", workflow, re.M) + assert match, "no `pytest ` invocation found in unit-tests.yml" + assert collection_is_broad([match.group(1)], _REPO) + + +def test_automatic_pr_gates_are_fast_and_repeat_on_updates(): + system = repo_path(".github", "workflows", "system-tests.yml").read_text() + unit = repo_path(".github", "workflows", "unit-tests.yml").read_text() + trigger = "types: [opened, synchronize, reopened]" + assert trigger in system + assert trigger in unit + assert "args = ['-m', 'build_packages']" in system + assert "runs-on: ubuntu-latest" in unit + assert "run: pytest tests/ -m unit" in unit + + +def test_report_uses_the_revision_that_was_actually_tested(): + workflow = repo_path(".github", "workflows", "system-tests.yml").read_text() + assert "tested_sha: ${{ steps.identity.outputs.tested_sha }}" in workflow + assert "test-results-${{ steps.identity.outputs.tested_sha }}" in workflow + assert "test-results-${{ needs.run-tests.outputs.tested_sha }}" in workflow + assert "Number('${{ needs.run-tests.outputs.pr_number }}')" in workflow + + +def test_pr_head_check_is_finalized_after_metrics(): + workflow = repo_path(".github", "workflows", "system-tests.yml").read_text() + assert workflow.index("- name: Finalize check on PR head") > workflow.index( + "- name: Fail on regression" + ) + assert "ref: ${{ needs.run-tests.outputs.tested_sha }}" in workflow + assert "conclusion: '${{ job.status }}'" not in workflow + + +def test_cross_run_baseline_uses_supported_download_inputs(): + workflow = repo_path(".github", "workflows", "system-tests.yml").read_text() + explicit = workflow.split( + "- name: Download baseline results (manual, explicit run ID)", 1 + )[1].split("- name:", 1)[0] + assert "github-token:" in explicit + assert 'pattern: "test-results-*"' in explicit + assert "name_is_regexp:" not in explicit + + +def test_injection_actually_produced_items(request): + """Every discovered unit-test file contributed at least one collected item. + + Catches breakage below the guard — YAML drift, a glob change, an import error that + turns a module into a collection error rather than tests. + """ + if not getattr(request.config, "airstack_unit_tests_injected", False): + pytest.skip("narrowed run — co-located tests are not injected by design") + + collected = {Path(str(item.path)).resolve() for item in request.session.items} + missing = [f for f in unit_test_files() if f.resolve() not in collected] + assert not missing, "discovered but not collected: " + ", ".join( + str(f.relative_to(_REPO)) for f in missing + ) diff --git a/tests/meta/test_metrics_reporting_contract.py b/tests/meta/test_metrics_reporting_contract.py new file mode 100644 index 000000000..f016ec71f --- /dev/null +++ b/tests/meta/test_metrics_reporting_contract.py @@ -0,0 +1,263 @@ +# Copyright (c) 2024 Carnegie Mellon University +# MIT License - see LICENSE in the repository root for full text. +"""Contracts that keep infrastructure failures out of simulation metrics.""" + +import json +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from harness.run_meta import ( + build_run_meta, + campaign_fingerprint, + classify_run, + simulation_metrics_comparable, +) +from parse_metrics import generate_report, merge_metrics + + +pytestmark = pytest.mark.unit + + +def _write_junit(run_dir: Path, testcase: str) -> None: + run_dir.mkdir() + (run_dir / "results.xml").write_text( + '' + f'{testcase}' + ) + + +def _item(nodeid: str, *, failed=False, skipped=False): + report = SimpleNamespace( + failed=failed, + skipped=skipped, + passed=not failed and not skipped, + ) + return SimpleNamespace( + nodeid=nodeid, + _rep_setup=SimpleNamespace(failed=False, skipped=False, passed=True), + _rep_call=report, + _rep_teardown=SimpleNamespace(failed=False, skipped=False, passed=True), + ) + + +def _setup_failed_item(nodeid: str): + return SimpleNamespace( + nodeid=nodeid, + _rep_setup=SimpleNamespace(failed=True, skipped=False, passed=False), + _rep_call=None, + _rep_teardown=None, + ) + + +def _phase_report(nodeid: str, when: str, outcome: str): + return SimpleNamespace( + nodeid=nodeid, + when=when, + failed=outcome == "failed", + skipped=outcome == "skipped", + passed=outcome == "passed", + ) + + +def test_completed_simulation_failure_is_a_valid_campaign(): + meta = build_run_meta( + [_item( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_circle", + failed=True, + )], + exitstatus=1, + mark_expression="autonomy", + ) + assert meta["outcome"] == "simulation" + assert meta["simulation_completed"] == 1 + assert meta["failed"] == 1 + + +def test_collection_exit_is_not_simulation_performance(): + meta = build_run_meta([], exitstatus=2, mark_expression="optitrack") + assert meta["outcome"] == "collection_error" + assert meta["simulation_completed"] == 0 + + +def test_interrupted_partial_simulation_is_not_comparable(): + meta = build_run_meta( + [_item( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_takeoff", + )], + exitstatus=2, + mark_expression="autonomy", + ) + assert meta["outcome"] == "incomplete" + assert meta["complete"] is False + + +def test_setup_only_failure_is_not_policy_performance(): + nodeid = ( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_takeoff" + ) + meta = build_run_meta( + [_setup_failed_item(nodeid)], + exitstatus=1, + mark_expression="autonomy", + reports=[_phase_report(nodeid, "setup", "failed")], + ) + assert meta["outcome"] == "simulation_not_executed" + assert meta["simulation_completed"] == 0 + + +def test_fail_fast_partial_campaign_is_not_comparable(): + first = ( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_takeoff" + ) + meta = build_run_meta( + [ + _item(first, failed=True), + SimpleNamespace( + nodeid=( + "system/test_fixed_trajectory.py::" + "TestFixedTrajectory::test_circle" + ), + _rep_setup=None, + _rep_call=None, + _rep_teardown=None, + ), + ], + exitstatus=1, + mark_expression="autonomy", + reports=[_phase_report(first, "call", "failed")], + ) + assert meta["outcome"] == "incomplete" + assert meta["simulation_completed"] == 1 + assert meta["simulation_selected"] == 2 + + +def test_teardown_error_makes_campaign_incomplete(): + nodeid = ( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_circle" + ) + meta = build_run_meta( + [_item(nodeid)], + exitstatus=1, + mark_expression="autonomy", + reports=[ + _phase_report(nodeid, "call", "passed"), + _phase_report(nodeid, "teardown", "failed"), + ], + ) + assert meta["outcome"] == "incomplete" + assert meta["complete"] is False + + +def test_only_identical_campaigns_are_comparable(): + current = build_run_meta( + [_item( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_circle[a]", + )], + exitstatus=0, + mark_expression="autonomy", + ) + same = dict(current) + different = dict(current, campaign_fingerprint="different") + assert simulation_metrics_comparable(current, same) + assert not simulation_metrics_comparable(current, different) + + +def test_campaign_fingerprint_matches_pytest_and_junit_ids(): + pytest_id = ( + "system/test_fixed_trajectory.py::TestFixedTrajectory::test_circle[a]" + ) + junit_id = ( + "system.test_fixed_trajectory.TestFixedTrajectory.test_circle[a]" + ) + assert campaign_fingerprint([pytest_id]) == campaign_fingerprint([junit_id]) + + +def test_collection_error_report_suppresses_pass_rates(tmp_path): + run_dir = tmp_path / "collection-error" + _write_junit( + run_dir, + '' + "", + ) + + assert classify_run(run_dir)["outcome"] == "collection_error" + markdown, regressed = generate_report(run_dir) + assert "Simulation metrics are not comparable" in markdown + assert "Pass rates" not in markdown + assert regressed is False + + +def test_sim_setup_error_report_is_not_policy_performance(tmp_path): + run_dir = tmp_path / "setup-error" + _write_junit( + run_dir, + '' + '', + ) + + markdown, regressed = generate_report(run_dir) + assert "Simulation metrics are not comparable" in markdown + assert "Pass rates" not in markdown + assert regressed is False + + +def test_incomplete_artifact_is_not_simulation_performance(tmp_path): + run_dir = tmp_path / "incomplete" + run_dir.mkdir() + (run_dir / "metrics.json").write_text("{}") + + markdown, regressed = generate_report(run_dir) + assert "timeout or cancellation" in markdown + assert "Pass rates" not in markdown + assert regressed is False + + +def test_truncated_junit_is_not_simulation_performance(tmp_path): + run_dir = tmp_path / "truncated" + run_dir.mkdir() + (run_dir / "results.xml").write_text("") + + markdown, regressed = generate_report(run_dir) + assert "Simulation metrics are not comparable" in markdown + assert "Pass rates" not in markdown + assert regressed is False + + +def test_real_sim_failure_keeps_metrics_and_pass_rate(tmp_path): + run_dir = tmp_path / "sim-failure" + _write_junit( + run_dir, + '' + '', + ) + (run_dir / "metrics.json").write_text(json.dumps({ + "system/test_fixed_trajectory.TestFixedTrajectory." + "test_circle[isaacsim-iter1]": { + "cross_track_error_mean_m": { + "value": 4.2, + "unit": "m", + "direction": "lower_is_better", + }, + }, + })) + + merged = merge_metrics(run_dir) + assert list(merged) == [ + "system.test_fixed_trajectory.TestFixedTrajectory." + "test_circle[isaacsim]" + ] + only_metrics = next(iter(merged.values())) + assert only_metrics["status"] == "failed" + assert only_metrics["cross_track_error_mean_m"]["value"] == 4.2 + + markdown, regressed = generate_report(run_dir) + assert "### Pass rates" in markdown + assert "cross_track_error_mean_m" in markdown + assert "0%" in markdown + assert "not comparable" not in markdown + assert regressed is False diff --git a/tests/parse_metrics.py b/tests/parse_metrics.py index f2267e627..9d8d77210 100644 --- a/tests/parse_metrics.py +++ b/tests/parse_metrics.py @@ -21,6 +21,9 @@ from tabulate import tabulate +from harness.run_meta import classify_run, simulation_metrics_comparable +from harness.test_ids import canonical_test_id + FLAG_SUFFIX = {"regression": " :red_circle:", "improved": " :green_circle:"} ITER_RE = re.compile(r"-iter(\d+)(?=\])") @@ -173,15 +176,20 @@ def parse_results_xml(path): tree = ET.parse(path) metrics = {} for tc in tree.iter("testcase"): - name = f"{tc.get('classname')}.{tc.get('name')}" - failed = tc.find("failure") is not None + name = canonical_test_id(f"{tc.get('classname')}.{tc.get('name')}") + if tc.find("failure") is not None or tc.find("error") is not None: + status = "failed" + elif tc.find("skipped") is not None: + status = "skipped" + else: + status = "passed" metrics[name] = { "duration_s": { "value": float(tc.get("time", 0)), "unit": "s", "direction": "lower_is_better", }, - "status": "failed" if failed else "passed", + "status": status, } return metrics @@ -210,7 +218,10 @@ def parse_passrates(path): def parse_metrics_json(path): if not path.exists(): return {} - return json.loads(path.read_text()) + return { + canonical_test_id(test_name): metrics + for test_name, metrics in json.loads(path.read_text()).items() + } def merge_metrics(run_dir): @@ -250,10 +261,9 @@ def _collapse_iterations(merged): bucket = out.setdefault(base, {}) for key, val in metrics.items(): if key == "status": - if val == "failed" or bucket.get("status") == "failed": - bucket["status"] = "failed" - else: - bucket["status"] = val + priority = {"passed": 0, "skipped": 1, "failed": 2} + previous = bucket.get("status", "passed") + bucket["status"] = max((previous, val), key=priority.get) continue if isinstance(val, dict) and "samples" in val: series.setdefault((base, key), []).append(val["samples"]) @@ -603,6 +613,92 @@ def render_passrates(mod): return "\n\n".join(sections), has_regression +def _non_comparable_report(meta): + outcome = meta.get("outcome", "unknown") + explanations = { + "collection_error": ( + "Pytest collection failed before a simulation campaign could run." + ), + "internal_error": ( + "Pytest exited with an internal or command-line error." + ), + "no_tests": "No tests were selected or executed.", + "simulation_not_executed": ( + "Simulation tests were selected, but none reached a recorded outcome." + ), + "incomplete": ( + "The runner stopped before pytest finalized its result artifacts " + "(for example, a timeout or cancellation)." + ), + "missing_results": "The test job produced no pytest result artifact.", + } + explanation = explanations.get( + outcome, meta.get("reason", "The run did not complete as a valid test campaign.") + ) + fields = [ + ("Outcome", outcome), + ("Pytest exit status", meta.get("pytest_exitstatus", "unavailable")), + ("Selected tests", meta.get("selected_tests", "unavailable")), + ("Completed tests", meta.get("completed_tests", "unavailable")), + ("Simulation tests completed", meta.get("simulation_completed", 0)), + ] + rows = "\n".join(f"- **{label}:** {value}" for label, value in fields) + return ( + "## Run status\n\n" + f"**Simulation metrics are not comparable.** {explanation}\n\n" + f"{rows}\n\n" + "Pass-rate and regression tables are suppressed because they would " + "misrepresent an infrastructure/collection failure as policy performance." + ) + + +def generate_report(current_dir, baseline_dir=None, threshold=20): + """Generate report markdown and whether a comparable regression exists.""" + current_dir = Path(current_dir) + current_meta = classify_run(current_dir) + if current_meta.get("outcome") not in ("simulation", "non_simulation"): + return _non_comparable_report(current_meta), False + + baseline_meta = classify_run(Path(baseline_dir)) if baseline_dir else None + diff_mode = bool( + baseline_dir + and simulation_metrics_comparable(current_meta, baseline_meta) + ) + + current = merge_metrics(current_dir) + baseline = merge_metrics(Path(baseline_dir)) if diff_mode else {} + current_pr = parse_passrates(current_dir / "results.xml") + baseline_pr = ( + parse_passrates(Path(baseline_dir) / "results.xml") if diff_mode else {} + ) + main_rows, hz_rows, compute_rows, iter_counts = build_rows(current, baseline) + md, has_regression = format_markdown( + main_rows, + hz_rows, + compute_rows, + iter_counts, + current_pr, + baseline_pr, + threshold, + diff_mode, + ) + + notices = [] + if current_meta.get("outcome") == "non_simulation": + notices.append( + "> This was a unit/build-only run. Simulation regression comparison " + "does not apply." + ) + elif baseline_dir and not diff_mode: + notices.append( + "> The baseline is not the same complete simulation campaign. " + "Showing current results without a regression comparison." + ) + if not md: + md = "_No per-test metrics were recorded._" + return "\n\n".join([*notices, md]), has_regression + + def main(): parser = argparse.ArgumentParser( description="Render a markdown report for a test run, or a diff if --baseline is supplied.") @@ -612,23 +708,28 @@ def main(): parser.add_argument("--output", help="Write markdown report to file") args = parser.parse_args() - current = merge_metrics(Path(args.current)) - baseline = merge_metrics(Path(args.baseline)) if args.baseline else {} - current_pr = parse_passrates(Path(args.current) / "results.xml") - baseline_pr = (parse_passrates(Path(args.baseline) / "results.xml") - if args.baseline else {}) - diff_mode = bool(args.baseline) - - main_rows, hz_rows, compute_rows, iter_counts = build_rows(current, baseline) - md, has_regression = format_markdown( - main_rows, hz_rows, compute_rows, iter_counts, - current_pr, baseline_pr, args.threshold, diff_mode) + try: + md, has_regression = generate_report( + args.current, + args.baseline, + args.threshold, + ) + except Exception as exc: + md = ( + "## Report generation failed\n\n" + f"`{type(exc).__name__}: {exc}`\n\n" + "The test result is not being interpreted as a policy regression." + ) + print(md) + if args.output: + Path(args.output).write_text(md) + sys.exit(2) print(md) if args.output: Path(args.output).write_text(md) - sys.exit(1 if diff_mode and has_regression else 0) + sys.exit(1 if has_regression else 0) if __name__ == "__main__": diff --git a/tests/robot/README.md b/tests/robot/README.md index 3961d90cc..facecdc61 100644 --- a/tests/robot/README.md +++ b/tests/robot/README.md @@ -1,7 +1,7 @@ # Robot-side unit tests Unit-test **source is co-located** with each ROS 2 package (the standard colcon -convention) and is collected by `pytest tests/`: +convention): ``` robot/ros_ws/src///test/test_.py ← source of truth @@ -10,8 +10,10 @@ robot/ros_ws/src///test/test_.py ← source of truth [`../colcon_unit_test_packages.yaml`](../colcon_unit_test_packages.yaml) lists which packages have unit tests; `tests/conftest.py` resolves each to its `test/` dir and collects the non-linter `test_*.py` files under `--import-mode=importlib`, tagging each -`@pytest.mark.unit`. Both `airstack test -m unit` and `colcon test --packages-select ` -run the same source. +`@pytest.mark.unit` by path — you do not write the mark yourself. + +Run them with `airstack test -m unit`, or `cd tests && pytest -m unit`. C++ gtests in the +same `test/` dir run under `colcon test --packages-select `. To add a package's unit tests, list it under `robot.packages` in the YAML — see the -`add-unit-tests` agent skill. The per-layer subdirectories here hold only documentation. +`add-unit-tests` agent skill. diff --git a/tests/robot/behavior/README.md b/tests/robot/behavior/README.md deleted file mode 100644 index 713fd31f2..000000000 --- a/tests/robot/behavior/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Unit tests — behavior layer - -Add `@pytest.mark.unit` tests for `robot/ros_ws/src/behavior/` packages here. diff --git a/tests/robot/global/README.md b/tests/robot/global/README.md deleted file mode 100644 index 280c41dec..000000000 --- a/tests/robot/global/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Unit tests — global layer - -Add `@pytest.mark.unit` tests for `robot/ros_ws/src/global/` packages here. diff --git a/tests/robot/interface/README.md b/tests/robot/interface/README.md deleted file mode 100644 index ea4ee8b5c..000000000 --- a/tests/robot/interface/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Unit tests — interface layer - -Add `@pytest.mark.unit` tests for `robot/ros_ws/src/interface/` packages here. diff --git a/tests/robot/local/README.md b/tests/robot/local/README.md deleted file mode 100644 index 118cc2071..000000000 --- a/tests/robot/local/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Unit tests — local layer - -Add `@pytest.mark.unit` tests for `robot/ros_ws/src/local/` packages here. diff --git a/tests/robot/perception/README.md b/tests/robot/perception/README.md deleted file mode 100644 index 350ef9fb0..000000000 --- a/tests/robot/perception/README.md +++ /dev/null @@ -1,3 +0,0 @@ -# Unit tests — perception layer - -Add `@pytest.mark.unit` tests for `robot/ros_ws/src/perception/` packages here. diff --git a/tests/robot/sensors/README.md b/tests/robot/sensors/README.md deleted file mode 100644 index 8a44129eb..000000000 --- a/tests/robot/sensors/README.md +++ /dev/null @@ -1,4 +0,0 @@ -# Unit tests — sensors layer - -Package-specific folders (for example `lidar_point_cloud_filter/`) mirror -`robot/ros_ws/src/sensors//`. diff --git a/tests/run_summary.py b/tests/run_summary.py index c8ebaac48..3e60e7112 100644 --- a/tests/run_summary.py +++ b/tests/run_summary.py @@ -14,6 +14,9 @@ import xml.etree.ElementTree as ET from pathlib import Path +from harness.run_meta import classify_run +from harness.test_ids import canonical_test_id + PARAM_RE = re.compile(r"\[(.+)\]$") ITER_RE = re.compile(r"-iter\d+$") ROBOT_METRIC_RE = re.compile(r"^robot_\d+\.(.+)$") @@ -61,24 +64,11 @@ } -def _canonical_test_id(name: str) -> str: - """Unify metrics.json path slashes with JUnit classname dots. - - metrics.json keys look like ``system/test_fixed_trajectory.Class.test_x[...]`` - (pytest nodeid). results.xml uses ``system.test_fixed_trajectory.Class.test_x[...]``. - """ - head, dot, rest = name.partition(".") - if "/" in head: - head = head.replace("/", ".") - return head + dot + rest if dot else head - return name - - def _normalize_keyed_map(raw: dict) -> dict: """Merge entries that differ only by path-slash vs dot classname form.""" out: dict = {} for key, value in raw.items(): - out[_canonical_test_id(key)] = value + out[canonical_test_id(key)] = value return out @@ -86,10 +76,14 @@ def _parse_results_xml(path: Path) -> tuple[dict[str, str], dict[str, float]]: """Return ({full_test_name: status}, {full_test_name: wall_time_s}).""" if not path.exists(): return {}, {} + try: + testcases = ET.parse(path).iter("testcase") + except (OSError, ET.ParseError): + return {}, {} statuses: dict[str, str] = {} durations: dict[str, float] = {} - for tc in ET.parse(path).iter("testcase"): - full = _canonical_test_id(f"{tc.get('classname')}.{tc.get('name')}") + for tc in testcases: + full = canonical_test_id(f"{tc.get('classname')}.{tc.get('name')}") if tc.find("failure") is not None or tc.find("error") is not None: statuses[full] = "FAILED" elif tc.find("skipped") is not None: @@ -116,7 +110,7 @@ def _param_id(test_name: str) -> str: def _module_name(test_name: str) -> str: - canonical = _canonical_test_id(test_name) + canonical = canonical_test_id(test_name) match = MODULE_RE.search(canonical) if match: return match.group(1) @@ -125,7 +119,7 @@ def _module_name(test_name: str) -> str: def _phase_name(test_name: str) -> str: """test_fixed_trajectory.TestFixedTrajectory.test_takeoff[...] -> test_takeoff""" - canonical = _canonical_test_id(test_name) + canonical = canonical_test_id(test_name) match = PHASE_RE.search(canonical) if match: return match.group(1) @@ -187,7 +181,7 @@ def _collect_scalar_metrics(metrics_blob: dict) -> dict[str, list[dict]]: def _metrics_blob(metrics: dict, test_name: str) -> dict: - canonical = _canonical_test_id(test_name) + canonical = canonical_test_id(test_name) return metrics.get(canonical, {}) @@ -245,7 +239,7 @@ def _group_tests( ) -> dict[tuple[str, str], list[str]]: """Group full test names by (module, base_param_id) across stress iterations.""" groups: dict[tuple[str, str], list[str]] = {} - all_names = {_canonical_test_id(name) for name in set(metrics) | set(statuses)} + all_names = {canonical_test_id(name) for name in set(metrics) | set(statuses)} for name in sorted(all_names): module = _module_name(name) param = _base_param_id(_param_id(name)) @@ -288,6 +282,7 @@ def _chain_status(test_names: list[str], statuses: dict[str, str]) -> str: def build_summary_lines(run_dir: Path) -> list[str]: metrics_path = run_dir / "metrics.json" results_path = run_dir / "results.xml" + run_meta = classify_run(run_dir) statuses, durations = _parse_results_xml(results_path) metrics = _load_metrics(metrics_path) @@ -302,6 +297,13 @@ def build_summary_lines(run_dir: Path) -> list[str]: f"Overall: {passed} passed, {failed} failed, {skipped} skipped ({total} tests)", "", ] + if run_meta.get("outcome") not in ("simulation", "non_simulation"): + reason = run_meta.get("reason", run_meta.get("outcome", "unknown")) + lines.extend([ + f"Run status: {run_meta.get('outcome', 'unknown')}", + f"Simulation metrics are not comparable: {reason}.", + "", + ]) groups = _group_tests(metrics, statuses, durations) if not groups: diff --git a/tests/sim/README.md b/tests/sim/README.md index 09f45f6a6..46344d94b 100644 --- a/tests/sim/README.md +++ b/tests/sim/README.md @@ -1,14 +1,20 @@ # Simulation-side unit tests -Tests for **simulation components** that are not part of the onboard ROS workspace -(for example an OptiTrack Motive / NatNet emulator, Isaac launch helpers, or -AirSim bridge utilities). +Unit-test **source is co-located** with each simulation component, the same way the +robot workspace works: -Mark fast, hermetic checks with `@pytest.mark.unit`. Tests that require a GPU, -full sim, or Docker belong in [`tests/system/`](../system/) instead. +``` +simulation/**//test/test_.py ← source of truth +``` -Suggested layout: +[`../colcon_unit_test_packages.yaml`](../colcon_unit_test_packages.yaml) lists which +components have unit tests, under the `sim:` key; `tests/conftest.py` resolves each to +its `test/` dir and tags the collected items `@pytest.mark.unit` by path. -| Directory | Purpose | -|-----------|---------| -| `motive_emulator/` | Motive / NatNet protocol emulation / parsing | +Run them with `airstack test -m unit`, or `cd tests && pytest -m unit`. These components +are not part of the onboard ROS workspace, so `colcon test` does not run them. + +Tests needing a GPU, a full sim, or Docker belong in [`../system/`](../system/) instead. + +Currently listed: `optitrack.natnet.emulator` +([source](../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/)). diff --git a/tests/sim/motive_emulator/README.md b/tests/sim/motive_emulator/README.md deleted file mode 100644 index 0e682c448..000000000 --- a/tests/sim/motive_emulator/README.md +++ /dev/null @@ -1,61 +0,0 @@ -# Motive / NatNet Emulator - -This directory is the future home of **integration tests** that drive a real -NatNet wire-protocol mock server against `natnet_ros2_node`. - -## Why here, not in the package test/ dir? - -Unit tests for pure logic live in -`tests/robot/perception/natnet_ros2/test_natnet_logic.cpp` and run via `colcon -test` with no network or SDK required (uses `FakeNatNetClient`). - -The emulator tests here will require an actual UDP server that speaks the NatNet -protocol, so they belong in the `sensors` mark of the system test suite alongside -other topic-streaming tests. - -## Planned implementation - -The mock server should: - -1. Open a UDP socket on the NatNet command port (default 1510). -2. Respond to `NAT_CONNECT` (message type 0) with a `NAT_SERVERINFO` (type 1) - packet containing a canned `sServerDescription`. -3. Respond to `NAT_REQUEST_MODELDEF` (type 4) with a `NAT_MODELDEF` (type 5) - packet describing one or more rigid bodies. -4. Stream `NAT_FRAMEOFDATA` (type 7) packets to the client's data port at a - configurable rate with synthetic pose data. - -### Reference - -The NatNet wire format is documented in the NatNet SDK developer notes and the -`PacketClient` example shipped with the SDK (available inside the robot Docker -container after `airstack setup --natnet`). - -## Relationship to `FakeNatNetClient` - -``` - ┌──────────────────────────────────────┐ - │ Test boundary │ - colcon gtest │ FakeNatNetClient (in-process) │ ← unit tests (no network) - │ test_natnet_logic.cpp │ - └──────────────────────────────────────┘ - - ┌──────────────────────────────────────┐ - │ Network boundary │ - pytest sensors │ MotiveEmulator (UDP server, Python) │ ← integration tests - │ NatNetClientAdapter → NatNetClient │ - │ natnet_ros2_node (full ROS node) │ - └──────────────────────────────────────┘ -``` - -The `FakeNatNetClient` seam (already implemented) lets unit tests verify all -connection-outcome logic paths. The emulator here will verify the full -end-to-end path including the NatNet SDK's own parser. - -## When to add this - -Implement the emulator when: -- The OptiTrack emulator service is placed under `simulation/optitrack-emulator/` - or `tests/sim/motive_emulator/` -- The `sensors` test mark is extended to include `natnet_ros2` topic checks -- CI has access to the robot container with the NatNet SDK installed From 9e2e0e3991cd5130c24f90966a95f1c3c5e2bf54 Mon Sep 17 00:00:00 2001 From: Andrew Jong Date: Thu, 20 Aug 2026 04:02:19 -0400 Subject: [PATCH 19/21] docs(skills): require dates and timestamps in feature notebook entries Add a 'date and timestamp everything' convention to the use-feature-notebook skill: Date started / Last updated in design_spec.md, run timestamps on stored test artifacts, and per-section run times in results_summary.md. Update both templates accordingly and add a pitfall for undated documents. Co-Authored-By: Claude Fable 5 --- .agents/skills/use-feature-notebook/SKILL.md | 9 ++++++++- .../use-feature-notebook/assets/design_spec_template.md | 2 +- .../assets/results_summary_template.md | 1 + 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/.agents/skills/use-feature-notebook/SKILL.md b/.agents/skills/use-feature-notebook/SKILL.md index 7a2f2dc8a..eb465b69e 100644 --- a/.agents/skills/use-feature-notebook/SKILL.md +++ b/.agents/skills/use-feature-notebook/SKILL.md @@ -37,6 +37,12 @@ Naming rules: - **Feature folder:** `NNN-short-kebab-slug`, where `NNN` is zero-padded three digits. Pick the next number by listing `notebook/` and incrementing the highest existing prefix (start at `001` if empty or missing — create `notebook/` yourself, it is not committed). - **Results subfolders:** one per lettered test section in `design_spec.md`, named `-` (e.g. section "(a) Planner core" → `results/a-planner-core/`). The letters MUST match the test-plan section letters in the spec so a reader can navigate spec ↔ results directly. +**Date and timestamp everything.** The notebook is a lab journal, and a journal entry without a date is unusable later. Every design doc, experiment, and results file records when it happened: + +- `design_spec.md` header: `Date started` and `Last updated` (update the latter whenever you revise the spec), as `YYYY-MM-DD`. +- Each test run stored under `results/-/`: record the run timestamp (`YYYY-MM-DD HH:MM` local time) — keep the harness's timestamped directory name when copying from `tests/results//`, or prefix artifact filenames / note the timestamp in the section of `results_summary.md`. +- `results_summary.md` header: the date written; each per-section **Setup** line: when that run was executed. + ## Workflow ### 1. On starting a feature — write `design_spec.md` @@ -66,7 +72,7 @@ Every test run that validates the feature drops its artifacts into the matching - Plots and screenshots (cross-track error curves, Foxglove/RViz captures, sim screenshots) - Relevant log excerpts — excerpts, not full container logs -Keep raw artifacts as-produced; interpretation belongs in the summary. +Keep raw artifacts as-produced; interpretation belongs in the summary. Preserve the run's timestamp with the artifacts (keep the `tests/results//` directory name, or timestamp-prefix the copied files) so repeated runs of the same section stay distinguishable and ordered. ### 4. After validation — write `results/results_summary.md` @@ -86,5 +92,6 @@ The PR body for the feature is built from the notebook, since reviewers cannot s - ❌ Stale status labels — a spec still marked `DESIGN/TODO` (or a section marked `WIP`) after the work shipped misleads the next reader; update statuses as you go. - ❌ Committing `notebook/` or referencing `notebook/...` paths from committed code, docs, or tests — it doesn't exist on other machines or in CI. - ❌ Results subfolder letters that don't match the spec's test-plan letters. +- ❌ Undated documents or results — a spec without `Date started`/`Last updated`, or test artifacts with no run timestamp, can't be sequenced against other runs or the code they tested. - ❌ A `results_summary.md` that just links to raw files — embed the tables and figures. - ❌ Confusing this with [capture-discovered-knowledge](../capture-discovered-knowledge): the notebook records *per-feature* design and evidence locally; durable repo-wide knowledge still goes to AGENTS.md/skills, and module documentation still follows [update-documentation](../update-documentation). diff --git a/.agents/skills/use-feature-notebook/assets/design_spec_template.md b/.agents/skills/use-feature-notebook/assets/design_spec_template.md index ecb184ba2..a50c2c92f 100644 --- a/.agents/skills/use-feature-notebook/assets/design_spec_template.md +++ b/.agents/skills/use-feature-notebook/assets/design_spec_template.md @@ -1,6 +1,6 @@ # Design Spec: -> Notebook entry: `notebook/NNN-feature-slug/` · Date started: YYYY-MM-DD · Branch: `` +> Notebook entry: `notebook/NNN-feature-slug/` · Date started: YYYY-MM-DD · Last updated: YYYY-MM-DD · Branch: `` > > **Status: `DESIGN/TODO`** diff --git a/.agents/skills/use-feature-notebook/assets/results_summary_template.md b/.agents/skills/use-feature-notebook/assets/results_summary_template.md index 5f3b0977f..7a5929a0f 100644 --- a/.agents/skills/use-feature-notebook/assets/results_summary_template.md +++ b/.agents/skills/use-feature-notebook/assets/results_summary_template.md @@ -9,6 +9,7 @@ ## (a)
**Setup:** +**Run at:** YYYY-MM-DD HH:MM | Metric | Value | Pass criterion | Pass? | |--------|-------|----------------|-------| From 262f12679af7c67cb4df40c074e8d2133d79f226 Mon Sep 17 00:00:00 2001 From: Andrew Jong Date: Thu, 20 Aug 2026 12:45:28 -0700 Subject: [PATCH 20/21] Pre-RFC workflow cleanup: intent-based launch, readiness gates, launch-script dedup, truthful logs (#386) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * refactor(isaac): dedupe launch scripts into shared PegasusApp base The six launch scripts were 80-90% copy-pasted boilerplate (extension enabling, wait_for_stage, scene prep, spawn calls, run loop) that had already drifted: livestream existed only in the *_one_* scripts (so the isaac-sim-livestream service silently black-screened with multi scripts), ISAAC_SIM_HEADLESS was honored only by the *_multi_* scripts, and barebones_pegasus_launch.py (the documented template) crashed with a NameError (os never imported). pegasus_app.py now owns the skeleton once: create_simulation_app() (livestream + headless env handling, uniform across all scripts), extension enabling, world/env loading, scene prep, drone/sensor spawning from config dicts, and the run loop. Scripts reduce to scenario declarations plus hooks (pre_scene_prep/post_scene_prep/post_spawn). Behavior preserved per script (spawn poses, prim/node names, sensor offsets, NatNet bodies, GPS origins), with three deliberate fixes: - ISAAC_SIM_HEADLESS and ISAAC_SIM_LIVESTREAM now work in every script - barebones template runs again - NATNET_BODY_NAME/NATNET_TARGET_NAME env overrides now work as the one-drone natnet script's docstring already claimed example_multi_drone_scene_import keeps its historical ZED offset [0.21, 0, 0.05] (drift vs the canonical [0.2, 0, -0.05] — now visible and annotated instead of buried). Co-Authored-By: Claude Fable 5 * feat(cli): intent flags on 'up', resolved-value preflight, and 'airstack ready' airstack up learns intent flags that derive the coordinated env-var sets users previously had to know by heart (they export leaf values only — compose interpolation gives shell env precedence, so .env is untouched): --sim isaac|airsim swap simulator profile + matching URDF --robots N NUM_ROBOTS + auto-select one/multi Isaac script (also natnet pair; warns on custom scripts) --headless ISAAC_SIM_HEADLESS + MS_AIRSIM_HEADLESS + QT offscreen --play/--no-play PLAY_SIM_ON_START --no-autolaunch AUTOLAUNCH=false --wait chain into 'airstack ready' after compose up --dry-run print + validate the resolved config, start nothing Every up prints the resolved launch config and dumps it to .airstack/runs//effective_config.env (gitignored; best-effort on read-only checkouts). Preflight now validates RESOLVED values (env > --env-file > .env), fixing the historical guard bypass where 'up --env-file overrides/...' was checked against .env only. New checks: NUM_ROBOTS>1 with the single-drone Isaac script (previously a silent 3-containers-1-drone failure) is a hard error; missing images are listed by name with an image-pull hint before compose starts a multi-GB implicit build; missing omni_pass.env / empty Pegasus submodule / docker<29 name-resolution are surfaced on the host instead of dying invisibly inside tmux. AIRSTACK_SKIP_PREFLIGHT=1 downgrades errors to warnings. 'airstack ready' (and 'up --wait') answers "can I press Takeoff yet?": staged gates mirroring the system-test budgets — containers (120s) → sim /clock (600s) → per-robot sentinel nodes (300s) → PX4 MAVROS connected + local_position/odom streaming (300s, the EKF-converged armable signal; connected alone fires ~25s early). --json for scripts; per-gate failures name the container/tmux window to inspect. tests/meta/test_launch_intent_contract.py pins the flag derivations, guard behavior, and exit codes (runs under the unit mark). Co-Authored-By: Claude Fable 5 * feat(docker): tee tmux pane output to container stdout Every service runs its real workload inside tmux, so 'docker logs' / 'airstack logs' were empty by construction — colcon build failures, Pegasus import errors, and scene downloads all landed in panes nobody attaches to. tmux hooks in the shared .tmux.conf (mounted into robot, gcs, isaac-sim, and ms-airsim containers) now pipe-pane every created session/window/split to /proc/1/fd/1, making container logs truthful. Co-Authored-By: Claude Fable 5 * docs: fix launch-workflow drift against actual code behavior Corrects statements the audit found wrong, and teaches the new flags: - getting_started: sim comes up PAUSED by default (PLAY_SIM_ON_START=false in .env, docs claimed auto-play), operator UI is Foxglove not RViz (DEBUG_RVIZ=false by default), adds 'airstack ready' / --wait and --sim/--robots variants - simulation index + isaac docker.md + key_concepts + docker_usage: ISAAC_SIM_SCENE does not exist — scene selection is ISAAC_SIM_SCRIPT_NAME (standalone) or ISAAC_SIM_GUI (USD path, non- standalone); defaults table now matches .env/compose (AUTOLAUNCH=true, PLAY_SIM_ON_START=false, ISAAC_SIM_USE_STANDALONE=true, 100 Hz physics) - simulation index: NUM_ROBOTS=3 alone does NOT put 3 drones in Isaac — documents --robots (auto script switch) and the preflight guard - docker_usage: the test service is robot-test, not autotest - gcs user_interface: gcs service is not in the deploy profile (gcs-real is) - ms-airsim: MAVROS connects on 14540+domain (24540+i is AirSim's own PX4 channel), camera FOV default is 90 not 110, vehicles are robot_ not drone - AGENTS.md: airstack stop/build are not registered commands (down / image-build); documents the new up flags and ready - .env: correct usage comment; PLAY_SIM_ON_START paused-by-default note Co-Authored-By: Claude Fable 5 * chore(release): bump VERSION to 0.19.0-alpha.18 and update CHANGELOG Image inputs are unchanged (all edits are bind-mounted or host-side), so docker-build should registry-retag rather than rebuild on merge. Co-Authored-By: Claude Fable 5 * docs(sim): document PegasusApp launch-script authoring; re-teach stale skills spawning_drones.md now documents the pegasus_app.PegasusApp base class as the way to write a launch script: import-order contract, constructor kwargs, the drone-config dict (incl. prim/node_name/sensor overrides), hooks (pre_scene_prep/post_scene_prep/post_spawn), and which reference subclass to study for scene-import and NatNet scenarios. pegasus_scene_setup.md points at it and drops the false 'PLAY_SIM_ON_START not supported in standalone mode' claim. docker_usage.md gains a 'Launch flags and readiness' section (--sim/--robots/--headless/--play/--wait/ --dry-run, effective-config dumps, airstack ready). The write-isaac-sim-scene skill was re-taught from scratch: it prescribed copy-pasting a ~240-line skeleton whose API had drifted to non-runnable (wrong add_zed_stereo_camera_subgraph signature, nonexistent SIMULATION_ENVIRONMENTS keys, low-level Multirotor API no shipped script uses). It now teaches scenario declaration on PegasusApp with an explicit 'do not copy-paste' rule. Other skills fixed where the old guidance became wrong or footgun-inducing: integrate-module-into-layer ('airstack stop' is not a command), test-in-simulation and configure-multi-robot (bare NUM_ROBOTS=N up now fails preflight with the single-drone script — use --robots), use-airstack-cli (new flags + ready in the reference), optitrack-development (single-drone NatNet body names are env-overridable now). Co-Authored-By: Claude Fable 5 --------- Co-authored-by: Claude Fable 5 --- .agents/skills/configure-multi-robot/SKILL.md | 6 +- .../integrate-module-into-layer/SKILL.md | 2 +- .agents/skills/optitrack-development/SKILL.md | 2 +- .agents/skills/test-in-simulation/SKILL.md | 5 +- .agents/skills/use-airstack-cli/SKILL.md | 9 +- .agents/skills/write-isaac-sim-scene/SKILL.md | 685 ++---------------- .airstack/modules/ready.sh | 245 +++++++ .env | 6 +- .gitignore | 3 + AGENTS.md | 10 +- CHANGELOG.md | 12 + airstack.sh | 314 +++++++- common/.tmux.conf | 12 + .../beginner/airstack-cli/docker_usage.md | 33 +- docs/development/beginner/key_concepts.md | 14 +- docs/gcs/usage/user_interface.md | 3 +- docs/getting_started/index.md | 17 +- docs/simulation/index.md | 38 +- docs/simulation/isaac_sim/docker.md | 21 +- .../isaac_sim/pegasus_scene_setup.md | 4 +- docs/simulation/isaac_sim/spawning_drones.md | 73 +- docs/simulation/ms-airsim/index.md | 6 +- .../barebones_pegasus_launch.py | 104 +-- .../example_multi_drone_scene_import.py | 248 ++----- ...example_multi_px4_pegasus_launch_script.py | 192 +---- ..._multi_px4_pegasus_natnet_launch_script.py | 178 +---- .../example_one_px4_pegasus_launch_script.py | 286 +------- ...le_one_px4_pegasus_natnet_launch_script.py | 219 +----- .../isaac-sim/launch_scripts/pegasus_app.py | 443 +++++++++++ tests/meta/test_launch_intent_contract.py | 180 +++++ 30 files changed, 1617 insertions(+), 1753 deletions(-) create mode 100644 .airstack/modules/ready.sh create mode 100644 simulation/isaac-sim/launch_scripts/pegasus_app.py create mode 100644 tests/meta/test_launch_intent_contract.py diff --git a/.agents/skills/configure-multi-robot/SKILL.md b/.agents/skills/configure-multi-robot/SKILL.md index 640b183bd..a6ffacef1 100644 --- a/.agents/skills/configure-multi-robot/SKILL.md +++ b/.agents/skills/configure-multi-robot/SKILL.md @@ -151,7 +151,7 @@ robot-desktop: So `NUM_ROBOTS=3 airstack up` produces **three** robot containers (`airstack-robot-desktop-1`, `-2`, `-3`), each with its own `ROBOT_NAME` and its own `ROS_DOMAIN_ID`. Each container runs the full autonomy stack independently. Cross-robot communication, when needed, goes through the DDS router (see [`onboard_all/config/dds_router.yaml`](../../../robot/ros_ws/src/autonomy_bringup/onboard_all/config/dds_router.yaml)) which bridges allowlisted topics from each per-robot domain into a shared GCS domain. ```bash -NUM_ROBOTS=3 airstack up +airstack up --sim isaac --robots 3 # sets NUM_ROBOTS and the multi-drone Isaac script together docker ps --format '{{.Names}}' | grep robot-desktop # airstack-robot-desktop-1 # airstack-robot-desktop-2 @@ -255,13 +255,13 @@ for i in range(1, NUM_ROBOTS + 1): spawn_drone(i) ``` -To use the multi-drone launcher, set in `.env`: +To use the multi-drone launcher, either launch with `airstack up --sim isaac --robots N` (which selects it automatically) or set in `.env`: ``` ISAAC_SIM_SCRIPT_NAME="example_multi_px4_pegasus_launch_script.py" ``` -(The default `example_one_px4_pegasus_launch_script.py` only spawns one.) +(The default `example_one_px4_pegasus_launch_script.py` only spawns one; `airstack up` preflight rejects `NUM_ROBOTS>1` with a single-drone script.) ### Test harness diff --git a/.agents/skills/integrate-module-into-layer/SKILL.md b/.agents/skills/integrate-module-into-layer/SKILL.md index c4f7bfb50..46166877b 100644 --- a/.agents/skills/integrate-module-into-layer/SKILL.md +++ b/.agents/skills/integrate-module-into-layer/SKILL.md @@ -244,7 +244,7 @@ Launch the complete autonomy stack to test integration: ```bash # Stop any running containers -airstack stop +airstack down # Launch with full autonomy AUTOLAUNCH=true airstack up robot-desktop diff --git a/.agents/skills/optitrack-development/SKILL.md b/.agents/skills/optitrack-development/SKILL.md index 359afa3cd..d5e656c9a 100644 --- a/.agents/skills/optitrack-development/SKILL.md +++ b/.agents/skills/optitrack-development/SKILL.md @@ -51,7 +51,7 @@ flowchart LR | [`example_one_px4_pegasus_natnet_launch_script.py`](../../../simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_natnet_launch_script.py) | Single drone + static ``Target`` | | [`example_multi_px4_pegasus_natnet_launch_script.py`](../../../simulation/isaac-sim/launch_scripts/example_multi_px4_pegasus_natnet_launch_script.py) | ``NUM_ROBOTS`` drones + shared ``Target`` (pair with 3-profile ``natnet_config.yaml``) | -Helpers: [`isaac/scene_setup.py`](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py) (`start_drone_natnet_server`, `author_static_target`). Drone body: single = ``Drone`` (id 1); multi = ``Drone`` (id ``i``); target = ``Target`` (id 100). These are **constants in the launch script**, not env vars — change them there AND in the matching ``natnet_config.yaml`` profile together. The client filters frames by numeric id, so a mismatch is silent: it connects and never publishes. Baseline Pegasus scripts (no NatNet) remain ``example_one_px4_pegasus_launch_script.py`` / ``example_multi_px4_pegasus_launch_script.py``. +Helpers: [`isaac/scene_setup.py`](../../../simulation/isaac-sim/extensions/optitrack.natnet.emulator/optitrack/natnet/emulator/isaac/scene_setup.py) (`start_drone_natnet_server`, `author_static_target`). Drone body: single = ``Drone`` (id 1); multi = ``Drone`` (id ``i``); target = ``Target`` (id 100). In the single-drone script these are overridable via ``NATNET_BODY_NAME``/``NATNET_TARGET_NAME`` env vars; in the multi script they are constants. Either way they must match the ``natnet_config.yaml`` profile — change both together. The client filters frames by numeric id, so a mismatch is silent: it connects and never publishes. Baseline Pegasus scripts (no NatNet) remain ``example_one_px4_pegasus_launch_script.py`` / ``example_multi_px4_pegasus_launch_script.py``. **Default client config:** unicast, `server_ip` → Motive/emulator (use `172.31.0.200` for Isaac container), ports 1510/1511. The config is per-robot: each `robots[$ROBOT_NAME]` profile lists the bodies it tracks (each a `rigid_body_name` + `id` mapped to a relative `topic`, with `pose`/`pose_cov` toggles and per-body covariance) and an optional `vision_pose` block that drives the MAVROS bridge. See [`natnet_config.yaml`](../../../robot/ros_ws/src/perception/natnet_ros2/config/natnet_config.yaml). diff --git a/.agents/skills/test-in-simulation/SKILL.md b/.agents/skills/test-in-simulation/SKILL.md index c9f84b8fd..c5882a312 100644 --- a/.agents/skills/test-in-simulation/SKILL.md +++ b/.agents/skills/test-in-simulation/SKILL.md @@ -395,8 +395,9 @@ Don't just test the happy path: If module supports multi-robot: ```bash -# Launch multi-robot simulation -NUM_ROBOTS=2 airstack up isaac-sim robot +# Launch multi-robot simulation (--robots also selects the multi-drone Isaac script; +# a bare NUM_ROBOTS=2 with the single-drone default script is rejected by preflight) +airstack up --sim isaac --robots 2 # Verify each robot runs independently docker exec airstack-robot-desktop-1 bash -c "ros2 node list | grep robot" diff --git a/.agents/skills/use-airstack-cli/SKILL.md b/.agents/skills/use-airstack-cli/SKILL.md index 7a3a6ead2..564ca5324 100644 --- a/.agents/skills/use-airstack-cli/SKILL.md +++ b/.agents/skills/use-airstack-cli/SKILL.md @@ -355,9 +355,14 @@ airstack config:git-hooks # Install git pre-commit hooks airstack install # Install Docker + nvidia-container-toolkit (one time) airstack setup # Add airstack to PATH (one time per shell) airstack up # Start default profile from .env +airstack up --sim isaac|airsim # Pick the simulator (profile + URDF + Isaac script derived) +airstack up --sim isaac --robots 2 # Multi-robot (keeps NUM_ROBOTS and the sim script consistent) +airstack up --play --wait # Auto-play sim, block until flight-ready +airstack up --dry-run --sim airsim # Print + validate resolved config; start nothing +airstack ready # Wait until flight-ready (--json for scripts) airstack up robot-desktop # Start one service -AUTOLAUNCH=false airstack up robot-desktop # Start idle (for development) — IMPORTANT -NUM_ROBOTS=2 AUTOLAUNCH=false airstack up # Multi-robot, idle +airstack up --no-autolaunch robot-desktop # Start idle (for development) — IMPORTANT +airstack up --no-autolaunch --robots 2 --sim isaac # Multi-robot, idle airstack status # List running containers airstack down # Stop and remove containers airstack clean # Stop, remove containers, prune volumes/networks diff --git a/.agents/skills/write-isaac-sim-scene/SKILL.md b/.agents/skills/write-isaac-sim-scene/SKILL.md index 2f46c75d3..c1de34ff9 100644 --- a/.agents/skills/write-isaac-sim-scene/SKILL.md +++ b/.agents/skills/write-isaac-sim-scene/SKILL.md @@ -1,674 +1,131 @@ --- name: write-isaac-sim-scene -description: Create custom simulation environments in Isaac Sim using standalone Python scripts with Pegasus extension. Use when creating test scenarios, multi-robot simulations, or custom environments for testing autonomy modules. +description: Create custom simulation scenarios in Isaac Sim by declaring them on top of the shared pegasus_app.PegasusApp base class. Use when creating test scenarios, multi-robot simulations, or custom environments for testing autonomy modules. license: Apache-2.0 metadata: author: AirLab CMU repository: AirStack --- -# Skill: Write Isaac Sim Scene in Standalone Python Mode +# Skill: Write an Isaac Sim Scene (Standalone Launch Script) ## When to Use Creating custom simulation environments for testing autonomy modules, multi-robot scenarios, or specific environmental conditions. -## Prerequisites - -- Isaac Sim container running or accessible -- Understanding of Pegasus Simulator extension for drones -- Knowledge of required sensors and vehicle configuration -- Familiarity with Python and basic Isaac Sim concepts +## The One Rule That Matters -## Isaac Sim Integration Overview +**Do NOT copy-paste an existing launch script wholesale.** All shared boilerplate (SimulationApp creation, extension enabling, Pegasus world + environment loading, stage prep, drone/sensor spawning, the run loop) lives once in `simulation/isaac-sim/launch_scripts/pegasus_app.py`. A launch script is a *scenario declaration*: an environment URL, a list of drone configs, sensor toggles, and (only if needed) hook overrides. If you find yourself copying more than ~50 lines, you are re-creating the duplication this base class removed. -AirStack uses NVIDIA Isaac Sim with the **Pegasus Simulator extension** for high-fidelity drone simulation. There are two ways to define scenes: +## Prerequisites -1. **USD Files:** Static scene description files (`.usd` format) -2. **Standalone Python Scripts:** Dynamic scene creation with full programmatic control (recommended for complex scenarios) +- Isaac Sim container image present (`airstack image-pull`) +- The scenario you want: which environment, how many drones, which sensors -This skill covers **standalone Python mode**. +## How a Scene Reaches the Simulator -## Script Structure Overview +`airstack up --sim isaac` starts the isaac-sim service, which (with `.env`'s default `ISAAC_SIM_USE_STANDALONE=true`) runs the Python file named by `ISAAC_SIM_SCRIPT_NAME` from `simulation/isaac-sim/launch_scripts/`. Scripts must live in that directory; set the variable to the filename only. -Standalone Python scripts follow this pattern: +Env vars every script honors automatically (via the base class — do not re-implement): -``` -1. Start SimulationApp (BEFORE any omni imports) -2. Import required modules -3. Enable necessary extensions -4. Create PegasusApp class - - Initialize Pegasus interface - - Load environment - - Spawn vehicles with sensors - - Setup physics and backends -5. Run simulation loop -6. Clean up -``` +| Env var | Effect | +|---|---| +| `ISAAC_SIM_HEADLESS` | run without a window | +| `ISAAC_SIM_LIVESTREAM` (+`_UDP_PORT`) | headless + WebRTC livestream | +| `PLAY_SIM_ON_START` | auto-play the timeline after setup (`airstack up --play`) | ## Steps -### 1. Create Script File +### 1. Create the script from the minimal template -**Location:** `simulation/isaac-sim/launch_scripts/.py` - -```bash -cd simulation/isaac-sim/launch_scripts/ -touch your_scene_name.py -chmod +x your_scene_name.py -``` - -### 2. Script Header and SimulationApp Initialization - -**Critical:** SimulationApp MUST be started before importing any `omni` modules. +Copy `barebones_pegasus_launch.py` (an environment, no drones) or start from this skeleton. The **import-order contract** is the only fragile part: Kit requires the `SimulationApp` to exist before any `omni.*`/`pegasus.*` import. ```python #!/usr/bin/env python -""" -Description: Brief description of your simulation scene -Author: Your Name -Date: YYYY-MM-DD - -This script creates a simulation environment for testing . -- Number of drones: X -- Sensors: Camera, LiDAR, etc. -- Environment: Description -""" - -import carb -from isaacsim import SimulationApp - -# MUST start SimulationApp before importing omni modules -# Set headless=False for GUI, headless=True for automated testing -simulation_app = SimulationApp({"headless": False}) - -# Now safe to import omni and other modules -import rclpy -print(f"[Launcher] SUCCESS: rclpy imported from {rclpy.__file__}") -``` - -### 3. Import Required Modules - -```python -import omni.kit.app -import omni.timeline -import omni.ui -from omni.isaac.core.world import World -from datetime import datetime -from pxr import UsdLux, Gf, UsdGeom - -# Pegasus imports -from pegasus.simulator.params import SIMULATION_ENVIRONMENTS, ROBOTS -from pegasus.simulator.logic.interface.pegasus_interface import PegasusInterface -from pegasus.simulator.ogn.api.spawn_multirotor import spawn_px4_multirotor_node -from pegasus.simulator.ogn.api.spawn_zed_camera import add_zed_stereo_camera_subgraph -from pegasus.simulator.ogn.api.spawn_rtx_lidar import add_rtx_lidar_subgraph -from pegasus.simulator.logic.vehicles.multirotor import Multirotor, MultirotorConfig -from pegasus.simulator.logic.state import State -from pegasus.simulator.logic.backends.px4_mavlink_backend import ( - PX4MavlinkBackend, - PX4MavlinkBackendConfig -) -from pegasus.simulator.logic.backends.ros2_backend import ROS2Backend -from scipy.spatial.transform import Rotation -import numpy as np +"""One-line description of the scenario.""" import os -import subprocess -import threading -import signal -import atexit -import time - -# Scene preparation utilities (scaling, collision, lighting, export) -# NOTE: importlib is used instead of a normal import because Isaac Sim's -# script runner does not reliably set __file__, making sys.path manipulation -# fragile. Loading the module by absolute file path is the robust approach. -import importlib.util as _ilu, os as _os -_scene_prep_path = _os.path.join(_os.path.dirname(_os.path.abspath(__file__)), "..", "utils", "scene_prep.py") -_spec = _ilu.spec_from_file_location("scene_prep", _os.path.normpath(_scene_prep_path)) -_scene_prep = _ilu.module_from_spec(_spec); _spec.loader.exec_module(_scene_prep) -scale_stage_prim = _scene_prep.scale_stage_prim -add_colliders = _scene_prep.add_colliders -add_dome_light = _scene_prep.add_dome_light -save_scene_as_contained_usd = _scene_prep.save_scene_as_contained_usd -``` - -### 4. Enable Required Extensions +import sys -```python -# Explicitly enable required extensions -ext_manager = omni.kit.app.get_app().get_extension_manager() - -# Required extensions for Pegasus and OmniGraph -required_extensions = [ - "omni.graph.core", # Core runtime for OmniGraph engine - "omni.graph.action", # Action Graph framework - "omni.graph.action_nodes", # Built-in Action Graph node library - "isaacsim.core.nodes", # Core helper nodes for OmniGraph - "omni.graph.ui", # UI scaffolding for graph tools - "omni.graph.visualization.nodes", # Visualization helper nodes - "omni.graph.scriptnode", # Python script node support - "omni.graph.window.action", # Action Graph editor window - "omni.graph.window.generic", # Generic graph UI tools - "omni.graph.ui_nodes", # UI node building helpers - "pegasus.simulator", # Pegasus Simulator extension -] - -for ext in required_extensions: - if not ext_manager.is_extension_enabled(ext): - print(f"[Launcher] Enabling extension: {ext}") - ext_manager.set_extension_enabled_immediate(ext, True) - print(f"[Launcher] Successfully enabled extension: {ext}") - else: - print(f"[Launcher] Extension already enabled: {ext}") -``` +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from pegasus_app import create_simulation_app -### 5. Create PegasusApp Class +simulation_app = create_simulation_app() # FIRST — before any omni/pegasus import -```python -class YourSceneApp: - """ - Simulation application for your specific scenario. - """ - - def __init__(self): - print("[YourScene] Initializing simulation...") - - # Start Pegasus interface - self.pg = PegasusInterface() - - # Create Isaac Sim world - self.world = World(**self.pg.world_settings) - self.pg.world = self.world - - # Dictionary to store vehicle instances - self.vehicles = {} - - # PX4 process handles (if using PX4 SITL) - self.px4_processes = [] - - # Load environment - self.load_environment() - - # Prepare environment (scale, colliders, lighting) - stage = omni.usd.get_context().get_stage() - self._prepare_environment(stage) - - # Spawn vehicles - self.spawn_vehicles() - - # Setup simulation - self.world.reset() - - print("[YourScene] Simulation initialized successfully") - - def load_environment(self): - """Load or create the simulation environment.""" - print("[YourScene] Loading environment...") - - # Option 1: Load pre-defined environment - # Available: "Grid", "Outdoor", "Office", etc. - # See SIMULATION_ENVIRONMENTS in Pegasus for options - stage = self.pg.load_environment(SIMULATION_ENVIRONMENTS["Grid"]["usd"]) - - # Option 2: Add ground plane only - # self.world.scene.add_default_ground_plane() - - # Option 3: Load custom USD environment - # stage = self.pg.load_environment("/path/to/your/environment.usd") - - # Add obstacles or other static objects - self._add_environment_objects() - - def _prepare_environment(self, stage): - """Scale, add collisions, and light the environment.""" - stage_prim = stage.GetPrimAtPath("/World/stage") - if stage_prim.IsValid(): - # STAGE_SCALE: use 0.01 for Nucleus assets authored in cm, 1.0 if already in meters - scale_stage_prim(stage, "/World/stage", STAGE_SCALE) - add_colliders(stage_prim) - # Allow physics to settle after adding colliders - for _ in range(10): - omni.kit.app.get_app().update() - # add_dome_light defaults: intensity=3500, exposure=-3 - # Override via kwargs, e.g. add_dome_light(stage, intensity=5000, exposure=-2) - add_dome_light(stage) - - def _add_environment_objects(self): - """Add obstacles or other objects to the environment.""" - # Example: Add a cube obstacle - stage = omni.usd.get_context().get_stage() - - # cube_prim = stage.DefinePrim("/World/Obstacle1", "Cube") - # UsdGeom.Xformable(cube_prim).AddTranslateOp().Set(Gf.Vec3d(5.0, 0.0, 0.5)) - # UsdGeom.Xformable(cube_prim).AddScaleOp().Set(Gf.Vec3d(1.0, 1.0, 1.0)) - - pass - - def spawn_vehicles(self): - """Spawn drone vehicles with sensors and backends.""" - print("[YourScene] Spawning vehicles...") - - # Vehicle 1: Primary drone - self._spawn_vehicle( - vehicle_id=0, - vehicle_name="drone1", - position=[0.0, 0.0, 1.0], # [x, y, z] - orientation=[0.0, 0.0, 0.0, 1.0], # quaternion [x, y, z, w] - px4_autostart_id=4001, # PX4 vehicle type (4001 = quadrotor) - mavlink_tcp_port=4560, # PX4 MAVLink port - px4_instance=0, - sensors={ - "camera": True, - "lidar": False - } - ) - - # Vehicle 2: Second drone (optional, for multi-robot) - # self._spawn_vehicle( - # vehicle_id=1, - # vehicle_name="drone2", - # position=[5.0, 0.0, 1.0], - # orientation=[0.0, 0.0, 0.0, 1.0], - # px4_autostart_id=4001, - # mavlink_tcp_port=4561, - # px4_instance=1, - # sensors={"camera": True, "lidar": True} - # ) - - def _spawn_vehicle(self, vehicle_id, vehicle_name, position, orientation, - px4_autostart_id, mavlink_tcp_port, px4_instance, - sensors=None): - """ - Spawn a single vehicle with specified configuration. - - Args: - vehicle_id: Unique vehicle ID - vehicle_name: Name for the vehicle - position: [x, y, z] spawn position - orientation: [x, y, z, w] quaternion orientation - px4_autostart_id: PX4 vehicle type ID - mavlink_tcp_port: MAVLink TCP port for PX4 communication - px4_instance: PX4 instance number - sensors: Dict of sensors to add {"camera": bool, "lidar": bool} - """ - if sensors is None: - sensors = {"camera": True, "lidar": False} - - # Configure multirotor - config = MultirotorConfig() - - # PX4 MAVLink backend configuration - px4_backend_config = PX4MavlinkBackendConfig({ - "vehicle_id": vehicle_id, - "px4_autostart": px4_autostart_id, - "px4_dir": os.environ.get("PX4_DIR", "/PX4-Autopilot"), - "px4_instance": px4_instance, - "mavlink_tcp_port": mavlink_tcp_port, - "enable_lockstep": True, - "update_rate": 250.0 # Hz - }) - - # Add ROS 2 backend for ROS communication - ros2_backend = ROS2Backend( - vehicle_id=vehicle_id, - config={ - "namespace": vehicle_name, - "pub_sensors": True, - "pub_state": True - } - ) - - # Attach backends - config.backends = [ - PX4MavlinkBackend(px4_backend_config), - ros2_backend - ] - - # Create vehicle - vehicle = Multirotor( - stage_prefix="/World", - prim_path=f"/World/{vehicle_name}", - name=vehicle_name, - usd_model=ROBOTS["Iris"]["usd"], # or other model - init_pos=position, - init_orientation=orientation, - config=config - ) - - # Add sensors - if sensors.get("camera", False): - self._add_camera_sensor(vehicle) - - # RTX LiDAR uses OmniGraph: spawn_px4_multirotor_node() returns graph_handle, - # then call self._add_lidar_sensor(vehicle, graph_handle). See - # example_one_px4_pegasus_launch_script.py for the full pattern. - - # Initialize vehicle in world - self.world.scene.add(vehicle) - self.vehicles[vehicle_name] = vehicle - - print(f"[YourScene] Spawned vehicle: {vehicle_name}") - - def _add_camera_sensor(self, vehicle): - """Add stereo camera to vehicle.""" - add_zed_stereo_camera_subgraph( - camera_prim_path=vehicle.prim_path + "/ZedCamera", - parent_prim_path=vehicle.prim_path, - config={ - "graph_evaluator": "execution", # or "push" - "resolution": (1280, 720), - "position": (0.3, 0.0, -0.1), # Relative to vehicle - "orientation": (0.0, 0.0, 0.0, 1.0), - } - ) - - def _add_lidar_sensor(self, vehicle, graph_handle): - """Add RTX LiDAR (OmniGraph subgraph) to vehicle.""" - add_rtx_lidar_subgraph( - parent_graph_handle=graph_handle, - drone_prim=vehicle.prim_path, - robot_name="robot_1", - lidar_config="ouster_os1", - lidar_offset=[0.0, 0.0, 0.025], - lidar_rotation_offset=[0.0, 0.0, 0.0], - min_range=0.75, - ) - - def run(self): - """Main simulation loop.""" - print("[YourScene] Starting simulation loop...") - - # Optionally auto-start timeline - # omni.timeline.get_timeline_interface().play() - - step_count = 0 - while simulation_app.is_running(): - # Step the simulation - self.world.step(render=True) - - # Optional: Add periodic logic - if step_count % 100 == 0: - # print(f"[YourScene] Simulation step: {step_count}") - pass - - step_count += 1 - - print("[YourScene] Simulation loop ended") - - def cleanup(self): - """Clean up resources.""" - print("[YourScene] Cleaning up...") - - # Stop PX4 processes - for process in self.px4_processes: - if process.poll() is None: # Process still running - process.terminate() - process.wait() - - self.px4_processes.clear() -``` +from pegasus.simulator.params import SIMULATION_ENVIRONMENTS # noqa: E402 +from pegasus_app import PegasusApp, row_spawn_configs # noqa: E402 -### 6. Main Entry Point -```python def main(): - """Main entry point for the simulation.""" - try: - # Create and run simulation - app = YourSceneApp() - app.run() - except Exception as e: - print(f"[YourScene] Error: {e}") - import traceback - traceback.print_exc() - finally: - # Clean up - if 'app' in locals(): - app.cleanup() - simulation_app.close() + PegasusApp( + env_url=SIMULATION_ENVIRONMENTS["Default Environment"], + drone_configs=row_spawn_configs(int(os.environ.get("NUM_ROBOTS", "1"))), + enable_lidar=os.environ.get("ENABLE_LIDAR", "false").lower() == "true", + ).run() + if __name__ == "__main__": main() ``` -### 7. Configure in .env - -Update the main `.env` file to use your script: +### 2. Declare the scenario via constructor kwargs -```bash -# Set to standalone script mode -ISAAC_SIM_USE_STANDALONE="true" +The full list with defaults is in `PegasusApp.__init__`'s signature and docstring; the ones you'll set: -# Specify your script name -ISAAC_SIM_SCRIPT_NAME="your_scene_name.py" -``` +| Kwarg | Purpose | +|---|---| +| `env_url` | A `SIMULATION_ENVIRONMENTS[...]` entry or any `omniverse://` / file USD URL | +| `drone_configs` | Per-drone dicts (below); `row_spawn_configs(n, spacing_m, z_m)` for the standard row | +| `stage_scale` | `0.01` for cm-authored Nucleus assets, `1.0` for metric scenes | +| `enable_camera`, `camera_offset` | ZED stereo subgraph per drone (default on, offset `[0.2, 0, -0.05]`) | +| `enable_lidar`, `lidar_min_range`, ... | RTX Ouster lidar subgraph per drone | +| `dome_light` | `True` (defaults), `False`, or `{"prim_path":…, "intensity":…, "exposure":…}` | +| `world_gps_origin` | `(lat, lon, alt)` — writes per-drone PX4 GPS homes before SITL boots (see [spawning_drones.md](../../../docs/simulation/isaac_sim/spawning_drones.md)) | +| `scale_spawn_positions` | `True` when spawn meters must be converted into non-metric stage units | +| `save_scene_to` | Directory to export a self-contained USD of the prepared scene | +| `extra_extensions` | Additional Kit extensions to enable | -Alternatively, override from command line: -```bash -ISAAC_SIM_USE_STANDALONE=true ISAAC_SIM_SCRIPT_NAME="your_scene_name.py" airstack up isaac-sim -``` +Per-drone config dict keys: `domain_id` (required — ROS domain and default vehicle id; MAVLink port `14540 + vehicle_id`), `x_m`/`y_m`/`z_m`, `orient` (quaternion `[x,y,z,w]`), and optional overrides `prim`, `node_name`, `lidar`, `lidar_min_range`, `camera_offset`. -### 8. Test the Scene +### 3. Custom behavior goes in hooks, not copied blocks -Launch Isaac Sim with your script: - -```bash -# Start Isaac Sim container with your scene -airstack up isaac-sim - -# Check logs for errors -airstack logs isaac-sim - -# If errors occur, connect to container for debugging -airstack connect isaac-sim -``` +Subclass `PegasusApp` and override (each receives the loaded USD stage): -### 9. Document the Scene +- `pre_scene_prep(stage)` — right after the environment loads (e.g. `dedupe_physics_scenes`, `reference_root_prims_under_world` for imported scenes) +- `post_scene_prep(stage)` — after scale/colliders/dome light, before drones (e.g. overhead map camera) +- `post_spawn(stage)` — after all drones exist (e.g. author the NatNet mocap interface) -Create a README.md next to your script: +Reference subclasses to study (not copy): `example_multi_drone_scene_import.py` (Nucleus scene import, explicit poses, overhead camera, GPS origins) and `example_multi_px4_pegasus_natnet_launch_script.py` (`post_spawn` mocap authoring). -**File:** `simulation/isaac-sim/launch_scripts/your_scene_name.md` +Stage-prep helpers live in `simulation/isaac-sim/utils/scene_prep.py` (`add_colliders`, `scale_stage_prim`, `add_dome_light`, `add_orthographic_camera`, …) — documented in [spawning_drones.md](../../../docs/simulation/isaac_sim/spawning_drones.md). -```markdown -# Your Scene Name +Scene-level things that need to happen **before Pegasus imports** (e.g. overriding the Nucleus asset root via `carb.settings`) go at script top level right after `create_simulation_app()` — see the top of `example_multi_drone_scene_import.py`. -## Overview -Brief description of the simulation scene. - -## Purpose -Why this scene was created and what it tests. - -## Configuration - -### Vehicles -- Number of drones: X -- Vehicle types: Quadrotor, fixed-wing, etc. -- Initial positions: List positions - -### Sensors -- Cameras: Resolution, FoV -- LiDAR: Model, range -- Other sensors - -### Environment -Description of the environment, obstacles, lighting. - -## Usage +### 4. Run it ```bash -# Launch scene -ISAAC_SIM_SCRIPT_NAME="your_scene_name.py" airstack up isaac-sim - -# With robot autonomy -airstack up isaac-sim robot -``` - -## Parameters -Any configurable parameters in the script. - -## Known Issues -Any limitations or known problems. +ISAAC_SIM_SCRIPT_NAME=my_scenario.py airstack up --sim isaac --play --wait ``` -## Advanced Topics +`--wait` (or `airstack ready`) blocks until the sim publishes `/clock`, the autonomy nodes are up, and PX4 is armable — so a hang here tells you which layer is broken. Watch script output from the host with `airstack logs isaac-sim` (tmux panes are mirrored to docker logs) or attach with `airstack connect isaac-sim`. -### Scene Preparation Utilities - -**File:** `simulation/isaac-sim/utils/scene_prep.py` - -Four reusable helpers that cover the most common environment setup tasks. Import them as shown in Step 3. - -| Function | When to use | -|----------|-------------| -| `scale_stage_prim(stage, prim_path, scale)` | Nucleus assets authored in centimeters need `STAGE_SCALE=0.01`; assets already in meters use `1.0`. | -| `add_colliders(stage_prim)` | **Must** be called for physics to interact with environment meshes. Without it drones fall through the floor. Call after scaling. | -| `add_dome_light(stage, **kwargs)` | Adds uniform hemisphere lighting. Defaults: `intensity=3500`, `exposure=-3`. Pass kwargs to override, e.g. `add_dome_light(stage, intensity=5000)`. | -| `save_scene_as_contained_usd(src_url, output_dir)` | Copies a Nucleus-hosted stage (and all its textures/MDLs) to a local directory using `omni.kit.usd.collect.Collector`. Useful for archiving or offline replay. | - -**Two-step save pattern** used internally by `save_scene_as_contained_usd`: -1. `export_as_stage_async` — writes a flat `.usd` of the live stage -2. `Collector` — resolves and copies all referenced Nucleus assets locally - -Set `SAVE_SCENE_TO = None` in your script to skip saving entirely. - ---- - -### Multi-Robot Scenarios - -For multiple robots, spawn additional vehicles with unique IDs and ports: - -```python -def spawn_vehicles(self): - for i in range(num_robots): - self._spawn_vehicle( - vehicle_id=i, - vehicle_name=f"drone{i}", - position=[i * 5.0, 0.0, 1.0], # Space them out - orientation=[0.0, 0.0, 0.0, 1.0], - px4_autostart_id=4001, - mavlink_tcp_port=4560 + i, # Unique port per vehicle - px4_instance=i, - sensors={"camera": True, "lidar": False} - ) -``` - -### Custom Sensor Configuration - -Create custom sensor configurations: - -```python -def _add_custom_camera(self, vehicle, config): - """Add camera with custom parameters.""" - add_zed_stereo_camera_subgraph( - camera_prim_path=vehicle.prim_path + "/CustomCamera", - parent_prim_path=vehicle.prim_path, - config={ - "resolution": config.get("resolution", (1920, 1080)), - "horizontal_fov": config.get("fov", 90.0), - "position": config.get("position", (0.3, 0.0, 0.0)), - "orientation": config.get("orientation", (0.0, 0.0, 0.0, 1.0)), - } - ) -``` - -### Dynamic Obstacles - -Add moving obstacles: - -```python -def _add_dynamic_obstacle(self): - """Add a moving obstacle to the scene.""" - from omni.isaac.core.objects import DynamicCuboid - - obstacle = DynamicCuboid( - prim_path="/World/DynamicObstacle", - position=[10.0, 0.0, 1.0], - scale=[1.0, 1.0, 1.0], - color=[1.0, 0.0, 0.0] # Red - ) - self.world.scene.add(obstacle) - - # In simulation loop, update position - # obstacle.set_world_pose(position=[x, y, z]) -``` - -## Common Pitfalls - -### SimulationApp Import Order -- ❌ **Importing omni modules before SimulationApp** - - ✅ ALWAYS create SimulationApp first, then import omni modules - -### Extension Loading -- ❌ **Missing required extensions** - - ✅ Enable all required extensions before using their features - - ✅ Check extension status with `ext_manager.is_extension_enabled()` - -### PX4 Port Conflicts -- ❌ **Using same MAVLink port for multiple vehicles** - - ✅ Each vehicle needs unique mavlink_tcp_port - - ✅ Increment port number for each vehicle: 4560, 4561, 4562, ... - -### Sensor Configuration -- ❌ **Incorrect sensor placement (inside vehicle mesh)** - - ✅ Position sensors outside vehicle collision geometry - - ✅ Typical camera position: forward of vehicle center - -### Missing Colliders on Environment Meshes -- ❌ Loading a Nucleus environment without calling `add_colliders()` - - ✅ Call `add_colliders(stage_prim)` after scaling — drones will fall through the floor otherwise - -### World Reset -- ❌ **Not calling world.reset()** - - ✅ Call world.reset() after adding all objects before stepping - -## Debugging - -### View Scene in GUI - -Run with headless=False to see the scene: -```python -simulation_app = SimulationApp({"headless": False}) -``` - -### Print Vehicle Info - -```python -def run(self): - while simulation_app.is_running(): - self.world.step(render=True) - - # Print vehicle state - for name, vehicle in self.vehicles.items(): - pos, ori = vehicle.get_world_pose() - print(f"{name}: pos={pos}, ori={ori}") -``` - -### Check ROS 2 Topics - -```bash -# From another terminal, check topics are publishing -docker exec airstack-isaac-sim-1 bash -c "ros2 topic list" -docker exec airstack-isaac-sim-1 bash -c "ros2 topic hz /drone1/sensors/camera/image" -``` +For multi-drone scenarios, `airstack up --sim isaac --robots N` keeps `NUM_ROBOTS` (robot containers) and the launch script consistent; if your custom script reads `NUM_ROBOTS`, say so in its docstring — preflight warns when `--robots > 1` is used with a custom script name. -## References +### 5. Verify -- **Pegasus Simulator:** - - [Pegasus GitHub](https://github.com/PegasusSimulator/PegasusSimulator) - - [Pegasus Documentation](https://pegasussimulator.github.io/PegasusSimulator/) +1. `python3 -m py_compile simulation/isaac-sim/launch_scripts/my_scenario.py` +2. `airstack up --dry-run --sim isaac` with your `ISAAC_SIM_SCRIPT_NAME` — preflight validates the config +3. Full bring-up with `--wait`; then `ros2 topic hz` the sensor topics per drone (see the debug-module skill) +4. For scenarios meant to gate CI: run the relevant system-test marks (`airstack test -m liveliness --sim isaacsim ...`) -- **Isaac Sim:** - - [Isaac Sim Documentation](https://docs.omniverse.nvidia.com/isaacsim/latest/index.html) - - [USD Introduction](https://docs.omniverse.nvidia.com/py/isaacsim/source/extensions/omni.isaac.core/docs/index.html) +## Pitfalls -- **AirStack Examples:** - - Single drone: `simulation/isaac-sim/launch_scripts/example_one_px4_pegasus_launch_script.py` - - Multiple drones: `simulation/isaac-sim/launch_scripts/example_multi_px4_pegasus_launch_script.py` +- ❌ Importing anything `omni.*`/`pegasus.*` before `create_simulation_app()` — Kit crashes or hangs +- ❌ Copying the extension-enable loop / run loop / stage-prep blocks into your script — they're in the base class +- ❌ Duplicate `domain_id`s in `drone_configs` — port and domain collisions, silent MAVROS failures +- ❌ Hardcoding a drone count while robot containers scale with `NUM_ROBOTS` — extra robots will wait forever for a PX4 that doesn't exist +- ❌ Forgetting `scale_spawn_positions=True` for cm-authored scenes — drones spawn 100× too far apart +- ❌ Re-reading `PLAY_SIM_ON_START`/`ISAAC_SIM_HEADLESS` yourself — the base class already does -- **Scene Preparation Utilities:** - - `simulation/isaac-sim/utils/scene_prep.py` +## Documentation -- **Related Skills:** - - [test-in-simulation](../test-in-simulation) - Testing modules in Isaac Sim - - [debug-module](../debug-module) - Debugging simulation issues +Follow [update-documentation](../update-documentation): a scenario intended for others should be mentioned in `docs/simulation/isaac_sim/index.md` and, if it introduces new patterns, documented alongside [spawning_drones.md](../../../docs/simulation/isaac_sim/spawning_drones.md). diff --git a/.airstack/modules/ready.sh b/.airstack/modules/ready.sh new file mode 100644 index 000000000..12ceb2897 --- /dev/null +++ b/.airstack/modules/ready.sh @@ -0,0 +1,245 @@ +#!/bin/bash +# Readiness gates for a running AirStack stack. +# +# `airstack up` reports success the moment `docker compose up -d` returns — +# before workspaces build, the sim loads, or PX4 boots. `airstack ready` +# answers the question users otherwise guess at: "can I press Takeoff yet?" +# +# Gates and budgets mirror the system-test suite (the source of truth for +# real-world timings — tests/system/test_liveliness.py and +# tests/system/test_takeoff_hover_land.py): +# 1. containers Running (120 s) +# 2. sim publishing /clock (600 s — Isaac scene loads are slow) +# 3. sentinel ROS 2 nodes per robot (300 s — includes the colcon build in dev mode) +# 4. PX4 ready per robot: MAVROS connected (300 s) +# then local_position/odom streaming (EKF converged = armable; connected +# alone fires ~25 s too early and takeoff returns "failed to arm") + +# Defaults match the system-test budgets; overridable from the environment +# (e.g. READY_CLOCK_TIMEOUT=60 airstack ready). +: "${READY_CONTAINERS_TIMEOUT:=120}" +: "${READY_CLOCK_TIMEOUT:=600}" +: "${READY_NODES_TIMEOUT:=300}" +: "${READY_PX4_TIMEOUT:=300}" +: "${READY_POLL_INTERVAL:=5}" + +# Sentinel nodes expected per robot domain (matches tests/system/test_liveliness.py). +READY_SENTINEL_TEMPLATES=( + "/robot_%d/interface/mavros/mavros" + "/robot_%d/robot_state_publisher" + "/robot_%d/trajectory_controller/trajectory_control_node" +) + +function _ready_now { date +%s; } + +function _ready_elapsed { + echo "$(( $(_ready_now) - $1 ))" +} + +# Run a ros2 command inside a robot container on a given domain, sourcing the +# workspace if it is built yet (mavros msgs need it). +function _ready_ros2_exec { + local container="$1" domain="$2" cmd="$3" timeout_s="${4:-10}" + docker exec "$container" bash -c " + source /opt/ros/jazzy/setup.bash >/dev/null 2>&1 + [ -f /root/AirStack/robot/ros_ws/install/setup.bash ] && source /root/AirStack/robot/ros_ws/install/setup.bash >/dev/null 2>&1 + export ROS_DOMAIN_ID=$domain + timeout $timeout_s $cmd" 2>/dev/null +} + +# List running robot containers (compose replicas), one per line. +function _ready_robot_containers { + docker ps --format '{{.Names}}' | grep -E -- '-robot-' | sort +} + +# domain for robot container (via the same .bashrc resolution airstack status uses) +function _ready_domain_of { + local container="$1" vars + vars=$(docker exec "$container" bash --login -c \ + 'printf "AIRSTACK_VARS:%s:%s\n" "$ROBOT_NAME" "$ROS_DOMAIN_ID"' 2>/dev/null \ + | grep "^AIRSTACK_VARS:" | tail -1) + [ -z "$vars" ] && return 1 + echo "${vars##*:}" +} + +function _ready_robot_name_of { + local container="$1" vars + vars=$(docker exec "$container" bash --login -c \ + 'printf "AIRSTACK_VARS:%s:%s\n" "$ROBOT_NAME" "$ROS_DOMAIN_ID"' 2>/dev/null \ + | grep "^AIRSTACK_VARS:" | tail -1) + [ -z "$vars" ] && return 1 + vars="${vars#AIRSTACK_VARS:}" + echo "${vars%%:*}" +} + +# Poll a predicate function until it returns 0 or the timeout expires. +# Usage: _ready_poll