diff --git a/DeepClean.sh b/DeepClean.sh index 90f8c12..7c4f019 100644 --- a/DeepClean.sh +++ b/DeepClean.sh @@ -17,6 +17,14 @@ if [ -r "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/color.sh" ]; th source "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/color.sh" fi +# Repository transport guard. DeepClean's apt branch runs +# `apt-get autoremove --purge`, which resolves dependency chains and can +# pull packages from a mirror, so HTTPS is enforced before it. +# shellcheck disable=SC1091 +if [ -r "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/apt-https.sh" ]; then + source "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}")")/lib/apt-https.sh" +fi + # If lib/color.sh was not sourced or did not set USE_COLOR, run the canonical # gate inline so _c is safe to call in all execution paths. if [ -z "${USE_COLOR:-}" ]; then @@ -66,6 +74,20 @@ PM=$(pkg_mgr) USED_BEFORE_KB=$(df -kP / | tail -1 | awk '{print $3}') msg "Detected package manager: ${PM:-none}" + +# Precaution: `apt-get autoremove --purge` below resolves dependencies and +# can fetch from a mirror, so force repository traffic over HTTPS first. +# No-op once per process, and a silent no-op when the lib is unavailable +# (curl|bash of DeepClean.sh on its own). +if declare -F apt_https_guard >/dev/null 2>&1 && [ "$PM" = "apt" ]; then + apt_https_guard "DeepClean" || true +elif [ "$PM" = "apt" ]; then + # Reached when this file is curl|bash'd on its own, with no lib/ next to + # it. Say so rather than silently skipping the precaution. + echo "[!] lib/apt-https.sh not found next to this script; skipping the" >&2 + echo "[!] repository transport guard before 'apt-get autoremove --purge'." >&2 +fi + msg "Starting DeepClean..." # 1. Systemd Journal Logs diff --git a/README.md b/README.md index 9217630..dd9c803 100644 --- a/README.md +++ b/README.md @@ -30,7 +30,14 @@ the package name right per distro. This script: - **Rollback log per file.** Every config it edits is backed up to a timestamped copy; the index lives at `/var/log/linux-install-rollback.log` and is one `cp` away from a full undo. -- **Three security profiles + a 20-tool maintenance suite.** From +- **HTTPS-only package transport, enforced before every download.** A + plaintext `http://` mirror lets anyone on the path swap a `.deb`/`.rpm`/wheel + for their own. The guard audits **every** app store on the box — apt, dnf, + yum, zypper, pacman, apk, flatpak, snap, docker, brew, pip, npm, cargo, + gem, nix, fwupd — rewrites apt automatically with rollback if a mirror + can't speak TLS, and reports the rest. See + [Package transport security](#package-transport-security-https). +- **Three security profiles + a 21-tool maintenance suite.** From "Recommended" (firewall + updates, 6 steps, no SSH risk) to "Full" (Tor + IPv6 disable + ASR + deep clean, 12 steps). Maintenance menu re-runs any step on a live box without re-hardening. @@ -45,9 +52,11 @@ the package name right per distro. This script: | SSH hardening | `PasswordAuthentication no` gated on validated pubkey; port never changed | | Fail2ban, sysctl profile, AppArmor/SELinux check | per-distro package names | | Tor, dnscrypt-proxy, unattended-upgrades, DeepClean | optional per profile | +| **HTTPS-only package transport** | enforced before every `apt`/`dnf`/`yum`/`zypper`/`pacman`/`apk`/`pip`/`npm`/… download — `--apt-https` | +| **Disconnect-safe** | SSH runs auto-wrap in tmux; subscripts get their own session; the reattach command is always printed | | **Rollback log** | `/var/log/linux-install-rollback.log` — `original\tbackup` per file | | **SSH self-heal** | `--install-self-heal` — systemd timer or cron, every 60s | -| **20-tool maintenance suite** | Re-runs any step, lists keys, tails logs, dumps config | +| **21-tool maintenance suite** | Re-runs any step, lists keys, tails logs, dumps config | ## Quick start @@ -82,8 +91,6 @@ the in-script `restore_ssh` routine or Tailscale SSH gets you back in. > → yum → apt`, so Arch derivatives pick `pacman`, SUSE picks `zypper`, > RHEL/Fedora pick `dnf`, Debian/Ubuntu pick `apt`. No manual flag required. -<<<<<<< HEAD -======= ## One-step automated setup Run the general interactive script directly from the repo — it prompts you @@ -114,7 +121,218 @@ SSH can get you back in. `━━━ PROGRESS ████████████░░░░ 12/17 (70%) ━━━`) before every step, so you always see what's already done and what's coming. ->>>>>>> origin/main +### Package transport security (HTTPS) + +A plaintext `http://` mirror means anyone who can intercept the route — a +hostile Wi-Fi, a compromised router, an upstream CDN node — can swap the +`.deb` / `.rpm` / `.pkgz` / wheel you just downloaded for one of theirs. +Signature checks catch *forged* packages, but they do not stop a +*downgrade* to an older, genuinely-signed, vulnerable build. Only TLS on +the transport closes that gap. + +So every entry point in this repo runs a guard **before** anything is +fetched. It is the first workflow step (on every profile, including +Custom), it runs once per process, and it is idempotent, so re-runs and +`--auto` are cheap. + +```bash +# Apply it on its own (normally automatic) +sudo bash linuxinstall.sh --apt-https + +# Report only; never modifies anything. Exit 1 if anything is plaintext. +sudo bash linuxinstall.sh --apt-https-audit + +# Undo: restore every backup and remove the policy drop-in. +sudo bash linuxinstall.sh --apt-https-off + +# Or standalone, no installer needed: +sudo bash lib/apt-https.sh # enforce +sudo bash lib/apt-https.sh --report # audit +sudo bash lib/apt-https.sh --revert # undo +``` + +#### What is checked + +This is **not** apt-specific. Every distro and language has its own "app +store", and several ship plaintext HTTP by default. The audit spans all of +them and only lists the ones actually installed: + +| Store | Where the transport is configured | +|---|---| +| `apt` | `/etc/apt/sources.list`, `sources.list.d/*.list`, DEB822 `*.sources` | +| `dnf` / `yum` | `/etc/yum.repos.d/*.repo` (`baseurl`, `metalink`, `mirrorlist`, `gpgkey`) | +| `zypper` | `/etc/zypp/repos.d/*.repo` | +| `pacman` | `/etc/pacman.d/*` (`Server=`) | +| `apk` (Alpine) | `/etc/apk/repositories` — **plaintext `http://` by default on many images** | +| `flatpak` | `flatpak remotes` | +| `snap` | store is snapd-managed; no operator-configurable transport | +| `docker` | `/etc/docker/daemon.json` (`registry-mirrors`, `insecure-registries`) | +| `brew` | `HOMEBREW_BREW_GIT_REMOTE`, `HOMEBREW_API_DOMAIN`, … | +| `pip` | `PIP_INDEX_URL`, `PIP_EXTRA_INDEX_URL`, `pip.conf` | +| `npm` | `NPM_CONFIG_REGISTRY`, `.npmrc` | +| `cargo` | `CARGO_REGISTRIES_CRATES_IO_INDEX`, `~/.cargo/config.toml` | +| `gem` | `GEM_SOURCE`, `.gemrc` | +| `nix` | `nix.conf` (`substituters`, `channel`) | +| `fwupd` | `/etc/fwupd/remotes.d/*.conf` (LVFS `UpdateURI`) | + +Detection is deliberately format-agnostic: **any non-comment line carrying an +`http://` URL** is reported. A per-dialect key list would miss `gpgkey=`, +`metalink=`, and whatever the next release adds — and a miss is exactly the +failure this is meant to prevent. The one exception is JSON, which has no +comment syntax: there, `http://` must sit at the start of a JSON string, so a +`"_comment": "see http://docs.internal"` note is not mistaken for a registry. + +`--apt-https-audit` prints a per-store verdict: + +``` +━━━ App-store transport security (HTTPS) ━━━ + Package manager: apt + [x] apt policy drop-in active: /etc/apt/apt.conf.d/99neohiro-force-https + + Store Result + apt (Debian/Ubuntu/Mint/Pop!/Kali) https only + apk (Alpine) 2 plaintext endpoint(s) + /etc/apk/repositories:1: http://dl-cdn.alpinelinux.org/alpine/v3.19/main + pip index 1 plaintext endpoint(s) + /etc/pip.conf:2: index-url = http://pypi.internal/simple +``` + +#### What is changed automatically + +Only **apt** is rewritten unattended, because it is the one store where the +result can be *verified*: after rewriting, a real `apt-get update` runs, and if +a mirror turns out not to serve the same paths over TLS the rewrite is **rolled +back automatically**. You are never left with a broken package manager. + +1. Installs `/etc/apt/apt.conf.d/99neohiro-force-https`: + + ```text + Acquire::https::AllowRedirect "true"; // https -> https redirects are fine + Acquire::http::AllowRedirect "false"; // https -> http is REFUSED, not followed + Acquire::https::Verify-Peer "true"; + Acquire::https::Verify-Host "true"; + Acquire::Retries "3"; + ``` + + The anti-downgrade line is the important one: without it, an + `https://` mirror can silently bounce you down to `http://`. + +2. Rewrites `http://` → `https://` on active lines in + `/etc/apt/sources.list`, `sources.list.d/*.list` (classic) and + `sources.list.d/*.sources` (DEB822 `URIs:` field only). + Commented-out lines and DEB822 structural fields (`Suites:`, + `Components:`, `Signed-By:`) are left byte-identical. + +3. Backs every file up to `/var/backups/neohiro-apt-https/` **before** the + first edit and records each in the rollback log, so + `bash linuxinstall.sh --rollback --apply` can undo it too. + +4. Applies each change with a **staging file in the same directory followed by + `rename(2)`**, not `cp` into place. `cp` truncates first, so a crash + mid-copy leaves a truncated `sources.list`; a rename is atomic, so a reader + sees either the whole old file or the whole new one. Original mode and + ownership are preserved. + +5. Warns if no CA trust store is present, since HTTPS verification is + worthless without one. + +#### Changing the others + +For every other store the guard **reports but does not rewrite**. Whether +`https://` actually exists cannot be known without a +network round trip, and silently breaking a working mirror is worse than the +threat it prevents. Two options: + +```bash +# Repoint the endpoint at an https-capable mirror (recommended), or +NEOHIRO_APT_HTTPS_REWRITE=1 sudo bash linuxinstall.sh --apt-https +``` + +The opt-in sweeps **every** installed store (apk, pip, npm, cargo, gem, nix, +docker, fwupd, and the RPM/Arch/SUSE repo formats) in one pass, backing each +up and honouring the same atomic replace. Re-audit afterwards, because some +mirrors genuinely do not serve the same paths over TLS. Stores whose tool is +not installed are left alone. + +`NEOHIRO_APT_HTTPS_STRICT=1` turns any leftover plaintext endpoint into a +hard error, which is what you want in CI. + +#### Where it is wired in + +`pkg_update` / `pkg_install` / `pkg_upgrade` / `pkg_autoremove`, +`update_system`, `update_kernel`, `updates_only_mode`, `restore_ssh.sh` +(before installing `openssh-server`), `DeepClean.sh` (before +`apt-get autoremove --purge`), the **Maintenance** submenu option 1, the +`--step apt_https` mode, and the `--apt-https*` flags. + +In `lib/updater.sh` it guards the `_run_all_updates` dispatcher plus every +sub-step that touches the network: `_update_apt`, `_update_dnf`, +`_update_yum`, `_update_zypper`, `_update_pacman`, `_update_snap`, +`_update_flatpak`, `_update_docker`, `_update_brew`, `_update_firmware`, +`_update_geoip`, `_update_pihole`. (`_update_virsh`, `_update_suse_snapper` +and `_update_btrfs_balance` only read or write local state, so they are +deliberately not guarded.) + +Two subtleties worth knowing: + +- **Fetched subscripts get the guard too.** `run_remote_script` pulls + `DeepClean.sh` / `OptimizeLinuxASR.sh` into a temp directory, and a script + resolves its helpers relative to its own location — so without help it + would find no `lib/`, and its `apt-get autoremove --purge` would run + unguarded even though the parent had already enforced. The installer + therefore prefetches `lib/apt-https.sh` next to the subscript. If a + subscript is `curl | bash`'d entirely on its own and finds no library, it + says so out loud rather than skipping the precaution silently. +- **`GEOIP_URL` must be `https://`.** It is the one operator-supplied + download target in the update engine, and a GeoIP database decides which + country a packet counts as being in — a tampered copy is a + traffic-tunneling primitive. An `http://` value is refused outright. + +`OptimizeLinuxASR.sh` does not download packages, so it needs no guard. + +#### Under `curl | sudo bash` + +`curl ... | sudo bash` has no `lib/` directory next to it. Rather than carry +a second copy of this logic (which is how fixes silently fail to reach the +most common install path), the script resolves `lib/apt-https.sh` from disk +if present, otherwise fetches it from the same raw base it already trusts for +`DeepClean.sh`, and sources that. If neither is possible it says so loudly +and runs with the guard inactive — it never pretends to be enforcing. + +| Variable | Effect | +|---|---| +| `NEOHIRO_APT_HTTPS=1` | enforce (default) | +| `NEOHIRO_APT_HTTPS=audit` | report only, never modify | +| `NEOHIRO_APT_HTTPS=0` | disable the guard entirely | +| `NEOHIRO_APT_HTTPS_REWRITE=1` | also rewrite the non-apt stores | +| `NEOHIRO_APT_HTTPS_NOVERIFY=1` | skip the post-rewrite `apt-get update` check | +| `NEOHIRO_APT_HTTPS_STRICT=1` | treat leftover plaintext as a hard error | +| `NEOHIRO_APT_BLOCK_PORT80=1` | also `ufw deny out 80/tcp` (opt-in, see below) | +| `NEOHIRO_APT_FAMILY=apt\|dnf\|yum\|zypper\|pacman\|none` | pin the detected family (CI containers, testing) | + +### Optional: block port 80 entirely + +If you want belt-and-braces so that *no* process can open an unencrypted +package connection, add an outbound deny rule: + +```bash +sudo bash linuxinstall.sh --apt-https # make sure every repo is https first +sudo ufw deny out 80/tcp && sudo ufw reload +# or: NEOHIRO_APT_BLOCK_PORT80=1 sudo bash linuxinstall.sh --apt-https +``` + +This is **opt-in** because a blanket outbound block also breaks unrelated +plaintext protocols (local registries, metrics endpoints, captive-portal +checks). The guard refuses to add the rule while any plaintext repo is +still configured, since that would only break those repos. + +Verify: + +```bash +sudo ufw status | grep -E '80/tcp|Status' # outbound DENY present +sudo bash linuxinstall.sh --apt-https-audit # exit 0 = no plaintext repos left +``` + ### Cross-distro kernel update `linuxinstall.sh` auto-detects the package manager and updates the kernel @@ -185,6 +403,35 @@ automatically and there's nothing to re-attach to. When the script finishes successfully, the tmux session closes itself; if it fails, the session is left intact for inspection. +### You can always get back to a running install + +This is the property that matters when something scary happens mid-run, so +it is enforced rather than hoped for: + +- **The reattach command is printed before the wrap, not after.** The script + `exec`s into tmux, so anything printed afterwards would never be seen. You + get the exact command — including a PID-suffixed name if the standard one + is taken — on screen at the moment you need it. +- **A leftover session is never hijacked.** If a `linux-setup` session + already exists (a previous run that did not exit cleanly), the script says + so and starts `linux-setup-` instead. You are never silently dropped + into an old, differently-flagged run while yours never starts. +- **Your flags survive the wrap.** The re-exec carries `--auto`, `--step`, + `--dry-run` and friends through, shell-quoted, so wrapping cannot change + what the run does. +- **Sessions are reaped on success.** A clean exit tears the session down, so + the next run starts clean. A failed run leaves it alive — that is your + inspection point. +- **Script hops get their own session.** `DeepClean.sh` / + `OptimizeLinuxASR.sh` run in a dedicated `neohiro-sub--` tmux + session with the reattach command printed, plus a heartbeat every 30s so a + long step does not look frozen (a frozen screen invites a Ctrl-C that + orphans the work). If there is no tmux to protect the hop, the script says + plainly that the step cannot be recovered. + +`tmux ls` lists sessions; `tmux attach -t ` reattaches; `Ctrl-b` then +`d` detaches without stopping anything. + ## Reconnecting after a reboot or lockout **Tailscale SSH bypasses OpenSSH settings** — it authenticates via the @@ -265,14 +512,15 @@ The Maintenance suite itself is expanded to include: | # | Option | What it does | |---|---|---| -| 1–13 | system / dns / firewall / tor / ssh / fail2ban / unattended / ipv6 / sysctl / apparmor / pam / OptimizeLinuxASR / DeepClean | Re-run any step on demand | -| 14 | SSH diagnostics & lockout fix | Same routine as `--restore-ssh` | -| 15 | Authorized keys | List all keys in every user's `authorized_keys` | -| 16 | SSH config review | Print every key directive from `sshd_config` and drop-ins | -| 17 | SSH self-heal guard | Install / remove / status of the per-minute watchdog | -| 18 | Logs | Tail `/var/log/linux-install-rollback.log` and `/var/log/neohiro-ssh-watchdog.log` | -| 19 | System info | Uptime, load, memory, disk, CPU, listening ports | -| 20 | Back to main menu | — | +| 1 | Force HTTPS for package repos | Enforces TLS for every repo before any download; prints the resulting state | +| 2–14 | system / dns / firewall / tor / ssh / fail2ban / unattended / ipv6 / sysctl / apparmor / pam / OptimizeLinuxASR / DeepClean | Re-run any step on demand | +| 15 | SSH diagnostics & lockout fix | Same routine as `--restore-ssh` | +| 16 | Authorized keys | List all keys in every user's `authorized_keys` | +| 17 | SSH config review | Print every key directive from `sshd_config` and drop-ins | +| 18 | SSH self-heal guard | Install / remove / status of the per-minute watchdog | +| 19 | Logs | Tail `/var/log/linux-install-rollback.log` and `/var/log/neohiro-ssh-watchdog.log` | +| 20 | System info | Uptime, load, memory, disk, CPU, listening ports | +| 21 | Back to main menu | — | The self-heal guard runs as root via `systemd` or cron and **never modifies `authorized_keys` or any credentials** — it only fixes config @@ -541,11 +789,20 @@ dumps — add to `/etc/security/limits.conf`: ### Testing ```bash -bash tests/test_linuxinstall.sh # 65 tests: parse, logic, UX coverage, snapshot -bash tests/test_updater.sh # 43 tests: dispatcher, race safety, version floor +bash tests/run-all.sh # every suite, with a summary +bash tests/test_linuxinstall.sh # 67 tests: parse, logic, UX coverage, snapshot +bash tests/test_apt_https.sh # 275 tests: transport guard, run continuity, encoding +bash tests/test_updater.sh # 45 tests: dispatcher, race safety, version floor shellcheck -S warning *.sh lib/*.sh tests/*.sh # lint ``` +`test_apt_https.sh` is hermetic: it relocates the whole `/etc` tree into a +temp sandbox, stubs `sudo`/`ufw`/`apk` on `PATH`, and never makes a network +call. It also enforces repository encoding hygiene (valid UTF-8, no +double-encoding artifacts, LF endings), because these scripts are full of +box-drawing characters and a silent re-encode is otherwise invisible until a +snapshot diff catches it. + To regenerate snapshot fixtures after a deliberate UX change: ```bash bash tests/gen_snapshots.sh # re-captures print_welcome + print_metrics_summary diff --git a/SECURITY.md b/SECURITY.md index b6b627d..d5ced1b 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -28,6 +28,59 @@ multiple distribution families (Debian, RHEL/Fedora, SUSE, Arch). Always review what will be applied, keep backups/restoration points, and test on non-critical systems first. +### Package and app-store transport + +Every package download, update, and upgrade is preceded by +`apt_https_guard` (see `lib/apt-https.sh`). The audit spans every app +store on the host, not just the distro package manager: + +`apt`, `dnf`, `yum`, `zypper`, `pacman`, `apk`, `flatpak`, `snap`, +`docker`, `brew`, `pip`, `npm`, `cargo`, `gem`, `nix`, `fwupd`. + +Detection is format-agnostic: any non-comment line carrying an `http://` +URL in a store's configuration, or a plaintext transport URL exported in +the environment, is reported. Signature verification catches *forged* +packages; only TLS prevents a *downgrade* to an older, genuinely-signed, +vulnerable build. + +On apt the guard: + +* installs `/etc/apt/apt.conf.d/99neohiro-force-https`, whose + `Acquire::http::AllowRedirect "false"` prevents an `https://` mirror + from silently downgrading a fetch to plaintext HTTP; +* rewrites `http://` repo URLs to `https://` in `sources.list`, + `sources.list.d/*.list`, and DEB822 `*.sources`; +* backs every file up to `/var/backups/neohiro-apt-https/` first and logs + it to `/var/log/linux-install-rollback.log`; +* applies each change by staging in the target's own directory and + `rename(2)`, so an interrupted run cannot leave a truncated repo file; +* runs a verification `apt-get update` and rolls the rewrite back + automatically if a mirror does not serve the same paths over HTTPS. + +Other stores are reported rather than rewritten, because whether +`https://` exists is not knowable offline and +silently breaking a working mirror is worse than the threat. Opt in with +`NEOHIRO_APT_HTTPS_REWRITE=1`. + +Escape hatches and audit tooling: + +```bash +sudo bash linuxinstall.sh --apt-https-audit # read-only, exit 1 if plaintext remains +sudo bash linuxinstall.sh --apt-https-off # restore backups, remove the drop-in +NEOHIRO_APT_HTTPS=0 # disable the guard for one run +``` + +The optional `NEOHIRO_APT_BLOCK_PORT80=1` adds `ufw deny out 80/tcp`. +It is off by default because a blanket outbound block also affects +unrelated plaintext protocols, and the guard declines to add it while +any plaintext endpoint is still configured. + +Under `curl | sudo bash` the guard is loaded from `lib/apt-https.sh` +(fetched from the same base the script already trusts) rather than +duplicated inline, so there is one implementation to audit. If it cannot +be loaded the script says so and runs with the guard inactive — it never +reports success for a precaution that is not running. + --- Maintained by **[neohiro](https://github.com/neohiro)**. diff --git a/lib/README.md b/lib/README.md index 82e698a..11797e0 100644 --- a/lib/README.md +++ b/lib/README.md @@ -10,6 +10,30 @@ each file for the API contract. EXIT trap cleans everything. ERR trap in STRICT_RUN / CI mode logs the failing command to `NEOHIRO_DEBUG_LOG` (default `/var/log/neohiro-debug.log`). +- `updater.sh` — the comprehensive cross-distro update engine + (`_run_all_updates` plus one `_update_*` per tool). Sourceable and + runnable standalone (`sudo bash lib/updater.sh`). +- `apt-https.sh` — repository/app-store transport guard. `apt_https_guard` + is called by every package entry point so nothing is ever fetched over + plaintext HTTP. It audits **every** app store, not just apt: `apt`, `dnf`, + `yum`, `zypper`, `pacman`, `apk`, `flatpak`, `snap`, `docker`, `brew`, + `pip`, `npm`, `cargo`, `gem`, `nix`, `fwupd` (see `_apt_https_source_ids`). + Detection is format-agnostic — any non-comment line carrying an `http://` + URL — so a new key in a new distro release cannot silently slip past. + On apt it installs `/etc/apt/apt.conf.d/99neohiro-force-https` (refusing + HTTPS→HTTP downgrade redirects), rewrites `http://` to `https://` across + `sources.list`, `sources.list.d/*.list` and DEB822 `*.sources`, verifies + with a real `apt-get update`, and rolls the rewrite back if a mirror does + not speak TLS. Every write is a same-directory staging file plus + `rename(2)`, so a crash cannot leave a truncated repo file. Other stores + are reported, and rewritten only on `NEOHIRO_APT_HTTPS_REWRITE=1`, + because whether `https://` exists is not knowable + offline. API: `apt_https_guard`, `apt_https_enforce`, `apt_https_report`, + `apt_https_revert`, `apt_https_status_text`, `apt_https_plaintext_for`; + also runnable standalone. `NEOHIRO_APT_ETC_DIR` / `NEOHIRO_APT_BACKUP_DIR` + relocate the whole tree, so tests never touch the real `/etc`. +- `sync-inline.sh`, `color-gate.sh` — shared inline-fallback sources + consumed by the `curl | bash` path of the top-level scripts. Top-level scripts (`linuxinstall.sh`, `restore_ssh.sh`, `DeepClean.sh`, `OptimizeLinuxASR.sh`) source these automatically @@ -17,5 +41,14 @@ when run from a clone. When run via `curl ... | bash`, the lib directory is not on disk, so each script falls back to inline definitions of the same helpers. -To regenerate the inline fallbacks, copy the body of each lib file -into the `else` branch in the top-level script. +`apt-https.sh` is the exception: it is never duplicated inline. +`linuxinstall.sh` resolves the canonical file — from disk, or by +fetching it from the same `REPO_RAW_BASE` it already trusts for +`DeepClean.sh` — and sources that, so there is exactly one +implementation. If it cannot be loaded the public entry points become +loud no-ops rather than undefined functions. `tests/test_apt_https.sh` +asserts both properties: the resolver is present, and no library +helper is redefined inline. + +To regenerate the inline fallbacks for the other helpers, copy the body +of each lib file into the `else` branch in the top-level script. diff --git a/lib/apt-https.sh b/lib/apt-https.sh new file mode 100644 index 0000000..36ddbd3 --- /dev/null +++ b/lib/apt-https.sh @@ -0,0 +1,1247 @@ +#!/usr/bin/env bash +# lib/apt-https.sh - HTTPS-only repository enforcement (precautionary guard). +# +# Purpose: BEFORE any package manager in this repo downloads, updates, or +# upgrades anything, make sure repository traffic is authenticated/TLS +# instead of plaintext HTTP. An on-path attacker who can rewrite an +# http:// mirror URL can otherwise inject arbitrary .deb/.rpm/.pkgz files +# into an otherwise fully-hardened machine. +# +# What "enforce" means per family: +# +# apt (Debian / Ubuntu / Mint / Pop!_OS / Kali / derivatives) +# 1. Install a managed policy drop-in +# /etc/apt/apt.conf.d/99neohiro-force-https: +# - Acquire::https::AllowRedirect "true" -> https mirrors that +# redirect within https are allowed. +# - Acquire::http::AllowRedirect "false" -> a redirect from +# https:// down to http:// is REFUSED instead of silently +# followed. This is the anti-downgrade control. +# - Acquire::https::Verify-Peer/Verify-Host "true" +# - Acquire::Retries "3" +# 2. Rewrite http:// -> https:// in every repo definition: +# /etc/apt/sources.list +# /etc/apt/sources.list.d/*.list (classic one-line format) +# /etc/apt/sources.list.d/*.sources (DEB822: URIs: field only) +# Comments are left untouched. Every touched file is backed up to +# /var/backups/neohiro-apt-https/ before the first edit. +# 3. Verify with a real `apt-get update`. If the rewrite broke the +# mirrors, the backups are restored automatically so the machine is +# never left with an unusable package manager. +# +# dnf / yum / zypper / pacman / flatpak +# No blind rewrite. Many upstream mirrors do not serve the same paths +# over TLS, and silently breaking a working repo is worse than the +# threat. Instead: audit, report every plaintext entry, and offer an +# opt-in rewrite via NEOHIRO_APT_HTTPS_REWRITE=1. +# +# Optional hardening (NOT default, because a global outbound port-80 block +# also breaks legitimate plaintext tooling such as local registries and +# metrics endpoints): +# NEOHIRO_APT_BLOCK_PORT80=1 -> ufw deny out 80/tcp +# +# Full environment surface: +# NEOHIRO_APT_HTTPS=1 enforce (default) +# NEOHIRO_APT_HTTPS=audit report only, never write +# NEOHIRO_APT_HTTPS=0 disable entirely +# NEOHIRO_APT_HTTPS_REWRITE=1 allow http->https rewrite on non-apt families +# NEOHIRO_APT_HTTPS_NOVERIFY=1 skip the post-rewrite `apt-get update` check +# NEOHIRO_APT_BLOCK_PORT80=1 add `ufw deny out 80/tcp` +# NEOHIRO_APT_ETC_DIR=path relocate the /etc tree (used by the tests) +# NEOHIRO_APT_BACKUP_DIR=path relocate the backup tree (used by the tests) +# +# Usage: +# source lib/apt-https.sh +# apt_https_guard # idempotent, once per process; use in hot paths +# apt_https_enforce # force a full pass (menus / CLI) +# apt_https_report # read-only status; rc 0 = all repos are https +# apt_https_revert # restore every backup taken by this lib +# +# Bash 4.0+ is required, matching the rest of the repo. + +if [ "${BASH_VERSINFO[0]:-0}" -lt 4 ]; then + printf '%s\n' "lib/apt-https.sh requires Bash 4.0+; found ${BASH_VERSION:-unknown}." >&2 + return 1 2>/dev/null || exit 1 +fi + +# Guard against double-sourcing. +[ -n "${__NEOHIRO_APT_HTTPS_INIT:-}" ] && return 0 2>/dev/null || true +__NEOHIRO_APT_HTTPS_INIT=1 + +# Re-entrancy / once-per-process latch. Set BEFORE any work so a nested +# call (e.g. the verification `apt-get update` re-entering pkg_update) +# short-circuits instead of recursing. +_APT_HTTPS_DONE="${_APT_HTTPS_DONE:-}" + +# Print helpers. Reuse the host's when available, otherwise define +# self-contained fallbacks so this file works standalone. +if ! declare -F _c >/dev/null 2>&1; then + _c() { if [ "${USE_COLOR:-0}" = "1" ]; then printf '\033[%s%s\033[0m' "$1" "$2"; else printf '%s' "$2"; fi; } +fi +if ! declare -F info >/dev/null 2>&1; then + info() { printf ' %s\n' "$*"; } +fi +if ! declare -F msg >/dev/null 2>&1; then + msg() { echo "=> $*"; } +fi +if ! declare -F ok >/dev/null 2>&1; then + ok() { printf '%s %s\n' "$(_c '1;32m' '[OK]')" "$*"; } +fi +if ! declare -F warn >/dev/null 2>&1; then + warn() { printf '%s %s\n' "$(_c '1;33m' '[WARNING]')" "$*" >&2; } +fi +if ! declare -F err >/dev/null 2>&1; then + err() { printf '%s %s\n' "$(_c '1;31m' '[ERROR]')" "$*" >&2; } +fi + +# ── Paths ──────────────────────────────────────────────────────────────────── + +APT_HTTPS_CONF_NAME="99neohiro-force-https" + +apt_https_etc_dir() { + printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}" +} + +apt_https_backup_dir() { + printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}" +} + +# _apt_https_conf_file — absolute path of the managed apt policy drop-in. +apt_https_conf_file() { + printf '%s' "$(apt_https_etc_dir)/apt/apt.conf.d/${APT_HTTPS_CONF_NAME}" +} + +# ── Privilege / dry-run plumbing ───────────────────────────────────────────── + +_apt_https_is_root() { + [ "${EUID:-$(id -u 2>/dev/null || echo 1000)}" = "0" ] +} + +# _apt_priv — run a privileged command, honouring DRY_RUN. +# Returns 1 when privileges are unavailable so callers can degrade. +_apt_priv() { + if [ "${DRY_RUN:-0}" = "1" ]; then + printf ' DRY: %s\n' "$*" + return 0 + fi + if _apt_https_is_root; then + "$@" + return $? + fi + if command -v sudo >/dev/null 2>&1; then + sudo "$@" + return $? + fi + return 1 +} + +# _apt_https_privileged_ok — true when we can actually write to /etc. +_apt_https_privileged_ok() { + _apt_https_is_root && return 0 + command -v sudo >/dev/null 2>&1 && return 0 + return 1 +} + +# ── Family detection ───────────────────────────────────────────────────────── +# Same precedence as detect_distro() in linuxinstall.sh and pkg_mgr() in +# DeepClean.sh: pacman -> zypper -> dnf -> yum -> apt. +# +# NEOHIRO_APT_FAMILY pins the result. That is what lets the test suite (and +# a CI container running a foreign base image) exercise the apt, dnf, +# zypper, pacman and "none" code paths deterministically. + +apt_https_family() { + if [ -n "${NEOHIRO_APT_FAMILY:-}" ]; then + printf '%s' "$NEOHIRO_APT_FAMILY" + return 0 + fi + if command -v pacman >/dev/null 2>&1 && [ -f /etc/pacman.conf ]; then + printf 'pacman' + elif command -v zypper >/dev/null 2>&1; then + printf 'zypper' + elif command -v dnf >/dev/null 2>&1; then + printf 'dnf' + elif command -v yum >/dev/null 2>&1; then + printf 'yum' + elif command -v apt-get >/dev/null 2>&1 || [ -f /etc/apt/sources.list ]; then + printf 'apt' + else + printf 'none' + fi +} + +# ── Repo file discovery ────────────────────────────────────────────────────── + +# _apt_https_repo_files — one repo-definition path per line (apt only). +_apt_https_repo_files() { + local etc f + etc="$(apt_https_etc_dir)" + [ -f "${etc}/apt/sources.list" ] && printf '%s\n' "${etc}/apt/sources.list" + for f in "${etc}"/apt/sources.list.d/*.list "${etc}"/apt/sources.list.d/*.sources; do + [ -f "$f" ] && printf '%s\n' "$f" + done + return 0 +} + +# _apt_https_foreign_repo_files — repo definitions for the other +# families, so the audit has something to read even when the detected family +# is different (dual-boot / container with a foreign sources dir). +_apt_https_foreign_repo_files() { + local etc f d + etc="$(apt_https_etc_dir)" + case "$1" in + dnf|yum) + for f in "${etc}"/yum.repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done + ;; + zypper) + for f in "${etc}"/zypp/repos.d/*.repo; do [ -f "$f" ] && printf '%s\n' "$f"; done + ;; + pacman) + for f in "${etc}"/pacman.d/*; do [ -f "$f" ] && printf '%s\n' "$f"; done + ;; + *) : ;; + esac + return 0 +} + +# ── Plaintext detection ────────────────────────────────────────────────────── + +# _apt_https_plaintext_in_apt_file +# Prints the offending (line-number + trimmed-line) pairs. DEB822 files are +# matched on the URIs: field only, so a commented-out http:// note never +# counts. Classic files skip comment lines entirely. +_apt_https_plaintext_in_apt_file() { + local f="$1" + [ -f "$f" ] || return 0 + case "$f" in + *.sources) + awk '/^[[:space:]]*URIs:[[:space:]]/ && /http:\/\// { + gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null + ;; + *) + awk '/^[[:space:]]*#/ { next } /http:\/\// { + gsub(/^[[:space:]]+/, "", $0); printf "%s:%d: %s\n", FILENAME, NR, $0 }' "$f" 2>/dev/null + ;; + esac + return 0 +} + +# _apt_https_plaintext_in_repo_file +# Generic scan for the yum/dnf/zypper/pacman repo layouts. $2 is an ERE +# applied per line; the line must also contain http:// to be reported. +_apt_https_plaintext_in_repo_file() { + local f="$1" pat="$2" + [ -f "$f" ] || return 0 + awk -v pat="$pat" ' + /^[[:space:]]*#/ { next } + $0 ~ pat && /http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + return 0 +} + +# _apt_https_plaintext_generic +# +# The format-agnostic detector: any non-comment line carrying an http:// URL. +# Used for every non-apt store (apk repositories, docker daemon.json, pip.conf, +# .npmrc, cargo config.toml, .gemrc, nix.conf, fwupd remotes, ...). +# +# Being generic is the point. A per-dialect key list would miss gpgkey=, +# metalink=, and whatever the next distro release adds, and a miss is exactly +# the failure this library exists to prevent. +# +# One deliberate exception: JSON has no comment syntax, so skipping "#" lines +# cannot distinguish a doc link from a fetch target. In a .json file we +# therefore require http:// to sit at the start of a JSON string, which is +# where a URL value lives. Without this, docker daemon.json entries like +# {"_comment": "see http://docs.internal"} would be reported as plaintext +# registries and train users to ignore the report. +_apt_https_plaintext_generic() { + local f="$1" id="${2:-}" + [ -f "$f" ] || return 0 + case "$f" in + *.json) + awk ' + /"http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + ;; + *) + awk ' + /^[[:space:]]*(#|;)/ { next } + /http:\/\// { + line = $0 + gsub(/^[[:space:]]+/, "", line) + printf "%s:%d: %s\n", FILENAME, NR, line + }' "$f" 2>/dev/null + ;; + esac + return 0 +} + +# _apt_https_rewrite_generic — http:// -> https:// on non-comment lines. +# Same substitution the apt rewrite uses, so behaviour is identical. +_APT_HTTPS_GENERIC_RE='/^[[:space:]]*(#|;)/! s|http://|https://|g' + +# ── Backups ────────────────────────────────────────────────────────────────── + +# _apt_https_backup_path — deterministic, collision-resistant name. +# +# The path is flattened so it is a legal filename inside the backup +# directory, but flattening alone is NOT injective: /a/b/c and /a_b/c both +# become _a_b_c. A collision here means reverting one file restores another +# file's contents, which is a silent, hard-to-trace corruption of a package +# manager. So append a checksum of the real path. +# +# cksum (POSIX) is used rather than a shell hash so the value is stable +# across shells, runs, and machines -- these backups may be inspected or +# moved during an incident. +_apt_https_backup_path() { + local flat sum + flat="$(printf '%s' "$1" | tr -c 'A-Za-z0-9._-' '_')" + sum="$(printf '%s' "$1" | cksum 2>/dev/null | tr -d ' ' | cut -d' ' -f1)" + if [ -n "$sum" ]; then + printf '%s/%s-%s.orig' "$(apt_https_backup_dir)" "$flat" "$sum" + else + # cksum unavailable: the flattened name alone is still better than + # hashing nothing, and this is recorded so it is not mistaken for a bug. + printf '%s/%s.orig' "$(apt_https_backup_dir)" "$flat" + fi +} + +# _apt_https_backup_once — copy aside the first time we touch it. +# +# The copy is staged inside the backup directory and renamed into place, for +# the same reason the rewrite is: `cp` into a destination that does not exist +# yet leaves a truncated `.orig` if it dies partway. A truncated backup is +# worse than no backup, because `--apt-https-off` would then confidently +# restore a broken file. rename(2) means the backup is either absent or +# complete. +_apt_https_backup_once() { + local src="$1" dst stage + [ -f "$src" ] || return 0 + dst="$(_apt_https_backup_path "$src")" + [ -f "$dst" ] && return 0 + _apt_priv mkdir -p "$(apt_https_backup_dir)" || return 1 + stage="${dst}.partial.$$" + _apt_priv rm -f "$stage" 2>/dev/null || true + if ! _apt_priv cp -p "$src" "$stage"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + warn "Could not back up $src — refusing to edit it." + return 1 + fi + if ! _apt_priv mv -f "$stage" "$dst"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + warn "Could not store the backup for $src — refusing to edit it." + return 1 + fi + # Hook into the host's rollback log when one exists so `linuxinstall.sh + # --rollback` can undo this too. + if declare -F record_backup >/dev/null 2>&1; then + record_backup "$src" "$dst" 2>/dev/null || true + fi + info "Backed up $src -> $dst" + return 0 +} + +# ── apt policy drop-in ─────────────────────────────────────────────────────── + +# _apt_https_write_conf — install the managed apt.conf.d policy file. +# Idempotent: byte-identical content is left alone (no backup churn). +_apt_https_write_conf() { + local conf tmp rc + conf="$(apt_https_conf_file)" + tmp="$(_apt_https_stage)" + { + printf '%s\n' '// Managed by neohiro/linux (lib/apt-https.sh). Do not edit.' + printf '%s\n' '// Re-running the installer overwrites this file safely;' + printf '%s\n' '// `bash linuxinstall.sh --apt-https-off` removes it.' + printf '%s\n' '' + printf '%s\n' '// Prefer TLS, and never silently downgrade to plaintext.' + printf '%s\n' 'Acquire::https::AllowRedirect "true";' + printf '%s\n' 'Acquire::http::AllowRedirect "false";' + printf '%s\n' 'Acquire::ftp::AllowRedirect "false";' + printf '%s\n' '' + printf '%s\n' '// Authenticate the mirror, not just the channel.' + printf '%s\n' 'Acquire::https::Verify-Peer "true";' + printf '%s\n' 'Acquire::https::Verify-Host "true";' + printf '%s\n' '' + printf '%s\n' '// Transparent mirrors usually need more than one try.' + printf '%s\n' 'Acquire::Retries "3";' + printf '%s\n' '' + printf '%s\n' '// Pin a corporate proxy by uncommenting and editing:' + printf '%s\n' '// Acquire::http::Proxy "http://proxy.corp.example:3128";' + printf '%s\n' '// Acquire::https::Proxy "http://proxy.corp.example:3128";' + } > "$tmp" + # Already correct? Then there is nothing to do. + if [ -f "$conf" ] && cmp -s "$tmp" "$conf"; then + rm -f "$tmp" + return 0 + fi + if ! _apt_https_backup_once "$conf"; then + rm -f "$tmp" + return 1 + fi + _apt_priv mkdir -p "$(apt_https_etc_dir)/apt/apt.conf.d" || { rm -f "$tmp"; return 1; } + # Atomic rename into place; see _apt_https_atomic_replace for why. + if _apt_https_atomic_replace "$conf" 0644 _apt_https_prep_conf_stage "$tmp"; then + rc=0 + else + rc=1 + fi + rm -f "$tmp" + [ "$rc" = "0" ] || return 1 + info "apt policy: $conf" + return 0 +} + +# apt_https_available — is the real guard loaded? +# +# Callers that infer "the repositories are fine" from an empty +# apt_https_status_text must check this first. When the library cannot be +# loaded, a stub takes its place and status_text returns empty for "found +# nothing" -- indistinguishable from "checked, all https". Reporting that as +# verified would be an unverified security claim printed as fact. +apt_https_available() { + return 0 +} + +# _apt_https_stage — private scratch path for generated files. +_apt_https_stage() { + if declare -F _tmpfile >/dev/null 2>&1; then + _tmpfile apt-https + return 0 + fi + mktemp "${TMPDIR:-/tmp}/neohiro-apt-https.XXXXXX" +} + +# _apt_https_atomic_replace [prep-args...] +# +# Runs [prep-args...] to build a staging file that lives +# in the *target's own directory*, then renames it over the target. +# +# Why not just `cp tmp target`: cp truncates the destination and then +# writes. If the process dies mid-copy — OOM, SIGKILL, a container being +# stopped — the repo file is left truncated, i.e. a broken package manager. +# rename(2) within a single filesystem is atomic, so a reader (apt itself, +# or a concurrent run of this script) sees either the whole old file or the +# whole new one, never a half-written mix. Staging in the target's own +# directory guarantees the same filesystem, which is what makes the rename +# atomic instead of a slow cross-device copy. +# +# The staging name ends in the PID and therefore never ends in ".list" or +# ".sources", so apt's own sources.list.d globs cannot pick it up. +# +# is a chmod mode for the result ("-" = leave it alone, which is how +# the repo rewrite preserves the original file's mode and ownership). +_apt_https_atomic_replace() { + local target="$1" mode="$2" prep="$3"; shift 3 + local stage + stage="${target}.neohiro-rewrite.$$" + _apt_priv rm -f "$stage" 2>/dev/null || true + if ! "$prep" "$stage" "$@"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + return 1 + fi + if [ "$mode" != "-" ]; then + if ! _apt_priv chmod "$mode" "$stage"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + return 1 + fi + fi + if ! _apt_priv mv -f "$stage" "$target"; then + _apt_priv rm -f "$stage" 2>/dev/null || true + return 1 + fi + return 0 +} + +# _apt_https_prep_conf_stage — plain content copy. +_apt_https_prep_conf_stage() { + _apt_priv cp "$2" "$1" +} + +# _apt_https_prep_repo_stage +# +# Builds the staged rewrite so the renamed result keeps the original file's +# mode and ownership. `cp -p` clones those attributes onto the staging file, +# then `cp` over the *existing* staging file replaces only its content -- +# POSIX only applies the source's permissions when cp CREATES the +# destination, so the cloned attributes survive. +# +# This deliberately avoids `sed -i`: whether GNU sed, busybox sed, or a BSD +# variant preserve the mode across an in-place edit is implementation +# detail, and this file is edited while root on a machine whose package +# manager must keep working. +_apt_https_prep_repo_stage() { + _apt_priv cp -p "$2" "$1" && _apt_priv cp "$3" "$1" +} + +# ── apt source rewriting ───────────────────────────────────────────────────── +# +# The rewriters below report their result through globals +# (_APT_HTTPS_VERDICT / _APT_HTTPS_CHANGED / _APT_HTTPS_REVERTED / +# _APT_HTTPS_REFUSED) instead of stdout. They emit progress messages of +# their own, and a command substitution would fold that text into the value +# it returns -- which then lands inside an arithmetic expansion and turns a +# word like "Restored" into a variable lookup. Globals keep the message +# channel and the value channel separate. + +_APT_HTTPS_VERDICT="" +_APT_HTTPS_CHANGED=0 +_APT_HTTPS_REVERTED=0 +# Files that DID contain plaintext but could not be rewritten (no writable +# backup, or the privileged copy failed). Tracked separately so +# apt_https_enforce never claims "already https" when it actually gave up. +_APT_HTTPS_REFUSED=0 +# Latch for the CA-trust-store warning, which would otherwise repeat once per +# call site (enforce AND report both check it). This MUST be initialised here, +# at the top level: a report-only run never calls the rewrite path, so +# initialising it inside a function left it unbound under `set -u`. +_APT_HTTPS_CA_WARNED="" + +# _apt_https_rewrite_file +# Sets _APT_HTTPS_VERDICT to "changed" or "clean". Backs the file up before +# the first edit and refuses to edit a file it could not back up. +_apt_https_rewrite_file() { + local f="$1" tmp expr="${2:-}" + _APT_HTTPS_VERDICT="clean" + [ -f "$f" ] || return 0 + + if [ -z "$expr" ]; then + # DEB822 must only be rewritten on the URIs: field; everything else is a + # flat key=value or URL-per-line format, where any active line counts. + case "$f" in + *.sources) expr='/^[[:space:]]*URIs:[[:space:]]/ s|http://|https://|g' ;; + *) expr="$_APT_HTTPS_GENERIC_RE" ;; + esac + fi + + tmp="$(_apt_https_stage)" + if ! sed -E "$expr" "$f" > "$tmp" 2>/dev/null; then + rm -f "$tmp" + _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) + return 0 + fi + if cmp -s "$tmp" "$f"; then + rm -f "$tmp" + return 0 + fi + # Refuse to edit anything we cannot first back up. + if ! _apt_https_backup_once "$f"; then + rm -f "$tmp" + _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) + return 0 + fi + # Atomic replace that keeps the original file's mode and ownership. + if ! _apt_https_atomic_replace "$f" - _apt_https_prep_repo_stage "$f" "$tmp"; then + rm -f "$tmp" + _APT_HTTPS_REFUSED=$((_APT_HTTPS_REFUSED + 1)) + return 0 + fi + rm -f "$tmp" + _APT_HTTPS_VERDICT="changed" + return 0 +} + +# _apt_https_rewrite_apt_sources — rewrite every apt repo definition. +# Sets _APT_HTTPS_CHANGED to the number of files rewritten. +_apt_https_rewrite_apt_sources() { + local f + _APT_HTTPS_CHANGED=0 + _APT_HTTPS_REFUSED=0 + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + if [ "$_APT_HTTPS_VERDICT" = "changed" ]; then + _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + info "Rewrote plaintext repo URLs -> https: $f" + fi + done < <(_apt_https_repo_files) + return 0 +} + +# _apt_https_revert_all_sources — undo the rewrite for EVERY source, not just +# apt. Sets _APT_HTTPS_REVERTED to the number of files restored. +_apt_https_revert_all_sources() { + local f bak backupdir + _APT_HTTPS_REVERTED=0 + backupdir="$(apt_https_backup_dir)" + [ -d "$backupdir" ] || return 0 + while IFS= read -r f; do + [ -n "$f" ] || continue + bak="$(_apt_https_backup_path "$f")" + if [ -f "$bak" ] && ! cmp -s "$bak" "$f"; then + if _apt_priv cp "$bak" "$f"; then + _APT_HTTPS_REVERTED=$((_APT_HTTPS_REVERTED + 1)) + info "Restored $f from backup" + fi + fi + done < <(_apt_https_managed_files) + return 0 +} + +# Kept as an alias: the original name is referenced by the inline fallback and +# by callers that only ever touched apt. +_apt_https_revert_apt_sources() { + _apt_https_revert_all_sources +} + +# ── App-store source registry ──────────────────────────────────────────────── +# The precaution is not apt-specific: every distribution and language has its +# own "app store", and several of them default to plaintext HTTP. This +# registry is the single list of every source this library inspects. +# +# Three per-source questions: +# files — where the transport is configured on disk +# env — where it is configured through the environment +# family — the repo-file dialect (drives which keys hold URLs) +# +# Detection is deliberately generic: ANY non-comment line containing an +# http:// URL in one of those files is reported. Per-dialect key regexes +# (baseurl=, Server=, index-url, registry, substituters, ...) would miss +# gpgkey=, metalkink= and whatever the next release adds, and a miss is +# exactly the failure mode this library exists to prevent. +# +# Rewriting is a separate, opt-in step (see apt_https_enforce) because +# whether https:// actually exists is not knowable +# without a network round trip. Rewriting blind would turn a working mirror +# into a broken one, which is worse than the threat. apt is the exception: +# it is verified with a real `apt-get update` and rolled back on failure. + +# _apt_https_source_ids — every source id, one per line. +_apt_https_source_ids() { + printf '%s\n' \ + apt dnf yum zypper pacman apk \ + flatpak snap docker brew pip npm cargo gem nix fwupd +} + +# _apt_https_source_label — human name for the report. +_apt_https_source_label() { + case "$1" in + apt) printf 'apt (Debian/Ubuntu/Mint/Pop!/Kali)' ;; + dnf) printf 'dnf (RHEL 8+/Fedora/Alma/Rocky)' ;; + yum) printf 'yum (CentOS 7/RHEL 7)' ;; + zypper) printf 'zypper (openSUSE/SLES)' ;; + pacman) printf 'pacman (Arch/Manjaro)' ;; + apk) printf 'apk (Alpine)' ;; + flatpak) printf 'flatpak remotes' ;; + snap) printf 'snap (store)' ;; + docker) printf 'docker registry config' ;; + brew) printf 'Homebrew taps' ;; + pip) printf 'pip index' ;; + npm) printf 'npm registry' ;; + cargo) printf 'cargo registry' ;; + gem) printf 'gem sources' ;; + nix) printf 'nix substituters/channels' ;; + fwupd) printf 'fwupd remotes (LVFS)' ;; + *) printf '%s' "$1" ;; + esac +} + +# _apt_https_source_files — config files that carry this source's URLs. +_apt_https_source_files() { + local etc f home + etc="$(apt_https_etc_dir)" + home="${HOME:-}" + case "$1" in + apt) _apt_https_repo_files ;; + dnf|yum) _apt_https_foreign_repo_files dnf ;; + zypper) _apt_https_foreign_repo_files zypper ;; + pacman) _apt_https_foreign_repo_files pacman ;; + # Alpine: one URL per line, http:// by default on many images. + apk) + [ -f "${etc}/apk/repositories" ] && printf '%s\n' "${etc}/apk/repositories" + ;; + # Registry mirrors / insecure-registries live in daemon.json. + docker) + [ -f "${etc}/docker/daemon.json" ] && printf '%s\n' "${etc}/docker/daemon.json" + ;; + # pip: system config plus the two variables that usually win. + pip) + for f in "${etc}/pip.conf" "${etc}/xdg/pip/pip.conf"; do + [ -f "$f" ] && printf '%s\n' "$f" + done + # Per-user config only makes sense with a real HOME. Without this guard + # an unset HOME builds "/.config/pip/pip.conf" -- a path at the + # filesystem root, which would be probed and could match. + if [ -n "$home" ]; then + for f in "${home}/.pip/pip.conf" "${home}/.config/pip/pip.conf"; do + [ -f "$f" ] && printf '%s\n' "$f" + done + fi + ;; + npm) + [ -f "${etc}/npmrc" ] && printf '%s\n' "${etc}/npmrc" + [ -n "$home" ] && [ -f "${home}/.npmrc" ] && printf '%s\n' "${home}/.npmrc" + ;; + cargo) + if [ -n "$home" ] && [ -f "${home}/.cargo/config.toml" ]; then + printf '%s\n' "${home}/.cargo/config.toml" + fi + [ -f "${etc}/cargo/config.toml" ] && printf '%s\n' "${etc}/cargo/config.toml" + ;; + gem) + [ -f "${etc}/gemrc" ] && printf '%s\n' "${etc}/gemrc" + [ -n "$home" ] && [ -f "${home}/.gemrc" ] && printf '%s\n' "${home}/.gemrc" + ;; + nix) + [ -f "${etc}/nix/nix.conf" ] && printf '%s\n' "${etc}/nix/nix.conf" + ;; + fwupd) + for f in "${etc}"/fwupd/remotes.d/*.conf; do + [ -f "$f" ] && printf '%s\n' "$f" + done + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_source_env — transport configured through the environment. +# Printed as "VAR=value" so the user gets a copy-pasteable fix. +_apt_https_source_env() { + case "$1" in + pip) + [ -n "${PIP_INDEX_URL:-}" ] && printf 'PIP_INDEX_URL=%s\n' "$PIP_INDEX_URL" + [ -n "${PIP_EXTRA_INDEX_URL:-}" ] && printf 'PIP_EXTRA_INDEX_URL=%s\n' "$PIP_EXTRA_INDEX_URL" + ;; + npm) + [ -n "${NPM_CONFIG_REGISTRY:-}" ] && printf 'NPM_CONFIG_REGISTRY=%s\n' "$NPM_CONFIG_REGISTRY" + [ -n "${npm_config_registry:-}" ] && printf 'npm_config_registry=%s\n' "$npm_config_registry" + ;; + cargo) + [ -n "${CARGO_REGISTRIES_CRATES_IO_INDEX:-}" ] && \ + printf 'CARGO_REGISTRIES_CRATES_IO_INDEX=%s\n' "$CARGO_REGISTRIES_CRATES_IO_INDEX" + ;; + gem) + [ -n "${GEM_SOURCE:-}" ] && printf 'GEM_SOURCE=%s\n' "$GEM_SOURCE" + ;; + brew) + [ -n "${HOMEBREW_BREW_GIT_REMOTE:-}" ] && printf 'HOMEBREW_BREW_GIT_REMOTE=%s\n' "$HOMEBREW_BREW_GIT_REMOTE" + [ -n "${HOMEBREW_CORE_GIT_REMOTE:-}" ] && printf 'HOMEBREW_CORE_GIT_REMOTE=%s\n' "$HOMEBREW_CORE_GIT_REMOTE" + [ -n "${HOMEBREW_API_DOMAIN:-}" ] && printf 'HOMEBREW_API_DOMAIN=%s\n' "$HOMEBREW_API_DOMAIN" + [ -n "${HOMEBREW_ARTIFACT_DOMAIN:-}" ] && printf 'HOMEBREW_ARTIFACT_DOMAIN=%s\n' "$HOMEBREW_ARTIFACT_DOMAIN" + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_source_applicable [family] +# False when this host has no such store, so the report does not list eleven +# N/A rows on a minimal box. +# +# The detected family is passed in by callers rather than re-probed here. +# apt_https_family costs up to five `command -v` PATH scans, and the report +# loop called it once per RPM/Arch store -- twice over, since the report and +# the per-source query each ran the loop. That is wasted work and a latent +# inconsistency: if PATH changed mid-run a store could be "applicable" in one +# pass and missing from the next, and the same file could be reported twice. +_apt_https_source_applicable() { + local id="$1" fam="${2:-}" + case "$id" in + # Only the package manager actually present should be audited as such. + dnf|yum|zypper|pacman) + [ -n "$fam" ] || fam="$(apt_https_family)" + [ "$fam" = "$id" ] || return 1 + ;; + snap) + command -v snap >/dev/null 2>&1 || return 1 + ;; + flatpak) + command -v flatpak >/dev/null 2>&1 || return 1 + ;; + docker) + command -v docker >/dev/null 2>&1 || return 1 + ;; + brew) + command -v brew >/dev/null 2>&1 || return 1 + ;; + pip) + command -v pip3 >/dev/null 2>&1 || command -v pip >/dev/null 2>&1 || return 1 + ;; + npm) + command -v npm >/dev/null 2>&1 || return 1 + ;; + cargo) + command -v cargo >/dev/null 2>&1 || return 1 + ;; + gem) + command -v gem >/dev/null 2>&1 || return 1 + ;; + apk) + command -v apk >/dev/null 2>&1 || return 1 + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_source_hint — one short line on how to fix this store. +# +# Printed only for stores that actually have a plaintext endpoint. Without it +# the report says "repoint at https" and leaves the reader to work out where +# that setting lives -- and for flatpak and snap there is no config file this +# library can rewrite at all, so the generic advice is actively misleading. +_apt_https_source_hint() { + case "$1" in + apt) printf 'rewrite automatically: sudo bash linuxinstall.sh --apt-https' ;; + dnf|yum) printf 'edit baseurl=/metalink=/mirrorlist= under /etc/yum.repos.d/*.repo' ;; + zypper) printf 'edit baseurl=/uri= under /etc/zypp/repos.d/*.repo' ;; + pacman) printf 'edit Server= lines in /etc/pacman.d/mirrorlist' ;; + apk) printf 'rewrite with NEOHIRO_APT_HTTPS_REWRITE=1, or edit /etc/apk/repositories' ;; + docker) printf 'set registry-mirrors to https:// in /etc/docker/daemon.json, drop insecure-registries, then restart docker' ;; + brew) printf 'export HOMEBREW_API_DOMAIN / HOMEBREW_BREW_GIT_REMOTE with an https:// URL' ;; + pip) printf 'export PIP_INDEX_URL=https://... (and PIP_EXTRA_INDEX_URL) or fix pip.conf' ;; + npm) printf 'npm config set registry https://registry.npmjs.org/' ;; + cargo) printf 'set the crates-io registry to sparse+https:// in ~/.cargo/config.toml' ;; + gem) printf 'gem sources --remove && gem sources --add https://rubygems.org/' ;; + nix) printf 'use substituters = https://cache.nixos.org in nix.conf' ;; + # No operator-editable transport config; a URL rewrite cannot fix these. + flatpak) printf 'flatpak remote-modify --url=https://... (no config file to rewrite)' ;; + snap) printf 'snapd-managed store; cannot be repointed without a custom snapd build' ;; + *) : ;; + esac +} + +# _apt_https_source_plaintext — report this source's plaintext endpoints. +_apt_https_source_plaintext() { + local id="$1" f + case "$id" in + apt) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_apt_file "$f" + done < <(_apt_https_repo_files) + ;; + flatpak) + _apt_https_flatpak_plaintext + ;; + *) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_generic "$f" "$id" + done < <(_apt_https_source_files "$id") + ;; + esac + _apt_https_source_env "$id" + return 0 +} + +# _apt_https_source_prefer_https +# Rewrites http:// -> https:// on non-comment lines of every config file for +# this source. Sets _APT_HTTPS_CHANGED / _APT_HTTPS_REFUSED. Opt-in only, +# except for apt (see apt_https_enforce). +_apt_https_source_prefer_https() { + local id="$1" f + _APT_HTTPS_CHANGED=0 + _APT_HTTPS_REFUSED=0 + [ "$id" = "apt" ] && { _apt_https_rewrite_apt_sources; return 0; } + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + if [ "$_APT_HTTPS_VERDICT" = "changed" ]; then + _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + info "Rewrote plaintext URLs -> https: $f" + fi + done < <(_apt_https_source_files "$id") + return 0 +} + +# _apt_https_managed_files — every file this library may have edited. +_apt_https_managed_files() { + local id f + while IFS= read -r id; do + [ -n "$id" ] || continue + while IFS= read -r f; do + [ -n "$f" ] && printf '%s\n' "$f" + done < <(_apt_https_source_files "$id") + done < <(_apt_https_source_ids) + return 0 +} + +# ── Non-apt audit / optional rewrite ───────────────────────────────────────── + +_DNF_URL_RE='^([[:space:]]*)(baseurl|metalink|mirrorlist)[[:space:]]*=' +_ZYPPER_URL_RE='^([[:space:]]*)(baseurl|uri|mirrorlist)[[:space:]]*=' +_PACMAN_URL_RE='^([[:space:]]*)Server[[:space:]]*=' + +# _apt_https_audit_family — print plaintext repo lines. +_apt_https_audit_family() { + local fam="$1" f + case "$fam" in + apt) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_apt_file "$f" + done < <(_apt_https_repo_files) + ;; + dnf|yum) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" "$_DNF_URL_RE" + done < <(_apt_https_foreign_repo_files "$fam") + ;; + zypper) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" "$_ZYPPER_URL_RE" + done < <(_apt_https_foreign_repo_files "$fam") + ;; + pacman) + while IFS= read -r f; do + [ -n "$f" ] && _apt_https_plaintext_in_repo_file "$f" "$_PACMAN_URL_RE" + done < <(_apt_https_foreign_repo_files "$fam") + ;; + *) : ;; + esac + return 0 +} + +# _apt_https_rewrite_family — opt-in http->https for non-apt +# families (NEOHIRO_APT_HTTPS_REWRITE=1). +# Sets _APT_HTTPS_CHANGED to the number of files rewritten. +_apt_https_rewrite_family() { + local fam="$1" f + _APT_HTTPS_CHANGED=0 + case "$fam" in + dnf|yum|zypper|pacman) : ;; + *) return 0 ;; + esac + while IFS= read -r f; do + [ -n "$f" ] || continue + _apt_https_rewrite_file "$f" + [ "$_APT_HTTPS_VERDICT" = "changed" ] && _APT_HTTPS_CHANGED=$((_APT_HTTPS_CHANGED + 1)) + done < <(_apt_https_foreign_repo_files "$fam") + return 0 +} + +# ── flatpak remotes ────────────────────────────────────────────────────────── + +_apt_https_flatpak_plaintext() { + command -v flatpak >/dev/null 2>&1 || return 0 + flatpak remotes --show-details 2>/dev/null | awk '/http:\/\// { print }' || true + return 0 +} + +# ── ca-certificates sanity check ───────────────────────────────────────────── + +# HTTPS verification is worthless without a trust store. Warn (never fail) +# when it is missing, because installing it would itself need a download. +# +# Latched to once per process: apt_https_enforce and apt_https_report both +# check this, and the maintenance menu and --apt-https path call both, which +# used to print the identical warning two or three times in one run. +_apt_https_check_ca_certs() { + local fam="$1" p + case "$fam" in + apt) + if [ -e /etc/ssl/certs/ca-certificates.crt ] || [ -e /etc/pki/tls/certs/ca-bundle.crt ]; then + return 0 + fi + p="ca-certificates" + ;; + dnf|yum) [ -e /etc/pki/tls/certs/ca-bundle.crt ] && return 0; p="ca-certificates" ;; + zypper) [ -e /etc/ssl/ca-bundle.pem ] && return 0; p="ca-certificates" ;; + pacman) [ -e /etc/ssl/certs/ca-certificates.crt ] && return 0; p="ca-certificates" ;; + *) return 0 ;; + esac + [ -n "$_APT_HTTPS_CA_WARNED" ] && return 1 + _APT_HTTPS_CA_WARNED=1 + warn "No system CA trust store found. HTTPS verification of packages will fail." + info "Install '$p' after the sources are trusted, then re-run: apt_https_enforce" + return 1 +} + +# ── Optional outbound port-80 block ────────────────────────────────────────── + +# _apt_https_block_port80 — belt-and-braces so no process can open an +# unencrypted repo connection. Opt-in (NEOHIRO_APT_BLOCK_PORT80=1) because a +# blanket outbound block also affects unrelated plaintext protocols. +_apt_https_block_port80() { + [ "${NEOHIRO_APT_BLOCK_PORT80:-0}" = "1" ] || return 0 + if ! command -v ufw >/dev/null 2>&1; then + if command -v firewall-cmd >/dev/null 2>&1; then + warn "NEOHIRO_APT_BLOCK_PORT80=1 but only firewalld is present." + info "firewalld needs an explicit rich rule; skipping to avoid locking yourself out." + else + warn "NEOHIRO_APT_BLOCK_PORT80=1 but no UFW/firewalld found; skipping." + fi + return 0 + fi + if ufw status 2>/dev/null | grep -qE '^80/tcp[[:space:]]+DENY[[:space:]]+OUT'; then + info "ufw already denies outbound 80/tcp" + return 0 + fi + # Refuse while plaintext repos remain: the block would break them anyway. + if [ -n "$(apt_https_status_text)" ]; then + warn "Skipping the port-80 block: plaintext repos are still configured." + info "Fix them first (see the report above), then re-run with NEOHIRO_APT_BLOCK_PORT80=1." + return 0 + fi + if _apt_priv ufw deny out 80/tcp; then + ok "Blocked outbound TCP/80 (apt traffic is HTTPS-only from now on)." + info "Undo: sudo ufw delete deny out 80/tcp" + else + warn "Could not add the ufw outbound 80/tcp deny rule." + fi + return 0 +} + +# ── Status / report ────────────────────────────────────────────────────────── + +# apt_https_status_text — every plaintext endpoint across every app store. +# One line per offending config line, plus "VAR=value" for env-configured +# transports, so the output is directly actionable. +apt_https_status_text() { + local id fam + fam="$(apt_https_family)" + while IFS= read -r id; do + [ -n "$id" ] || continue + _apt_https_source_applicable "$id" "$fam" || continue + _apt_https_source_plaintext "$id" + done < <(_apt_https_source_ids) + return 0 +} + +# _apt_https_plaintext_for [family] — findings for one source only. +apt_https_plaintext_for() { + _apt_https_source_applicable "$1" "${2:-}" || return 0 + _apt_https_source_plaintext "$1" +} + +# apt_https_report — per-source state summary. +# Returns 0 when nothing anywhere uses plaintext, 1 when any source does, +# 2 when no package manager at all was detected. +apt_https_report() { + local fam conf findings id label n clean + fam="$(apt_https_family)" + printf '\n%s\n' "$(_c '1;36m' '━━━ App-store transport security (HTTPS) ━━━')" + printf ' %-34s %s\n' "Package manager:" "$fam" + if [ "$fam" = "apt" ]; then + conf="$(apt_https_conf_file)" + if [ -f "$conf" ]; then + printf ' %s %s\n' "$(_c '1;32m' '[x]')" "apt policy drop-in active: $conf" + else + printf ' %s %s\n' "$(_c '1;31m' '[ ]')" "apt policy drop-in MISSING: $conf" + fi + fi + printf '\n %s\n' "$(_c '1;37m' 'Store Result')" + + clean=1 + while IFS= read -r id; do + [ -n "$id" ] || continue + _apt_https_source_applicable "$id" "$fam" || continue + label="$(_apt_https_source_label "$id")" + findings="$(apt_https_plaintext_for "$id" "$fam")" + if [ -z "$findings" ]; then + printf ' %-30s %s\n' "$label" "$(_c '1;32m' 'https only')" + else + n="$(printf '%s\n' "$findings" | wc -l | tr -d ' ')" + printf ' %-30s %s\n' "$label" "$(_c '1;31m' "$n plaintext endpoint(s)")" + printf '%s\n' "$findings" | sed 's/^/ /' + printf ' %s\n' "$(_apt_https_source_hint "$id")" + clean=0 + fi + done < <(_apt_https_source_ids) + + _apt_https_check_ca_certs "$fam" || true + + printf '\n' + if [ "$clean" = "1" ]; then + printf ' %s %s\n' "$(_c '1;32m' '[x]')" "No plaintext endpoints in any configured app store." + else + printf ' %s %s\n' "$(_c '1;33m' '[!]')" "Some stores still fetch over plaintext HTTP." + printf ' %s\n' " Use the per-store hint above. apt needs nothing: --apt-https" + printf '%s\n' " rewrites it unattended (and rolls back if a mirror cannot speak" + printf '%s\n' " TLS). For the others, repoint the endpoint, or opt in to a blanket" + printf '%s\n' " rewrite with NEOHIRO_APT_HTTPS_REWRITE=1 and re-audit afterwards." + fi + printf '\n' + + if [ "$fam" = "none" ]; then return 2; fi + [ "$clean" = "1" ] && return 0 + return 1 +} + +# ── Enforce ────────────────────────────────────────────────────────────────── + +# _apt_https_verify_apt_sources — run a real `apt-get update` and roll back +# the rewrite if the mirrors turned out not to speak https. +_apt_https_verify_apt_sources() { + [ "${NEOHIRO_APT_HTTPS_NOVERIFY:-0}" = "1" ] && return 0 + [ "${DRY_RUN:-0}" = "1" ] && return 0 + command -v apt-get >/dev/null 2>&1 || return 0 + + info "Verifying the rewritten sources with a real apt-get update..." + if _apt_priv env DEBIAN_FRONTEND=noninteractive apt-get update -qq; then + ok "All apt repositories answered over HTTPS." + return 0 + fi + if [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ]; then + err "apt-get update failed after the https rewrite and NEOHIRO_APT_HTTPS_STRICT=1." + err "Nothing was rolled back. Fix the repo URLs, or re-run with --apt-https-off." + return 1 + fi + warn "apt-get update failed after the https rewrite — rolling back to the backups." + _apt_https_revert_apt_sources >/dev/null + warn "Reverted. A mirror does not serve the same paths over HTTPS." + info "Repoint that repo at an https-capable mirror, then re-run --apt-https." + return 1 +} + +# apt_https_enforce [label] — apply the precaution once, unconditionally. +# [label] is a short breadcrumb describing the caller, e.g. "pkg_install". +apt_https_enforce() { + local label="${1:-enforce}" mode fam conf_failed findings rc=0 rewrite id n + mode="${NEOHIRO_APT_HTTPS:-1}" + + fam="$(apt_https_family)" + if [ "$fam" = "none" ] && [ "$mode" != "audit" ]; then + info "No supported package manager detected — HTTPS guard not applicable." + return 0 + fi + + if [ "$mode" = "0" ]; then + info "NEOHIRO_APT_HTTPS=0 — repository transport guard disabled ($label)." + return 0 + fi + + msg "Repository transport guard ($label): package manager = $fam" + + if [ "$mode" = "audit" ]; then + info "NEOHIRO_APT_HTTPS=audit — reporting only, nothing is modified." + apt_https_report || true + return 0 + fi + + if ! _apt_https_privileged_ok; then + warn "Not root and no sudo available — cannot enforce HTTPS for repositories." + info "Re-run as root, or set NEOHIRO_APT_HTTPS=0 to silence this." + return 0 + fi + + # ---- automatic: apt only ------------------------------------------------- + # The one store we rewrite unattended, because a real `apt-get update` can + # verify the result and roll the rewrite back if the mirror cannot speak TLS. + if [ "$fam" = "apt" ]; then + conf_failed=0 + _apt_https_write_conf || conf_failed=1 + _apt_https_rewrite_apt_sources + if [ "$_APT_HTTPS_CHANGED" -gt 0 ]; then + ok "Rewrote $_APT_HTTPS_CHANGED apt source file(s) to https://" + _apt_https_verify_apt_sources || rc=1 + elif [ "$_APT_HTTPS_REFUSED" -gt 0 ]; then + warn "Could not rewrite $_APT_HTTPS_REFUSED apt source file(s): no writable backup." + info "Check that $(apt_https_backup_dir) is writable by root, then re-run." + rc=1 + else + ok "apt repositories already use https://" + fi + if [ "$conf_failed" = "1" ]; then + warn "Could not install the apt policy drop-in." + rc=1 + fi + fi + + # ---- opt-in: every other store ------------------------------------------- + # Whether https:// exists is unknowable without a + # network round trip, so this is never automatic. + if [ "${NEOHIRO_APT_HTTPS_REWRITE:-0}" = "1" ]; then + while IFS= read -r id; do + [ -n "$id" ] || continue + [ "$id" = "apt" ] && continue + _apt_https_source_applicable "$id" "$fam" || continue + _apt_https_source_prefer_https "$id" + n="$_APT_HTTPS_CHANGED" + if [ "${n:-0}" -gt 0 ] 2>/dev/null; then + ok "Rewrote $n $(_apt_https_source_label "$id") file(s) to https://" + fi + if [ "${_APT_HTTPS_REFUSED:-0}" -gt 0 ] 2>/dev/null; then + warn "Could not rewrite $_APT_HTTPS_REFUSED $(_apt_https_source_label "$id") file(s): no writable backup." + fi + done < <(_apt_https_source_ids) + info "Rewrote non-apt stores because NEOHIRO_APT_HTTPS_REWRITE=1." + info "Re-audit with --apt-https-audit; not every mirror serves the same paths over TLS." + fi + + # ---- report --------------------------------------------------------------- + findings="$(apt_https_status_text)" + if [ -n "$findings" ]; then + warn "Plaintext endpoints still configured:" + printf '%s\n' "$findings" | sed 's/^/ /' + [ "$fam" = "apt" ] || info "Repoint these at an https:// endpoint, or opt in to a" + [ "$fam" = "apt" ] || info "blind rewrite with NEOHIRO_APT_HTTPS_REWRITE=1." + if [ "${NEOHIRO_APT_HTTPS_STRICT:-0}" = "1" ]; then + rc=1 + fi + else + ok "Every configured app store uses https://" + fi + + _apt_https_check_ca_certs "$fam" || true + _apt_https_block_port80 + return "$rc" +} + +# apt_https_guard [label] — hot-path wrapper. Enforces at most once per +# process so every package entry point can call it unconditionally. +# +# It always returns 0. A guard must never be the reason a package operation +# fails: call sites are things like `pkg_install`, which run under `set -e` +# in some of the standalone scripts, and a non-zero return there would abort +# the caller before it ever reached the package manager. Enforcement failures +# are reported through apt_https_report and the enforce exit status, not by +# failing the hot path. +apt_https_guard() { + [ -n "$_APT_HTTPS_DONE" ] && return 0 + # Latch first: the verification `apt-get update` below can re-enter the + # package layer, which must not recurse. + _APT_HTTPS_DONE=1 + apt_https_enforce "${1:-guard}" || true + return 0 +} + +# apt_https_revert — undo everything this lib changed. +apt_https_revert() { + local conf n + conf="$(apt_https_conf_file)" + _apt_https_revert_apt_sources + n="$_APT_HTTPS_REVERTED" + if [ -f "$conf" ] && [ -f "$(_apt_https_backup_path "$conf")" ]; then + if _apt_priv cp "$(_apt_https_backup_path "$conf")" "$conf"; then + info "Restored $conf from backup" + fi + elif [ -f "$conf" ]; then + # No backup means we created it: removing it is the correct revert. + if _apt_priv rm -f "$conf"; then + ok "Removed $conf" + fi + fi + if [ "$_APT_HTTPS_REVERTED" -gt 0 ] 2>/dev/null; then + ok "Reverted $_APT_HTTPS_REVERTED repository file(s) to their pre-guard contents." + fi + _APT_HTTPS_DONE="" + ok "Repository transport guard disabled." + return 0 +} + +# ── Standalone entry point ─────────────────────────────────────────────────── +# sudo bash lib/apt-https.sh enforce +# sudo bash lib/apt-https.sh --report audit only +# sudo bash lib/apt-https.sh --revert undo +if [ "${BASH_SOURCE[0]:-$0}" = "$0" ]; then + # Standalone: this file sets no `set -e`, but capture the status anyway so + # the documented exit-code contract (0 = clean, 1 = plaintext remains, + # 2 = no package manager) is explicit rather than an accident of which + # command happened to run last. + _ap_https_rc=0 + case "${1:---enforce}" in + --report|--audit) apt_https_report || _ap_https_rc=$? ;; + --revert) apt_https_revert || _ap_https_rc=$? ;; + --enforce) apt_https_enforce "cli" || _ap_https_rc=$? ;; + *) printf 'usage: bash lib/apt-https.sh [--enforce|--report|--revert]\n' >&2 + _ap_https_rc=2 ;; + esac + exit "$_ap_https_rc" +fi \ No newline at end of file diff --git a/lib/updater.sh b/lib/updater.sh index a527af8..1562d1e 100644 --- a/lib/updater.sh +++ b/lib/updater.sh @@ -163,10 +163,21 @@ _cmd() { run "$@" } +# ── Repository transport guard ──────────────────────────────────────────────── +# Sourced here (after the print helpers and `run` exist) so the update engine +# can never fetch a package over plaintext HTTP, whether it is run standalone +# (`sudo bash lib/updater.sh`) or called from linuxinstall.sh. The guard is a +# no-op when it has already run in this process. +# shellcheck disable=SC1091 +if [ -r "$(dirname "${BASH_SOURCE[0]:-$0}")/apt-https.sh" ]; then + source "$(dirname "${BASH_SOURCE[0]:-$0}")/apt-https.sh" +fi + # ── Package managers ───────────────────────────────────────────────────────── _update_apt() { command -v apt >/dev/null 2>&1 || return 0 + apt_https_guard "_update_apt" || true msg "apt: updating package lists..." if ! run sudo env DEBIAN_FRONTEND=noninteractive apt-get update -qq; then err "apt update failed"; _track; return 1 @@ -192,6 +203,7 @@ _update_apt() { _update_dnf() { command -v dnf >/dev/null 2>&1 || return 0 + apt_https_guard "_update_dnf" || true msg "dnf: checking for updates..." if ! run sudo dnf upgrade --refresh -y -q; then err "dnf upgrade failed"; _track; return 1 @@ -203,6 +215,7 @@ _update_dnf() { _update_yum() { command -v yum >/dev/null 2>&1 || return 0 + apt_https_guard "_update_yum" || true msg "yum: checking for updates..." if ! run sudo yum update -y -q; then err "yum update failed"; _track; return 1 @@ -214,6 +227,7 @@ _update_yum() { _update_zypper() { command -v zypper >/dev/null 2>&1 || return 0 + apt_https_guard "_update_zypper" || true msg "zypper: refreshing + updating..." if ! run sudo zypper --quiet refresh; then err "zypper refresh failed"; _track; return 1 @@ -228,6 +242,7 @@ _update_zypper() { _update_pacman() { command -v pacman >/dev/null 2>&1 || return 0 + apt_https_guard "_update_pacman" || true msg "pacman: syncing + upgrading..." if ! run sudo pacman -Syu --noconfirm --quiet; then err "pacman update failed"; _track; return 1 @@ -242,6 +257,7 @@ _update_pacman() { _update_snap() { command -v snap >/dev/null 2>&1 || return 0 + apt_https_guard "_update_snap" || true msg "snap: refreshing all snaps..." # Track snap refresh outcome so the dispatcher summary is accurate. if _cmd sudo snap refresh; then @@ -285,6 +301,7 @@ _update_snap() { _update_flatpak() { command -v flatpak >/dev/null 2>&1 || return 0 + apt_https_guard "_update_flatpak" || true msg "flatpak: updating remote repos + all installations..." local rc=0 # `flatpak remote-ls --updates` exits 0 whether or not there are updates @@ -318,6 +335,7 @@ _update_flatpak() { _update_docker() { command -v docker >/dev/null 2>&1 || return 0 + apt_https_guard "_update_docker" || true msg "docker: pulling latest images..." local images images=$(docker images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null | grep -v '' || true) @@ -347,6 +365,7 @@ _update_docker() { _update_brew() { command -v brew >/dev/null 2>&1 || return 0 + apt_https_guard "_update_brew" || true msg "brew: updating..." local brew_failed=0 if ! HOMEBREW_NO_ANALYTICS=1 run brew update 2>/dev/null; then @@ -365,6 +384,7 @@ _update_brew() { _update_firmware() { command -v fwupdmgr >/dev/null 2>&1 || return 0 + apt_https_guard "_update_firmware" || true msg "fwupdmgr: refreshing metadata..." run sudo fwupdmgr refresh 2>/dev/null || true local available @@ -399,6 +419,7 @@ _update_geoip() { # Export: GEOIP_URL=https://your-mirror.example.com/GeoLite2-Country.mmdb # (default) — community-maintained fork (no account needed). local _geoip_url="" _key + apt_https_guard "_update_geoip" || true if [ -n "${MAXMIND_LICENSE_KEY:-}" ]; then _key=$(printf '%s' "${MAXMIND_LICENSE_KEY}" | tr -d '[:space:]') if [ -n "$_key" ]; then @@ -411,6 +432,24 @@ _update_geoip() { if [ -z "$_geoip_url" ]; then _geoip_url="https://raw.githubusercontent.com/maccurry/GeoIP-country/main/GeoLite2-Country.mmdb" fi + # GEOIP_URL is operator-supplied, so it is the one download target in this + # engine that can be pointed at plaintext. A GeoIP database decides which + # country a packet is "in", so a tampered copy is a traffic-tunneling + # primitive. Refuse http:// outright rather than silently fetching it. + case "$_geoip_url" in + https://*) : ;; + http://*) + err "GEOIP_URL is plaintext (http://). Refusing to download the GeoIP database over HTTP." + info "Use an https:// mirror, or unset GEOIP_URL to use the default fork." + _track + return 1 + ;; + *) + err "GEOIP_URL must be an https:// URL; got: ${_geoip_url%%\?*}" + _track + return 1 + ;; + esac # Common locations for GeoIP Country database. local _geoip_db geoip_candidates=( "/usr/share/GeoIP/GeoLite2-Country.mmdb" @@ -575,6 +614,7 @@ _update_btrfs_balance() { _update_pihole() { command -v pihole >/dev/null 2>&1 || return 0 + apt_https_guard "_update_pihole" || true if [ "$EUID" -ne 0 ]; then _log "pihole: requires root — skipping" return 0 @@ -619,6 +659,10 @@ _update_pihole() { # VERBOSE=2 — trace every command # DRY_RUN=1 — simulate all commands without running them # +# Repository transport: apt_https_guard runs before the first sub-step, so +# nothing in this dispatcher can pull a package over plaintext HTTP. It is +# idempotent, so the per-_update_* guards cost nothing. +# # Exit: 0 = all succeeded, 1 = one or more sub-steps had errors. _run_all_updates() { @@ -632,6 +676,9 @@ _run_all_updates() { esac done + # Precaution before a single byte is downloaded. + apt_https_guard "_run_all_updates" || true + msg "=== Comprehensive system update ===" local start_sec=$SECONDS @@ -763,6 +810,32 @@ else # Called as a script. Pass all arguments to _run_all_updates. set -euo pipefail SECONDS=0 + # Sub-commands handled here rather than by the dispatcher, because they + # only touch the repository transport and never run an update. + # + # `set -euo pipefail` is active in this branch, so a bare call that returns + # non-zero would exit before `exit $?` runs and skip the report. Capture + # the status explicitly instead of relying on that coincidence. + _ap_https_rc=0 + case "${1:-}" in + --apt-https|--enforce-https) + if [ "${2:-}" = "--audit" ]; then + apt_https_report || _ap_https_rc=$? + else + apt_https_enforce "updater --apt-https" || _ap_https_rc=$? + apt_https_report || true + fi + exit "$_ap_https_rc" + ;; + --apt-https-audit) + apt_https_report || _ap_https_rc=$? + exit "$_ap_https_rc" + ;; + --apt-https-off|--disable-https) + apt_https_revert || _ap_https_rc=$? + exit "$_ap_https_rc" + ;; + esac _run_all_updates "$@" exit $? fi diff --git a/linuxinstall.sh b/linuxinstall.sh index 444c072..c58b6b6 100644 --- a/linuxinstall.sh +++ b/linuxinstall.sh @@ -40,6 +40,10 @@ if [ -n "$_NEOHIRO_LIB_DIR" ] && [ -r "$_NEOHIRO_LIB_DIR/color.sh" ]; then if [ -r "$_NEOHIRO_LIB_DIR/updater.sh" ]; then source "$_NEOHIRO_LIB_DIR/updater.sh" fi + # shellcheck disable=SC1091 + if [ -r "$_NEOHIRO_LIB_DIR/apt-https.sh" ]; then + source "$_NEOHIRO_LIB_DIR/apt-https.sh" + fi else # Inline fallback for run-from-pipe (curl ... | bash) where the lib # directory is not on disk. Sources the canonical color-gate function from @@ -158,42 +162,53 @@ if ! declare -F _run_all_updates >/dev/null 2>&1; then VERBOSE="${VERBOSE:-0}"; UPDATED=0; FAILED=0 _log() { [ "$VERBOSE" = "1" ] && info "$*" || true; } _update_apt() { command -v apt >/dev/null 2>&1 || return 0 + apt_https_guard "_update_apt" || true run sudo env DEBIAN_FRONTEND=noninteractive apt-get update -qq || return 1 run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y -qq full-upgrade run sudo env DEBIAN_FRONTEND=noninteractive apt-get -y autoremove -qq run sudo apt-get clean -qq; } _update_dnf() { command -v dnf >/dev/null 2>&1 || return 0 + apt_https_guard "_update_dnf" || true run sudo dnf upgrade --refresh -y -q || return 1 run sudo dnf autoremove -y -q; } _update_yum() { command -v yum >/dev/null 2>&1 || return 0 + apt_https_guard "_update_yum" || true run sudo yum update -y -q || return 1 run sudo yum autoremove -y -q; } _update_zypper(){ command -v zypper >/dev/null 2>&1 || return 0 + apt_https_guard "_update_zypper" || true run sudo zypper --quiet refresh run sudo zypper update -y --quiet || return 1 run sudo zypper --quiet clean; } _update_pacman(){ command -v pacman >/dev/null 2>&1 || return 0 + apt_https_guard "_update_pacman" || true run sudo pacman -Syu --noconfirm --quiet || return 1 run sudo pacman -Scc --noconfirm -q; } _update_snap() { command -v snap >/dev/null 2>&1 || return 0 + apt_https_guard "_update_snap" || true run sudo snap refresh 2>/dev/null || true; } _update_flatpak(){ command -v flatpak >/dev/null 2>&1 || return 0 + apt_https_guard "_update_flatpak" || true run flatpak update -y --assumeyes 2>/dev/null || true run flatpak uninstall --unused -y --assumeyes 2>/dev/null || true; } _update_docker() { command -v docker >/dev/null 2>&1 || return 0 + apt_https_guard "_update_docker" || true while IFS= read -r img; do [ -z "$img" ] && continue docker pull "$img" >/dev/null 2>&1 || true done < <(docker images --format '{{.Repository}}:{{.Tag}}' 2>/dev/null | grep -v '') docker image prune -f >/dev/null 2>&1 || true; } _update_brew() { command -v brew >/dev/null 2>&1 || return 0 + apt_https_guard "_update_brew" || true HOMEBREW_NO_ANALYTICS=1 run brew update 2>/dev/null || true run brew upgrade 2>/dev/null || true run brew cleanup -s -q 2>/dev/null || true; } _update_firmware(){ command -v fwupdmgr >/dev/null 2>&1 || return 0 + apt_https_guard "_update_firmware" || true run sudo fwupdmgr refresh 2>/dev/null || true run sudo fwupdmgr update -y --no-reboot-check 2>/dev/null || true; } _run_all_updates() { + apt_https_guard "_run_all_updates" || true msg "=== Comprehensive system update ===" _update_apt || true _update_dnf || true @@ -208,6 +223,86 @@ if ! declare -F _run_all_updates >/dev/null 2>&1; then printf '\n'; ok "Update engine complete."; } fi +# ── lib/apt-https.sh resolution (curl|bash path) ───────────────────────────── +# `curl ... | sudo bash` has no lib/ directory next to this script, so this +# block used to carry a full inline copy of the repository transport guard. +# That was ~400 lines of security-critical code duplicated from a canonical +# source, which is the worst possible arrangement: fixes and new store +# coverage silently did not reach the most common install path. +# +# Instead, resolve the canonical library -- from disk if it is there, +# otherwise by fetching it from the same raw base this script already trusts +# for DeepClean.sh and OptimizeLinuxASR.sh -- and source that. One +# implementation, no drift, and the curl|bash path gets the same coverage as a +# clone (every distro package manager plus apk, flatpak, docker, brew, pip, +# npm, cargo, gem, nix and fwupd). +# +# If neither works (no network, filtered egress), define the public entry +# points as loud no-ops so every caller still works and the operator is told +# the precaution is inactive rather than being left to assume it is on. +if ! declare -F apt_https_guard >/dev/null 2>&1; then + _apt_https_resolved="" + for _ah_cand in \ + "${_NEOHIRO_LIB_DIR:-}/apt-https.sh" \ + "/usr/local/lib/neohiro/apt-https.sh" \ + "$(dirname "$(readlink -f "${BASH_SOURCE[0]:-$0}" 2>/dev/null)" 2>/dev/null)/lib/apt-https.sh"; do + if [ -n "$_ah_cand" ] && [ -r "$_ah_cand" ] && [ -s "$_ah_cand" ]; then + _apt_https_resolved="$_ah_cand" + break + fi + done + + if [ -z "$_apt_https_resolved" ]; then + _apt_https_dir="$(mktemp -d "${TMPDIR:-/tmp}/neohiro-https.XXXXXX" 2>/dev/null)" || _apt_https_dir="" + if [ -n "$_apt_https_dir" ]; then + if command -v curl >/dev/null 2>&1; then + curl -fsSL "${REPO_RAW_BASE}/lib/apt-https.sh" \ + -o "$_apt_https_dir/apt-https.sh" 2>/dev/null || true + elif command -v wget >/dev/null 2>&1; then + wget -qO "$_apt_https_dir/apt-https.sh" \ + "${REPO_RAW_BASE}/lib/apt-https.sh" 2>/dev/null || true + fi + if [ -s "$_apt_https_dir/apt-https.sh" ]; then + _apt_https_resolved="$_apt_https_dir/apt-https.sh" + else + rm -rf "$_apt_https_dir" 2>/dev/null || true + fi + fi + fi + + if [ -n "$_apt_https_resolved" ]; then + # shellcheck disable=SC1090 + . "$_apt_https_resolved" 2>/dev/null || _apt_https_resolved="" + fi + + if ! declare -F apt_https_guard >/dev/null 2>&1; then + # Latch the warning to once per process. This stub stands in for a hot-path + # helper called by every pkg_* / update_* entry point, so without a latch + # a single run printed the same four lines a dozen times and buried the + # actual installer output. + _apt_https_unavailable() { + [ -n "${_APT_HTTPS_DEGRADED_WARNED:-}" ] && return 0 + _APT_HTTPS_DEGRADED_WARNED=1 + printf '%s\n' "[WARNING] lib/apt-https.sh could not be loaded, so the repository" >&2 + printf '%s\n' "[WARNING] transport guard is INACTIVE for this run: packages may" >&2 + printf '%s\n' "[WARNING] still be fetched over plaintext HTTP. Run from a clone of" >&2 + printf '%s\n' "[WARNING] neohiro/linux to get the full guard." >&2 + return 0 + } + apt_https_guard() { _apt_https_unavailable; return 0; } + apt_https_enforce() { _apt_https_unavailable; return 0; } + apt_https_report() { _apt_https_unavailable; return 0; } + apt_https_revert() { _apt_https_unavailable; return 0; } + apt_https_status_text(){ return 0; } + apt_https_backup_dir() { printf '%s' "${NEOHIRO_APT_BACKUP_DIR:-/var/backups/neohiro-apt-https}"; } + apt_https_conf_file() { printf '%s' "${NEOHIRO_APT_ETC_DIR:-/etc}/apt/apt.conf.d/99neohiro-force-https"; } + # Reported so no caller can mistake "the guard found nothing" for + # "the guard checked and the repos are fine". See _auto_skip_if_done. + apt_https_available() { return 1; } + fi + unset _apt_https_resolved _apt_https_dir _ah_cand 2>/dev/null || true +fi + RECOVERY_CMD="tmux attach -t linux-setup # reconnect after SSH disconnect" ROLLBACK_LOG="${ROLLBACK_LOG:-/var/log/linux-install-rollback.log}" @@ -399,6 +494,24 @@ print_recovery_if_ssh() { print_recovery_cmd } +# Wrap an SSH run in a tmux session so a dropped socket cannot abort it. +# +# Three things this must get right, each of which was a bug: +# +# 1. Arguments must survive the re-exec. `bash "$SCRIPT_PATH"` with no +# "$@" silently dropped --auto / --step / --dry-run, so the run behaved +# differently after wrapping than the user asked for. +# 2. A stale session must never be hijacked. `tmux new-session -A` attaches +# to an existing session if one exists, which means the new run never +# starts and the user is dropped into someone else's (possibly older, +# differently-flagged) run with no way back to theirs. A leftover +# session from a previous run is exactly how this happens, so the +# session is torn down on clean exit and a name collision falls back to +# a PID-suffixed name instead of attaching. +# 3. The recovery command must be on screen BEFORE exec, because exec +# replaces this process and nothing after it can print. If the user +# detaches or the socket drops, "tmux attach -t " is the only way +# back and they must not have to remember it. ensure_tmux_if_ssh() { [ -n "${SSH_CONNECTION:-}${SSH_TTY:-}" ] || return 0 [ -z "${TMUX:-}" ] && [ -z "${STY:-}" ] || return 0 @@ -408,28 +521,63 @@ ensure_tmux_if_ssh() { fi fi command -v tmux >/dev/null 2>&1 || { - warn "tmux unavailable; SSH disconnect may kill the run. ${RECOVERY_CMD} will NOT exist - run the script from a local terminal instead." + warn "tmux unavailable; an SSH disconnect WILL kill this run and there is no way to reattach." + warn "Recovery: ${RECOVERY_CMD} will NOT exist -- run the script from a local terminal instead." return 0 } [ -n "$SCRIPT_PATH" ] || { warn "SCRIPT_PATH is empty; cannot re-exec inside tmux."; return 0; } - bold "SSH session detected. Wrapping this run in a tmux session so disconnects do not abort it." - # Quote everything via env+args (no string interpolation) so paths with spaces - # or shell metacharacters survive. The inner bash re-execs the same script - # by absolute path; on clean exit it tears the tmux session down. - local inner - inner=$(cat <<'INNER_EOF' -trap 'tmux kill-session -t linux-setup 2>/dev/null' EXIT -cd "$1" && shift -bash "$1" "$@" + + # Never attach to a pre-existing session: pick a free name instead. + local _sess="linux-setup" + if tmux has-session -t "$_sess" 2>/dev/null; then + warn "A tmux session named '$_sess' already exists (a previous run that did not exit cleanly)." + warn "Starting this run as '$_sess-$$' instead so it is never mixed up with the old one." + _sess="linux-setup-$$" + fi + + bold "SSH session detected. Wrapping this run in tmux so disconnects do not abort it." + # Shell-quote every argument so paths with spaces or metacharacters survive. + # printf '%q ' with zero arguments emits nothing, so a flagless run stays clean. + local _args="" _a + for _a in "$@"; do _args="$_args$(printf ' %q' "$_a")"; done + + # Print the recovery command BEFORE exec: after exec nothing in this process + # can print, and this is the only moment the user is guaranteed to see it. + printf '\n' + ok "This run is now inside tmux. If you detach or lose the SSH socket, resume with:" + bold " tmux attach -t $_sess" + printf ' (detach without stopping: Ctrl-b then d. list sessions: tmux ls)\n\n' + + # Inner wrapper: tears the session down on a clean exit so the next run is + # not poisoned by a leftover session. On failure the session is deliberately + # left alive -- that is the recovery path. + # Argument order: