From a974a3eda7de79d9e0f64b41a5cef775c5d78bee Mon Sep 17 00:00:00 2001 From: Pascal Zarrad Date: Sat, 6 Jun 2026 00:59:31 +0200 Subject: [PATCH] docs: add docs/ guide tree and slim README Add user-facing documentation under docs/: installation, configuration (config file + CONBA_ env reference), full command reference, container-label reference, and task guides (getting started, database dumps, restore, retention and filtering). Slim the README into an overview plus a quick start that links into docs/, removing the deep sections now covered there. --- README.md | 344 +++++-------------------- docs/README.md | 43 ++++ docs/commands.md | 178 +++++++++++++ docs/configuration.md | 127 +++++++++ docs/container-labels.md | 57 ++++ docs/guides/database-dumps.md | 86 +++++++ docs/guides/getting-started.md | 83 ++++++ docs/guides/restore.md | 69 +++++ docs/guides/retention-and-filtering.md | 118 +++++++++ docs/installation.md | 57 ++++ 10 files changed, 877 insertions(+), 285 deletions(-) create mode 100644 docs/README.md create mode 100644 docs/commands.md create mode 100644 docs/configuration.md create mode 100644 docs/container-labels.md create mode 100644 docs/guides/database-dumps.md create mode 100644 docs/guides/getting-started.md create mode 100644 docs/guides/restore.md create mode 100644 docs/guides/retention-and-filtering.md create mode 100644 docs/installation.md diff --git a/README.md b/README.md index 76e5b49..80f2a3c 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,9 @@ volumes, applies filtering rules, snapshots each volume (optionally running a pr command and streaming its output instead), and manages snapshot retention — all driven by a YAML config file with environment variable overrides and optional container labels. +Conba runs **on demand** — there is no built-in scheduler. Invoke it from cron, a systemd +timer, or CI/CD (the `run` command does a full init + backup + forget cycle in one shot). + ## Features | Feature | Description | @@ -34,283 +37,75 @@ a YAML config file with environment variable overrides and optional container la ## Requirements - Docker (or compatible runtime with Docker socket) -- restic (installed separately for host binary; bundled in container image) +- restic (installed separately for the host binary; bundled in the container image) -## Getting Started +## Quick start -Clone and build: +Build the binary (all Make targets run inside Docker — no local Go needed) or use the +container image; see [Installation](docs/installation.md) for both paths. ```sh git clone https://github.com/lazybytez/conba.git cd conba -make build +make build # -> ./bin/conba ``` -All Make targets run inside Docker containers — no local Go installation required. - -Create a config file (`conba.yaml`): +Create a minimal `conba.yaml`: ```yaml restic: repository: "s3:s3.amazonaws.com/my-bucket" password_file: "/run/secrets/restic-password" -runtime: - type: docker - docker: - host: "unix:///var/run/docker.sock" - -discovery: - opt_in_only: false - retention: keep_daily: 7 keep_weekly: 4 - keep_monthly: 6 - keep_yearly: 0 - -logging: - level: "info" - format: "human" ``` -Run a backup: - -```sh -./bin/conba backup -``` - -### Running the container image locally - -After `make docker/build`, run the built image with your local (gitignored) -config bind-mounted in. This is the recommended way to smoke-test conba -against the host's Docker daemon without installing the binary: +Then: ```sh -docker run --rm -it \ - --hostname "$(hostname)" \ - -v "$PWD/conba-config.test.yaml:/app/conba.yaml:ro" \ - -v /var/run/docker.sock:/var/run/docker.sock \ - -v /var/lib/docker/volumes:/var/lib/docker/volumes:ro \ - -v /tmp/conba-restic-test-repo:/tmp/conba-restic-test-repo \ - ghcr.io/lazybytez/conba:edge \ - backup --dry-run -``` - -Drop `--dry-run` to execute the backup. `--hostname "$(hostname)"` makes -snapshots carry the real host's name instead of a random container ID -(conba tags every snapshot with the hostname). The Docker socket mount -lets conba discover running containers; `/var/lib/docker/volumes` exposes -the actual volume contents so they can be read for snapshotting; -`/tmp/conba-restic-test-repo` is the writable local restic repository -(matching `restic.repository` in the test config); and the config is -mounted to `/app/conba.yaml`, the default lookup path inside the image's -working directory. - -### Backing up bind mounts - -Two things to know about bind mounts: - -1. **Container labels match the destination path.** Use the - container-side destination in `conba.exclude-mount-destinations` - (and other label values), not the host source. Destinations are - portable across hosts; sources are not. -2. **Conba opens the source path.** When conba runs in a container, - the host source of every bind mount you want backed up must be - visible inside conba's container — mount it at the same path. - -Example: a service with `-v /srv/myapp/data:/var/lib/myapp/data` is -only backed up when conba's container also has `/srv/myapp/data` -mounted at `/srv/myapp/data`: - -```sh -docker run --rm -it \ - ...existing mounts... \ - -v /srv/myapp/data:/srv/myapp/data:ro \ - ghcr.io/lazybytez/conba:edge backup -``` - -If the source isn't reachable, conba pre-flights, logs -`WARN: skipping /: source unreadable (...)`, -and continues with the remaining targets. - -## Container Labels - -Configure per-container behavior with Docker labels: - -| Label | Values | Default | Description | -|-------|--------|---------|-------------| -| `conba.enabled` | `true`, `false` | — | Override include/exclude filters | -| `conba.retention` | `Nd,Nw,Nm,Ny` | global | Override the global `retention:` policy for this container. Suffix-tagged, comma-separated, order-agnostic, case-insensitive. Example: `conba.retention: "7d,4w,6m,2y"`. Suffixes: `d` daily, `w` weekly, `m` monthly, `y` yearly. Missing components default to 0. | -| `conba.exclude-volumes` | comma-separated | — | Comma-separated list matched against `Mount.Name`. For named volumes that's the volume name; for bind mounts it's the host source path (which is rarely portable across hosts — prefer `conba.exclude-mount-destinations` for bind mounts). | -| `conba.exclude-bind-mounts` | `true`, `false` | `false` | Set to `true` on a container to exclude all of its bind-mounted paths from backup. Named volumes on the same container are not affected. Default: false (bind mounts are eligible). | -| `conba.exclude-mount-destinations` | comma-separated | — | Comma-separated list of container-side destination paths. Any mount (bind or named volume) whose destination matches an entry exactly is excluded from backup. Example: `conba.exclude-mount-destinations: "/var/log,/etc/myapp/cache"`. | -| `conba.pre-backup.command` | shell command | — | Required to enable a pre-backup command for the container; the shell string executed inside the container, whose stdout is streamed into restic as the snapshot. Requires `pre_backup_commands.enabled: true` in config. | -| `conba.pre-backup.mode` | `replace`, `alongside` | `replace` | `replace` substitutes the stream snapshot for the container's volume snapshots; `alongside` produces the stream snapshot plus the volume snapshots. | -| `conba.pre-backup.filename` | filename | labeled container name | Filename used for restic's `--stdin-filename` (e.g. `mysql.sql`). | -| `conba.pre-backup.restore-command` | shell command | — | Restore-side command, run inside the labeled container (locked, no sidecar override) by `conba restore` for stream snapshots when `--to-command` is not provided. Requires `pre_backup_commands.enabled: true` in config. | - -## Pre-backup commands - -Stateful services like databases produce inconsistent on-disk files -unless quiesced or routed through the engine's own export tool. Conba -can run a shell command inside a container at backup time and stream -its stdout into restic as the snapshot — for example, `mysqldump` -piped straight into a restic snapshot tagged for the mysql container. - -The feature is **off by default**. Label-driven command execution is a -qualitative change in conba's trust surface (anyone able to set labels -on a container can cause conba to execute arbitrary shell strings -inside it), so operators must opt in explicitly: - -```yaml -pre_backup_commands: - enabled: true -``` - -When `pre_backup_commands.enabled` is `false` or absent (the default), -all `conba.pre-backup.*` labels are ignored and volume backups proceed -as usual. - -### Example: consistent MySQL backups via mysqldump - -Label the mysql container with the dump command and (optionally) a -filename for the stream: - -```yaml -# compose.yaml -services: - mysql: - image: mysql:8 - environment: - MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}" - volumes: - - mysql-data:/var/lib/mysql - labels: - conba.pre-backup.command: 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mysqldump --all-databases -uroot' - conba.pre-backup.filename: "mysql.sql" - -volumes: - mysql-data: -``` - -The `MYSQL_PWD` env var is preferred over `-p` because the -`-p` form puts the password on the argv where any `ps` -invocation in the container's PID namespace can read it; `MYSQL_PWD` -keeps it in the env. - -Enable the feature in `conba.yaml`: - -```yaml -pre_backup_commands: - enabled: true -``` - -At backup time, conba runs `mysqldump` inside the mysql container -through the container runtime's API and streams its stdout into a -single restic snapshot tagged `container=mysql` and `kind=stream`. Conba tags every -snapshot with a `kind` — `kind=volume` for volume snapshots and -`kind=stream` for command-output (stream) snapshots — an internal -classification tag conba writes to tell the two apart, not a label -you set. In the default `replace` mode, the on-disk -`mysql-data` volume is **not** backed up as a separate snapshot — -the dump is the canonical representation of the database's state, so -the inconsistent at-rest files are skipped. Switch to -`conba.pre-backup.mode: alongside` if the container also holds -volumes you want backed up directly (e.g. an uploads directory next -to the database). - -### Example: restoring a MySQL backup - -`conba restore` is one command that handles both volume and stream -snapshots; conba inspects the resolved snapshot's tags and picks -the right restic primitive (`restic restore` for volume snapshots, -`restic dump` piped into an in-container command for stream snapshots, -run through the Docker API — no `docker` CLI required). -Operators describe *what* to restore via flags. Use `conba snapshots` -to enumerate candidates and pass `--snapshot ` for a -point-in-time restore; without it, conba selects the latest -matching snapshot. - -#### Volume restore - -Restore the latest `mysql-data` volume snapshot to a sidecar -directory for inspection: - -```sh -conba restore --container mysql --volume mysql-data --to /tmp/recovered -``` - -The operator owns the container lifecycle. Stop the mysql container -before overwriting the live volume; conba does not auto-stop, and -restoring into a path mounted by a running container is the -operator's risk to take. If the destination is non-empty, conba -refuses unless `--force` is passed. - -#### Stream restore via CLI flag - -Pipe the latest stream snapshot back into mysql via the standard -client: - -```sh -conba restore --container mysql \ - --to-command "MYSQL_PWD=\"$MYSQL_ROOT_PASSWORD\" mysql -uroot" -``` - -Stream restore requires the target container to be running (you -cannot exec a command into a stopped container). Conba refuses with -a clear error otherwise. - -#### Stream restore via label - -Add a `conba.pre-backup.restore-command` label alongside the -existing `conba.pre-backup.command` label so operators do not have -to retype the restore invocation: - -```yaml -# compose.yaml -services: - mysql: - image: mysql:8 - environment: - MYSQL_ROOT_PASSWORD: "${MYSQL_ROOT_PASSWORD}" - volumes: - - mysql-data:/var/lib/mysql - labels: - conba.pre-backup.command: 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mysqldump --all-databases -uroot' - conba.pre-backup.filename: "mysql.sql" - conba.pre-backup.restore-command: 'MYSQL_PWD="$MYSQL_ROOT_PASSWORD" mysql -uroot' - -volumes: - mysql-data: -``` - -With the label in place, the restore reduces to: - -```sh -conba restore --container mysql -``` - -The label form is gated by the same `pre_backup_commands.enabled: -true` feature flag as the backup-side command. When the flag is -false or absent, the label is ignored. When both `--to-command` and -the label are set, the CLI flag wins. - -## CLI Commands - -``` -conba backup # Discover, filter, and backup all matching volumes -conba backup --dry-run # Show what would be backed up without executing -conba restore # Restore a volume snapshot to a host path or pipe a stream snapshot back into a container -conba forget # Apply retention policies and prune -conba forget --dry-run # Show what would be forgotten without changes -conba run # One-shot init + backup + forget cycle (intended for CI/CD) -conba snapshots # List snapshots -conba diff # Show file differences between two snapshots -conba verify # Verify restic repository integrity -conba verify --read-data # Full data verification (slow) -conba version # Print version info -``` +conba init # initialise the repository +conba inspect # preview which containers/volumes will be backed up +conba backup --dry-run # confirm, then run for real: +conba backup +conba snapshots # list what you have +``` + +For scheduled backups, run `conba run` (init + backup + forget) from cron/CI. A full +walkthrough is in [Getting started](docs/guides/getting-started.md). + +## Documentation + +Full documentation lives in [`docs/`](docs/README.md): + +- **[Installation](docs/installation.md)** — host binary or container image. +- **[Getting started](docs/guides/getting-started.md)** — first backup, end to end. +- **[Configuration](docs/configuration.md)** — `conba.yaml` and `CONBA_*` env overrides. +- **[Commands](docs/commands.md)** — every subcommand, flags, and dry-run support. +- **[Container labels](docs/container-labels.md)** — per-container filtering, retention, and pre-backup behaviour. +- **[Database dumps](docs/guides/database-dumps.md)** — consistent DB backups via `mysqldump` and friends. +- **[Restoring](docs/guides/restore.md)** — restore volume and stream snapshots. +- **[Retention and filtering](docs/guides/retention-and-filtering.md)** — scope and snapshot lifetime. + +## Commands + +| Command | Purpose | +|---------|---------| +| `init` | Initialise the restic repository | +| `backup` | Back up all discovered volume targets (`--dry-run` supported) | +| `snapshots` | List snapshots (filter by container/volume/hostname) | +| `restore` | Restore a volume or stream snapshot (`--dry-run` supported) | +| `verify` | Check repository integrity (`--read-data` for a full scan) | +| `diff` | Show changes between two snapshots | +| `forget` | Apply retention / prune (`--dry-run` supported) | +| `run` | One-shot init + backup + forget cycle (for cron/CI) | +| `inspect` | Preview which containers/volumes would be backed up | +| `status` | Show repository status | +| `unlock` | Remove stale repository locks | +| `version` | Print version information | + +See [Commands](docs/commands.md) for full flag details. ## Development @@ -327,43 +122,22 @@ make clean # Remove build artifacts ### End-to-end tests -The `test/e2e/` package exercises the compiled `conba` binary against a real -Docker daemon and a real restic filesystem repository. A small Docker Compose -stack (`test/e2e/compose.yaml`) provides MySQL plus two Alpine services as -backup targets. Run the full suite with: +The `test/e2e/` package exercises the compiled `conba` binary against a real Docker daemon +and a real restic filesystem repository, using a small Docker Compose fixture +(`test/e2e/compose.yaml`). Run the full suite with: ```sh make e2e ``` -The target builds the test image, brings the compose fixture up, runs every -scenario inside the test image (with `/var/run/docker.sock` and -`/var/lib/docker/volumes` mounted), then unconditionally tears the fixture -down. Iterative loop: `make go/test-e2e/up` once, then `make go/test-e2e/run` -repeatedly. CI runs the same target on every PR via `.github/workflows/e2e.yml` -and publishes per-scenario pass/fail. - -### Branching - -| Branch | Purpose | -|--------|---------| -| `main` | Stable — all PRs target here | -| `feature/*` | New features | -| `fix/*` | Bug fixes | - -### Commit Messages - -Conventional commits enforced via [commitlint](https://commitlint.js.org/): - -``` -prefix(scope): subject -``` - -Prefixes: `feat`, `fix`, `build`, `chore`, `ci`, `docs`, `perf`, `refactor`, `revert`, `style`, `test`, `sec` +The target builds the test image, brings the fixture up, runs every scenario, then +unconditionally tears the fixture down. For an iterative loop: `make go/test-e2e/up` once, +then `make go/test-e2e/run` repeatedly. CI runs the same target on every PR. ## Contributing -Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md). +Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the branching model +(`main`, `feature/*`, `fix/*`) and the conventional-commit format enforced by commitlint. ## Useful Links diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..a2fb8af --- /dev/null +++ b/docs/README.md @@ -0,0 +1,43 @@ +# Conba Documentation + +Conba is a Go CLI that wraps [restic](https://restic.net/) to back up Docker +container volumes — either as raw volume snapshots or by streaming a command's +output (e.g. `mysqldump`) straight into a snapshot. It is invoked **on demand** +(there is no built-in scheduler); run it from cron, a systemd timer, or a CI/CD +pipeline. + +## Start here + +- [Installation](installation.md) — build from source or run the container image. +- [Getting started](guides/getting-started.md) — your first backup, end to end. + +## Reference + +- [Configuration](configuration.md) — the `conba.yaml` file and `CONBA_*` + environment overrides, field by field. +- [Commands](commands.md) — every subcommand, its flags, and dry-run support. +- [Container labels](container-labels.md) — per-container filtering, retention, + and pre-backup/restore behaviour driven by Docker labels. + +## Guides + +- [Database dumps (streaming backups)](guides/database-dumps.md) — back up + MySQL/PostgreSQL/etc. with a consistent dump instead of raw volume files. +- [Restoring](guides/restore.md) — restore volume and stream snapshots. +- [Retention and filtering](guides/retention-and-filtering.md) — choose what + gets backed up and how long snapshots are kept. + +## Concepts at a glance + +| Term | Meaning | +|------|---------| +| **Target** | A single container volume mount eligible for backup. | +| **Volume snapshot** | A restic snapshot of a volume's files (`kind=volume`). | +| **Stream snapshot** | A restic snapshot of a command's stdout (`kind=stream`), e.g. a `mysqldump`. | +| **Pre-backup command** | A command run inside a container at backup time whose stdout becomes a stream snapshot. | +| **Discovery** | Listing running containers and their mounts via the Docker API. | +| **Filter** | Include/exclude rules (config + labels) deciding which targets are backed up. | + +Every snapshot is tagged with `container=`, `volume=` (volume snapshots), +`hostname=`, and `kind=`, which is how `snapshots`, `restore`, and `forget` +locate and scope snapshots. diff --git a/docs/commands.md b/docs/commands.md new file mode 100644 index 0000000..2880c5a --- /dev/null +++ b/docs/commands.md @@ -0,0 +1,178 @@ +# Commands + +All commands read configuration as described in [Configuration](configuration.md) +and accept the global flag `-c, --config ` (default `conba.yaml`). + +| Command | Purpose | Dry-run | +|---------|---------|---------| +| [`init`](#init) | Initialise the restic repository | — | +| [`backup`](#backup) | Back up all discovered volume targets | ✅ | +| [`snapshots`](#snapshots) | List snapshots | n/a (read-only) | +| [`restore`](#restore) | Restore a volume or stream snapshot | ✅ | +| [`verify`](#verify) | Check repository integrity | n/a (read-only) | +| [`diff`](#diff) | Show changes between two snapshots | n/a (read-only) | +| [`forget`](#forget) | Apply retention / prune snapshots | ✅ | +| [`run`](#run) | One-shot init + backup + forget cycle | ✅ | +| [`inspect`](#inspect) | Preview which containers/volumes would be backed up | n/a (read-only) | +| [`status`](#status) | Show repository status | n/a (read-only) | +| [`unlock`](#unlock) | Remove stale repository locks | — | +| [`version`](#version) | Print version information | — | + +Mutating commands (`backup`, `restore`, `forget`, `run`) support `--dry-run`, +which prints the planned actions and performs none of them. Read-only commands +have no dry-run because they never change state. + +--- + +## init + +```sh +conba init +``` + +Initialises the configured restic repository. Safe to run against an +already-initialised repository (reported, not an error). + +## backup + +```sh +conba backup [--dry-run] +``` + +Discovers running containers, applies discovery filters, and backs up each +eligible volume mount as a `kind=volume` snapshot. If +`pre_backup_commands.enabled` is true, containers carrying +`conba.pre-backup.command` instead (or additionally) produce a `kind=stream` +snapshot — see [Database dumps](guides/database-dumps.md). + +| Flag | Description | +|------|-------------| +| `--dry-run` | Print what would be backed up; create no snapshots. | + +## snapshots + +```sh +conba snapshots [--container ] [--volume ] [--hostname ] +``` + +Lists snapshots, optionally filtered. Filters combine (AND). + +| Flag | Description | +|------|-------------| +| `--container` | Only snapshots tagged `container=`. | +| `--volume` | Only snapshots tagged `volume=`. | +| `--hostname` | Only snapshots tagged `hostname=`. | + +## restore + +```sh +conba restore --container [--volume ] [--snapshot ] \ + [--to ] [--to-command ] [--force] [--all-hosts] [--dry-run] +``` + +Restores the latest matching snapshot (or `--snapshot `). The mode is +auto-detected from the snapshot's `kind` tag. See [Restoring](guides/restore.md). + +| Flag | Description | +|------|-------------| +| `--container` | Container whose snapshot to restore (**required**). | +| `--volume` | Volume name; required when multiple volume snapshots match. | +| `--snapshot` | Restore a specific snapshot ID instead of the latest. | +| `--to` | Host directory to restore a **volume** snapshot into. | +| `--to-command` | In-container command to pipe a **stream** snapshot into. | +| `--force` | Overwrite a non-empty restore destination (volume mode). | +| `--all-hosts` | Consider snapshots from any hostname, not just this host. | +| `--dry-run` | Print the planned restore; touch nothing. | + +## verify + +```sh +conba verify [--read-data] +``` + +Wraps `restic check`. Exits non-zero if the repository is corrupt or missing. + +| Flag | Description | +|------|-------------| +| `--read-data` | Read and verify every data blob (slow, exhaustive). Without it, only repository structure is checked. | + +## diff + +```sh +conba diff +``` + +Wraps `restic diff` and prints its output verbatim. Snapshot identifiers may be +full IDs, short IDs, or the literal `latest`. + +## forget + +```sh +conba forget [--dry-run] [--no-prune] [--all-hosts] \ + [--container ] [--volume ] [--tag ]... +``` + +Applies the configured [retention](guides/retention-and-filtering.md) policy +(with per-container `conba.retention` overrides) and prunes by default. + +| Flag | Description | +|------|-------------| +| `--dry-run` | Show what would be forgotten; change nothing. | +| `--no-prune` | Forget snapshots but skip the (slow) prune step. | +| `--all-hosts` | Apply across all hostnames, not just this host. | +| `--container` | Restrict to one container. | +| `--volume` | Restrict to one volume. | +| `--tag` | Restrict to snapshots carrying the tag (repeatable). | + +## run + +```sh +conba run [--dry-run] [--all-hosts] [--no-forget] +``` + +The on-demand "do everything" command, intended for cron/CI: initialises the +repository if needed, runs a backup, then applies retention. This is the typical +scheduled entry point. + +| Flag | Description | +|------|-------------| +| `--dry-run` | Plan the full cycle without changing anything. | +| `--all-hosts` | Pass through to the forget phase. | +| `--no-forget` | Skip the retention/forget phase. | + +## inspect + +```sh +conba inspect +``` + +Read-only preview of discovery: lists containers and volumes that **would** be +backed up (Included) and those that would not (Excluded, with reasons), plus any +pre-backup label details. Use it to validate filters before backing up. + +## status + +```sh +conba status +``` + +Prints repository status (initialised/ready, repository path, snapshot/stats +summary). Exits 0 even when the repository is uninitialised, printing a friendly +"run 'conba init'" hint. + +## unlock + +```sh +conba unlock +``` + +Removes stale locks left behind by an interrupted restic operation. + +## version + +```sh +conba version +``` + +Prints the conba version plus the Go and bundled restic versions. Requires no +configuration or repository. diff --git a/docs/configuration.md b/docs/configuration.md new file mode 100644 index 0000000..63c76d7 --- /dev/null +++ b/docs/configuration.md @@ -0,0 +1,127 @@ +# Configuration + +Conba reads a YAML config file (default: `conba.yaml` in the working directory; +override with `-c/--config `). Every value can also be supplied via an +environment variable. + +## Environment overrides + +Any config key maps to an environment variable by upper-casing it, replacing +dots with underscores, and prefixing `CONBA_`: + +| Config key | Environment variable | +|------------|----------------------| +| `restic.repository` | `CONBA_RESTIC_REPOSITORY` | +| `restic.password` | `CONBA_RESTIC_PASSWORD` | +| `logging.level` | `CONBA_LOGGING_LEVEL` | +| `discovery.opt_in_only` | `CONBA_DISCOVERY_OPT_IN_ONLY` | +| `pre_backup_commands.enabled` | `CONBA_PRE_BACKUP_COMMANDS_ENABLED` | + +Environment variables take precedence over the file. They are convenient for +secrets (`CONBA_RESTIC_PASSWORD`) and for the container image. + +## Full example + +```yaml +restic: + repository: "s3:s3.amazonaws.com/my-bucket" + password_file: "/run/secrets/restic-password" + +runtime: + type: docker + docker: + host: "unix:///var/run/docker.sock" + +discovery: + opt_in_only: false + +retention: + keep_daily: 7 + keep_weekly: 4 + keep_monthly: 6 + keep_yearly: 0 + +pre_backup_commands: + enabled: false + +logging: + level: "info" + format: "human" +``` + +## Reference + +### `restic` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `repository` | string | — (**required**) | restic repository URL (`s3:…`, `/path`, `sftp:…`, etc.). | +| `password` | string | — | Repository password. Provide this **or** `password_file`. | +| `password_file` | string | — | Path to a file containing the repository password. | +| `binary` | string | `restic` | Path/name of the restic binary to invoke. | +| `extra_args` | list | — | Extra arguments appended to every restic invocation. | +| `environment` | map | — | Extra environment variables for restic (e.g. cloud credentials, `RESTIC_CACHE_DIR`). | + +One of `password` / `password_file` is required, as is `repository`. Missing +either fails fast at startup. + +> Pre-backup (stream) commands require restic to spawn a child process, which +> needs a cache directory. When using that feature, set `RESTIC_CACHE_DIR` (and +> `HOME`) under `restic.environment`, and ensure `PATH` is set so restic can +> find the runtime. See [Database dumps](guides/database-dumps.md). + +### `runtime` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `type` | string | `docker` | Container runtime. Docker is the only supported type. | +| `docker.host` | string | `unix:///var/run/docker.sock` | Docker daemon address. | + +### `discovery` + +Controls which containers/volumes become backup targets. See +[Retention and filtering](guides/retention-and-filtering.md) for worked examples. + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `opt_in_only` | bool | `false` | When `true`, only containers labelled `conba.enabled=true` are considered. | +| `include` | filter list | — | Force-include matching containers. | +| `exclude` | filter list | — | Exclude matching containers. | + +A **filter list** has four fields, each a list of strings: + +| Field | Matches on | +|-------|-----------| +| `names` | exact container name | +| `name_patterns` | container name (regular expression) | +| `ids` | exact container ID | +| `id_patterns` | container ID (regular expression) | + +### `retention` + +Global retention policy applied by `conba forget` (and the forget phase of +`conba run`). All default to `0` (= keep, i.e. that dimension is not enforced). +Per-container overrides are possible with the `conba.retention` label. + +| Key | Type | Default | +|-----|------|---------| +| `keep_daily` | int | `0` | +| `keep_weekly` | int | `0` | +| `keep_monthly` | int | `0` | +| `keep_yearly` | int | `0` | + +### `pre_backup_commands` + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| `enabled` | bool | `false` | Master switch for label-driven pre-backup/restore commands. **Off by default** because it lets anyone who can set container labels make conba run shell commands inside those containers. | + +When disabled (the default), all `conba.pre-backup.*` labels are ignored and +volume backups proceed as usual. + +### `logging` + +| Key | Type | Default | Values | +|-----|------|---------|--------| +| `level` | string | `info` | `debug`, `info`, `warn`, `error` | +| `format` | string | `human` | `human`, `json` | diff --git a/docs/container-labels.md b/docs/container-labels.md new file mode 100644 index 0000000..e618329 --- /dev/null +++ b/docs/container-labels.md @@ -0,0 +1,57 @@ +# Container labels + +Conba reads Docker labels off each container to drive per-container behaviour. +Set them in your `compose.yaml` (`labels:`) or `docker run --label`. + +## Filtering & retention labels + +These work regardless of configuration. + +| Label | Values | Default | Description | +|-------|--------|---------|-------------| +| `conba.enabled` | `true`, `false` | — | Force include/exclude this container. Required to be `true` for a container to be backed up when `discovery.opt_in_only` is on. | +| `conba.retention` | `Nd,Nw,Nm,Ny` | global | Per-container retention override. Suffix-tagged, comma-separated, order-agnostic, case-insensitive. Suffixes: `d` daily, `w` weekly, `m` monthly, `y` yearly; missing components default to 0. Example: `7d,4w,6m,2y`. | +| `conba.exclude-volumes` | comma-separated | — | Volume names (`Mount.Name`) to exclude. For bind mounts this is the host source path (rarely portable — prefer `conba.exclude-mount-destinations`). | +| `conba.exclude-bind-mounts` | `true`, `false` | `false` | Exclude all of the container's bind mounts. Named volumes are unaffected. | +| `conba.exclude-mount-destinations` | comma-separated | — | Container-side destination paths to exclude (bind or volume). Matched exactly. Example: `/var/log,/etc/app/cache`. | + +## Pre-backup / restore labels + +These are **only honoured when `pre_backup_commands.enabled: true`** in the +config (off by default — see [Configuration](configuration.md#pre_backup_commands)). +They let conba run a command inside the container at backup/restore time. See +[Database dumps](guides/database-dumps.md) and [Restoring](guides/restore.md). + +| Label | Values | Default | Description | +|-------|--------|---------|-------------| +| `conba.pre-backup.command` | shell command | — | Required to enable streaming for the container. Runs inside the container (`sh -c ""`); its stdout is streamed into restic as a `kind=stream` snapshot. | +| `conba.pre-backup.mode` | `replace`, `alongside` | `replace` | `replace`: the stream snapshot substitutes the container's volume snapshots. `alongside`: produce the stream snapshot **and** the volume snapshots. | +| `conba.pre-backup.filename` | filename | labeled container name | The `--stdin-filename` restic records for the stream (e.g. `mysql.sql`). Used as the dump filename on restore. | +| `conba.pre-backup.restore-command` | shell command | — | Command `conba restore` pipes the dump into when restoring a stream snapshot and `--to-command` is not given. Runs inside the labeled container. | + +### Trust note + +Label-driven commands are a meaningful trust-surface change: anyone who can set +labels on a container can make conba execute shell strings inside it. That is +why the feature is opt-in via `pre_backup_commands.enabled`. The command is only +ever interpreted by the **in-container** shell — conba runs it through the Docker +API exec, never by building a host shell string. + +## Example + +```yaml +# compose.yaml +services: + mysql: + image: mysql:8 + volumes: + - mysql-data:/var/lib/mysql + labels: + conba.pre-backup.command: "mysqldump --all-databases -uroot" + conba.pre-backup.mode: "replace" + conba.pre-backup.filename: "mysql.sql" + conba.pre-backup.restore-command: "mysql -uroot" + +volumes: + mysql-data: +``` diff --git a/docs/guides/database-dumps.md b/docs/guides/database-dumps.md new file mode 100644 index 0000000..8992573 --- /dev/null +++ b/docs/guides/database-dumps.md @@ -0,0 +1,86 @@ +# Database dumps (streaming backups) + +Stateful services like databases produce inconsistent on-disk files unless they +are quiesced or exported through the engine's own tool. Instead of snapshotting +raw volume files, conba can run a command **inside** the container at backup time +and stream its stdout straight into a restic snapshot — for example `mysqldump` +piped directly into a snapshot tagged for the MySQL container. + +## Enable the feature + +Label-driven command execution is **off by default** because it lets anyone who +can set container labels make conba run shell commands inside the container. Turn +it on explicitly: + +```yaml +# conba.yaml +pre_backup_commands: + enabled: true +``` + +restic spawns the dump as a child process, which needs a cache directory. When +this feature is on, make sure restic has one: + +```yaml +restic: + environment: + RESTIC_CACHE_DIR: /var/cache/restic + HOME: /root + PATH: /usr/local/bin:/usr/bin:/bin +``` + +## Label the container + +```yaml +# compose.yaml +services: + mysql: + image: mysql:8 + environment: + MYSQL_ROOT_PASSWORD: example + volumes: + - mysql-data:/var/lib/mysql + labels: + conba.pre-backup.command: 'mysqldump --all-databases -uroot -p"$MYSQL_ROOT_PASSWORD"' + conba.pre-backup.mode: "replace" + conba.pre-backup.filename: "mysql.sql" + conba.pre-backup.restore-command: 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' + +volumes: + mysql-data: +``` + +The command runs as `sh -c ""` inside the container, so shell features +(env-var expansion, pipes) work and are evaluated in the container's environment. + +## Modes + +| Mode | Result | +|------|--------| +| `replace` (default) | Only the **stream** snapshot is produced; the container's volume snapshots are skipped. Use this for databases where the raw files are not worth backing up. | +| `alongside` | Produce the stream snapshot **and** the normal volume snapshots. | + +## Run it + +```sh +conba backup --dry-run # shows "would run: in " +conba backup +conba snapshots --container mysql # you'll see a kind=stream snapshot +``` + +The same applies to any tool: `pg_dumpall` for PostgreSQL, `mongodump --archive` +for MongoDB, `redis-cli --rdb -` for Redis, etc. Any command that writes the +backup to stdout works. + +## Failure safety + +A stream snapshot is only finalised if the command exits 0. If the dump command +fails (non-zero exit) — even after emitting partial output — conba aborts the +restic side so **no truncated snapshot is stored**, and the overall backup cycle +reports the failure while continuing with the remaining targets. + +## Restoring a dump + +See [Restoring → stream snapshots](restore.md#stream-snapshots). In short, conba +pipes `restic dump` back into an in-container command (`--to-command` or the +`conba.pre-backup.restore-command` label), e.g. `mysql -uroot`. diff --git a/docs/guides/getting-started.md b/docs/guides/getting-started.md new file mode 100644 index 0000000..5958057 --- /dev/null +++ b/docs/guides/getting-started.md @@ -0,0 +1,83 @@ +# Getting started + +This walks through a first backup of your running containers' volumes to a local +restic repository. See [Installation](../installation.md) to get the binary or +image first. + +## 1. Write a config + +Create `conba.yaml` next to where you'll run conba: + +```yaml +restic: + repository: "/srv/conba-repo" # any restic repo URL works (s3:, sftp:, …) + password: "change-me" # or password_file: /run/secrets/... + +retention: + keep_daily: 7 + keep_weekly: 4 + +logging: + level: info + format: human +``` + +Prefer to keep the password out of the file? Drop `password` and export +`CONBA_RESTIC_PASSWORD` instead (see [Configuration](../configuration.md#environment-overrides)). + +## 2. Initialise the repository + +```sh +conba init +``` + +## 3. Preview what will be backed up + +```sh +conba inspect # lists Included / Excluded containers and volumes +conba backup --dry-run +``` + +`inspect` is the fastest way to confirm your [filters](retention-and-filtering.md) +select the right targets before you write any data. + +## 4. Back up + +```sh +conba backup +``` + +Each eligible volume becomes a snapshot tagged with its container, volume, and +hostname. + +## 5. List what you have + +```sh +conba snapshots +conba snapshots --container mysql # filter +``` + +## 6. Run it on a schedule (on demand) + +Conba has **no built-in scheduler** — it runs when invoked. The `run` command +does init + backup + forget in one shot, which is ideal for cron, a systemd +timer, or CI: + +```sh +conba run # init (if needed) + backup + apply retention +conba run --dry-run # see the whole cycle first +``` + +Example cron entry (daily at 02:00): + +```cron +0 2 * * * cd /srv/conba && /usr/local/bin/conba run >> /var/log/conba.log 2>&1 +``` + +## Next steps + +- [Database dumps](database-dumps.md) — consistent DB backups via `mysqldump`. +- [Restoring](restore.md) — get your data back. +- [Retention and filtering](retention-and-filtering.md) — fine-tune scope and + snapshot lifetime. +- [Verify](../commands.md#verify) the repository periodically: `conba verify`. diff --git a/docs/guides/restore.md b/docs/guides/restore.md new file mode 100644 index 0000000..b8b6e73 --- /dev/null +++ b/docs/guides/restore.md @@ -0,0 +1,69 @@ +# Restoring + +`conba restore` recovers data from an existing snapshot. One command handles both +kinds of snapshot: it inspects the resolved snapshot's `kind` tag and picks the +right restic primitive (`restic restore` for volume snapshots, `restic dump` +piped into an in-container command for stream snapshots — through the Docker API, +no `docker` CLI required). + +You describe **what** to restore via flags; conba selects the **latest** matching +snapshot by default. Use `conba snapshots` to enumerate candidates and +`--snapshot ` for a point-in-time restore. `--all-hosts` drops the hostname +filter. `--container` is always required. If multiple volume snapshots match, +conba asks you to disambiguate with `--volume`. + +> The operator owns the container lifecycle. Conba never stops or starts +> containers. For a volume restore that overwrites a live volume, stop the +> container first. + +## Volume snapshots + +Restore the latest `mysql-data` volume snapshot into a host directory for +inspection: + +```sh +conba restore --container mysql --volume mysql-data --to /tmp/recovered +``` + +- `--to ` is required for volume restores. +- A non-empty destination is refused unless you pass `--force`. + +## Stream snapshots + +A stream snapshot (e.g. a `mysqldump`) is restored by piping the dump back into a +command **inside a running container**. The target container must be running +(you cannot exec into a stopped container); conba refuses with a clear error +otherwise. + +The restore command comes from either source — the CLI flag wins if both are set: + +**Via flag** (always available): + +```sh +conba restore --container mysql \ + --to-command 'mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' +``` + +**Via label** (needs `pre_backup_commands.enabled: true`): if the container +carries `conba.pre-backup.restore-command`, you can omit `--to-command`: + +```sh +conba restore --container mysql +``` + +If a stream snapshot is selected and neither `--to-command` nor the label is +available, conba errors and names both options. + +## Dry run + +`--dry-run` prints the planned action and invokes neither restic nor the +in-container command (for stream mode it still checks the container is running): + +```sh +conba restore --container mysql --volume mysql-data --to /tmp/recovered --dry-run +``` + +## Flag summary + +See [`restore` in the command reference](../commands.md#restore) for the full +flag list. diff --git a/docs/guides/retention-and-filtering.md b/docs/guides/retention-and-filtering.md new file mode 100644 index 0000000..3380299 --- /dev/null +++ b/docs/guides/retention-and-filtering.md @@ -0,0 +1,118 @@ +# Retention and filtering + +Two related questions: **what gets backed up** (filtering) and **how long +snapshots are kept** (retention). + +## Filtering — what gets backed up + +By default conba discovers every running container and backs up all of its volume +mounts. You narrow that with config-level filters and per-container labels. + +### Opt-in mode + +To back up nothing unless explicitly enabled: + +```yaml +discovery: + opt_in_only: true +``` + +Then only containers labelled `conba.enabled=true` are considered. + +### Include / exclude by name or ID + +```yaml +discovery: + exclude: + names: ["jenkins"] + name_patterns: ["^tmp-.*"] + include: + names: ["important-db"] +``` + +Each of `include`/`exclude` accepts `names`, `name_patterns` (regex), `ids`, and +`id_patterns` (regex). The `conba.enabled` label overrides these per container. + +### Excluding specific mounts + +Even on an included container, you can drop individual mounts via labels: + +| Label | Effect | +|-------|--------| +| `conba.exclude-volumes` | Exclude named volumes (by `Mount.Name`). | +| `conba.exclude-mount-destinations` | Exclude by container-side path (e.g. `/var/log`). Best for bind mounts. | +| `conba.exclude-bind-mounts: true` | Exclude all bind mounts on the container. | + +See [Container labels](../container-labels.md) for the full reference. + +### Preview before backing up + +```sh +conba inspect +``` + +Lists **Included** targets and **Excluded** ones (with the reason each was +dropped). Run this whenever you change filters. + +### Bind mounts + +Two things to know: + +1. **Labels match the destination path.** Use the container-side destination in + `conba.exclude-mount-destinations`, not the host source — destinations are + portable across hosts, sources are not. +2. **Conba must be able to read the source.** When conba runs in a container, the + host source of every bind mount you want backed up must be visible inside + conba's container at the same path: + + ```sh + docker run --rm \ + ...existing mounts... \ + -v /srv/myapp/data:/srv/myapp/data:ro \ + ghcr.io/lazybytez/conba:edge backup + ``` + + If the source isn't reachable, conba logs + `WARN: skipping /: source unreadable (...)` and + continues with the remaining targets. + +## Retention — how long snapshots are kept + +Retention is applied by [`conba forget`](../commands.md#forget) (and the forget +phase of [`conba run`](../commands.md#run)). Set a global policy: + +```yaml +retention: + keep_daily: 7 + keep_weekly: 4 + keep_monthly: 6 + keep_yearly: 0 +``` + +Any dimension left at `0` is not enforced. `forget` prunes by default; use +`--no-prune` to skip the (slow) prune step, and always preview with `--dry-run`: + +```sh +conba forget --dry-run +conba forget +``` + +### Per-container overrides + +A container can override the global policy with the `conba.retention` label — +suffix-tagged, comma-separated, order-agnostic, case-insensitive: + +```yaml +labels: + conba.retention: "7d,4w,6m,2y" # daily, weekly, monthly, yearly +``` + +Missing components default to 0. This takes precedence over the global +`retention:` policy for that container. + +### Scoping a forget run + +`forget` is host-scoped by default (only this machine's snapshots). Adjust with: + +- `--all-hosts` — apply across every hostname. +- `--container ` / `--volume ` / `--tag ` — restrict the set. diff --git a/docs/installation.md b/docs/installation.md new file mode 100644 index 0000000..f8165db --- /dev/null +++ b/docs/installation.md @@ -0,0 +1,57 @@ +# Installation + +Conba needs three things at runtime: + +1. **Access to the Docker daemon** (to discover containers and their volumes). +2. **A restic binary** — installed separately when running the host binary; + **bundled** in the container image. +3. **Read access to the volume data** being backed up. + +## Option A — Container image (recommended) + +The published image bundles a pinned restic, so there is nothing else to install. + +```sh +docker run --rm -it \ + --hostname "$(hostname)" \ + -v "$PWD/conba.yaml:/app/conba.yaml:ro" \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v /var/lib/docker/volumes:/var/lib/docker/volumes:ro \ + ghcr.io/lazybytez/conba:edge \ + backup --dry-run +``` + +Drop `--dry-run` to execute. Notes on the mounts: + +- `--hostname "$(hostname)"` makes snapshots carry the real host's name (conba + tags every snapshot with the hostname) instead of a random container ID. +- The Docker socket lets conba discover running containers. +- `/var/lib/docker/volumes` exposes named-volume contents so they can be read + for snapshotting (read-only is fine). +- `conba.yaml` is mounted to `/app/conba.yaml`, the default lookup path. + +For **bind mounts**, the host source path must also be visible inside conba's +container at the same path — see +[Retention and filtering](guides/retention-and-filtering.md#bind-mounts). + +## Option B — Build from source + +All build operations run inside Docker via Make — no local Go toolchain needed. + +```sh +git clone https://github.com/lazybytez/conba.git +cd conba +make build # produces ./bin/conba +./bin/conba version +``` + +When running the host binary, install [restic](https://restic.net/) yourself and +ensure it is on `PATH` (or set `restic.binary` in the config). + +## Verify the install + +```sh +conba version # prints conba, go, and restic versions +``` + +Next: [Getting started](guides/getting-started.md).