Skip to content

feat: standard DB host ports, expose-by-default, unified provisioning + up/down fixes - #123

Merged
gustavobertoi merged 2 commits into
mainfrom
feat/standard-db-ports-and-engines
Jul 10, 2026
Merged

feat: standard DB host ports, expose-by-default, unified provisioning + up/down fixes#123
gustavobertoi merged 2 commits into
mainfrom
feat/standard-db-ports-and-engines

Conversation

@gustavobertoi

@gustavobertoi gustavobertoi commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

What & why

Builds on #122. Makes shared services reachable on the ports developers already expect, adds the common databases as first-class engines, and fixes the up/down reliability issues rooted in the old dual host-port design.

1. One standard host port per engine — no separate bands

Previously there were two host-port overlays fighting over the same shared containers' ports::

  • the expose overlay (standard ports), and
  • a separate provisioning band (45xxx via writeProvisionOverlay/engineTarget).

Every provision, db reset, snapshot, resource op and re-up applied a different subset, so compose kept recreating the shared container with a different port set — the root of "port already allocated", connection-reset races, and re-up flakiness.

Now collapsed into one overlay published on the engine's standard 127.0.0.1 port (ensureExposed/primaryExposePort). Provisioning, reset, snapshot and resource ops all resolve their admin endpoint from it — exactly one host port per engine, the well-known one. Idempotent (same ports + same overlay bytes → no recreate). Deleted engineOverlays, writeProvisionOverlay, and the pg-provision/45xxx band.

Engine Host port
postgres 5432
mysql / mariadb 3306
mongodb 27017
cassandra 9042
arangodb 8529
redis 6379 · minio 9000/9001 · nats 4222 · rabbitmq 5672 · kafka 49092*

*kafka keeps its fixed advertised external listener.

2. Expose by default on up

up auto-publishes every exposable shared engine on its standard loopback port, so GUI clients' defaults just work and provisioning has one port to dial. Opt out with up --no-expose; skipped on remote backends.

3. down actually brings things down + re-up works

  • down now stops every shared engine that fell to zero refs (Manager.GC), respecting cross-workspace ref-counting — no more warm engines left running.
  • It clears the saga phase records for what it stopped (compose-up per project; shared/provision/resources when the shared stack stops), so the next up restarts them instead of skipping on a stale fingerprint — fixes "re-up after down is a no-op".

4. Five new shared database engines

Built-in templates, each reachable at shared-<name>, exposable, with a connection URL: mysql, mariadb, mongodb, cassandra, arangodb. Plus the devdock-import map and README.

Verification

  • End-to-end against a real Docker daemon: up → auto-expose on 127.0.0.1:5432 → provision role/db (psql confirmed) → down stops shared → re-up restarts shared+provision+app and re-publishes ports. Standard ports throughout, zero collisions across the cycle.
  • Five new templates lint (compose-go/v2); ${ref} resolves to shared-mysql/3306 etc.
  • go vet, gofmt, make determinism, and the race detector on orchestrate/cli pass.

Notes

🤖 Generated with Claude Code

…rangoDB engines

Publish shared engines on their WELL-KNOWN host ports instead of a 5xxxx
offset band, so `defaultPort` (in-network) and the exposed host port match
and a GUI client's defaults just work: postgres 5432, mysql/mariadb 3306,
mongodb 27017, cassandra 9042, arangodb 8529, redis 6379, minio 9000/9001.

The standard port becomes the FreeHostPort search base. This is safe because
AllocatePort skips every already-allocated port across all owners, so a lone
engine lands on the standard port, a host-native server holding it makes the
allocator fall back to the next free port, and protocol twins (mysql+mariadb
on 3306, localstack+ministack on 4566) deconflict to base/base+1. Kafka keeps
its fixed advertised 49092.

Add five image-based shared database engines (built-in templates), each
reachable at shared-<name>, exposable, and with a client-ready connection URL:
mysql, mariadb, mongodb, cassandra, arangodb. Wire the devdock-import engine
map and the templates README.

Determinism, vet, gofmt, and the race detector on orchestrate/generate pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Base automatically changed from fix/expose-wsl2-excluded-ports to main July 10, 2026 20:15
…ix up/down

There were TWO host-port overlays fighting over the same shared containers'
`ports:` — the expose overlay (standard ports) and a separate provisioning band
(45xxx via writeProvisionOverlay/engineTarget). Every provision, `db reset`,
snapshot, resource op and re-up applied a DIFFERENT subset, so compose kept
recreating the shared container with a different port set. That churn is the root
of "port already allocated", the connection-reset races, and re-up flakiness.

Collapse the two into ONE unified overlay published on the engine's STANDARD
127.0.0.1 port:

- ensureExposed / primaryExposePort (expose.go): the single host-reachability
  primitive. Provisioning, reset, snapshot and resource ops all resolve their
  admin endpoint from it, so there is exactly one host port per engine — the
  well-known one — never a separate band. Idempotent: same ports + same overlay
  bytes → no container recreate.
- engineTarget, the resources phase, and the provision phase now go through
  ensureExposed. Deleted engineOverlays/perEngineOverlay, writeProvisionOverlay,
  and the provisionPortBase/provisionPurpose (pg-provision → pg-expose) band.
- Expose-by-default: `up` auto-publishes every exposable shared engine on its
  standard 127.0.0.1 port (loopback-only), so a GUI client's defaults just work
  and provisioning has one port to dial. Opt out with `up --no-expose`; skipped
  on a remote backend.
- `down` now actually brings things DOWN: after dropping a project's refs it
  stops every shared engine that fell to zero refs (Manager.GC stop=true),
  respecting cross-workspace ref-counting, and CLEARS the saga phase records for
  what it stopped (compose-up per project; shared/provision/resources when the
  shared stack is stopped) so the next `up` restarts them instead of skipping on
  a stale fingerprint — the "re-up after down is a no-op" bug.

Verified end-to-end against a real Docker daemon (up → provision on 127.0.0.1:5432
→ down stops shared → re-up restarts and re-publishes): standard ports throughout,
no collisions across the cycle. Unit suite + race + determinism pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@gustavobertoi gustavobertoi changed the title feat(expose): standard host ports + MySQL/MariaDB/MongoDB/Cassandra/ArangoDB engines feat: standard DB host ports, expose-by-default, unified provisioning + up/down fixes Jul 10, 2026
@gustavobertoi
gustavobertoi merged commit b215341 into main Jul 10, 2026
5 checks passed
@gustavobertoi
gustavobertoi deleted the feat/standard-db-ports-and-engines branch July 10, 2026 20:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant