feat(templates): cloud-emulation engines — localstack, nats, kafka (redpanda), rabbitmq (spec 28)

docs/specs/28-cloud-engine-templates.md

@@ -234,7 +234,7 @@ Because the AWS API surface is what matters, `ministack` reuses LocalStack's exa
 Consumes `internal/scaffold` (the `provides:`-required builder), `internal/template` (the `[[ ]]` engine + `go:embed`/store chain), `internal/generate` (`sharedAttr`/`sharedAlias` + the new non-secret export cases and the secondary-port lookup), `internal/workspace` (`(engine, major)` ledger keying, ref-counting/reconcile, `FreeHostPort(ctx, owner, purpose, base)`), `internal/state`+`internal/lock` (the `provisioned` free-text-`kind` ledger + flock), `internal/secrets` (valueless env injection for broker creds, [spec 04](04-secrets.md)). **Consumed by** [spec 29](29-resource-commands.md) (the `aws`/`nats`/`kafka`/`amqp` resource commands that provision buckets/queues/streams/topics inside these engines — the imperative sibling, with [spec 15](15-db-snapshot-restore.md) as the data-command precedent) and `internal/doctor` (the new `info`-level `bin.aws`/`bin.nats`/`bin.rpk`/`bin.rabbitmqadmin` probes, introduced with spec 29's verbs). **Thin (~1w):** LocalStack + NATS templates with health + host overlay + the new `endpoint`/`region` export attrs. **Full (~2.5w):** adds Kafka (Redpanda) + RabbitMQ, the secondary-port resolver lookup, the `ministack` AWS-emulation engine template (once ministack.org config is confirmed), and golden tests.
 
 ## Open questions
-**Q-MINISTACK — RESOLVED (owner):** `ministack` is the [ministack.org](https://ministack.org/) container image — a real AWS-local-emulation runtime, a lighter alternative to LocalStack. It is authored as a normal AWS-emulation **engine template** (`provides: aws`), interchangeable with LocalStack by template name (both key the ledger `engine=aws`). **Remaining sub-task before authoring:** confirm the exact image repo/tag, edge/gateway port, service-selection env knob, and health endpoint against the ministack.org docs (the recon found zero references; spec uses placeholders).
+**Q-MINISTACK — RESOLVED (owner):** `ministack` is the [ministack.org](https://ministack.org/) container image — a real AWS-local-emulation runtime, a lighter alternative to LocalStack. It is authored as a normal AWS-emulation **engine template** (`provides: aws`), interchangeable with LocalStack by template name (both key the ledger `engine=aws`). **Remaining sub-task before authoring:** confirm the exact image repo/tag, edge/gateway port, service-selection env knob, and health endpoint against the ministack.org docs (the recon found zero references; spec uses placeholders). **Implementation note (not yet authored):** the first cloud-engine batch ships `localstack`/`nats`/`kafka`/`rabbitmq` only; `templates/ministack/` is deliberately **deferred** because its image tag/edge port/`SERVICES` knob/health endpoint are unconfirmed offline — authoring it with a guessed image would key the `aws` engine to a non-existent container. It lands once a maintainer confirms the ministack.org config; LocalStack already satisfies the `aws` capability in the meantime.
 **Q-SECONDARY-PORTS** — how do `monitorPort`/`adminPort`/`mgmtPort` resolve, given the resolver tracks only one in-network port per shared service (`sharedPort map[string]int`)? **Recommendation:** add a small static per-template export-attr→port map in `internal/generate` (driven off the template's declared secondary ports), not a hardcode in `sharedAttr`. **Decision:** owner to confirm whether secondary-port exports ship in v2 or are deferred (host-published mgmt/monitor UIs work without a `${ref}` export).
 **Q-KAFKA-RUNTIME** — Redpanda vs Apache Kafka KRaft as the `kafka` engine. **Recommendation:** Redpanda (single static binary, no JVM/quorum, healthy in seconds; `provides: kafka` keeps consumers implementation-agnostic). **Decision:** owner to confirm Redpanda as the default `kafka` image, with an `image:` param escape hatch to `apache/kafka` for parity testing.
 **Q-LOCALSTACK-EDITION** — community default vs Pro opt-in. **Recommendation:** community (`localstack/localstack`, Apache-2.0) default; Pro only via owner-set `image`+`authToken` (token through `env.import`). **Decision:** owner to confirm community-only ships v2; Pro support is param-gated, not bundled.

internal/generate/cloud_engines_test.go

@@ -0,0 +1,144 @@
+package generate
+
+import (
+	"strings"
+	"testing"
+
+	"github.com/open-source-cloud/devstack/internal/config"
+	"github.com/open-source-cloud/devstack/internal/template"
+	"github.com/open-source-cloud/devstack/templates"
+)
+
+// TestCloudEngineTemplatesLint resolves each spec-28 cloud-emulation engine from
+// the embedded built-in set and validates it through compose-go (the same path
+// `template lint`/`template test` drive). Each must declare a non-empty provides:
+// and a defaultPort: and produce a compose-valid single-service fragment.
+func TestCloudEngineTemplatesLint(t *testing.T) {
+	src := template.NewFSSource(templates.FS)
+	cases := []struct {
+		name     string
+		provides string
+		port     int
+		wantIn   string // a literal that must survive into the rendered compose
+	}{
+		{"localstack", "aws", 4566, "_localstack/health"},
+		{"nats", "nats", 4222, "-js"},
+		{"kafka", "kafka", 9092, "advertise-kafka-addr=internal://shared-kafka:9092"},
+		{"rabbitmq", "amqp", 5672, "rabbitmq-diagnostics -q ping"},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			res, err := template.Resolve(src, tc.name, nil)
+			if err != nil {
+				t.Fatalf("resolve %s: %v", tc.name, err)
+			}
+			if res.Provides != tc.provides {
+				t.Errorf("provides = %q, want %q", res.Provides, tc.provides)
+			}
+			if res.DefaultPort != tc.port {
+				t.Errorf("defaultPort = %d, want %d", res.DefaultPort, tc.port)
+			}
+			compose, err := LintResolved(tc.name, res)
+			if err != nil {
+				t.Fatalf("lint %s: %v", tc.name, err)
+			}
+			if !strings.Contains(string(compose), tc.wantIn) {
+				t.Errorf("compose missing %q:\n%s", tc.wantIn, compose)
+			}
+		})
+	}
+}
+
+// TestRabbitMQSecretIsValueless asserts RABBITMQ_DEFAULT_PASS is emitted as a
+// valueless env key (no plaintext) — the §7.5 secret coupling for broker creds.
+func TestRabbitMQSecretIsValueless(t *testing.T) {
+	src := template.NewFSSource(templates.FS)
+	res, err := template.Resolve(src, "rabbitmq", nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+	compose, err := LintResolved("rabbitmq", res)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !strings.Contains(string(compose), "RABBITMQ_DEFAULT_PASS: null") {
+		t.Errorf("RABBITMQ_DEFAULT_PASS should be a valueless (null) key:\n%s", compose)
+	}
+}
+
+// cloudResolver builds a graphResolver whose workspace declares the four spec-28
+// engines so the new export-attr cases can be exercised directly.
+func cloudResolver() *graphResolver {
+	m := &config.Model{
+		Workspace: config.Workspace{
+			Name: "w",
+			Shared: map[string]config.SharedSvc{
+				"localstack": {Template: "localstack", Params: map[string]any{"region": "eu-west-1"}},
+				"awsdefault": {Template: "localstack"},
+				"nats":       {Template: "nats"},
+				"kafka":      {Template: "kafka"},
+				"rabbitmq":   {Template: "rabbitmq"},
+			},
+		},
+		Projects: map[string]config.Project{},
+	}
+	return &graphResolver{
+		model: m,
+		sharedPort: map[string]int{
+			"localstack": 4566,
+			"awsdefault": 4566,
+			"nats":       4222,
+			"kafka":      9092,
+			"rabbitmq":   5672,
+		},
+		curProject: "api",
+	}
+}
+
+// TestSharedAttr_EndpointAndRegion covers the new non-secret AWS-emulation export
+// attrs: endpoint resolves to the in-network alias URL, region to the param (or
+// the us-east-1 default when unset).
+func TestSharedAttr_EndpointAndRegion(t *testing.T) {
+	r := cloudResolver()
+	if got, err := r.sharedAttr("localstack", "endpoint"); err != nil || got != "http://shared-localstack:4566" {
+		t.Errorf("endpoint = %q err=%v, want http://shared-localstack:4566", got, err)
+	}
+	if got, err := r.sharedAttr("localstack", "region"); err != nil || got != "eu-west-1" {
+		t.Errorf("region = %q err=%v, want eu-west-1 (the param)", got, err)
+	}
+	if got, err := r.sharedAttr("awsdefault", "region"); err != nil || got != "us-east-1" {
+		t.Errorf("region default = %q err=%v, want us-east-1", got, err)
+	}
+}
+
+// TestSharedAttr_SecondaryPorts covers the per-template export-attr→secondary-port
+// lookup (monitorPort/adminPort/mgmtPort). Attrs reach sharedAttr lowercased.
+func TestSharedAttr_SecondaryPorts(t *testing.T) {
+	r := cloudResolver()
+	cases := []struct {
+		name, attr, want string
+	}{
+		{"nats", "monitorport", "8222"},
+		{"kafka", "adminport", "9644"},
+		{"rabbitmq", "mgmtport", "15672"},
+	}
+	for _, tc := range cases {
+		got, err := r.sharedAttr(tc.name, tc.attr)
+		if err != nil || got != tc.want {
+			t.Errorf("%s.%s = %q err=%v, want %s", tc.name, tc.attr, got, err, tc.want)
+		}
+	}
+	// An admin port is engine-specific: nats has no adminPort.
+	if _, err := r.sharedAttr("nats", "adminport"); err == nil {
+		t.Error("nats.adminPort should be unknown")
+	}
+}
+
+// TestSharedAttr_SecretRejected proves an inline ${ref:...secret} attribute on a
+// cloud engine is rejected at resolve time (must flow through env.import).
+func TestSharedAttr_SecretRejected(t *testing.T) {
+	r := cloudResolver()
+	if _, err := r.Ref("workspace.shared.rabbitmq.password"); err == nil || !strings.Contains(err.Error(), "secret") {
+		t.Errorf("rabbitmq.password should be rejected as a secret, got %v", err)
+	}
+}

internal/generate/resolver.go

@@ -76,11 +76,32 @@ func (r *graphResolver) refAttr(ref config.Reference) (string, error) {
 	}
 }
 
+// defaultAWSRegion is the region advertised when a shared AWS-emulation engine
+// (LocalStack/ministack) declares no `region` param (spec 28).
+const defaultAWSRegion = "us-east-1"
+
+// secondaryPorts maps a shared engine's template name to its non-default
+// (secondary) export attrs and the in-network container port each resolves to
+// (spec 28 Q-SECONDARY-PORTS). The resolver tracks only one in-network port per
+// shared service (sharedPort); these extra admin/monitor/mgmt ports are static
+// per the template's declared service command and resolved from this lookup so a
+// consumer's ${ref:...<engine>.monitorPort} resolves without a hardcode in the
+// generic sharedAttr switch.
+var secondaryPorts = map[string]map[string]int{
+	"nats":     {"monitorport": 8222},
+	"kafka":    {"adminport": 9644},
+	"rabbitmq": {"mgmtport": 15672},
+}
+
 // sharedAttr resolves an attribute of a shared service. host/port are stable
 // (DNS alias + the engine's default port); user/database default to the CONSUMER
 // project name — the per-project role/db provisioned on the shared engine in M2.
+// endpoint/region (AWS-emulation engines) and the per-engine secondary admin/
+// monitor/mgmt ports are non-secret extras resolved from the alias, the default
+// port, the `region` param, and the static secondaryPorts lookup (spec 28).
 func (r *graphResolver) sharedAttr(name, attr string) (string, error) {
-	if _, ok := r.model.Workspace.Shared[name]; !ok {
+	svc, ok := r.model.Workspace.Shared[name]
+	if !ok {
 		return "", fmt.Errorf("shared service %q does not exist%s", name, suggestShared(r.model))
 	}
 	switch attr {
@@ -91,11 +112,27 @@ func (r *graphResolver) sharedAttr(name, attr string) (string, error) {
 			return strconv.Itoa(p), nil
 		}
 		return "", fmt.Errorf("shared service %q exposes no default port", name)
+	case "endpoint":
+		p := r.sharedPort[name]
+		if p == 0 {
+			return "", fmt.Errorf("shared service %q exposes no default port for its endpoint", name)
+		}
+		return fmt.Sprintf("http://%s:%d", sharedAlias(name), p), nil
+	case "region":
+		if v, ok := svc.Params["region"].(string); ok && v != "" {
+			return v, nil
+		}
+		return defaultAWSRegion, nil
 	case "user", "accesskey":
 		return r.curProject, nil
 	case "database", "db":
 		return r.curProject, nil
 	default:
+		if ports, ok := secondaryPorts[svc.Template]; ok {
+			if p, ok := ports[attr]; ok {
+				return strconv.Itoa(p), nil
+			}
+		}
 		return "", fmt.Errorf("unknown attribute %q on shared service %q", attr, name)
 	}
 }

templates/embed.go

@@ -7,7 +7,7 @@ package templates
 
 import "embed"
 
-//go:embed all:postgres all:redis all:minio all:php.nginx all:php.laravel.nginx all:node.vite
+//go:embed all:postgres all:redis all:minio all:php.nginx all:php.laravel.nginx all:node.vite all:localstack all:nats all:kafka all:rabbitmq
 var builtinFS embed.FS
 
 // FS is the embedded built-in templates root: template-name directories at the

templates/kafka/golden.yaml

@@ -0,0 +1,34 @@
+name: devstack-lint
+services:
+  kafka:
+    command:
+      - redpanda
+      - start
+      - --mode=dev-container
+      - --smp=1
+      - --default-log-level=warn
+      - --kafka-addr=internal://0.0.0.0:9092,external://0.0.0.0:19092
+      - --advertise-kafka-addr=internal://shared-kafka:9092,external://127.0.0.1:49092
+    healthcheck:
+      test:
+        - CMD-SHELL
+        - rpk cluster health -x admin.hosts=localhost:9644 | grep -q 'Healthy:.*true'
+      timeout: 5s
+      interval: 10s
+      retries: 8
+      start_period: 15s
+    image: redpandadata/redpanda:v24.2
+    networks:
+      default: null
+    restart: unless-stopped
+    volumes:
+      - type: volume
+        source: kafkadata
+        target: /var/lib/redpanda/data
+        volume: {}
+networks:
+  default:
+    name: devstack-lint_default
+volumes:
+  kafkadata:
+    name: devstack-lint_kafkadata

templates/kafka/template.yaml

@@ -0,0 +1,35 @@
+schemaVersion: 1
+description: "Shared Kafka-compatible broker (Redpanda), reached over the shared network at shared-kafka:9092."
+provides: kafka
+exports: [host, port, adminPort]
+defaultPort: 9092
+params:
+  image:
+    type: string
+    default: "redpandadata/redpanda:v24.2"
+    description: "Single-binary Kafka-API broker; pin a MAJOR-stable tag for (engine,major) ledger keying."
+
+service:
+  image: "[[ .params.image ]]"
+  restart: unless-stopped
+  command:
+    - "redpanda"
+    - "start"
+    - "--mode=dev-container"
+    - "--smp=1"
+    - "--default-log-level=warn"
+    # TWO advertised listeners: in-network clients use the DNS alias; host tools
+    # use 127.0.0.1. A single advertised listener is the #1 Kafka-local footgun.
+    - "--kafka-addr=internal://0.0.0.0:9092,external://0.0.0.0:19092"
+    - "--advertise-kafka-addr=internal://shared-kafka:9092,external://127.0.0.1:49092"
+  volumes:
+    - "kafkadata:/var/lib/redpanda/data"
+  healthcheck:
+    test: ["CMD-SHELL", "rpk cluster health -x admin.hosts=localhost:9644 | grep -q 'Healthy:.*true'"]
+    interval: 10s
+    timeout: 5s
+    retries: 8
+    start_period: 15s
+
+volumes:
+  kafkadata: {}

templates/localstack/golden.yaml

@@ -0,0 +1,31 @@
+name: devstack-lint
+services:
+  localstack:
+    environment:
+      AWS_DEFAULT_REGION: us-east-1
+      DEBUG: "0"
+      PERSISTENCE: "1"
+      SERVICES: s3,sqs,sns,kinesis,dynamodb
+    healthcheck:
+      test:
+        - CMD-SHELL
+        - curl -sf http://localhost:4566/_localstack/health | grep -q running
+      timeout: 5s
+      interval: 10s
+      retries: 8
+      start_period: 20s
+    image: localstack/localstack:3
+    networks:
+      default: null
+    restart: unless-stopped
+    volumes:
+      - type: volume
+        source: localstackdata
+        target: /var/lib/localstack
+        volume: {}
+networks:
+  default:
+    name: devstack-lint_default
+volumes:
+  localstackdata:
+    name: devstack-lint_localstackdata

templates/localstack/template.yaml

@@ -0,0 +1,40 @@
+schemaVersion: 1
+description: "Shared LocalStack AWS-emulation engine, reached over the shared network at shared-localstack:4566."
+provides: aws
+exports: [host, port, endpoint, region]
+defaultPort: 4566
+params:
+  image:
+    type: string
+    default: "localstack/localstack:3"
+    description: "LocalStack community image; pin a major for (engine,major) ledger keying."
+  services:
+    type: string
+    default: "s3,sqs,sns,kinesis,dynamodb"
+    description: "Comma list for the SERVICES env (lazy-loads only these for fast, light startup)."
+  region:
+    type: string
+    default: "us-east-1"
+    description: "Default AWS region advertised to in-network consumers (the region export)."
+
+service:
+  image: "[[ .params.image ]]"
+  restart: unless-stopped
+  environment:
+    SERVICES: "[[ .params.services ]]"
+    DEBUG: "0"
+    AWS_DEFAULT_REGION: "[[ .params.region ]]"
+    # Community persistence is best-effort; treat LocalStack data as recreatable.
+    PERSISTENCE: "1"
+  volumes:
+    - "localstackdata:/var/lib/localstack"
+  healthcheck:
+    # /_localstack/health reports per-service readiness; gate on the edge being up.
+    test: ["CMD-SHELL", "curl -sf http://localhost:4566/_localstack/health | grep -q running"]
+    interval: 10s
+    timeout: 5s
+    retries: 8
+    start_period: 20s
+
+volumes:
+  localstackdata: {}