Support per-node-type activity worker routing (dedicated Docker images/tools per node type)

[tbrandenburg]

Problem / Motivation

Today, every node type (trigger, decision, ai-agent, visualize, and any future type) executes as an Activity inside the same execution-worker process, built from the same Docker image. This means:

• Every node type's dependencies must be installed into one shared image — no per-tool isolation.
• A future node type that needs heavyweight or exotic tooling (a coding-agent CLI such as Claude Code, Docker-in-Docker, GPU/CUDA, a specific Python toolchain, ffmpeg, etc.) would bloat the shared image for every node type, or conflict with existing dependencies.
• All node types scale together — you can't run many replicas of a cheap node executor and one replica of an expensive one independently.
• No isolation/blast-radius boundary — a misbehaving dependency for one node type risks the whole worker process.

Goal

Allow specific node types to be executed by a dedicated worker process, running from its own Docker image with its own tools installed, while all other node types keep running on the existing general-purpose worker — with no changes to the workflow/interpreter code.

Non-goals

• Not proposing a container-per-activity-execution model (spinning up a fresh container per single node run). This is container-per-node-type-pool, i.e. a small number of long-lived specialized worker deployments, matching how Temporal is designed to scale.
• Not changing the graph interpreter (runGraph) or the deterministic workflow contract at all — this is purely a worker-topology/deployment change.

Proposed Design: Temporal Task-Queue Routing

Temporal already supports this natively: any proxyActivities(...) call can target a specific task queue name. Workers subscribe to (poll) task queues; Temporal server only ever dispatches a task to a worker polling the matching queue. So:

1. Introduce a taskQueue field on the per-node-type activity profile, so specific node types can be pinned to a non-default queue.
2. Stand up one additional worker deployment (own Dockerfile, own image, own installed tools) that polls that queue and registers only the executor(s) for that node type.
3. Everything else keeps running exactly as today, on the existing default queue/worker/image.

No change needed to: runGraph, the graph JSON model, the UI, the DB schema, or the backend's submit() call — the workflow code doesn't know or care which container ran an activity.

Required Changes

1. packages/temporal — extend the activity-profile type

packages/temporal/src/workflow/activity-profiles.ts:

export type ActivityProfile = {
  startToCloseTimeout: DurationString;
  retry: { maximumAttempts: number };
  taskQueue?: string;   // NEW — defaults to the workflow's own task queue if omitted
};

Keep DEFAULT_NODE_ACTIVITY_PROFILE unchanged (no taskQueue = current behavior, zero risk to existing node types).

2. packages/temporal — wire it into proxyActivities

packages/temporal/src/workflow/run-workflow.ts, inside runner.executeNode, pass the resolved profile's taskQueue straight through to proxyActivities(...) (structurally compatible with Temporal's real ActivityOptions already). resolveFromValidatedProfiles/profile-validation.ts need to accept and pass through the optional taskQueue.

3. apps/execution-worker — a second, minimal entrypoint

A new activity-only worker entrypoint (no workflowsPath needed — Temporal supports activity-only workers) that registers only the specialized executor(s) and polls the new task queue name.

4. New Docker image for the specialized worker

• A new Dockerfile/build stage installing whatever the new node type needs, on top of the shared runtime layer.
• A new service in the production docker-compose.yml alongside the existing worker service, connecting to the same Temporal server/namespace on a different task queue.
• A local dev script (own .env if needed) to run it standalone.

5. Validation

profile-validation.ts (already validated at Worker.create time and inside the sandbox) should also guard against a node type routed to a taskQueue with no worker polling it — otherwise the activity silently sits pending forever, which is an easy and hard-to-diagnose misconfiguration.

6. Testing

• A test that a node profile's taskQueue actually produces a ScheduleActivityTask command targeting that queue (inspect the recorded Event History, akin to the existing replay tests).
• A replay-determinism check confirming taskQueue is pure routing metadata and doesn't leak into the sandbox's decision-making.
• A manual/integration check: run a diagram with the specialized node type, confirm in Temporal UI that its task lands on the new queue, not the default one.

Rollout / Compatibility

Fully additive and backward compatible: existing node types with no taskQueue in their profile behave exactly as today. Can be introduced node-type by node-type.

Open Questions

1. Do specialized workers need their own .env/credentials, separate from the main worker?
2. Should long-running specialized activities (e.g. a CLI agent run) use Activity heartbeats rather than just a longer startToCloseTimeout?
3. Should a specialized activity's stdout/stderr integrate into the existing emitEvent/execution-log pipeline as-is, or does it need a richer event shape (streaming partial output)?
4. Should replica counts per worker type be independently configurable in the deploy compose file, given very different load/cost profiles per node type?

Summary

Add an optional taskQueue to the existing per-node-type ActivityProfile, thread it through to proxyActivities, and support standing up one additional worker process/Docker image per specialized node type that polls that queue — no changes to the graph interpreter, the workflow contract, or any other node type's behavior.

[tbrandenburg]

Open Question 1 — Worker secrets/config

Verdict: Yes — each specialized worker gets its own .env/.env.example and its own env.ts, following the pattern already established by apps/backend and apps/execution-worker today.

Justification:

This repo's existing convention is one deployable unit = one env contract, not a shared/global config:

• apps/backend/.env.example and apps/execution-worker/.env.example are separate files with overlapping-but-distinct keys (OPENROUTER_API_KEY, AI_MODEL appear in both, independently), and the comment in apps/backend/.env.example:16 is explicit: "The execution worker keeps its own key for running workflows" — this is the precedent question already answered once, for a different key.
• apps/execution-worker/src/env.ts implements requireEnv/envOr as a local, fail-fast module scoped to that package — not imported from apps/backend or any shared @workflow-builder/* package. There's no shared env module in the monorepo today; each app owns its full contract.
• deploy/ai-studio/docker-compose.yml reinforces this at the ops layer: each service gets its own environment: block wired at the compose level, with unit-specific required-vs-optional treatment (e.g. OPENROUTER_API_KEY: ${OPENROUTER_API_KEY:?set ...} for backend).

A specialized worker (e.g. apps/execution-worker-coding-agent) will need its own credentials (its own model/API key for its dedicated image, possibly a different provider or scoped token) that the main worker has no reason to hold or fail-fast-check for. Making it depend on or extend the main worker's env.ts would either force irrelevant vars into the main worker's contract or create an implicit coupling between two independently-deployed images — contradicting the "each Docker image is a separate deployable unit" premise of this issue itself.

Recommendation: Repeat the pattern exactly — new apps/<name>/ directory, own package.json, own .env.example, own env.ts with requireEnv/envOr. Do not add conditional/optional secrets to the shared execution-worker env module. taskQueue routing determines which worker picks up an activity; it says nothing about how that worker is configured, and this repo's existing convention already treats those as orthogonal.

[tbrandenburg]

Open Question 2 — Heartbeats vs. timeout

Verdict: Yes — heartbeats, and they should be a first-class ActivityProfile field, not bolted on separately.

Why timeout alone is insufficient here:

DEFAULT_NODE_ACTIVITY_PROFILE already picks a generous startToCloseTimeout with a low retry cap specifically to bound cost on partial failures (activity-profiles.ts:25-30). That reasoning holds for an LLM call that either finishes or errors quickly. A CLI-agent subprocess is a different failure mode: it can hang — wedged on stdin, a stuck child process, a network partition mid-stream — without erroring and without finishing. With only startToCloseTimeout, Temporal has no signal that the activity is dead until the full timeout elapses, so a 30-minute profile means up to 30 minutes wasted per attempt before a single retry fires, multiplied by retry.maximumAttempts. A heartbeatTimeout lets the activity self-report liveness on a much shorter cadence (e.g. every 30s) so a stalled worker slot is reclaimed and retried in seconds, not the entire timeout window — while startToCloseTimeout still bounds the case where it's alive but genuinely slow.

Fits the existing pattern:

node-activity-options.ts:37-53 and profile-validation.ts already treat a profile as a closed, whole-object shape (PROFILE_KEYS, RETRY_KEYS) explicitly to avoid silently inheriting Temporal defaults (README:96). heartbeatTimeout should follow the same rule — optional, but validated the same way as startToCloseTimeout (same DurationString + protobuf-Duration bounds check) — rather than becoming an implicit default that only long-running node types remember to set.

Concrete change:

// activity-profiles.ts
export type ActivityProfile = {
  startToCloseTimeout: DurationString;
  heartbeatTimeout?: DurationString; // required for activities that call Context.current().heartbeat()
  retry: { maximumAttempts: number };
};

Add heartbeatTimeout to PROFILE_KEYS in profile-validation.ts, pass it through in resolveFromValidatedProfiles, and document in the README that any node executor doing long-running subprocess/stream work must call heartbeat() on an interval shorter than its profile's heartbeatTimeout — otherwise Temporal cancels it as stalled.

[tbrandenburg]

Open Question 3 — Event shape for CLI output

Verdict: reuse the existing event pipeline as-is (option a) — no new event type needed for the MVP.

Why:

• execution_events.payload_json is already an untyped jsonb column (apps/backend/src/db/schema.ts:55) and emitEvent(executionId, type, payload?: unknown, nodeId?) (apps/execution-worker/src/database.ts:15) accepts any JSON-serializable payload. A captured stdout/stderr blob fits without a migration.
• The graph runner's event model is strictly binary per node today: node_started → one of node_completed / node_failed / node_skipped (packages/execution-core/src/graph-runner.ts:312-321). There is no existing notion of a partial/intermediate event for a single node — adding one would be a new concept threaded through the runner, the SSE formatEvent (apps/backend/src/routes/executions.ts:237), and the frontend execution-log renderer, not just the worker.
• Precedent: ai-agent.ts already returns the entire result as one { output: { response: ... } } payload on completion (apps/execution-worker/src/activities/ai-agent.ts:59), and errors are logged in one shot via logger?.error('llm call failed', {...}) (:63) rather than streamed. A CLI-agent activity should follow the same convention: buffer stdout/stderr for the run and attach it to the single node_completed/node_failed payload.

Recommendation:
Ship the CLI-agent node with buffered output on node_completed/node_failed (zero schema change, zero new SSE/UI work) as the MVP for this issue's routing work. Track true incremental streaming (a node_progress/partial event type, new SSE handling, log-panel append-as-you-go UI) as an explicit, separate follow-up once per-node-type worker routing has landed — same incremental-scope pattern the replay-test backlog already uses for "scenarios still worth adding." Don't couple the routing change to a streaming-UX change; they're independently valuable and independently risky.

[tbrandenburg]

Open Question 4 — Replica configurability

Verdict: No — don't add a static replica-count config field; use docker compose up --scale.

Why:

• deploy/ai-studio/docker-compose.yml has zero deploy:/replicas blocks today — backend and worker are both defined as strictly single-instance services (restart: unless-stopped only, no scaling primitive at all). There's no existing pattern to extend, only one to introduce.
• The worker service (docker-compose.yml:99-115) already differs from backend purely by command:, sharing one ai-studio-runtime image built from a single runtime target (Dockerfile:19-30, comment: "backend + worker, command chosen per compose service"). That's this repo's established precedent for "N behaviors, one image" — it argues for adding new specialized worker services (new command:/task-queue per node-type group), not a new config schema for how many of each to run.
• Docker Compose already ships the mechanism for this: docker compose up --scale worker-coding-agent=3 --scale worker-default=1. That's free, needs zero code or schema changes, and works today with plain docker compose (no Swarm/deploy: block required for scaling itself, just for resource limits if ever needed).
• No Kubernetes/Nomad/Helm manifests exist anywhere in the repo — introducing a bespoke replica-count config or orchestration layer would go against this project's stated KISS/YAGNI stance ("don't overengineer," "justify every line of code you write").

Recommendation: When this issue lands, give each specialized worker its own compose service (own command:/task-queue env, same shared ai-studio-runtime image), and document docker compose up --scale <service>=N per service as the scaling story. Revisit only if/when the project actually adopts a real orchestrator beyond docker-compose — it doesn't today.