Add ai-studio/agent-harness node type for delegating steps to external autonomous agent CLIs/SDKs

[tbrandenburg]

Summary

Add a new node type, ai-studio/agent-harness, that delegates a workflow step to an external autonomous coding/agentic CLI or SDK (starting with GitHub Copilot CLI, with a config surface designed for parity across multiple providers) — as opposed to the existing ai-studio/ai-agent node, which performs a single bounded LLM completion via the Vercel AI SDK (generateText, single-shot, maxRetries: 0, 10-minute activity timeout).

These two node types serve fundamentally different execution models and should not share a config schema:

	ai-studio/ai-agent (existing)	ai-studio/agent-harness (proposed)
Duration	Seconds	Minutes to tens of minutes
Execution	Single generateText call in one activity	Long-lived external CLI/SDK session
Side effects	None (pure completion)	Mutates files, runs shell commands, calls tools
Retry safety	Cheap to retry (maximumAttempts: 2)	Unsafe to blindly retry once side effects have occurred
Config	systemPrompt, model, provider, webSearch	Provider selection, task/prompt, tool allow/deny list, plus a full cross-provider config surface (see below)

Why "agent-harness" and not "coding-agent"

The underlying providers are coding-agent CLIs today, but the mechanism (spawn/embed an external autonomous agent runtime that owns its own tool loop, permissions, and session state) generalizes beyond coding tasks. Naming it around "coding" would misrepresent the node and force a rename later. "Harness" is the established industry term for an opinionated, full-loop runtime wrapper — as distinct from a bare LLM completion, and as distinct from an "agent framework" a developer assembles by hand — and it avoids a naming collision with the existing ai-agent node.

v1 scope

• Provider: GitHub Copilot CLI/SDK only for the first working adapter. The config schema and capability model should be designed for multiple providers from the start (see "Cross-provider config parity" below), but only Copilot needs a functional backend for v1.
• "Done" for v1 = a demo workflow in apps/ai-studio runs one agent-harness node against Copilot and returns a result, and the node type is available in the workflow builder UI, ready to be wired into any workflow (palette entry, config form, executor, registered end-to-end).
• Explicitly deferred (see "Out of scope for v1" below): session persistence/resume, live streaming of intermediate output, structured-output enforcement, a human-in-the-loop approval gate, dedicated-worker isolation.

Workdir

Model this the same way a proven reference design for this pattern does: the working directory a node's external agent process operates in is not a per-node config field. It is resolved from a workflow-level (or future "project"/"codebase"-level) context — if the workflow defines a working directory/project root, the node uses it; if none is defined, it falls back invisibly to a freshly created scratch temp directory with no author-facing config at all. No git checkout is required — a plain, non-repo scratch directory is a first-class, fully supported mode, not a fallback hack. For v1, since workflowbuilder has no project/codebase concept yet, every run uses the scratch-temp-dir fallback; the workflow-level context lookup is a natural extension point for later, not something to build now.

Secrets / credential delivery

Never a literal token/API key in node config or workflow YAML/JSON — that much matches ai-agent's existing pattern. But a plain "one env var per provider" model (as ai-agent uses for TAVILY_API_KEY/OPENROUTER_API_KEY) does not generalize to multiple agent-harness providers and should not be the long-term shape, even though only Copilot needs a working delivery in v1:

• Different providers want fundamentally different delivery mechanisms, not just different env var names. Some need one env var; some need a different env var for an API-key credential vs. an OAuth/subscription credential (same vendor, same node, different variable names depending on how the credential was obtained); some need a file written to disk (a provider-specific auth file in a provider-specific JSON shape) plus a pointer env var telling the provider where to find it. A design that only models "env var name per provider" cannot express the file-based case at all — and that case is common enough among coding-agent CLIs that it must be in the abstraction from day one, not retrofitted when provider fix(tools): restore missing dev scripts referenced by package.json #2 needs it.
• Credential identity should be vendor-canonical, not provider/node-canonical. One underlying credential (e.g. a single Anthropic key) can power more than one harness provider's requests. Keying storage/lookup by "which harness provider" rather than "which upstream vendor" forces duplicate credential entry for providers that share a vendor.
• Credential kind changes the delivery, not just the value. An API-key credential and an OAuth/subscription credential for the same vendor typically need to be delivered differently (different env var names, or an env var vs. a file). The delivery mechanism needs an explicit kind discriminator (e.g. { kind: 'api_key' } vs. { kind: 'oauth' }), not a single opaque string.
• File-based deliveries need a run-scoped location. Any credential delivered as a file must be written under the same per-run scratch/artifacts directory the node's workdir already uses (see "Workdir" above) — scoped to one run, cleaned up with it, never shared across runs or leaked into a persistent location.
• Injected credentials must be protected from being overridden by workflow-author config. The env bag passed to a provider mixes injected credentials with ordinary node config; the names used for injected credentials must be tracked (e.g. a protectedEnvKeys list) so custom node config can never select or overwrite them — without this, a workflow author could unintentionally (or deliberately) redirect a credential value.

Proposed shape: a small, pure CredentialDeliveryPort-style function — (vendorId, credential) → { env: Record<string,string>, files?: { path: string; contents: string }[] } — with one case per supported vendor, throwing on an unrecognized vendor id rather than silently delivering nothing. For v1, only the Copilot case needs a real implementation (a single env var, no files), but the port's shape (env + optional files, kind-discriminated credential input, vendor-canonical keying, protectedEnvKeys) must exist from the start so adding a second provider is a new case in an existing function, not a redesign of the mechanism and everything that calls it.

Credential storage stays out of v1 scope. The credential value itself can keep coming from worker/deployment configuration (an env var the worker process reads at startup, matching how TAVILY_API_KEY already works) rather than building an encrypted multi-tenant credential vault — that part of a more elaborate reference design is genuinely over-sized for this single-tenant reference stack. It's specifically the delivery mechanism (env vs. file, per-vendor shape, kind-awareness) that needs to be right, not where the raw credential is sourced from.

Liveness / timeouts / cancellation

Validated by a throwaway spike (real Copilot CLI call from inside a real Temporal activity, via @temporalio/testing): a minimal heartbeat is required in v1 — not for early-stall-detection sophistication, but because without it, cancelling the workflow does nothing at all. A cancelled workflow run silently let a 60+ second Copilot call finish untouched; only once the activity called ctx.heartbeat() periodically did handle.cancel() actually reach the running activity. So:

• The executor must heartbeat on an interval (e.g. every 1s) while the subprocess runs.
• heartbeatTimeout must be set explicitly on the activity's proxyActivities options (e.g. '5s') — without it, the SDK throttles the cancel-completion RPC and a workflow caller doesn't observe the cancellation for ~30s even though the subprocess is already dead; with it, that drops to ~4s.
• startToCloseTimeout stays as the outer bound (generous default, e.g. 45 minutes) for the case where the subprocess genuinely hangs and heartbeats stop entirely.

A second, more serious finding: killing the subprocess correctly is not optional, it's a real operational hazard if skipped. The Copilot CLI is a Node wrapper that spawns a native grandchild binary. A plain child.kill() (or an execFile-based spawn with a detached option, which Node silently ignores) only kills the wrapper — the grandchild survives as an orphaned process that keeps running and keeps burning AI credits, confirmed directly in the spike (grandchild kept emitting output 15+ seconds after the wrapper process was gone). The fix, also verified in the spike: use spawn(..., { detached: true }) and kill the whole process group on cancellation (process.kill(-child.pid, 'SIGTERM')), not just the child handle. This must be in the v1 executor from the start, not deferred — an orphaned, still-billing subprocess is a cost/security issue, not a polish item.

No other liveness/idle-detection sophistication is needed for v1 beyond these two mechanisms (heartbeat-for-cancellation, process-group kill) — early-stall detection ahead of the outer timeout remains a legitimate, separate follow-up.

Retry / idempotency

• Default to zero automatic retries for this node type (maximumAttempts: 1 in its activity profile), since the external agent may have already mutated files/run commands and a naive retry could compound the damage. This is a config convention using the framework's existing per-node-type activity-profile mechanism — no new enforcement logic needed.
• Design note for whenever retry is enabled later: don't just raise the node's maximumAttempts. A framework-level activity retry restarts the whole activity (a fresh subprocess, no continuity), which is wrong once session-resume exists — retry-with-backoff (including any special-cased handling for rate-limit-style failures) should be a bounded loop the executor manages within a single activity attempt, so a resumed session's context isn't thrown away by the retry itself. Not required for v1 (no retries at all), but worth writing down now so nobody "fixes" this later by just bumping a Temporal-level retry count.
• Failure classification should use the existing PermanentNodeExecutionError / TransientNodeExecutionError / unclassified NodeExecutionError family already available to every node executor — no new classification mechanism needed, just apply it correctly (auth/config errors → permanent, timeouts/rate-limits/5xx → transient).

Structured output

Opt-in only, gated on the node config actually declaring an expected output shape — if no schema is configured, the node just returns free text and no JSON-repair/validation machinery is invoked at all. For v1, the config surface may include the field (for cross-provider parity, see below) but the enforcement pipeline (prompt augmentation, tiered parse-repair, schema validation, bounded re-ask) is out of scope — if a workflow author sets it in v1, the node should either ignore it with a visible warning or explicitly reject it as "not yet supported," rather than silently pretending to enforce something it doesn't.

Cross-provider configuration parity

The node's config schema should be designed against the full field surface a mature, multi-provider agent-harness design exposes, not just what Copilot happens to support — even though only Copilot is functionally wired in v1. Concretely, the schema/UI should account for:

• Prompt/task: inline prompt text, or a named reusable command/template with parameter bindings
• Model & provider selection: provider, model (with an 'auto' option where the provider supports it), effort/reasoning-depth
• Tool access control: allowedTools / deniedTools allow/deny lists
• Extensibility: mcp (external MCP server config reference), skills (named skill bundles), agents (inline sub-agent/custom-agent definitions)
• Structured output: an outputFormat/schema field (present in the schema, not functionally enforced in v1 — see above)
• Session/context control: a context mode (fresh / shared / resume) and persistSession flag — present in the schema and UI for parity, but only fresh is functional in v1; selecting shared/resume should surface a clear "not yet supported" state rather than silently no-op
• Cost/budget control: an optional budget cap field, understanding that not every provider (including Copilot) supports enforcing it — a provider that can't honor it should warn, not silently ignore
• Provider-specific advanced knobs: fields that only make sense for specific providers (e.g. a Claude-style settingSources/hooks equivalent) should still exist in the shared contract as optional/typed-per-provider extension points, so adding a second provider later doesn't require reshaping the schema — but they can remain functionally inert until that provider is built
• Retry/execution controls already generic to every node type: always_run, mutatesCheckout (a declared-no-mutation guard, see below), error policy

The design principle: a provider that doesn't support a given field should get a clear warning (not a silent drop) if a workflow author sets it — mirroring how capability mismatches should surface, not fail invisibly.

Note the two-axis distinction this implies: a provider capability flag describes what the underlying agent runtime can do, while our wiring describes what this repo has implemented so far. A field can be capability-supported by the provider but not yet backed by our adapter (v1: mcp, skills, agents, context beyond fresh, outputFormat). Those must render in the UI with a clear "not yet supported" state, and must not be papered over by lying in the capability flags.

UI implementation note

This repo's node config UI is JSONForms-driven (schema.ts + uischema.ts, an existing custom-control pattern like the ai-agent node's tool-configuration control), not a hand-rolled form component tree. Achieving the field surface above will very likely require one new custom JSONForms control — a JSON-textarea-with-parse-validation control for the handful of free-form/structured fields (outputFormat schema, any raw hook/sandbox config) that don't fit a simple text/select input, following the same pattern used for the existing tool-configuration custom control rather than inventing an ad hoc form mechanism.

mutatesCheckout guard (worth porting as a cheap idempotency win)

A lightweight guard pattern: a node can declare it should not touch the working tree; the executor snapshots working-tree state (e.g. git status --porcelain, when a workdir happens to be a repo) before and after, and forces a non-retryable failure if the declaration is violated. Cheap to implement, catches a real class of bugs, and sits naturally alongside the zero-retry default above. Worth including even in a minimal v1 if the workdir is ever a real repo; a no-op against a scratch temp dir either way.

What NOT to copy from prior art without adapting

Several known design patterns for this kind of node exist in other coding-agent orchestration systems, and are a reasonable reference — but three things about how they're built don't transfer directly to this codebase's architecture and should be adapted, not ported verbatim:

1. Determinism boundary. Some reference designs run node scheduling and node execution in the same process/loop with no replay-determinism constraint. Here, workflow scheduling runs inside the workflow engine's deterministic sandbox — everything this node does (spawning the external process, waiting on its output, any structured-output repair logic) must live entirely inside the node's executor/activity function, never inside the graph scheduler itself.
2. Binary/SDK resolution complexity. Some reference implementations go through an elaborate multi-step binary-resolution fallback chain because they ship as a compiled, self-contained binary where dynamic imports and filesystem-relative paths break at runtime. This repo's worker runs as a normal Node.js/Docker process with node_modules — the Copilot SDK should be a plain top-level dependency with normal resolution; there's no need to replicate that defensive complexity here.
3. Credential storage. Some reference implementations maintain a full encrypted, multi-tenant, multi-vendor credential store (per-user rows, envelope encryption, an OAuth token vault) because they're built as a multi-tenant SaaS product. This repo's reference backend is single-tenant — sourcing the raw credential value from worker/deployment env config (not a database-backed vault) is the correctly-sized equivalent. This is distinct from the credential delivery mechanism (see "Secrets / credential delivery" above), which does need to be a proper per-vendor abstraction regardless of how simple the storage side is.

Explicitly out of scope for v1

• Session persistence/resume — the context/persistSession fields exist in the schema for parity but are non-functional beyond fresh in v1.
• Live streaming of intermediate output — v1 is a blocking, final-result-only node; streaming tool calls/partial text back to the UI mid-run is a follow-up.
• Structured-output enforcement pipeline — the field may exist in config, but no parsing/validation/repair logic ships in v1.
• Human-in-the-loop approval gate — worth a future, separate node type and issue; not part of this one at all.
• Dedicated worker/task-queue isolation — nice-to-have for operational/cost isolation later; not required for v1 to be considered done.
• Container/sandboxed process execution — host-subprocess execution only for now.
• Any new engine-agnostic port abstraction (abort-propagation ports, session-storage ports) — not needed given the "session persistence" decision above and the fact that heartbeating/cancellation is achievable directly against the workflow engine's own existing activity-context API (see "Liveness / timeouts / cancellation" above), with no new port required; revisit only if a future engine adapter needs the same capability through a different mechanism.

Pre-work spike: validated

A throwaway spike (real Copilot CLI call from inside a real Temporal activity, isolated from this repo) was run before committing to this design, to answer "does this mechanism actually work inside a worker activity, or does something about that execution model break it?" Findings (folded into the sections above):

• A Copilot CLI call via spawn inside a Temporal activity completes cleanly — correct, non-truncated output, no interference between the subprocess's stdout/stderr and the Temporal worker's own server connection.
• Two concurrent calls (two parallel workflow executions) showed no resource contention, port conflicts, or session-file locking issues, and ran genuinely in parallel (not serialized).
• Timing: a small real prompt took ~8-9s end-to-end through the activity (Temporal overhead <1s versus running the CLI bare); a longer prompt ran 56-62s uncancelled.
• The two real gotchas — cancellation requiring a heartbeat, and subprocess kill requiring process-group semantics — are exactly what's now specified in "Liveness / timeouts / cancellation" above. Without them: cancellation is silently ignored, and killing the wrapper process orphans a still-running, still-billing native subprocess.

No further spike is needed before implementation; the two behavioral requirements it surfaced are now explicit, non-optional parts of this issue's scope rather than open questions.

Implementation approach

This is a port of an existing, proven implementation, not a from-scratch design. A detailed file-by-file implementation plan exists, mapping each module to its reference source, with milestones and per-milestone verification gates. The provider contract, credential-delivery model, event/chunk translation, error taxonomy, idle-timeout handling, and the checkout-mutation guard all have a working reference to follow closely — deviation is limited to documented host-architecture mismatches (deterministic graph-runner sandbox, Temporal activity lifecycle, heartbeat-driven cancellation, process-group subprocess kill, JSONForms UI, single-tenant credential storage).

The PR implementing this must credit the reference implementation it derives from.

Suggested build order

1. Config schema + capability model designed for cross-provider parity (types only, Copilot-functional subset implemented).
2. Credential-delivery port ((vendorId, credential) → { env, files? }, kind-discriminated, protectedEnvKeys-aware) with a working Copilot case.
3. Copilot provider adapter: prompt in, scratch-workdir execution, credential via the delivery port, tool allow/deny translation, plain-text result out, token usage captured where available — with heartbeat-while-running and process-group-kill-on-cancel from the start (see "Liveness / timeouts / cancellation").
4. Node registration: palette entry, schema.ts/uischema.ts (incl. the new JSON-textarea custom control if needed), executor wiring, activity profile (45 min startToCloseTimeout, 5s heartbeatTimeout, maximumAttempts: 1).
5. mutatesCheckout guard (cheap, worth including from the start).
6. Demo workflow template in apps/ai-studio/src/data/, registered alongside the existing templates.
7. Documentation: workspace READMEs updated to describe the new node type, its config surface, and its v1 limitations.

Acceptance criteria

• Pre-work spike validated the mechanism works inside a worker activity (see "Pre-work spike: validated" above) — no further spike needed before implementation.
• New node type ai-studio/agent-harness registered independently from ai-studio/ai-agent, with its own config schema, palette entry, and executor.
• Config schema/UI expose the full cross-provider field surface described above (prompt/command, provider/model/effort, tool allow/deny, mcp/skills/agents, output format, context/session mode, budget cap, mutatesCheckout), with non-functional-for-v1 fields (context modes beyond fresh, output-format enforcement) surfacing a clear "not yet supported" state rather than silently no-opping.
• A credential-delivery function/port exists with the shape (vendorId, credential) → { env, files? }, kind-discriminated (api_key vs oauth) input, vendor-canonical keying, and protectedEnvKeys support — with a working Copilot case (single env var) and the shape proven extensible to a file-based case without a redesign.
• Copilot provider adapter fully wired end-to-end: config → scratch-workdir execution → credential delivered via the delivery port → plain-text result → token usage captured (where the SDK reports it).
• Executor heartbeats while the subprocess runs, and the activity profile sets heartbeatTimeout (e.g. 5s) alongside startToCloseTimeout (~45 min) and maximumAttempts: 1 — cancelling the workflow must actually terminate the running Copilot call within a few seconds, verified with a real test.
• Subprocess is spawned with process-group semantics (detached: true) and killed via process.kill(-pid, 'SIGTERM') on cancellation/timeout — verified no orphaned copilot process survives after cancellation.
• mutatesCheckout guard implemented and enforced outside the (currently disabled) retry path.
• A demo workflow in apps/ai-studio runs one agent-harness node against Copilot and returns a result, using it as the reference wiring example.
• Documentation added to the relevant workspace READMEs (apps/ai-studio/README.md, apps/execution-worker/README.md) describing the new node type, its config surface, and its v1 limitations (no retries, no session resume, no streaming, no structured-output enforcement).