test(cross-language): compare adk-js against adk-python on the upstream samples

[kalenkevich]

Link to Issue or Description of Change

Problem:

We had no way to answer "does adk-js actually behave like adk-python?" beyond
reading both sources. Behavioural drift between the two SDKs was invisible
until someone hit it.

Solution:

Run the same scenario through both runtimes' adk run CLIs and diff what
happened. Test cases are the adk-python
contributing/samples.

Why the CLIs and not an in-process harness. Both already speak the same two
dialects: adk run --replay <file> takes {"state": …, "queries": […]} in
each runtime, and --save_session dumps the resulting Session as JSON in the
same camelCase shape. So the harness drives the real CLIs and compares
structured events — never scraped stdout.

Wording is never a failure. The model phrases things differently between
two calls, let alone two SDKs. What must agree is what the framework decides:

Fails the suite (structural)	Reported only (cosmetic / infrastructure)	Not compared
tool calls and their argument names	event count / how text splits across events	answer wording
which agents produced events	same agents or tools in a different order	timestamps, ids
agent transfer chain	whether thought parts are surfaced	modelVersion, nodeInfo
session state keys and values	a transient model/API failure on one side	token usage
artifacts, escalation, long-running signals

Final answers go side by side in the report with a token-overlap score, so a
human can spot answers differing in substance without that ever failing a run.

The model is pinned identically on both sides. The two runtimes ship
different defaults, so a sample that omits model= would compare two different
models and blame the framework for the difference.

One run of an LLM-backed case proves nothing. Measured across two full runs
with neither framework changing, 18% of cases changed verdict. So each case
is compared --repeats times (default 3) and a difference is reported only if
a majority of repeats saw it; anything seen but not carried is listed under
Reproducibility as noise. One case alone shed five such dimensions, including
a spurious "blocked". A --repeats 1 report says in bold that it produces
leads, not findings.

Likewise a 429, 503, timeout or empty completion is classified
infrastructure: retried once and excluded from the verdict. The one
exception is deliberate — a deterministic 4xx that only one runtime provokes
stays a finding, because that is the interesting case.

Coverage: 71 cases catalogued, 56 runnable. Each has a TS port under
agents/ts/ and a Python shim under agents/py/ that imports the real
upstream sample and pins its model — the vendored checkout is never edited, so
git -C adk-python status stays clean. The 15 that cannot run carry a specific
reason; "adk-js has no equivalent for this" is recorded as a finding, not
quietly skipped. Families needing MCP, A2A, OAuth, BigQuery, live audio or
non-Gemini models are out of scope (A2A already has its own suite next door).

What it found. Three reproducible divergences, all stable at 3 repeats:

1. Workflow event authorship (19 cases) — adk-python attributes node events
   to the workflow (_workflow.py:226), adk-js to each node
   (function_node.ts:214). Breaks session portability.
2. transfer_to_agent signature — agent_name vs agentName, plus a
   missing enum constraint. Fixed in fix(core): make transfer and runtime tool confirmation work across runtimes #808.
3. Tool confirmation broken in both, oppositely — adk-js crashed on
   approval (fixed in fix(core): make transfer and runtime tool confirmation work across runtimes #808); adk-python re-raises the gate forever and the
   model then narrates success it never achieved.

Plus 12 documented capability gaps (no staticInstruction, no planners, no
logprobs on LlmResponse, 5 plugin modules vs adk-python's 15, …).

Setup and use:

tests/cross_language/python_parity/setup.sh   # clones adk-python at a pinned ref, builds a venv
npm run test:parity                           # full run, writes PARITY_REPORT.md
npm run test:parity -- --filter workflows     # one family; the report stays whole
npm run ts:check:parity                       # type-check this tree

The rendered report is not committed — it is a snapshot of one run against
one pair of versions, so a checked-in copy is stale the moment either side
moves. It is gitignored alongside runs/, the vendored checkout and the venv.

Testing Plan

Unit Tests:

• I have added or updated unit tests for my change.
• All unit tests pass locally.

✓ tests/cross_language/python_parity/python_parity_test.ts (6 tests | 1 skipped)
  Tests  5 passed | 1 skipped (6)

Under vitest this is two tiers, so CI gets value for free:

• Always — validate the catalogue (unique ids, every case names a real
  upstream sample, every runnable case has both halves, every non-runnable one
  has a reason) and load all 112 agents through both runtimes' real loaders.
  No model calls.
• ADK_PARITY_LIVE=1 — the actual side-by-side comparison.

The live suite is ~19 minutes and ~340 model calls at 3 repeats, so it is
deliberately opt-in and not wired into any workflow by this PR.

npm run ts:check:parity and eslint/prettier are clean. The tree is
excluded from the root tsconfig because it imports with explicit .ts
extensions — the harness runs under Node's --experimental-strip-types, which
resolves the real file and will not rewrite a .js specifier onto a .ts file.

Manual End-to-End (E2E) Tests:

./tests/cross_language/python_parity/setup.sh
npm run build
npm run test:parity -- --filter core --repeats 3

Needs a Gemini backend: GOOGLE_CLOUD_PROJECT (Vertex, via ADC) or
GOOGLE_API_KEY. Results quoted above are adk-python 2.7.1 (pinned at
1d89e0ff) against adk-js 1.6.0 on gemini-2.5-flash via Vertex.

Notes for reviewers

• The diff is large but shallow: 216 files, of which ~193 are the ported
  sample agents (57 TS agents + 69 three-line Python shims). The reviewable
  surface is harness/ (7 files), cases/ (9), and the README.
• This vendors an adk-python checkout into tests/cross_language/ at a
  pinned ref, provisioned by setup.sh and gitignored. Worth an explicit
  opinion if you would rather it lived elsewhere.
• No production code changes. package.json gains two scripts and
  tsconfig.json one exclude; nothing else outside the new directory.

Checklist

• I have read the contribution guide.
• My code follows the style of this project.
• I have added tests that prove my change is effective.
• New and existing unit tests pass locally with my changes.

[AmaadMartin]

The harness is test-only and well-built. I verified it adds no any, no instanceof, and no public API, and the one no-control-regex suppression (ANSI stripping) is legitimate. One blocker: package.json downgrades the version from 2.0.0 to 1.6.0, so a merge reverts the version on main. CI is green at this SHA, but run-tests does not check the version, so the fix belongs in the diff.

[AmaadMartin]

Not a nit. Restore the version to 2.0.0.

"version": "1.6.0",

The merge base has 2.0.0; this branch sets 1.6.0. A merge downgrades the version on main. The PR description says package.json only gains two scripts, so this reads as a stale rebase artifact. Set it back to "version": "2.0.0".