feat(benchmark): collect harbor atif trajectories

.gitignore

@@ -218,3 +218,6 @@ __marimo__/
 .streamlit/secrets.toml
 
 references/
+
+# Harbor benchmark results
+/jobs/

benchmarks/harbor/README.md

@@ -9,13 +9,13 @@ workspace's lockfile.
 
 ## Run a benchmark
 
-Set the same `ANTHROPIC_*` environment variables used by the CLI. For a
-third-party or proxy endpoint, set all three explicitly:
+Set the connection credentials used by the CLI. For a third-party or proxy
+endpoint, configure the API key and base URL; select the model with Harbor's
+`--model` option shown below:
 
 ```bash
 export ANTHROPIC_API_KEY="..."
 export ANTHROPIC_BASE_URL="https://gateway.example"
-export ANTHROPIC_MODEL="provider-model-name"
 ```
 
 From the repository root, run Harbor through this workspace and pin the agent
@@ -35,17 +35,29 @@ uv run --project benchmarks/harbor harbor run \
 Use `--agent-kwarg version=<released-version>` instead of `git_ref` to install a
 published PyPI release. The two pins are mutually exclusive. If neither is
 provided, the adapter installs the latest published release. For reproducible
-benchmark results, always provide one of them.
+benchmark results, always provide one of them. Prefer a full 40-character commit
+SHA for `git_ref`; Harbor may parse an unquoted abbreviated SHA such as `83e6271`
+as a number. To use an abbreviated revision, preserve its string type with
+`--agent-kwarg 'git_ref="83e6271"'`.
 
 The adapter sends the instruction through stdin, runs the agent in the task
 container's current directory, and saves combined stdout/stderr to
 `/logs/agent/nanopycodeagent.txt`. It uses the CLI's 50-turn default; override
 that with `--agent-kwarg max_turns=20`.
 
-`ANTHROPIC_MODEL` takes precedence over Harbor's `--model`. When it is absent,
-the adapter strips the first provider prefix from `--model`. Harbor-native
-provider credentials and configured base URLs are also normalized to the
-`ANTHROPIC_*` variables expected by nanoPyCodeAgent's SDK.
+The adapter also asks the agent to write an ATIF-v1.7 trajectory directly to
+`/logs/agent/trajectory.json`. Harbor collects that file as the trial's native
+ATIF output and backfills prompt, completion, cache-token, and cost totals into
+the agent result. Missing, invalid, or partial trajectories remain explicitly
+diagnosed; unknown usage or cost is not reported as zero.
+
+By default, the adapter strips the first provider prefix from `--model` and
+passes the result to nanoPyCodeAgent as `ANTHROPIC_MODEL`. Set
+`ANTHROPIC_MODEL` only when a custom endpoint requires an actual model name that
+differs from Harbor's `provider/model` identity; this explicit override takes
+precedence. Harbor-native provider credentials and configured base URLs are
+also normalized to the `ANTHROPIC_*` variables expected by nanoPyCodeAgent's
+SDK.
 
 ## Test the adapter
 

benchmarks/harbor/README.zh-CN.md

@@ -9,13 +9,12 @@
 
 ## 运行 benchmark
 
-设置与 CLI 相同的 `ANTHROPIC_*` 环境变量。使用第三方或代理 endpoint 时，显式
-设置以下三个变量：
+设置 CLI 使用的连接凭据。使用第三方或代理 endpoint 时，配置 API key 和 base
+URL；模型由下方 Harbor 命令中的 `--model` 选择：
 
 ```bash
 export ANTHROPIC_API_KEY="..."
 export ANTHROPIC_BASE_URL="https://gateway.example"
-export ANTHROPIC_MODEL="provider-model-name"
 ```
 
 从仓库根目录通过这个 workspace 运行 Harbor，并固定安装到 task 容器中的 agent：
@@ -34,15 +33,24 @@ uv run --project benchmarks/harbor harbor run \
 如需安装已经发布到 PyPI 的版本，请使用
 `--agent-kwarg version=<released-version>` 代替 `git_ref`。这两个版本参数互斥；
 如果都未提供，adapter 会安装最新发布版本。为了让 benchmark 结果可复现，请始终
-提供其中一个参数。
+提供其中一个参数。`git_ref` 最好使用完整的 40 位 commit SHA；Harbor 可能把
+`83e6271` 这样的无引号短 SHA 解析成数字。如果需要使用短 revision，请通过
+`--agent-kwarg 'git_ref="83e6271"'` 保留其字符串类型。
 
 adapter 通过 stdin 发送任务指令，在 task 容器的当前目录中运行 agent，并把合并后
 的 stdout/stderr 保存到 `/logs/agent/nanopycodeagent.txt`。它默认沿用 CLI 的 50
 轮限制；可以通过 `--agent-kwarg max_turns=20` 覆盖此设置。
 
-`ANTHROPIC_MODEL` 的优先级高于 Harbor 的 `--model`。未设置该环境变量时，adapter
-会移除 `--model` 中的第一个 provider 前缀。Harbor 原生的 provider 凭证和已经配置
-的 base URL 也会转换为 nanoPyCodeAgent SDK 所需的 `ANTHROPIC_*` 变量。
+adapter 还会要求 agent 将 ATIF-v1.7 trajectory 直接写入
+`/logs/agent/trajectory.json`。Harbor 会把该文件作为 trial 的原生 ATIF 输出采集，
+并将 prompt、completion、cache token 和 cost 汇总回填到 agent result。trajectory
+缺失、无效或指标不完整时会保留明确诊断；未知 usage 或 cost 不会被记成零。
+
+默认情况下，adapter 会移除 `--model` 中的第一个 provider 前缀，并将结果作为
+`ANTHROPIC_MODEL` 传给 nanoPyCodeAgent。只有自定义 endpoint 要求的实际模型名与
+Harbor 的 `provider/model` 身份不同时，才显式设置 `ANTHROPIC_MODEL`；该覆盖值优先。
+Harbor 原生的 provider 凭证和已经配置的 base URL 也会转换为 nanoPyCodeAgent SDK
+所需的 `ANTHROPIC_*` 变量。
 
 ## 测试 adapter
 

benchmarks/harbor/src/harbor_adapter/adapter.py

@@ -13,17 +13,20 @@
 from harbor.agents.model_connection import ModelConnectionSpec
 from harbor.environments.base import BaseEnvironment
 from harbor.models.agent.context import AgentContext
+from harbor.models.trajectories.trajectory import Trajectory
 
 _DEFAULT_MAX_TURNS = 50
 _PACKAGE_NAME = "nanoPyCodeAgent"
 _REPOSITORY_URL = "https://github.com/minixalpha/nanoPyCodeAgent.git"
 _UV_VERSION = "0.9.11"
 _PATH_SETUP = 'export PATH="$HOME/.local/bin:$PATH"; '
+_TRAJECTORY_PATH = "/logs/agent/trajectory.json"
 
 
 class NanoPyCodeAgent(BaseInstalledAgent):
     """Install and run nanoPyCodeAgent inside a Harbor task environment."""
 
+    SUPPORTS_ATIF = True
     MODEL_CONNECTION = ModelConnectionSpec(
         api_key_envs=("ANTHROPIC_API_KEY",),
         base_url_envs=("ANTHROPIC_BASE_URL",),
@@ -39,6 +42,11 @@ class NanoPyCodeAgent(BaseInstalledAgent):
 
     def __init__(self, *args, git_ref: str | None = None, **kwargs):
         if git_ref is not None:
+            if not isinstance(git_ref, str):
+                raise ValueError(
+                    "git_ref must be a string; use a full commit SHA or quote "
+                    "an abbreviated revision in --agent-kwarg"
+                )
             git_ref = git_ref.strip()
             if not git_ref:
                 raise ValueError("git_ref must not be blank")
@@ -131,7 +139,70 @@ async def run(
                 f"unset {instruction_env_var}; "
                 f'printf "%s" "${instruction_shell_var}" | '
                 f"nanoPyCodeAgent {cli_flags} "
+                f"--trajectory {_TRAJECTORY_PATH} "
                 "2>&1 | tee /logs/agent/nanopycodeagent.txt"
             ),
             env=env,
         )
+
+    @override
+    def populate_context_post_run(self, context: AgentContext) -> None:
+        trajectory_path = self.logs_dir / "trajectory.json"
+        try:
+            trajectory = Trajectory.model_validate_json(
+                trajectory_path.read_text(encoding="utf-8")
+            )
+            if trajectory.schema_version != "ATIF-v1.7":
+                raise ValueError(
+                    f"expected ATIF-v1.7, got {trajectory.schema_version}"
+                )
+        except FileNotFoundError:
+            self._record_trajectory_diagnostic(context, "missing")
+            self.logger.warning("No ATIF trajectory found at %s", trajectory_path)
+            return
+        except (OSError, UnicodeError, ValueError) as exc:
+            self._record_trajectory_diagnostic(
+                context,
+                "invalid",
+                error=f"{type(exc).__name__}: {exc}",
+            )
+            self.logger.warning(
+                "Failed to read ATIF trajectory at %s: %s",
+                trajectory_path,
+                exc,
+            )
+            return
+
+        metrics = trajectory.final_metrics
+        extra = metrics.extra if metrics and metrics.extra else {}
+        is_partial = extra.get("usage_complete") is False or (
+            extra.get("cost_is_partial") is True
+        )
+        self._record_trajectory_diagnostic(
+            context,
+            "partial" if is_partial else "complete",
+            total_steps=len(trajectory.steps),
+            usage_complete=extra.get("usage_complete"),
+            cost_is_partial=extra.get("cost_is_partial"),
+            known_cost_usd=extra.get("known_cost_usd"),
+            missing_generation_ids=extra.get("missing_generation_ids"),
+        )
+        if metrics is None:
+            return
+        context.n_input_tokens = metrics.total_prompt_tokens
+        context.n_output_tokens = metrics.total_completion_tokens
+        context.n_cache_tokens = metrics.total_cached_tokens
+        context.cost_usd = metrics.total_cost_usd
+
+    @staticmethod
+    def _record_trajectory_diagnostic(
+        context: AgentContext,
+        status: str,
+        **details: object,
+    ) -> None:
+        context.metadata = context.metadata or {}
+        context.metadata["trajectory"] = {
+            "format": "ATIF-v1.7",
+            "status": status,
+            **{key: value for key, value in details.items() if value is not None},
+        }

benchmarks/harbor/tests/test_adapter.py

@@ -1,10 +1,12 @@
 """Contract tests for the repository-local Harbor installed-agent adapter."""
 
 import asyncio
+import json
 from pathlib import Path
 from types import SimpleNamespace
 
 import pytest
+from harbor.models.agent.context import AgentContext
 
 from harbor_adapter import NanoPyCodeAgent
 
@@ -26,6 +28,32 @@ def make_adapter(tmp_path: Path, **kwargs) -> NanoPyCodeAgent:
     return NanoPyCodeAgent(logs_dir=tmp_path, **kwargs)
 
 
+def write_trajectory(tmp_path: Path, final_metrics: dict) -> None:
+    (tmp_path / "trajectory.json").write_text(
+        json.dumps(
+            {
+                "schema_version": "ATIF-v1.7",
+                "session_id": "run-1",
+                "trajectory_id": "run-1",
+                "agent": {
+                    "name": "nanoPyCodeAgent",
+                    "version": "0.8.0",
+                    "model_name": "test-model",
+                },
+                "steps": [
+                    {
+                        "step_id": 1,
+                        "source": "user",
+                        "message": "fix it",
+                    }
+                ],
+                "final_metrics": final_metrics,
+            }
+        ),
+        encoding="utf-8",
+    )
+
+
 def test_install_pins_the_requested_release_and_checks_its_version(tmp_path):
     adapter = make_adapter(tmp_path, version="0.8.0")
     environment = RecordingEnvironment()
@@ -67,6 +95,14 @@ def test_install_source_rejects_ambiguous_or_blank_pins(tmp_path):
         make_adapter(tmp_path, git_ref="  ")
 
 
+def test_install_source_rejects_a_revision_coerced_to_a_number(tmp_path):
+    with pytest.raises(
+        ValueError,
+        match="use a full commit SHA or quote an abbreviated revision",
+    ):
+        make_adapter(tmp_path, git_ref=float("inf"))
+
+
 def test_run_pipes_the_instruction_and_forwards_anthropic_configuration(tmp_path):
     instruction = 'fix "quoted" input; echo $TOKEN\nthen run the tests'
     adapter = make_adapter(
@@ -88,6 +124,7 @@ def test_run_pipes_the_instruction_and_forwards_anthropic_configuration(tmp_path
     assert instruction not in command
     assert 'printf "%s" "$harbor_nanopycodeagent_instruction_' in command
     assert "nanoPyCodeAgent --max-turns 20" in command
+    assert "--trajectory /logs/agent/trajectory.json" in command
     assert command.endswith("2>&1 | tee /logs/agent/nanopycodeagent.txt")
 
     run_env = run_call["env"]
@@ -123,3 +160,105 @@ def test_run_normalizes_harbor_provider_configuration_for_the_anthropic_sdk(
     assert run_env["ANTHROPIC_BASE_URL"] == "https://openrouter.example/api"
     assert run_env["ANTHROPIC_MODEL"] == "deepseek/deepseek-v4-flash-0731"
     assert "OPENROUTER_API_KEY" not in run_env
+
+
+def test_adapter_declares_atif_support_and_populates_complete_context(tmp_path):
+    adapter = make_adapter(tmp_path)
+    write_trajectory(
+        tmp_path,
+        {
+            "total_prompt_tokens": 120,
+            "total_completion_tokens": 30,
+            "total_cached_tokens": 20,
+            "total_cost_usd": 0.0042,
+            "total_steps": 1,
+        },
+    )
+    context = AgentContext()
+
+    adapter.populate_context_post_run(context)
+
+    assert adapter.SUPPORTS_ATIF is True
+    assert context.n_input_tokens == 120
+    assert context.n_output_tokens == 30
+    assert context.n_cache_tokens == 20
+    assert context.cost_usd == 0.0042
+    assert context.metadata == {
+        "trajectory": {
+            "format": "ATIF-v1.7",
+            "status": "complete",
+            "total_steps": 1,
+        }
+    }
+
+
+def test_partial_trajectory_preserves_known_values_and_completeness(tmp_path):
+    adapter = make_adapter(tmp_path)
+    write_trajectory(
+        tmp_path,
+        {
+            "total_steps": 1,
+            "extra": {
+                "usage_complete": False,
+                "known_cost_usd": 0.001,
+                "cost_is_partial": True,
+                "missing_generation_ids": ["generation-1"],
+            },
+        },
+    )
+    context = AgentContext()
+
+    adapter.populate_context_post_run(context)
+
+    assert context.n_input_tokens is None
+    assert context.n_output_tokens is None
+    assert context.n_cache_tokens is None
+    assert context.cost_usd is None
+    assert context.metadata == {
+        "trajectory": {
+            "format": "ATIF-v1.7",
+            "status": "partial",
+            "total_steps": 1,
+            "usage_complete": False,
+            "cost_is_partial": True,
+            "known_cost_usd": 0.001,
+            "missing_generation_ids": ["generation-1"],
+        }
+    }
+
+
+@pytest.mark.parametrize(
+    ("contents", "expected_status"),
+    [
+        (None, "missing"),
+        ("not JSON", "invalid"),
+        (
+            json.dumps(
+                {
+                    "schema_version": "ATIF-v1.6",
+                    "agent": {"name": "agent", "version": "1"},
+                    "steps": [
+                        {"step_id": 1, "source": "user", "message": "task"}
+                    ],
+                }
+            ),
+            "invalid",
+        ),
+    ],
+)
+def test_missing_or_invalid_trajectory_records_a_diagnostic(
+    tmp_path,
+    contents,
+    expected_status,
+):
+    adapter = make_adapter(tmp_path)
+    if contents is not None:
+        (tmp_path / "trajectory.json").write_text(contents, encoding="utf-8")
+    context = AgentContext()
+
+    adapter.populate_context_post_run(context)
+
+    diagnostic = context.metadata["trajectory"]
+    assert diagnostic["format"] == "ATIF-v1.7"
+    assert diagnostic["status"] == expected_status
+    assert ("error" in diagnostic) is (expected_status == "invalid")

docs/changelogs/0.8.x.md

@@ -42,6 +42,11 @@ All notable changes in the **0.8.x** release series are documented here.
   generation ID with bounded retries, support both API-key and bearer-token
   authentication, and retain per-attempt diagnostics when reconciliation
   remains incomplete.
+- Harbor ATIF-v1.7 collection and metric reporting. The adapter requests the
+  agent trajectory at Harbor's standard log path, validates and exposes it to
+  Harbor, populates step, token, cache-token, and cost totals without inventing
+  zeroes for unknown values, and records explicit diagnostics for partial,
+  missing, or invalid trajectories.
 
 ### Fixed
 - Declare `httpx` as a direct runtime dependency so clean and containerized