Skip to content

Expose response length on BenchmarkRun - #8

Open
amargupta0428 wants to merge 1 commit into
epoch-research:mainfrom
amargupta0428:feat/expose-response-length
Open

Expose response length on BenchmarkRun#8
amargupta0428 wants to merge 1 commit into
epoch-research:mainfrom
amargupta0428:feat/expose-response-length

Conversation

@amargupta0428

Copy link
Copy Markdown

Expose response length on BenchmarkRun

Closes #1

What and why

Computing model response length currently requires manually downloading and
parsing the eval logs referenced by BenchmarkRun.logs (typically on S3) with
Inspect's read_eval_log. This PR adds two methods to BenchmarkRun so the
client library handles that directly:

  • BenchmarkRun.read_eval_log(**kwargs) — fetches and parses the run's eval
    log from the URL in the logs field using Inspect's read_eval_log. Extra
    keyword arguments are passed through.
  • BenchmarkRun.response_lengths() — returns the response length, in output
    tokens, for each sample in the run. Samples without recorded token usage
    (e.g. failed samples) count as 0.

inspect-ai is imported lazily, so it remains an optional dependency: it is
only required when these methods are called, and a clear error message is
raised if it is missing. Existing users who don't need eval logs are
unaffected.

Usage

from epochai.airtable.models import BenchmarkRun

runs = BenchmarkRun.all(memoize=True)

for run in runs:
    lengths = run.response_lengths()
    print(run.inspect_run_id, sum(lengths) / len(lengths) if lengths else None)

Implementation notes

  • inspect_ai.log.read_eval_log resolves the logs URL itself, including
    s3:// paths (S3 credentials must be configured as usual for Inspect).
  • No changes to existing fields or models; no new required dependencies.

Tests

Added tests/test_benchmark_run.py (plus tests/conftest.py providing dummy
Airtable env vars). Tests stub out the inspect_ai.log module, so the suite
runs fully offline:

.venv/bin/python -m pytest tests/ -v
# 4 passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose response length in API

1 participant