Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
a666624
cleanup(sprint3): remove dead code, redundant tests, memoir comments,…
maltsev-dev Aug 11, 2026
d065ade
cleanup(sprint4): trim VCS bloat - Dockerfile fix + drop orphans + ti…
maltsev-dev Aug 11, 2026
ea77e21
cleanup(sprint5): trim long docstrings/memoirs + scrub Cyrillic from …
maltsev-dev Aug 11, 2026
7877aa4
cleanup(sprint5): dedupe sync/async wrappers + dead code in src/nullrun
maltsev-dev Aug 11, 2026
93f5586
cleanup(sprint5): CHANGELOG order + Makefile CI parity + error-code docs
maltsev-dev Aug 11, 2026
a491923
chore(release): 0.14.10 — Sprint 5 internal cleanup
maltsev-dev Aug 11, 2026
370d5f5
fix(sdk): route /auth/verify non-200 through canonical envelope parse…
maltsev-dev Aug 11, 2026
83ca75d
Revert "fix(sdk): route /auth/verify non-200 through canonical envelo…
maltsev-dev Aug 11, 2026
700b0af
Revert "cleanup(sprint5): trim long docstrings/memoirs + scrub Cyrill…
maltsev-dev Aug 11, 2026
2df6b3a
fix(sdk): restore branch-coverage tests deleted by sprint3 cleanup (a…
maltsev-dev Aug 11, 2026
d7e6866
chore(release): 0.14.11 — partial revert of sprint-5 cleanup
maltsev-dev Aug 11, 2026
f6aaca7
feat(sdk): ADR-009 P1 governance audit read surface (0.15.0)
maltsev-dev Aug 12, 2026
c6cf99c
chore(release): 0.15.0 — ADR-009 P1 governance audit read surface
maltsev-dev Aug 12, 2026
7f92b82
fix(sdk): defer runtime.py annotations to avoid AuditProxy.list shado…
maltsev-dev Aug 12, 2026
16d9daf
fix(sdk): ruff I001 + UP037 cleanup after adding __future__ annotations
maltsev-dev Aug 12, 2026
d8fc265
fix(sdk): mypy valid-type + arg-type cleanups in audit/runtime
maltsev-dev Aug 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,551 changes: 211 additions & 1,340 deletions CHANGELOG.md

Large diffs are not rendered by default.

11 changes: 4 additions & 7 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build stage for Python SDK
FROM python:3.11-slim as builder
FROM python:3.11-slim AS builder

WORKDIR /app

Expand Down Expand Up @@ -31,10 +31,7 @@ RUN pip install /app/dist/*.whl --force-reinstall
RUN useradd -m -u 1000 nullrun
USER nullrun

# Install optional dependencies
# Sprint 1.3 (B9): the previous `nullrun-breaker[langgraph]` package
# does not exist in `pyproject.toml` (only `nullrun[langgraph]`).
# Installing the non-existent package would make `docker build` fail.
# Install optional dependencies.
# `nullrun[langgraph]` is the canonical extras name — the previous
# `nullrun-breaker[langgraph]` package does not exist in pyproject.toml.
RUN pip install "nullrun[langgraph]"

ENTRYPOINT ["python", "-m", "nullrun.breaker"]
17 changes: 0 additions & 17 deletions Dockerfile.dev

This file was deleted.

23 changes: 14 additions & 9 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
.PHONY: install test lint type-check coverage clean build publish-test publish
.PHONY: install test lint type-check coverage clean build publish-test publish smoke-test

# ── Setup ─────────────────────────────────────────────────────
install:
Expand All @@ -12,18 +12,26 @@ install:
# with ``No such file or directory``. Re-introduce it ONLY
# when gRPC is unblocked (see README §"gRPC transport").

# Sprint 5: the ``run-example`` target was removed. The
# ``examples/`` directory was deleted along with the gRPC
# transport in 0.3.1, and the target referenced the now-missing
# ``examples/basic.py``. Local smoke-testing uses ``smoke-test``
# below instead.

# ── Tests ─────────────────────────────────────────────────────
test:
pytest tests/ -v

test-watch:
pytest tests/ -v --tb=short -f

# Sprint 5: align with CI (.github/workflows/ci.yml:82).
# ``coverage run -m pytest`` only traced the xdist coordinator,
# so every parallel run uploaded 0 hits. pytest-cov starts coverage
# in every worker and combines the data before producing the XML.
coverage:
coverage run -m pytest tests/
coverage report
coverage html
@echo "HTML report: htmlcov/index.html"
pytest tests/ --cov=src/nullrun --cov-branch --cov-report=xml:coverage.xml --cov-report=term
@echo "XML report: coverage.xml"

# ── Code quality ──────────────────────────────────────────────
lint:
Expand All @@ -41,7 +49,7 @@ check: lint type-check test

# ── Build & Publish ───────────────────────────────────────────
clean:
rm -rf dist/ build/ *.egg-info htmlcov/ .coverage
rm -rf dist/ build/ *.egg-info htmlcov/ .coverage coverage.xml

build: clean
pip install build
Expand All @@ -56,9 +64,6 @@ publish: build
twine upload dist/*

# ── Dev helpers ───────────────────────────────────────────────
run-example:
python examples/basic.py

smoke-test: build
pip install dist/*.whl --force-reinstall
python -c "from nullrun import protect; print('OK')"
93 changes: 66 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,14 @@
<div align="center">

<!-- HERO -->
<img src="https://raw.githubusercontent.com/nullrunio/nullrun-sdk-python/master/docs/assets/banner.svg" alt="NullRun — Runtime decision layer for AI agents" width="100%"/>

# NullRun

**Ship AI agents with real-time budget, policy, and human-approval gates.**

Zero-refactor cost control, tool policy enforcement, and audit trail for any
LLM-powered agent works with OpenAI, Anthropic, LangGraph, CrewAI, AutoGen,
LLM-powered agent - works with OpenAI, Anthropic, LangGraph, CrewAI, AutoGen,
LlamaIndex, and your own stack.

[Quickstart](#-quickstart) · [Docs](https://docs.nullrun.io) · [Examples](https://github.com/nullrunio/nullrun-examples)
[Quickstart](https://docs.nullrun.io/getting-started/onboarding/) · [Docs](https://docs.nullrun.io) · [Examples](https://github.com/nullrunio/nullrun-examples)

<!-- BADGES: project health -->
<div align="center">
Expand Down Expand Up @@ -40,7 +37,8 @@ LlamaIndex, and your own stack.

---

> ⚠️ **Status: alpha (v0.14.7, protocol v3.31.6).** The public API may shift between minor versions. Pin your dependency and read the [CHANGELOG](https://github.com/nullrunio/nullrun-sdk-python/blob/master/CHANGELOG.md) before upgrading.
> ⚠️ **Status: alpha (v0.15.0).** The public API may shift between minor versions.
> Pin your dependency and read the [CHANGELOG](https://github.com/nullrunio/nullrun-sdk-python/blob/master/CHANGELOG.md) before upgrading.

---

Expand All @@ -63,11 +61,11 @@ Existing observability tools tell you **after** the fact. NullRun enforces **bef

| | |
|---|---|
| **Hard & soft budget gates** — atomic Redis-enforced, no client-trust model | **Tool policy enforcement** — block dangerous tools before execution |
| **Hard & soft budget gates** — atomic Redis-enforced | **Tool policy enforcement** — block dangerous tools before execution |
| **Human-in-the-loop approvals** — pause agent and await `approval_resolved` via WS push | **Immutable audit trail** — every decision, every tool call, every cent |
| **Zero-code instrumentation** — `nullrun.init()` patches `httpx` once for any vendor | **LangGraph, CrewAI, AutoGen, LlamaIndex** — first-class integrations |
| **Memory-safe streaming** — 16 MiB response body cap (anti-OOM); full body for usage extraction | **Lightweight** — no LLM-key storage, no proxy required |
| **Server-authoritative cost** — wire protocol v3.31, server-minted execution IDs | **MCP support** — expose tools to agents via Model Context Protocol |
| **Memory-safe streaming** — 16 MiB response body; full body for usage extraction | **Lightweight** — no LLM-key storage, no proxy required |
| **Server-authoritative cost** — server-minted execution IDs | **MCP support** — expose tools to agents via Model Context Protocol |

---

Expand Down Expand Up @@ -218,7 +216,7 @@ def my_agent(prompt: str) -> str:

| | **NullRun** | LangChain callbacks | Helicone | Portkey | OpenLLMetry |
|---|---|---|---|---|---|
| **Enforce before execution** | ✅ | ❌ observe-only | ⚠️ async | ⚠️ async | ❌ |
| **Enforce before execution** | ✅ | ❌ | ⚠️ async | ⚠️ async | ❌ |
| **Server-authoritative budget** | ✅ | ❌ | ❌ | ❌ | ❌ |
| **Tool-call policy** | ✅ | ❌ | ❌ | ⚠️ limited | ❌ |
| **Human-in-the-loop approvals** | ✅ | ❌ | ❌ | ❌ | ❌ |
Expand All @@ -230,31 +228,76 @@ def my_agent(prompt: str) -> str:
> NullRun is the only option that **blocks** expensive or dangerous calls *before* they happen, not just observes them.


---

## Querying the audit log

Every gate decision, approval resolution, and execution lifecycle event
is written to the org's hash-chained `audit_events` table on the backend.
The SDK surfaces a typed read API at `runtime.audit.*` so backends on
ADR-009 (`schema_version = 3`) return typed dataclasses — not raw dicts.

```python
from nullrun import NullRunRuntime, AuditQuery
from datetime import datetime, timezone, timedelta

runtime = NullRunRuntime(api_key="nr_...")

# 1) Last 50 governance decisions in the last 24h.
since = (datetime.now(timezone.utc) - timedelta(hours=24)).isoformat()
page = runtime.audit.list(
AuditQuery(event_type="authorization_decision", since=since, limit=50)
)
for entry in page.entries:
print(entry.timestamp, entry.decision, entry.tool_name, entry.reason_code)
```

Available surfaces:

| Method | Returns | Endpoint |
|---|---|---|
| `runtime.audit.list(query=...)` | `AuditLogPage` (entries + meta) | `GET /api/v1/orgs/{org}/audit-log` |
| `runtime.audit.verify(since=...)` | `AuditVerifyResult` (chain head/tail/reason) | `GET /api/v1/orgs/{org}/audit-log/verify` |
| `runtime.audit.list_exports()` | `list[AuditExportJob]` | `GET /api/v1/orgs/{org}/audit-log/export` |
| `runtime.audit.create_export()` | `dict` (`job_id`, `status`) | `POST /api/v1/orgs/{org}/audit-log/export` |
| `runtime.audit.export_status(job_id)` | `AuditExportStatus` | `GET /api/v1/orgs/{org}/audit-log/export/{job_id}/status` |

`AuditQuery` filters on the canonical ADR-009 columns: `event_type`
(`authorization_decision` / `approval_decision` / `execution_lifecycle`),
`decision`, `policy_id`, `execution_id`, `actor`, `since`, `until`, `limit`.
Pre-ADR-009 backends return legacy fields only — `AuditEntry.is_governance`
is `False` for those rows, and the 13 governance columns default to `None`.

If you call `runtime.audit.*` before `nullrun.init()` (no org binding),
the proxy raises `NullRunAuthenticationError` — not a silent 404 — so a
misconfigured CI step fails loudly at the audit call site rather than
silently dropping the query.

---

## Examples

Runnable, copy-pastable examples live in a separate repo so you can adapt without cloning the SDK source:

- **LangGraph** — multi-node agent with budget + approval [→](https://github.com/nullrunio/nullrun-examples/tree/main/langgraph)
- **CrewAI** — multi-agent crew with shared budget [→](https://github.com/nullrunio/nullrun-examples/tree/main/crewai)
- **AutoGen** — group-chat agent with policy gating [→](https://github.com/nullrunio/nullrun-examples/tree/main/autogen)
- **LlamaIndex** — RAG pipeline with cost-per-query enforcement [→](https://github.com/nullrunio/nullrun-examples/tree/main/llama-index)
- **Custom tools** — register your own tools for policy [→](https://github.com/nullrunio/nullrun-examples/tree/main/custom-tools)
- **Multi-agent** — shared budget across sub-agents [→](https://github.com/nullrunio/nullrun-examples/tree/main/multi-agent)
- **[LangGraph](https://docs.nullrun.io/how-to/langgraph/)** — multi-node agent with budget + approval
- **[CrewAI](https://docs.nullrun.io/how-to/crewai/)** — multi-agent crew with shared budget
- **[AutoGen](https://docs.nullrun.io/how-to/autogen/)** — group-chat agent with policy gating
- **[LlamaIndex](https://docs.nullrun.io/how-to/llama-index/)** — RAG pipeline with cost-per-query enforcement
- **[Custom tools](https://docs.nullrun.io/how-to/fastapi/)** — register your own tools for policy
- **[Multi-agent](https://docs.nullrun.io/how-to/multi-agent/)** — shared budget across sub-agents

---

## Roadmap

| Version | Status | Highlights |
|---|---|---|
| **v0.14.x** (current) | ✅ alpha | Wire protocol v3.31, server-minted execution IDs, MCP, anti-OOM streaming cap |
| **v0.15** | 🚧 in progress | OpenTelemetry exporter, Redis-backed offline queue, hardened init contract |
| **v0.16** | 📋 planned | Cost prediction from prompt, semantic tool policy (regex → AST) |
| **v0.14.x** | ✅ alpha | Wire protocol v3.31, server-minted execution IDs, MCP, anti-OOM streaming cap |
| **v0.15** (current) | ✅ alpha | ADR-009 governance audit surface, typed `runtime.audit.*`, capability probes for `/audit-log/verify` |
| **v0.16** | 📋 planned | OpenTelemetry exporter, Redis-backed offline queue, hardened init contract |
| **v1.0** | 🎯 beta target | Stable wire contract, full async support, type-safe decisions |

[Full roadmap & RFCs →](https://docs.nullrun.io/roadmap)
[Full roadmap & RFCs →](https://nullrun.io/roadmap)

---

Expand All @@ -277,21 +320,17 @@ require tests for new public API, and run `ruff` + `mypy` in CI.

NullRun does **not** store or proxy your LLM provider keys — it sits beside your existing clients and observes the calls. The gate is **server-authoritative** for cost: even a malicious SDK cannot inflate spend by sending a fake `cost_cents` to `/track`.

See the security policy at <https://github.com/nullrunio/nullrun-sdk-python/security/policy> for the threat model and disclosure policy.

To report a vulnerability: **support@nullrun.io**.
See the security [policy](https://github.com/nullrunio/nullrun-sdk-python/security/policy) for the threat model and disclosure policy.

---

## Community & support

- **GitHub Issues**: <https://github.com/nullrunio/nullrun-sdk-python/issues>
- **GitHub Discussions**: <https://github.com/nullrunio/nullrun-sdk-python/discussions>
- **Enterprise support**: support@nullrun.io
- [GitHub Issues](https://github.com/nullrunio/nullrun-sdk-python/issues)
- [Support](support@nullrun.io)

---

---

<div align="center">

Expand Down
230 changes: 0 additions & 230 deletions docs/assets/banner.svg

This file was deleted.

70 changes: 70 additions & 0 deletions docs/errors/NR-A004.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
# NR-A004 — Approval flow anomaly

| Field | Value |
|---|---|
| **Code** | `NR-A004` |
| **Category** | Authentication (approval subsystem) |
| **Exception class** | `NullRunBlockedException` |
| **Retryable** | No |
| **Default `user_action`** | "Approval flow is in an invalid state: the server returned no approval_id, the operator denied the request, the approval timed out, or the approved action was rejected on re-check. Re-arm the workflow and contact support if it persists." |

## When

Raised by `runtime.execute()` when the approval-required gate fails for one
of these reasons:

1. **No `approval_id` in the `require_approval` response.** The backend
asked for approval but did not mint an id — should never happen on a
healthy backend.
2. **Operator denied the request.** The dashboard showed the prompt and
the operator clicked Deny.
3. **Approval timed out** before the operator responded. The default
timeout is the per-execution `approval_timeout_seconds` (300s unless
overridden).
4. **Approved action was not accepted on re-check.** The SDK re-runs
`/execute` with the `approval_id` and the backend returns
`require_approval` again — typically a stale approval or a context
drift between approve and re-check.

## Common causes

- **Operator denied the request** — verify with the approver.
- **Approval timeout exceeded** — raise `approval_timeout_seconds` in
the workflow policy or speed up the approval path.
- **Workflow was killed between approve and execute** — restore the
workflow before retrying.
- **Stale `approval_id`** — re-submit the original tool call (the
approval_id is single-use and bound to the original action digest).

## How to fix

1. Inspect `exc.tool_name` and `exc.workflow_id` to identify the call.
2. If the operator denied, redesign the call or have the operator
pre-approve the policy.
3. If it was a timeout, raise the timeout in the policy or shorten the
prompt → approval loop.
4. If the issue is on a healthy workflow with no operator action,
capture `exc` and open a support ticket with `error_code` and the
full request id from the gate response.

## Catch pattern

```python
from nullrun.breaker.exceptions import NullRunBlockedException

try:
runtime.execute(tool_name="charge_card", ...)
except NullRunBlockedException as exc:
if exc.error_code == "NR-A004":
# Surface to the operator: "Approval was denied or timed out."
log.warning("approval failed", extra={"tool": exc.tool_name, "reason": exc.reason})
return render_approval_failed_page(exc.reason)
raise
```

## Related codes

- `NR-A001` — `/auth/verify` returned non-200.
- `NR-A002` — `/auth/verify` response missing `organization_id`.
- `NR-W002` — workflow killed by control plane.
- `NR-W003` — workflow paused.
66 changes: 66 additions & 0 deletions docs/errors/NR-B003.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# NR-B003 — Sensitive-tool impact extractor failed

| Field | Value |
|---|---|
| **Code** | `NR-B003` |
| **Category** | Backend (tool extraction) |
| **Exception class** | `NullRunBlockedException` |
| **Retryable** | No |
| **Default `user_action`** | "The @sensitive decorator could not extract a business_impact envelope for this tool. Pass an explicit `impact=` argument to `@sensitive(...)` (see ToolParameters rules) or add a `ToolParamsExtractor` to the function. The default `include_all=True` extractor failed because the function signature is not introspectable (e.g. wrapped in C code or a non-Python callable)." |

## When

Raised when `@sensitive` cannot derive a `BusinessImpact` envelope for
the wrapped function. The decorator runs four extractors in priority
order:

1. Explicit `impact=` argument to `@sensitive(...)`.
2. Pre-registered `ToolParamsExtractor` on the function.
3. Per-function fallback (`include_all=True`).
4. Constant-extractor fallback (legacy).

If all four fail, the decorator blocks the call (fail-CLOSED) with
`NR-B003` so a missing impact never silently widens to a different
policy.

## Common causes

- **Wrapped in a non-introspectable callable** — `functools.partial`,
a `ctypes` function, or anything that hides its signature.
- **Decorator chain obscures the function** — a third-party decorator
replaced `__wrapped__` with something that lacks `__signature__`.
- **Custom `ToolParamsExtractor` raised** — your extractor has a bug;
the chain fails fast rather than passing the call through.

## How to fix

1. **Prefer the explicit form**: `@sensitive(impact=tool_params({...}))`
for static schemas, or `@sensitive(impact=BusinessImpact.tool_call(...))`
for runtime-built envelopes.
2. **Wrap before `@sensitive`** — place `@sensitive` as the OUTERMOST
decorator so it sees the un-wrapped signature.
3. **Provide a custom extractor** — `ToolParamsExtractor` with a
`params_for(func, args, kwargs)` method that returns a fixed
`ToolCallParams`.

## Catch pattern

```python
from nullrun.breaker.exceptions import NullRunBlockedException

try:
@sensitive
def my_op(x):
return do_it(x)
except NullRunBlockedException as exc:
if exc.error_code == "NR-B003":
# Surface the misconfiguration early — at decorator application
# time, not on the first call.
log.error("sensitive missing impact: %s", exc.user_action)
```

## Related codes

- `NR-B001` — network error during transport.
- `NR-B002` — 5xx from the NullRun backend.
- `NR-W002` — workflow killed.
Loading
Loading