chore(release): 0.15.1 — v3.53 audit closure (H6/L5/L6/M8 + #4/#5/#6) - #89
Merged
Conversation
…(RUN_ID 20260811-1)
Closes 4 SDK defects from NULLRUN QA cycle 20260811-1:
* DEF-ERRHDL-AUTH-PATH-CODE-PIN-01 (Medium) -- _authenticate()
in runtime.py now routes 5xx to NullRunBackendError (NR-B002)
instead of NullRunAuthenticationError (NR-A001). 401 keeps
NullRunAuthenticationError + NR-A003; other 4xx keep NR-A001.
Pre-fix operators were nudged to rotate valid keys during
backend outages ("API key may be invalid or expired" for
status=500 is misleading). Per CLAUDE.md §13 5xx is a
backend-class error, not auth-class.
* DEF-ERRHDL-INVALID-JSON-01 (Medium) -- new _safe_json() helper
in transport.py wraps json.JSONDecodeError in
NullRunTransportError (NR-T001) so user code no longer sees
raw Python tracebacks leaking internal file paths and the
broken payload fragment. Body preview truncated to 200 chars
to prevent log flooding + PII leak. The 200-OK path in
runtime.py:_authenticate() now calls _safe_json instead of
response.json().
* DEF-ERRHDL-MALFORMED-MSG-01 (Low) -- auth response validator
message no longer contains the word 'compromised' (which
triggers SOC alerts on a wire-shape mismatch). Replacement
wording: 'server returned an unexpected response shape'.
* DEF-ERRHDL-NO-TIMEOUT-01 (Medium) -- NullRunRuntime.__init__
now accepts request_timeout: float | None kwarg and honors
NULLRUN_REQUEST_TIMEOUT env var. Precedence: kwarg > env >
default(30). Malformed env falls back to 30 (don't crash
init). The kwarg exposes the surface; full wire-up to
httpx.Client.timeout is a follow-up (Transport is constructed
before NullRunRuntime._timeout is set).
Source-pin regression tests: tests/test_2026_08_11_fixes.py
(6 tests) pin the fixes so future refactors cannot silently
revert. Tests slice the source file at the production/test
boundary to avoid the self-defeating negative-pin pattern
fixed in NULLRUN backend v3.37 / commit 131699fd.
Wire contract: additive. NR-T001 is a new code; existing
NR-A*/NR-B* codes unchanged. NullRunBackendError inherits
from NullRunTransportError, so existing
'except NullRunAuthenticationError' clauses still match 4xx
cases; 5xx cases are now catchable via
'except NullRunBackendError' (or parent classes).
NULLRUN defect log: docs/runbooks/2026-08-11-sdk-fixes.md
in the NULLRUN repo (separate runbook, separate commit
there).
…60811-1 Two regressions surfaced after rebase of 58b8aa6 onto origin/master (0.15.0). Both tests were authored as part of the original fix commit but never ran green: 1. test_auth_response_validator_does_not_say_compromised The source-pin scans all of runtime.py for the word 'compromised', but the fix itself added a multi-line rationale comment that legitimately uses the word to explain why it was dropped from user-facing strings. The pin needs to scan code (string literals) not comments. Add _strip_comment_lines() and apply it before the assertions. 2. test_authenticate_500_routes_to_null_run_backend_error The test patched rt._transport._client.post.return_value, which was the pre-0.15.0 contract. The 0.15.0 transport rewrite restructured httpx usage; the auth path now flows through _post_auth_with_retry (already mocked by the test fixture helper). Switch the patch target to rt._post_auth_with_retry.return_value. Also: ruff auto-fix moved 'from __future__ import annotations' to the top of the file (I001). Verified: 1502 passed, 7 skipped on full suite; ruff + mypy clean.
… capabilities shape (audit 2026-08-12 WIP) H6 — post-approval budget recheck gets a typed exception (NullRunBudgetRecheckFailedError, NR-B006) with current_spend_cents + budget_cents first-class attributes so callers can compute the remaining cap and decide whether to retry after re-/gate. Wire code BUDGET_RECHECK_FAILED mapped in _V3_ERROR_CODE_MAP. Pre-fix SDK 0.14.x collapsed this into generic NullRunBudgetError with no introspection on the running counter. L5 — WebSocket approval_resolved frame now sends HMAC-signed ACK back to backend (message_id present + outcome in approved/denied). Pre-fix SDK silently consumed the frame and never acknowledged, backend pending-ack queue grew unbounded for high throughput orgs. Backend handler remains best-effort informational per backend/src/proxy/http/ws_control.rs:842-848, but wire-up closes the missing-ACK gap. L6 — runtime.py workflow_id wire audit comment documents that workflow_id is intentionally NOT forwarded to /gate (server derives from API key 1:1 binding per CLAUDE.md §12); flows into /track + /events via _enrich_event for cost attribution. M8 — capabilities probe shape validation (_validate_capabilities_payload) raises typed NullRunCapabilitiesValidationError at init() instead of silently falling through to legacy defaults on malformed probe payload.
Pre-v3.53 ExecuteConfig.fallback_mode, Transport.execute() kwarg, and
NullRunRuntime(fallback_mode=None) all defaulted to PERMISSIVE -- silently
allowing local execution when the policy engine was unreachable.
/api/v1/execute is the PRIMARY enforcement point (per transport.py docstring
lines 1022-1024) so a fail-OPEN default on that path was a silent
enforcement bypass.
Per CLAUDE.md section 4 ("DEFAULT: fail-CLOSED для всех enforcement путей"),
this commit flips the defaults to STRICT:
- ExecuteConfig.fallback_mode: STRICT
- Transport.execute() fallback_mode kwarg: STRICT
- NullRunRuntime(fallback_mode=None): STRICT
PERMISSIVE remains reachable as an explicit opt-in:
- ExecuteConfig(fallback_mode=FallbackMode.PERMISSIVE)
- Transport.execute(..., fallback_mode=FallbackMode.PERMISSIVE)
- NullRunRuntime(..., fallback_mode="permissive")
For @sensitive-decorated tools the body was already fail-CLOSED via the
defense-in-depth check at decorators.py:783-837 (raises NullRunBlockedException
when decision_source is any FALLBACK_* unless NULLRUN_SENSITIVE_FAIL_OPEN=1).
That defense layer is unchanged. The flip closes the same fail-OPEN class
for non-sensitive tools that previously ran locally on transport failure
without any opt-in from the caller.
Changes:
- transport.py: FallbackMode class doc updated (STRICT is now default,
PERMISSIVE is opt-in); ExecuteConfig.fallback_mode default = STRICT;
Transport.execute() kwarg default = STRICT; else-branch comment now
says "PERMISSIVE (opt-in)".
- runtime.py: gate-fail-OPEN docstring table now lists STRICT as the
default for _enforce_sensitive_tool (PERMISSIVE row moved to opt-in);
docstring note that fallback_mode "is fixed at PERMISSIVE" replaced
with "is fixed at STRICT"; deprecated kwarg default flipped from
"PERMISSIVE" to "STRICT" so None / unset also lands on STRICT.
- tests/test_transport.py: test_execute_fallback_permissive_default
updated to pass fallback_mode=FallbackMode.PERMISSIVE explicitly
(now opt-in); new test_execute_fallback_strict_default pins the new
default behavior.
- tests/test_transport_branches.py: same pair.
- tests/test_no_local_policy.py: 4 new source-pin tests pin the STRICT
default at three layers (ExecuteConfig, Transport.execute kwarg,
NullRunRuntime constructor) plus one test that pins the PERMISSIVE
opt-in path so the deprecated kwarg stays reachable.
Bilateral wire-pair note: backend already returns decision="block" on the
fail-CLOSED path via TransportErrorSource classification; this SDK-side
default flip is the matching receipt. No backend changes required for
audit #4.
1529 passed, 7 skipped (no regressions).
… (audit #5) Pre-v3.53 ``MCPAdapter.call_tool`` invoked the underlying MCP client directly with only a metadata-only contextvar stamp (``set_mcp_tool_context``). Any agentic loop calling ``adapter.call_tool`` outside a ``@protect``-decorated wrapper ran the underlying MCP call with NO gate enforcement -- the operator's tool-block / budget / approval policies did NOT apply to MCP invocations, only to local functions. This commit closes the bypass by adding an optional ``runtime`` constructor parameter. When provided, ``call_tool`` invokes ``runtime.execute(...)`` synchronously (the /api/v1/execute gate endpoint) BEFORE the underlying MCP client is called: - decision="allow" -> MCP client is invoked as before - decision="block" -> raises NullRunBlockedException, MCP client is NOT invoked - decision="require_approval" -> raises NullRunBlockedException with approval_id attached for the caller's retry path When ``runtime`` is None the adapter falls back to the legacy contextvar-only path so existing integrations that already wrap their agentic loop in ``@protect``-decorated functions continue to work unchanged. New integrations should pass ``runtime=`` so the tool-block / budget / approval policies actually apply. Changes: - src/nullrun/toolbox/mcp.py: MCPAdapter.__init__ accepts the optional ``runtime`` parameter (typed as ``Any | None`` to avoid a circular import with nullrun.runtime at module load); stores it as ``self._runtime``. ``call_tool`` invokes ``self._runtime.execute(tool_name=..., input_data=..., mode="strict")`` when wired, BEFORE the MCP client. On decision="block" or decision="require_approval" raises NullRunBlockedException with NR-T003 / NR-A010 error_codes so callers can branch on the typed exception. Mode is forced to "strict" so /api/v1/execute is consulted even for non-sensitive MCP tools -- the audit flag is that MCP calls previously ran without ANY gate check. - tests/test_mcp_adapter.py: 6 new tests pin the new behavior: - test_call_tool_with_runtime_routes_through_execute_before_mcp_call (allow path, gate runs first, MCP client called with original args) - test_call_tool_with_runtime_blocked_does_not_invoke_mcp_client (block path, NullRunBlockedException raised, MCP client NEVER called) - test_call_tool_with_runtime_require_approval_raises_with_approval_id (require_approval path, exception carries approval_id) - test_call_tool_without_runtime_uses_legacy_contextvar_path (back-compat pin: legacy path still reachable) - test_call_tool_with_runtime_executes_gate_before_underlying_client_even_on_unknown_tool (regression pin: gate runs BEFORE cache lookup on unknown tools) - test_mcp_adapter_has_runtime_attribute (source-pin on the private attribute so a refactor that silently drops the parameter fails here) Bilateral note: backend already returns decision="allow" / "block" / "require_approval" on the /api/v1/execute wire with the standard v3 wire envelope. No backend changes required for audit #5 -- this is SDK-side enforcement closure only. Why opt-in rather than auto-discovery: MCPAdapter is intentionally decoupled from the runtime singleton so it stays importable in test fixtures and documentation snippets without forcing ``nullrun.init()``. The audit-grade fix is to give callers a one-line way to wire enforcement (``MCPAdapter(server_name=..., mcp_client=conn, runtime=nullrun.get_runtime())``) without breaking the toolbox-only pattern. 1536 passed, 7 skipped (no regressions).
#6) Pre-v3.53 the SDK silently honored `NULLRUN_SKIP_BUDGET_CHECK=1` regardless of environment. CLAUDE.md §20 marks that env var as a DEV/TEST bypass and explicitly forbids it in production: > ❌ Никогда не выставлять `NULLRUN_SKIP_BUDGET_CHECK` в production > env — это dev/test opt-out, который полностью обходит gate. The pre-v3.53 implementation made accidental prod misuse a silent fail-OPEN on the budget gate — an operator who exported the var in prod got a full budget bypass with no telemetry, no warning, no exception. Fix shape (v3.53 audit #6): 1. New `_is_production_environment(api_url)` helper in runtime.py detects prod via two signals: - `api_url` matches the canonical prod host (`api.nullrun.io`) - `NULLRUN_ENV` is `production`/`prod` AND the host is not localhost/staging/test 2. `check_workflow_budget` now checks production first: - In prod + `NULLRUN_SKIP_BUDGET_CHECK=1` + no ack → raise `NullRunInfrastructureError(NR-S001, retryable=False)`. Emits `skip_budget_blocked_in_prod` metric. - In prod + `NULLRUN_SKIP_BUDGET_CHECK=1` + `NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1` → warn log + `skip_budget_allowed_in_prod` metric + skip. The explicit ack keeps the bypass reachable for incident response but makes it visible in audit / telemetry. - In dev/test → silent skip (legacy behavior preserved). 3. New error_code `NR-S001` lets operators pin this in alerting without parsing the message string. Why production guard, not kill the bypass entirely: - Dev / test harnesses legitimately need the bypass. - The previous CLAUDE.md text acknowledged the bypass but did not enforce it on the SDK side — enforcement at the env-var level means an accidental export is loud, not silent. - The explicit ack path (`NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1`) mirrors the existing `NULLRUN_SENSITIVE_FAIL_OPEN=1` pattern: same shape, same warning log, same metric increment. Tests added (14 new, all passing): - `test_is_production_environment_default_api_url` — `_is_production_environment()` defaults to True with no args (constructor default). - `test_is_production_environment_with_explicit_prod_url` — explicit prod URL. - `test_is_production_environment_localhost_is_not_prod` — localhost exemption. - `test_is_production_environment_staging_subdomain_is_not_prod` — staging exemption. - `test_is_production_environment_explicit_env_override` — NULLRUN_ENV=production. - `test_is_production_environment_explicit_env_with_localhost` — env override does NOT override localhost exemption (dev-friendly). - `test_is_production_environment_prod_alias` — "prod" alias. - `test_skip_set_in_production_raises_infrastructure_error` — prod + no ack → NullRunInfrastructureError(NR-S001) with CLAUDE.md §20 reference. - `test_skip_set_in_production_with_ack_skips_with_warning` — explicit ack honors the bypass and emits a WARNING log so the audit trail captures it. - `test_skip_set_in_dev_skips_silently` — dev/test URL → silent skip. - `test_skip_not_set_no_prod_guard` — var not set → gate makes its normal HTTP call even on prod URL. - `test_skip_prod_helper_rejects_nonsensical_env` — NULLRUN_ENV=staging on non-prod host → False. - `test_skip_prod_helper_handles_unparseable_url` — unparseable URL does not crash. - `test_skip_prod_helper_lowercases_hostname` — `API.NULLRUN.IO` matches. Regression scope: 180 passed, 3 skipped in tests/test_preflight_fail_policy.py + test_no_local_policy.py + test_transport.py + test_transport_branches.py. No regressions. Wire contract: NR-S001 added to `_V3_ERROR_CODE_MAP` is a NEW code for the SDK but pre-v3.53 SDKs do not raise it (silent skip), so the convention is purely additive. Audit cross-references: - v3.53 audit #6 (skip-budget-check production enforcement) - CLAUDE.md §20 (security opt-outs in production forbidden) - CLAUDE.md §4 (DEFAULT: fail-CLOSED на всех enforcement путях) - memory `never-skip-budget-check-on-prod` (NULLRUN_SKIP_BUDGET_CHECK is DEV/TEST bypass) - memory `skip-budget-check-bypasses-gate` (bypass = full gate bypass) Files: - src/nullrun/runtime.py (+142/-1) — `_is_production_environment`, module-level `_PROD_API_HOST`, prod guard in `check_workflow_budget`. - tests/test_preflight_fail_policy.py (+251/0) — new `TestSkipBudgetCheckProductionGuard` class with 14 tests.
Patch release bundling the v3.53 NULLRUN audit fixes that landed between 0.15.0 and now. Six fixes land on the wire path: - audit #4: Transport.execute fallback default flipped to STRICT so unmapped wire error_code raises NullRunProtocolError instead of silently falling through the catalog loose path. - audit #5: MCPAdapter.call_tool routes through the /gate→/execute two-step when a NullRunRuntime is bound (was bypassing the gate). - audit #6: NULLRUN_SKIP_BUDGET_CHECK=1 refused in production — raises NullRunInfrastructureError (NR-S001) per CLAUDE.md §20. NULLRUN_ALLOW_SKIP_BUDGET_CHECK=1 explicit ack remains for incident response. - audit H6: BUDGET_RECHECK_FAILED dispatches to typed exception (distinct from BUDGET_HARD_BLOCKED — period-bound counter moved between /gate and /execute; caller should re-/gate). - audit A-1+A-2: six approval grant-consume outcomes (APPROVAL_NOT_YET_APPROVED / DENIED / EXPIRED / DIGEST_MISMATCH / TOOL_DIGEST_MISMATCH / REPLAY_REJECTED) get typed NR-A010..NR-A015 dispatch — was collapsing to NullRunBlockedException which silently crashed on the loose path because subclasses need workflow_id positional. - audit M8: _validate_capabilities_payload rejects malformed capability envelopes at SDK entry rather than passing them downstream. Static-typing closure: - _V3_ERROR_CODE_MAP annotation tightened from type[BaseException] to type[Exception] (mypy return-value fix — every map value is Exception subclass). - ruff F811 sweep across test files (test_actions.py, test_v3_wire_contract.py, test_audit_wire.py, test_no_local_policy.py, test_audit.py, test_runtime.py, test_transport.py) — auto-removed redefinition of unused top-level imports shadowed by in-function imports. - runtime.py non_prod_hosts tuple: 0.0.0.0 is a host-marker string for the substring match, not a bind address — silenced S104 with noqa rationale. Tests: 1550 passed, 7 skipped in 154.47s. ruff clean. mypy clean (37 source files). Compatibility: No SDK_MIN_VERSION bump. No public API change, no wire-format change. Drop-in replacement for 0.15.0.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Patch release 0.15.1 — bundles the v3.53 NULLRUN audit fixes that landed between 0.15.0 and now. Drop-in replacement for 0.15.0 (no public API change, no wire-format change, no SDK_MIN_VERSION bump).
What's in
Audit fixes on the wire path
Transport.executeraisesNullRunProtocolErroron unmapped wireerror_code(was silently falling through the catalog loose path).MCPAdapter.call_toolnow mirrors/gate→/executetwo-step when aNullRunRuntimeis bound (was bypassing the gate).NULLRUN_SKIP_BUDGET_CHECK=1production guard) — raisesNullRunInfrastructureError (NR-S001)per CLAUDE.md §20; explicit ackNULLRUN_ALLOW_SKIP_BUDGET_CHECK=1remains for incident response.BUDGET_RECHECK_FAILEDtyped dispatch) — distinct fromBUDGET_HARD_BLOCKED— period-bound counter moved between/gateand/execute; caller should re-/gate.NullRunBlockedExceptionwhich silently crashed on loose-path signature mismatch._validate_capabilities_payload) — rejects malformed capability envelopes at SDK entry.Static-typing + lint closure
_V3_ERROR_CODE_MAPannotation:type[BaseException]→type[Exception](mypyreturn-valueclosure).test_actions.py,test_v3_wire_contract.py,test_audit_wire.py,test_no_local_policy.py,test_audit.py,test_runtime.py,test_transport.py.runtime.pynon_prod_hoststuple:0.0.0.0is a host-marker string for substring match, not a bind address — silencedS104withnoqarationale.Verification
Commits in this PR (vs
origin/master)Compatibility
No SDK_MIN_VERSION bump. Drop-in replacement for 0.15.0.