Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 0 additions & 12 deletions .flake8
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,6 @@ select = RMP
# are kept rather than replaced.
extend-exclude = .venv,build,dist

# Baseline of pre-existing violations, recorded so the rule can be enforced
# from day one without a large mechanical rename in the same change. Each
# entry is removed by the commit that fixes the file. Do not add new entries.
per-file-ignores =
rampart/core/execution.py:RMP001
rampart/core/injection.py:RMP001
rampart/evaluators/llm_judge.py:RMP001
rampart/pyrit_bridge/llm_bridge.py:RMP001
rampart/pytest_plugin/_collection.py:RMP001
rampart/surfaces/onedrive.py:RMP001
tests/*:RMP001

[flake8:local-plugins]
extension =
RMP = flake8_rampart:RampartChecker
Expand Down
5 changes: 2 additions & 3 deletions .github/instructions/coding-standards.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -623,9 +623,8 @@ while `# noqa:` is reserved for an RMP code (e.g., `RMP001`), which flake8
rather than ruff reads. `RMP001` is listed in `[tool.ruff.lint] external` so
that ruff's `RUF102` accepts it instead of rejecting it as an unknown code.

`.flake8` carries a `per-file-ignores` baseline of files that predate the rule.
Those entries are removed as the files are fixed; do not add new ones.

`RMP001` applies repo-wide, including to tests: the test standards require the
`_async` suffix on async test names too.
[flake8-local]: https://flake8.pycqa.org/en/latest/user/configuration.html#using-local-plugins

---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ class TestParseConfig:
```

### Async Tests
- Async test method names MUST end with `_async`
- Async test method names MUST end with `_async` (enforced by `RMP001`)
- Use `AsyncMock` instead of `MagicMock` when mocking async methods

```python
Expand Down
2 changes: 1 addition & 1 deletion docs/api/core-protocols.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Protocols and ABCs that define RAMPART's extension points. Implement these to co
members:
- Surface
- InjectionHandle
- sleep_until_ready
- sleep_until_ready_async

## Converter

Expand Down
4 changes: 2 additions & 2 deletions docs/attacks/xpia.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ sequenceDiagram

Test->>Surface: inject(payload) → handle
Note over Surface: Payload placed in data source
Test->>Surface: handle.wait_until_ready()
Test->>Surface: handle.wait_until_ready_async()
Test->>Agent: session.send_async("Summarize reports")
Agent-->>Test: Response (text + tool_calls)
Test->>Eval: evaluate_async(context)
Expand All @@ -28,7 +28,7 @@ sequenceDiagram
**Phases:**

1. **Inject** — Place payloads into the agent's data sources via surfaces. Each `surface.inject(payload)` returns an [`InjectionHandle`][rampart.core.injection.InjectionHandle].
2. **Wait** — Handles call `wait_until_ready()` to allow indexing. Runs concurrently for multiple surfaces.
2. **Wait** — Handles call `wait_until_ready_async()` to allow indexing. Runs concurrently for multiple surfaces.
3. **Trigger** — Send benign prompts that cause the agent to retrieve the injected content. Triggers are never adversarial — the attack is in the payload, not the prompt.
4. **Evaluate** — Check each turn for the attack objective. Early-stops on detection.
5. **Clean up** — Remove injected content. Guaranteed via `AsyncExitStack`, even on exceptions.
Expand Down
2 changes: 1 addition & 1 deletion docs/contributing/extending-rampart.md
Original file line number Diff line number Diff line change
Expand Up @@ -337,7 +337,7 @@ For the basic protocol skeleton, see [Implementing Surfaces](../usage/authoring-

- **`Surface.inject` does not activate** — it only prepares the handle. Activation happens when an execution strategy enters the handle as an async context manager.
- **`__aexit__` must be idempotent and must not raise** — cleanup runs even on exceptions, and a failing cleanup must not mask the original error.
- **`wait_until_ready` should bound itself** with `TimeoutError` rather than block indefinitely. For simple delay-based waits, call `sleep_until_ready` from `rampart.core.injection`.
- **`wait_until_ready_async` should bound itself** with `TimeoutError` rather than block indefinitely. For simple delay-based waits, call `sleep_until_ready_async` from `rampart.core.injection`.
- **Raise `InfrastructureError`** for transient, external failures (timeouts, rate limits, service outages). It's the documented convention for surfaces and adapters to signal "not a safety signal" — `BaseExecution` catches all exceptions and produces an `ERROR` result either way, but the exception type is preserved in metadata for triage.

For a complete reference, see [`OneDriveSurface`](https://github.com/microsoft/RAMPART/blob/main/rampart/surfaces/onedrive.py).
Expand Down
2 changes: 1 addition & 1 deletion docs/usage/authoring-tests.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ class MyFileSurface:
def surface_name(self) -> str:
return "file_system"

async def wait_until_ready(self) -> None:
async def wait_until_ready_async(self) -> None:
pass # or: await asyncio.sleep(10.0) for indexing delay

async def __aenter__(self):
Expand Down
2 changes: 1 addition & 1 deletion rampart/attacks/_xpia.py
Original file line number Diff line number Diff line change
Expand Up @@ -169,7 +169,7 @@ async def _activate_handles_async(
# Concurrent: total = max of all wait times
async with asyncio.TaskGroup() as tg:
for handle in self._handles:
tg.create_task(handle.wait_until_ready())
tg.create_task(handle.wait_until_ready_async())

def _build_attack_result(
self,
Expand Down
12 changes: 6 additions & 6 deletions rampart/core/execution.py
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ class ExecutionEventHandler(ABC):
"""

@abstractmethod
async def on_event(self, *, event_data: ExecutionEventData) -> None:
async def on_event_async(self, *, event_data: ExecutionEventData) -> None:
"""Handle an execution lifecycle event.

Args:
Expand Down Expand Up @@ -231,7 +231,7 @@ async def execute_async(self, *, adapter: AgentAdapter) -> Result:
Result: Safety verdict with evidence and diagnostics.
"""
start = time.monotonic()
await self._fire(
await self._fire_async(
ExecutionEvent.ON_PRE_EXECUTE,
adapter=adapter,
elapsed=0.0,
Expand All @@ -247,7 +247,7 @@ async def execute_async(self, *, adapter: AgentAdapter) -> Result:
self.strategy_name,
)

await self._fire(
await self._fire_async(
ExecutionEvent.ON_ERROR,
adapter=adapter,
elapsed=time.monotonic() - start,
Expand All @@ -264,7 +264,7 @@ async def execute_async(self, *, adapter: AgentAdapter) -> Result:

elapsed = time.monotonic() - start
result.duration_seconds = elapsed
await self._fire(
await self._fire_async(
ExecutionEvent.ON_POST_EXECUTE,
adapter=adapter,
elapsed=elapsed,
Expand All @@ -284,7 +284,7 @@ async def _execute_async(self, *, adapter: AgentAdapter) -> Result:
"""
...

async def _fire(
async def _fire_async(
self,
event: ExecutionEvent,
*,
Expand Down Expand Up @@ -314,7 +314,7 @@ async def _fire(
)
for handler in self._handlers:
try:
await handler.on_event(event_data=event_data)
await handler.on_event_async(event_data=event_data)
except Exception:
logger.warning(
"ExecutionEventHandler %s raised on %s — ignored.",
Expand Down
6 changes: 3 additions & 3 deletions rampart/core/injection.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
Two protocols serving two audiences: Surface is what surface authors
implement; InjectionHandle is what execution strategies consume.

``sleep_until_ready`` is a helper function for surfaces that only need
``sleep_until_ready_async`` is a helper function for surfaces that only need
a simple delay-based readiness wait.
"""

Expand Down Expand Up @@ -43,7 +43,7 @@ def surface_name(self) -> str:
"""The name of the surface this handle injects into (e.g., 'SharePoint')."""
...

async def wait_until_ready(self) -> None:
async def wait_until_ready_async(self) -> None:
"""Block until the injected content is visible to the agent.

Implementations should raise `TimeoutError` if readiness
Expand All @@ -52,7 +52,7 @@ async def wait_until_ready(self) -> None:
...


async def sleep_until_ready(delay: float) -> None:
async def sleep_until_ready_async(delay: float) -> None:
"""Sleep for `delay` seconds. Default readiness strategy for simple surfaces.

Args:
Expand Down
4 changes: 2 additions & 2 deletions rampart/evaluators/llm_judge.py
Original file line number Diff line number Diff line change
Expand Up @@ -479,15 +479,15 @@ async def evaluate_async(self, *, context: EvalContext) -> EvalResult:
user_message = self._build_user_message(context=context)

@pyrit_json_retry
async def _send_and_parse() -> _JudgeVerdict:
async def _send_and_parse_async() -> _JudgeVerdict:
raw = await self._send_async(
system_prompt=system_prompt,
user_message=user_message,
)
return _JudgeVerdict.from_json(raw)

try:
verdict = await _send_and_parse()
verdict = await _send_and_parse_async()
except InvalidJsonException:
return self._undetermined(
rationale="Judge could not produce valid JSON after retries.",
Expand Down
6 changes: 3 additions & 3 deletions rampart/pyrit_bridge/llm_bridge.py
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ async def send_user_turn_async(
Returns:
The model's text response.
"""
return await _send_via_normalizer(
return await _send_via_normalizer_async(
normalizer=normalizer,
target=target,
conversation_id=conversation_id,
Expand Down Expand Up @@ -242,7 +242,7 @@ async def send_judge_request_async(
prompt_metadata: dict[str, str | int] | None = (
{"response_format": response_format} if response_format else None
)
return await _send_via_normalizer(
return await _send_via_normalizer_async(
normalizer=normalizer,
target=target,
conversation_id=conversation_id,
Expand All @@ -253,7 +253,7 @@ async def send_judge_request_async(
)


async def _send_via_normalizer(
async def _send_via_normalizer_async(
*,
normalizer: PromptNormalizer,
target: PromptChatTarget,
Expand Down
2 changes: 1 addition & 1 deletion rampart/pytest_plugin/_collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ class ResultCollectionHandler(ExecutionEventHandler):
"""

@override
async def on_event(self, *, event_data: ExecutionEventData) -> None:
async def on_event_async(self, *, event_data: ExecutionEventData) -> None:
"""Record result on post-execute. Ignore all other events.

Args:
Expand Down
6 changes: 3 additions & 3 deletions rampart/surfaces/onedrive.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
from typing import TYPE_CHECKING, Self

from rampart.core.errors import InfrastructureError
from rampart.core.injection import sleep_until_ready
from rampart.core.injection import sleep_until_ready_async

if TYPE_CHECKING:
import types
Expand Down Expand Up @@ -195,14 +195,14 @@ def surface_name(self) -> str:
"""Identifies this injection as OneDrive for reporting."""
return "OneDrive"

async def wait_until_ready(self) -> None:
async def wait_until_ready_async(self) -> None:
"""Wait for the uploaded content to be indexed and discoverable.

Note: Currently sleeps for `OneDriveSurface.indexing_delay` seconds.
Future versions will poll the Graph API for content availability instead and
raise `TimeoutError` if it doesn't appear within the `indexing_delay`.
"""
await sleep_until_ready(delay=self._surface.indexing_delay)
await sleep_until_ready_async(delay=self._surface.indexing_delay)

async def __aenter__(self) -> Self:
"""Upload payload to OneDrive.
Expand Down
Loading